Information Technology: DOD Needs to Leverage Lessons Learned	 
from Its Outsourcing Projects (25-APR-03, GAO-03-371).		 
                                                                 
Given the magnitude of its reported spending on information	 
technology (IT) services--more than $6.2 billion in fiscal year  
2001--it is critical that the Department of Defense (DOD) adopt  
effective practices for acquiring IT services. GAO researched	 
leading commercial practices for the outsourcing of IT services, 
and, in November 2001, published a framework consisting of seven 
phases that span the full range of activities that are performed 
during the outsourcing of those services (this is an acquisition 
in which a client organization transfers responsibility for	 
performing services to an external provider). GAO was asked to	 
determine (1) the extent to which selected DOD projects for	 
outsourcing IT services use leading commercial practices as	 
specified in GAO's framework and (2) whether DOD is sharing	 
lessons learned from its IT outsourcing projects across the	 
department.							 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-03-371 					        
    ACCNO:   A06721						        
  TITLE:     Information Technology: DOD Needs to Leverage Lessons    
Learned from Its Outsourcing Projects				 
     DATE:   04/25/2003 
  SUBJECT:   Information technology				 
	     Defense procurement				 
	     Internal controls					 
	     Best practices					 
	     Procurement practices				 
	     Procurement policy 				 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-03-371

                                       A

Letter

April 25, 2003 The Honorable John Ensign Chairman The Honorable Daniel K.
Akaka Ranking Minority Member Subcommittee on Readiness and Management
Support Committee on Armed Services United States Senate

The Department of Defense (DOD) is the government*s largest purchaser of
information technology (IT) services, such as desktop support, network
operations, and software development services. In fiscal year 2001, DOD
reportedly obligated more than $6.2 billion on IT services, 1 and this
amount is expected to grow substantially. Given the magnitude of DOD*s
spending on such services, it is critical that the department adopt
effective practices for acquiring IT services.

Since 1996, we have conducted a series of studies for the Senate Committee
on Armed Services concerning how DOD can improve its acquisition processes
by adopting proven practices of leading commercial organizations. In this
vein, in November 2001, we issued a guide that

organized leading commercial practices for the outsourcing 2 of IT
services into a framework of seven phases that span the full range of
activities that are performed during IT services outsourcing. 3 This
report responds to your request that we determine (1) the extent to

which selected DOD IT services outsourcing projects use leading commercial
practices as specified in our framework and (2) whether DOD is sharing
lessons learned from its IT outsourcing projects across the department. To
address the first objective, we selected five projects from a group of
projects to outsource IT services that were

1 This figure is from the Federal Procurement Data System, which contains
detailed information on contract actions over $25,000. 2 IT services
outsourcing is a type of acquisition in which a client organization
transfers responsibility for performance of one or more IT services to one
or more external providers. 3 U. S. General Accounting Office, Information
Technology: Leading Commercial Practices for Outsourcing of Services, GAO-
02- 214 (Washington, D. C.: Nov. 30, 2001).

identified by the military services and other DOD components. 4 We then
asked the component responsible for each project to perform a
selfassessment against selected practices in our framework for outsourcing
IT services. 5 Next, we obtained and reviewed agencies* supporting
documentation and interviewed the appropriate agency and provider project
officials to independently determine whether a practice was met. To
address the second objective, we reviewed applicable DOD approaches for
capturing and disseminating lessons learned from IT services outsourcing
projects and interviewed the applicable acquisition and IT officials.
Details of our objectives, scope, and methodology are discussed in
appendix I.

Results in Brief The projects in our review substantially used leading
commercial practices as specified in our framework on outsourcing IT
services. Specifically, the

agencies fully implemented 88 percent of the practices. 6 This framework
consists of practices organized into seven phases that span the full range
of activities that are performed during IT outsourcing: (I) determine
sourcing strategy, (II) define operational model, (III) develop the
contract, (IV)

select the provider( s), (V) transition to provider( s), (VI) manage
provider( s) performance, and (VII) ensure services are provided. Figure 1
illustrates the percentage of practices that were followed in each phase.
Collectively, the projects fully implemented from 70 to 97 percent of the
practices in each phase.

4 We asked the military services and other DOD components to identify
candidate projects because DOD does not maintain a central list of IT
services outsourcing projects. We chose each project on the basis of the
following criteria: (1) no more than one project from each military
service and two agencies, (2) illustrative example of DOD IT outsourcing,
(3) dollar value greater than $10 million, and (4) enough time elapsed for
services to have been delivered and performance measured.

5 We identified 70 practices in our November 2001 report on leading
commercial practices that (1) are typically applied at the project level
and (2) were verifiable through documentation and interviews. 6 This
calculation does not include practices that were not applicable to a
particular project.

Figure 1: Percentage of Practices Implemented in Each Phase

Note: Not applicable* The practice was not relevant to the project*s
particular circumstances. No* The agency did not implement the practice.
Limited* The agency fully implemented some but not all aspects of the
practice and did not take alternative actions that fully satisfied the
practice. Yes* The agency fully implemented the practice or took an
alternative action that fully satisfied its intent.

Although implementing the leading commercial practices in our framework
does not guarantee the success of an outsourcing project, the consensus
view of the leading commercial activities that we studied is that these
practices are the most critical to success when acquiring IT services. 7
In addition, not implementing or only partially implementing particular
practices can produce negative consequences or add risk to a project. For
example, the Department of the Navy project*s baseline of its existing
environment (a phase I practice) was limited because it did not include an
assessment of its legacy applications. Instead, project officials decided
to rely on a preexisting inventory developed to address the Year 2000
challenge. The Navy subsequently found that it had substantially
underestimated the number of legacy applications, which, according to
program officials, later contributed to the transition period slipping
from 2- 1/2 years to 3- 1/2 years. As DOD gathers more experience in
implementing projects for outsourcing

IT services, it can benefit from leveraging the lessons derived from these
initiatives. For example, the projects in our review have identified
lessons learned in such areas as transitioning to the provider and
partnering with the provider. Although DOD has taken action to gather and
disseminate lessons learned and best practices on general acquisition
issues, these efforts generally do not focus on outsourcing or include
sharing the lessons learned from IT outsourcing projects across the
department. By not capturing and disseminating such information in a
systematic manner across the department, DOD is losing the opportunity to
leverage the knowledge gained on IT services projects like those in our
review.

We are making recommendations to the Secretary of Defense aimed at
leveraging lessons learned across the department from its components* IT
services outsourcing experiences. In written comments on a draft of this
report signed by the DOD Principal

Director, Deputy Assistant Secretary of Defense (Deputy Chief Information
Officer), DOD agreed that capturing lessons learned in the development and
implementation of its IT outsourcing initiatives is important to
continually improving its outsourcing methods and results achieved. The

7 This consensus view was based on interviews with managers in leading
commercial organizations, discussions with academic and professional
authorities, and extensive research on IT acquisition practices.

department also stated that it intends to explore a variety of mechanisms
for best exploiting lessons learned from its IT outsourcing initiatives.
We agree that it is prudent to consider alternative means to leveraging
these

lessons learned, and we believe that this is consistent with our
recommendations.

Background To protect the security of the United States, DOD relies on a
complex array of computer- dependent and mutually supportive
organizational components, including the military services, Commanders in
Chief, and Defense agencies. As such, it invests tens of billions of
dollars each year in a broad array of computer systems, which include
weapon systems, command and control systems, satellite systems, inventory
management

systems, transportation management systems, health systems, financial
systems, personnel systems, and payment systems. In addition, DOD spends
billions of dollars annually on IT services, which include database
management, help- desk operations, software maintenance, and network
services. In fiscal year 2001, DOD reportedly obligated more than $6. 2
billion on IT services alone. 8

Decisions regarding the purchasing of services are critical to ensuring
the effectiveness of DOD*s operations as well as those of the government
as a whole. Our November 2001 report recognizes the importance of such
sourcing decisions and provides a framework that spans the full range of
activities that are performed during IT services outsourcing. 9 At the
same time, governmentwide policies, initiatives, and challenges exist that
significantly influence the government*s sourcing decisions.

8 This figure is from the Federal Procurement Data System, which contains
detailed information on contract actions over $25,000. 9 GAO- 02- 214.

GAO*s Framework for Outsourcing of IT services has become increasingly
popular in both the

Outsourcing IT Services public and private sector. For example, according
to the Giga Information Group, Inc., a leading research firm, such
outsourcing is expected to grow

an average of 5 to 6 percent in 2003. 10 The federal sector*s outsourcing
is predicted to rise at an even greater rate. For example, INPUT, an IT
market research firm, forecasts that defense IT outsourcing will increase
about 143

percent between fiscal years 2002 and 2007. 11 IT outsourcing involves the
activities associated with acquiring services from one or more external
providers. During outsourcing, a client organization transfers
responsibilities for performing one or more IT

services to one or more external providers. This responsibility is
executed through control and management of the processes, people, and
technology associated with these services.

Figure 2 depicts the roles of the client and provider organizations in an
outsourcing relationship.

Figure 2: Roles of the Client and Provider in an Outsourcing Relationship

10 Giga Information Group, Inc., IT Trends 2003: IT Services (Dec. 19,
2002). We did not independently verify these data. 11 INPUT, The Federal
IT Outsourcing Market View (December 2002). We did not independently
verify these data.

Our November 2001 guide on leading commercial practices for outsourcing IT
services provides a generic framework of practices from leading commercial
organizations that can improve purchasing decisions and manage the
resulting government/ provider relationship. 12 The framework is
represented in figure 3 as a hierarchy of phases, practices, and critical
success factors.

Figure 3: GAO*s Framework for Outsourcing IT Services

Note: The arrow from phase VII to phase I represents the need to reflect
on lessons learned from previous phases. The arrows between phase III and
IV represent the iterative nature of developing the contract and selecting
the provider. Although there is a logical order to the sequence of the
common phases, the order of the practices within each phase does not imply
any priority or sequence. Table 1 provides a definition of each phase of
the framework. Each of the

phases has specific practices associated with it. Implementing these
practices does not guarantee the success of an outsourcing project.
However, our November 2001 study reflected a consensus view that these
practices were the most critical to success when IT services are being
acquired. 13 Restated, application of these practices increases the
probability of a successful outsourcing project.

12 GAO- 02- 214. 13 This consensus view was based on interviews with
managers in leading commercial organizations, discussions with academic
and professional authorities, and extensive research on IT acquisition
practices.

Table 1: Definition of Phases for IT Outsourcing Phase number Title
Definition

I Determine sourcing strategy Client organizations determine whether
internal capability or external expertise can more effectively meet IT
needs.

II Define operational model Client organizations formalize executive
leadership, team composition, client responsibilities, and operating
relationships between client and provider organizations.

III Develop the contract Client organizations establish the legal terms
for the IT outsourcing relationship. IV Select the provider( s) Client
organizations find one or more providers who can help them reach their

IT outsourcing goals. V Transition to provider( s) Client organizations
transfer responsibility of IT functions to one or more providers.

VI Manage provider( s) performance Client organizations make sure each
provider is meeting performance requirements.

VII Ensure services are provided Client organizations make sure that
services are provided and end- user needs are met.

Source: GAO.

The organizations that we studied also identified certain capabilities
(identified as critical success factors) that were essential for
implementing the practices identified in our framework. First, executive
leadership strengthens the interaction between executive management and
the

employees of the client organization. Second, partner alignment
strengthens the interaction between the client and provider organization
at the executive level, which ensures that the goals and objectives of
these organizations support each other. Third, relationship management

strengthens the interaction between the client and provider organization
at the operational level.

Influences on Government The federal government is one of the world*s
largest users of services.

Sourcing Decisions Because of the large dollar value and the number of
private- and publicsector

jobs involved, deciding whether the public or private sector would be the
most appropriate provider of the services the government needs (IT or
otherwise) is an important, and often highly charged, question. Among the
factors that agencies must consider as they determine how best to meet
their missions is whether the public or private sector would be the most

appropriate provider of the services the government needs. Phase I of our
framework, determine the sourcing strategy, addresses the client*s

assessment of whether expertise from within or outside of the organization
can more effectively meet the client*s needs.

The National Defense Authorization Act for Fiscal Year 2001 required the
Comptroller General of the United States to convene a panel of experts to
study the current process used by the government to make sourcing
decisions. The resulting Commercial Activities Panel conducted a year-
long

study and heard repeatedly about the importance of competition and its
central role in fostering economy, efficiency, and continuous performance
improvement. In particular, the panel reviewed the government*s
implementation of the Office of Management and Budget*s (OMB) Circular

A- 76, which sets forth federal policy for determining whether federal
employees or private contractors will perform commercial activities for
the government. 14 Circular A- 76 (1) outlines conditions under which
agencies are permitted to perform a commercial activity with government
employees or by contract and (2) provides guidance for whether, and if so,
how, agencies should conduct a cost comparison when they are considering
transferring the performance of commercial activities from the public to
the private sector (or vice versa). The panel reported that there

were positive elements to Circular A- 76 but that both federal employees
and private firms complained that it does not meet the standard of a
clear, transparent, and consistently applied process. For example, both
federal

employees and private firms criticized the Circular A- 76 process as
unequal and therefore unfair.

The Commercial Activities Panel strongly supported continued emphasis on
competition and concluded that whenever the government is considering
converting work from one sector to another, public/ private competitions
should be the norm. In addition, the panel made four recommendations,
including that all sourcing decisions be consistent with the principles
adopted unanimously by the panel, such as the principle that federal
policy provide for accountability in connection with all sourcing
decisions. 15 14 Office of Management and Budget, Circular A- 76,
Performance of Commercial Activities (Washington, D. C.: June 14, 1999).
In November 2002, OMB issued proposed revisions that

would substantially change this circular. As of April 8, 2003, these
revisions have not yet been made final.

15 Final report of the Commercial Activities Panel, Improving the Sourcing
Decisions of the Government (Washington, D. C.: Apr. 30, 2002).

As part of the administration*s efforts to implement the recommendations
of the Commercial Activities Panel, OMB has published proposed changes to
Circular A- 76. Key highlights of the proposed changes include presuming
that all functions are commercial in nature unless they are justified as
inherently governmental; 16 limiting the length of time for competitions;
and emphasizing awarding contracts on the basis of best value, not just
lowest cost. Best value allows the contracting official to consider
technical superiority, quality, innovation, and past performance as well
as price.

However, we reported that there are several areas in which the proposed
revisions to the circular are not consistent with the principles or
recommendations of the Commercial Activities Panel. 17 Specifically, the
proposed revision does not include a link between sourcing policy and
agency missions, has unnecessarily complicated source selection
procedures, contains certain unrealistic time frames, and includes
insufficient guidance on calculating savings.

Beyond the Commercial Activities Panel, other bodies have identified
challenges that the federal government faces in reaching and executing
effective sourcing decisions. For example, members of the Coalition for
Government Procurement, the Professional Services Council, and the
Information Technology Association of America told us that organizational
culture is one of the biggest differences between the commercial sector
and the federal government and one of the greatest barriers to the

government*s use of commercial practices. Also, as we have previously
reported, moving to outsourcing solutions can involve a cultural change
for government organizations because it may require a change to an
agency*s operating model, such as using a contractor to provide IT
services

previously performed by government staff or using a performance- based
contract. 18 This view was echoed by a 2001 study of DOD competitive

16 Section 5 of P. L. 105- 270, codified at 31 U. S. C. 501 note (1998),
defines an inherently governmental function as a *function that is so
intimately related to the public interest as to require performance by
Federal Government employees.* 17 U. S. General Accounting Office,
Proposed Revisions to OMB Circular A- 76, GAO- 03- 391R (Washington, D.
C.: Jan. 16, 2003).

18 U. S. General Accounting Office, Desktop Outsourcing: Positive Results
Reported, but Analyses Could Be Strengthened, GAO- 02- 329 (Washington, D.
C.: Mar. 29, 2002).

sourcing that found cultural, process, execution, and training barriers.
19 The study stated that these barriers need to be understood and
mitigated before the benefits of outsourcing can be fully realized.
Barriers such as these can be overcome by strong executive leadership,
which is a critical success factor in our framework. Another challenge is
creating a productive agency/ provider relationship* another critical
success factor in our framework. According to a report sponsored by the
PricewaterhouseCoopers Endowment for the Business of Government, such
public/ private partnerships are based on trust, commitment to problem or
conflict resolution, and the recognition that

flexibility is necessary and that the relationship will evolve and change
over time. 20 If deadlines are not met, or public agency goals change with
differing political climates, the partners need to discuss the basis of
the partnership and construct a different relationship. Our prior report
on desktop outsourcing found that developing a productive agency/
contractor relationship is not always easy. 21 Both sides must recognize
and understand

each other*s underlying motives and strive to achieve established
expectations.

Finally, human capital issues are another challenge facing federal
agencies that affect their ability to implement outsourcing. Our framework
recognizes the importance of having the right skills in place to support
the outsourcing relationship. However, as we have previously reported,
procurement reforms and technological changes have placed unprecedented
demands on the acquisition workforce. 22 Contracting personnel are now
expected to have a much greater knowledge of market conditions, industry
trends, and the technical details of the commodities and services they
procure. The Commercial Activities Panel report stated that developing and
maintaining a skilled acquisition workforce is the

critical first step in managing this more complex procurement environment.
The panel also reported that DOD bore the brunt of a 22

19 The Industrial College of the Armed Forces, Case Study: Complex
Business Management for Competitive Sourcing (2001). 20
PricewaterhouseCoopers Endowment for the Business of Government,
Contracting for the 21 st Century: A Partnership Model (January 2002).

21 GAO- 02- 329. 22 U. S. General Accounting Office, Contract Management:
Taking a Strategic Approach to Improving Service Acquisitions, GAO- 02-
499T (Washington, D. C.: Mar. 7, 2002).

percent downsizing of the federal acquisition workforce in the last
decade, going from 96,000 staff in 1991 to about 68,000 in fiscal year
2001.

Addressing human capital issues is not just a matter of the size of the
workforce; it is also a knowledge and skills issue. According to the
Commercial Activities Panel, it is critically important that federal
agencies adequately address human capital needs in meeting the current and
emerging needs of government and its citizens in the most effective,
efficient, and economical manner possible. This will require increased

emphasis on training and development, particularly in the area of
technology. Description of Five Projects

The five projects in our study varied in how they approached outsourcing
Reviewed

IT services, such as in using various solicitation methods, including
holding a public/ private competition under the policies outlined in OMB
Circular A76 or carrying out a negotiated competitive procurement. In
addition, the types of services being outsourced differed: services ranged
from the narrowly focused (e. g., help- desk services) to the very broad
(e. g., enterprisewide end- to- end information services); contract terms
ranged from 5 to 15 years (assuming all option years are exercised); and
estimated contract values ranged from $23 million to $8.8 billion. Table 2
provides information on the variety of IT services and outsourcing
approaches taken by the projects.

Table 2: Profile of Outsourcing Projects Reviewed Date of Solicitation
contract

Contract Contract Estimated total Agency/ Project method

award term type contract value a Project description

Air Force/ Kirtland Air Competitive, April 2000 1 year, with

Firm, fixedprice $23 million Management, operations, and Force Base*s
under OMB

4 option maintenance of command, Command, Control, Circular A- 76 years

control, communications, and Communications, and and small computer
systems, multimedia Computer (C4)

business setaside services, and information

Services rules

management for the Kirtland Air Force Base.

Army/ Network Negotiated March 2001 1 year, with

Firm, fixedprice $204 million Operation and maintenance

Enterprise Technology competitive 4 option services, including repair,

Command*s solicitation years

installation, and supply, for (NETCOM) Total Army communications equipment
in Communications*

Southwest Asia. Southwest Asia (TACSWA)

Military Health Negotiated

June 2001 1 year, with Firm, fixedprice

$71 million Call and help- desk services for System/ Information

competitive 7 option with all MHS software applications.

Technology solicitation

years incentive

Organization awards (MHS/ ITO) Help Desk

Department of the Negotiated October

7 years, Firm, fixedprice $8.8 billion Department of the Navy- wide

Navy/ Navy and Marine competitive 2000 with an

with end- to- end information services

Corps Intranet (NMCI) solicitation

option for 3 incentive through a common computing

additional awards

and communication years environment.

National Imagery and Sole- source,

December 1 year, with

Cost plus $2.1 billion NIMA- wide IT/ IS support

Mapping Agency*s using a 2001 14 option award fees services for printing,
digital

(NIMA) Information statutory

years replication, networks, distributed

Technology/ preferential

and centralized systems and Information Services

provider services, video and voice

(IT/ IS) (Alaska native communications, information

corporation) b research, and help- desk

operations. c Source: DOD.

a Estimated value if all option years are exercised. b NIMA performed an
OMB Circular A- 76 analysis and, on the basis of this analysis,
implemented a direct conversion to a preferential provider rather than
holding a public/ private competition or obtaining an agency cost-
comparison waiver. c As of February 24, 2003, NIMA had transitioned four
of these functions to the provider: printing, digital

replication, video and voice communications, and help- desk operations.

Projects Substantially As illustrated in figure 4, the five IT services
projects substantially used

leading commercial practices. Specifically, each project used at least 76
Used Leading percent of the practices. 23 Reasons for projects
implementing different Commercial Practices

percentages of the practices include differences in their individual
circumstances and objectives. For example, the Army*s Total Army
Communications* Southwest Asia (TAC- SWA) project, which used the

fewest number of practices and had the largest number of practices that
were not applicable, was largely a continuation of an existing approach
that already relied on the private sector but with fewer providers. In
contrast, the National Imagery and Mapping Agency*s (NIMA) Information
Technology/ Information Services (IT/ IS) project, which fully or
partially implemented all of the applicable practices, involved a
significant operational shift (e. g., functions previously performed by
NIMA staff are now performed by a contractor) and was intended to result
in substantial process improvements. In addition, the three projects that
implemented the largest percentage of practices also used third- party
assistance* including employing a contractor with sourcing expertise* to
help formulate their sourcing strategy, which could account for the extent
of their compliance.

23 This calculation does not include practices that were not applicable to
a particular project.

Figure 4: Use of Leading Commercial Practices for Outsourcing of IT
Services, by Project

Note: Not applicable* The practice was not relevant to the project*s
particular circumstances. No* The agency did not implement the practice.
Limited* The agency fully implemented some but not all aspects of the
practice and did not take alternative actions that fully satisfied the
practice. Yes* The agency fully implemented the practice or took an
alternative action that fully satisfied its intent.

In addition, figure 5 illustrates that project compliance extended to each
of the phases of our framework. Collectively, the projects fully
implemented from 70 to 97 percent of the practices in each phase. Phase I,
determine sourcing strategy, had the lowest percentage of practices
implemented by

the projects (70 percent). This result is not inconsistent with a recent
Giga Information Group, Inc., survey, which found that only half of the

respondent organizations had documented an IT sourcing strategy. 24 This
approach carries risk since phase I sets the tone for the outsourcing
initiative within the client organization.

Figure 5: Percentage of Practices Implemented in Each Phase

Note: Not applicable* The practice was not relevant to the project*s
particular circumstances. No* The agency did not implement the practice.
Limited* The agency fully implemented some but not all aspects of the
practice and did not take alternative actions that fully satisfied the
practice. Yes* The agency fully implemented the practice or took an
alternative action that fully satisfied its intent.

24 Giga Information Group, Inc., Optimizing IT Sourcing Strategy: Key
Stages and Phases of the IT Sourcing Process (Jan. 31, 2003).

In addition, collectively the projects fully implemented 88 percent of the
practices (see table 3). 25 Table 3: Percentage of Practices Implemented,
by Project

Percentage of practices implemented a Air Force

Army Phase

C4 Services TAC- SWA

MHS/ ITO Help Navy NMCI

NIMA IT/ IS project

project Desk project project project Overall

Phase I: Determine sourcing strategy (6 practices). 50 33 83 83 100 70

Phase II: Define operational model (13 practices). 92 83 100 77 100 90

Phase III: Develop the contract (16 practices). 64 83 100 94 100 89

Phase IV: Select provider( s) (7 practices). 86 86 100 100 71 89

Phase V: Transition to provider( s) (11 practices). 82 88 100 82 100 90

Phase VI: Manage provider( s) performance (11 practices). 73 73 100 91 82
84

Phase VII: Ensure services are provided (6 practices). 83 100 100 100 100
97

Overall 76 79 99 89 94 88

Source: GAO. a These calculations do not include practices that were not
applicable to a particular project.

The following provides additional information on the projects*
implementation of each phase of our framework.

25 This calculation does not include practices that were not applicable to
a particular project.

 Phase I: Determine sourcing strategy. In the first phase of our
outsourcing framework, the client organization determines whether internal
capability or external expertise can more effectively meet its IT needs.
The purpose of a sourcing strategy is to achieve the optimal balance
between internal and external capabilities, activities, processes, and
services to ensure the achievement of strategic business objectives at the
lowest risk. 26 The five projects* implementation of this phase was
uneven. In particular, two of the six practices in this phase were fully
implemented by all five projects, but the other four practices were not.
Among the practices that were implemented by all of the projects was
determining the business reasons for outsourcing. In addition, the three
projects that implemented the largest percentage of practices in our
framework* the Military Health System/ Information Technology Organization
(MHS/ ITO) Help Desk, the Department of the Navy*s Navy and Marine Corps
Intranet (NMCI), and NIMA IT/ IS projects* used the third- party
assistance practice in this phase to help formulate their sourcing
strategy, which could account for the extent of their compliance. For
example, the MHS/ ITO Help Desk project, which implemented the largest
percentage of practices, worked with the Department of the Interior*s
GovWorks Program, 27 the Defense Acquisition University staff, and a
private- sector contractor to obtain expertise on sourcing strategies.

The practice in this phase that was the most unevenly implemented was the
benchmarking 28 and baselining of existing internal services. Of the five
projects in our review, (1) one fully benchmarked and baselined the
productivity of the activity being outsourced before making the final
sourcing decision, (2) two partially baselined their existing activities,
and (3) two did not perform benchmark and baseline analyses at all. The
agencies* reasons for not fully implementing this practice included that
an executive decision had been made to conduct a public/ private
competition following the OMB Circular A- 76 policy, so such an analysis
would not have affected the sourcing decision, or that

26 Gartner, Inc., Strategic Analysis Report, How to Build a Sourcing
Strategy, Research Note R- 18- 1099 (Sept. 23, 2002). 27 The Department of
the Interior*s GovWorks Program is a federal fee- for- service acquisition
center that helps other federal agencies acquire supplies and services for
their programs on a project- by- project basis.

28 Gartner, Inc., defines benchmarking as a method to compare the cost or
price of an IT environment to peer groups with the same workload
characteristics.

available documentation to perform such an analysis was limited. Leading
research firms suggest benchmarking and baselining the entity*s current
processes before outsourcing because only then would it be able to
determine whether the arrangement has been successful. 29 In addition, the
risk of not fully baselining the existing environment is

illustrated by the NMCI project. Specifically, the NMCI project*s baseline
of its existing environment was limited because it did not include an
assessment of its legacy applications since project officials decided to
rely on a preexisting inventory developed to address the Year 2000
challenge. The Navy subsequently found that it had substantially
underestimated its number of legacy applications, that,

according to program officials, later contributed to the transition period
slipping from 2- 1/2 years to 3- 1/2 years. Appendix II provides
additional information on projects* implementation of the practices
comprising this phase.

 Phase II: Define operational model. The operational model is an
important mechanism for an organization to compare its plans with the
expectations that were set when the decision to outsource was made and to
ascertain whether these plans will enable the organization to meet
expectations. The five projects had largely implemented the 13 practices
contained in this phase. Specifically, about 90 percent 30 of the

practices were implemented. For example, all projects implemented the
practice that executive leadership be established to facilitate the
outsourcing effort. NIMA, for instance, formed a strategic sourcing office
to oversee the IT/ IS project. Another practice* training the provider on
the organization*s business environment and goals* was fully implemented
by one project (in two cases, the practice was not applicable). One
project that did not fully implement this practice was the Air Force*s
Kirtland Air Force Base*s Command, Control, Communications, and Computer
(C4) Services project. Although some training was provided (e. g.,
Kirtland held an orientation session for potential bidders), provider
officials stated that they did not receive adequate training, which made
the transition period more difficult. Appendix III provides additional
information on projects*

implementation of the practices comprising this phase. 29 Giga Information
Group, Inc., Payment and Incentives for Outsourcing Management (July 27,
2000) and Gartner, Inc., Benchmarking Helps Outsourcing Deals Stay
Competitive, Research Note COM- 16- 8055 (June 14, 2002).

30 This calculation does not include practices that were not applicable to
a particular project.

 Phase III: Develop the contract. A well- written contract is necessary
for the outsourcing organization to meet its requirements while allowing
the service provider to make a fair profit. It sets the expectations for
service levels, delivery of essential services, and continuous improvement
and should protect the interests of all parties. The five projects largely
implemented the practices in this phase. Specifically, about 89 percent 31
of the practices were implemented in this phase, and two projects (the
MHS/ ITO Help Desk and NIMA IT/ IS projects)

implemented all of the practices. Several practices in this phase address
performance requirements. For example, all five projects implemented the
practices that called for basing performance requirements on business
requirements and reviewing and updating them periodically. One practice
that was not fully implemented by two projects was including performance
measures that address both technical and enduser satisfaction aspects of
performance. For example, the Army TACSWA project included technical
performance measures in its contract but not measures related to end- user
satisfaction, even though the contract included help- desk services.
According to the project official that developed the performance work
statement in the contract, the command did not include customer
satisfaction measures because it did not think that it was necessary to
have a performance standard for the help- desk service. However, without
such measures, the agency does not have a contractual standard with which
to judge the provider*s performance. Appendix IV provides additional
information on projects*

implementation of the practices comprising this phase.  Phase IV: Select
the provider( s). Critical to the success of any

outsourcing project for IT services is identifying potential providers and
ultimately selecting a provider( s) that will best meet the needs of the
agency. The five projects had largely implemented the seven practices
contained in this phase. About 89 percent of the practices were
implemented in this phase and two projects (the MHS/ ITO Help Desk and
Navy NMCI projects) used each applicable practice. For example,

the five projects implemented the practice related to conducting due
diligence activities to verify provider capabilities before signing the
contract. In the case of the Army TAC- SWA project, the Network Enterprise
Technology Command (NETCOM) evaluated the provider*s financial and past
performance information. The Department of the

Navy*s NMCI project also evaluated bidders* past performance and 31 This
calculation does not include practices that were not applicable to a
particular project.

performed reference checks. Moreover, as part of its due diligence
activities, the Navy required all bidders to demonstrate that they had
experience in implementing large seat management contracts. 32 A practice
that was not implemented by two projects was using third- party assistance
when selecting the provider. Projects that did not implement this practice
believed that they had adequate in- house expertise with outsourcing,
making third- party assistance unnecessary. However, because third- party
assistance provides an independent resource that

can suggest options or processes that the client organization may not be
aware of, these projects may have missed an opportunity to implement their
outsourcing projects more effectively. Appendix V provides additional
information on projects* implementation of the practices comprising this
phase.

 Phase V: Transition to provider( s). This phase focuses on the client
organization*s transfer of the IT function to one or more providers. As
part of this transition, the clear definition of responsibilities and the
careful consideration of employees* needs matched against the client
organization*s needs enable both the client and provider to focus on
execution and give staff confidence in their future employment. The five

projects largely implemented the 11 practices associated with this phase.
Specifically, about 90 percent 33 of the practices were implemented in
this phase, and two projects implemented all of the practices (the MHS/
ITO Help Desk and NIMA IT/ IS projects). Several practices in this phase
address dealing with employees affected by the outsourcing projects. For
example, in the four projects in which federal employees were affected,
the projects provided assistance to those who did not want to transfer to
the provider, including helping to place them

in other positions and helping with resume writing. A related practice is
to clearly communicate to all employees what is going to happen and when
it is going to happen. Two projects did not fully implement this practice.
For example, the Navy used its normal chain of command to communicate
transition information, but found that implementation of

this practice was uneven. As a result, some staff did not know current
information about how NMCI would affect them until the provider was ready
to contact them regarding their possible transition to the 32 Seat
management generally refers to service provision arrangements in which
contractorowned desktop and other computing hardware, software, and
related services are bundled and provided to a client organization at a
fixed price per unit (or seat).

33 This calculation does not include practices that were not applicable to
a particular project.

contractor. However, according to the NMCI*s Director*s office, this
problem was somewhat mitigated by the provider*s Web site that provides
transition information to all NMCI customers/ users. Appendix VI provides
additional information on projects* implementation of the practices
comprising this phase.

 Phase VI: Manage provider( s) performance. The effectiveness with which
the provider( s) performance is managed is critical to the successful
implementation of an outsourcing project. Indeed, according to Gartner,
Inc., a leading research firm, an outsourcing project can be thwarted by
poorly designed, funded, and delivered processes for

managing the delivery of services. 34 The five projects generally
implemented the 11 practices contained in this phase, with about 84
percent of the practices being implemented in this phase. For example, the
practices related to obtaining feedback on provider performance

were largely implemented. This is important because different levels of an
agency can have different perceptions about the value of the outsourcing
project. For example, an outsourcing project may be considered successful
by the agency*s executive management if it is focused on controlling
costs, but be considered inadequate by business managers and users who may
be expecting higher levels of service. Each of these viewpoints is valid
and should be taken into account when the

provider*s performance is evaluated. Two other practices* including
incentives and penalties in contracts* were fully implemented by two
projects. Incentives and penalties are important because they can help
motivate the provider to exceed or meet performance requirements.

Nonetheless, two projects did not include monetary incentives and two
projects did not include monetary penalties in their contracts. For
example, the Army TAC- SWA project did not include monetary incentives,
although the contracting officer stated that incentives might have been
useful to motivate the provider to exceed performance requirements.
Incentives can also help control risks. According to a guide on
performance- based services acquisition, if the incentives in the

contract are right and if the provider and agency share the same goals,
risk is largely controlled and effective performance is *almost the 34
Gartner, Inc., Retain Enough Resources to Manage Outsourcing Deals,
Research Note COM- 16- 8425 (June 17, 2002).

inevitable outcome.* 35 Appendix VII provides additional information on
projects* implementation of the practices comprising this phase.  Phase
VII: Ensure services are provided. Although outsourcing

transfers responsibility for performing the service to the provider( s),
the client organization is ultimately responsible for ensuring that
services are provided and that end- user needs are met. Accordingly, it is
critical that the agency ensure that services are provided. The projects
had implemented 97 percent of these practices, and four projects
implemented all of them. For example, every project monitored the
providers* work. In the case of the Air Force*s C4 Services project,
quality assurance evaluators monitored the provider*s work to identify
problems or trends in accordance with the project*s quality assurance
surveillance plan. The results were reported to the contracting officer
and to the functional area chief for resolution. Another practice, using
customer satisfaction surveys, was fully implemented by four of the five
projects. However, the Air Force project did not conduct, or require its
contractor to conduct, customer satisfaction surveys. Although the
provider surveys staff annually, the Air Force is nevertheless relying on
the provider to voluntarily implement an important practice for
determining how customers view the services being delivered and whether
changes need to be made. Appendix VIII provides additional

information on projects* implementation of the practices comprising this
phase.

Leveraging Lessons We have previously reported on the importance of
collecting and

disseminating lessons learned. 36 For example, a critical activity in IT
Learned DOD- wide

investment management is establishing a process for developing and Could
Assist Other

capturing lessons learned in a written product or knowledge base and DOD
Projects

disseminating them to decision- makers. 37 In addition, one of the
practices in our framework for outsourcing IT services addresses
incorporating

35 An Interagency- Industry Partnership in Performance, Seven Steps to
Performance- Based Services Acquisition, Benchmark Version (January 2002).
36 U. S. General Accounting Office, NASA: Better Mechanisms Needed for
Sharing Lessons Learned, GAO- 02- 195 (Washington, D. C.: Jan. 30, 2002).
37 U. S. General Accounting Office, Information Technology Investment
Management: A Framework for Assessing and Improving Process Maturity, GAO/
AIMD- 10- 1. 23, Exposure Draft (Washington, D. C.: May 2000).

lessons learned from peers who have engaged in similar sourcing decisions.
Use of lessons learned is a principal component of an organizational
culture committed to continuous improvement. Sharing such information
serves to communicate acquired knowledge more

effectively and to ensure that beneficial information is factored into
planning, work processes, and activities. Lessons learned can be based on
positive experiences or on negative experiences that result in undesirable
outcomes.

Although DOD has taken action to gather and disseminate lessons learned
and best practices on general acquisition issues, these efforts generally
do not focus on outsourcing or include sharing the lessons learned from IT
outsourcing projects across the department. Specifically, a number of DOD
Web sites provide guidance, lessons learned, and best practices related to
general acquisition issues. However, using these sites to locate specific
information on IT outsourcing best practices and lessons learned can be
time- consuming and difficult because so many topics and information

sources are provided. Specifically, MHS/ ITO Help Desk project officials
said that searching numerous Web sites to get relevant information to
address questions and concerns about outsourcing IT services can consume
hours. For example, when we entered the keywords *IT outsourcing* and
*best practices* into the search feature on the Office of the
Undersecretary for Defense of Acquisition, Technology, and Logistics site,
ACQWeb (www. acq. osd. mil) in early March, it provided us with links to
1,251 documents. Likewise, lessons learned covers so many topics that it

is difficult to search for an applicable IT lesson. For example, when we
used the phrase *lessons learned in IT outsourcing* no documents were
identified on the ACQWeb, but when we inserted *lessons learned* and *IT
outsourcing* links to more than 1,700 documents were produced.

One DOD Web site, Share A- 76!, was established to address one of our
previous recommendations, 38 that is, to establish a framework for
identifying and analyzing best practices and lessons learned from
competitive sourcing studies and disseminating them DOD- wide. Share A76!
promotes the sharing of best practices and lessons learned related to one
form of outsourcing that was conducted under the OMB Circular A- 76
competitive sourcing process. Among other things, the site contains
guidance, links to other relevant sites, sample documents, and a best

38 U. S. General Accounting Office, DOD Competitive Sourcing: Lessons
Learned System Could Enhance A- 76 Study Process, GAO/ NSIAD- 99- 152
(Washington, D. C.: July 21, 1999).

practices library that communicates field staff experiences and advice
about the Circular A- 76 process. A NIMA project official said that NIMA*s
staff routinely accesses the Web site because it contains a wealth of
information on policies, procedures, lessons learned, and links to other
outsourcing sites. The DOD analyst responsible for Share A- 76! estimated
that the site receives about 12,000 visits per month and said that on the
basis of E- mails and anecdotes, user satisfaction is favorable. However,
this Web site is specific to the OMB Circular A- 76 process, which may not
apply to other types of outsourcing. For example, Circular A- 76*s policy
pertains to public/ private competitions and requires that the final
evaluation

between the government and the private sector be based exclusively on
cost. DOD acquisition and IT officials acknowledged that there is no
mechanism in DOD to easily share and leverage lessons learned relating to
outsourcing IT services. However, these officials agreed that a
departmentwide effort to identify, capture, and disseminate lessons
learned and leading practices of projects with experience in carrying out
IT outsourcing could offer valuable insights and new ideas that would
benefit others. Moreover, officials from three of the projects in our
review told us that there is value

in collecting and disseminating the knowledge acquired from IT outsourcing
projects in a systematic manner across the department.

Each of the projects in our review identified knowledge and experience
gained from their approaches to outsourcing IT services that could offer
insights and practices for other ongoing and future projects to consider.
For example:

 MHS developed specific guidance and lessons learned for implementing a
performance- based incentive contract for help- desk operations.  The
Department of the Navy*s NMCI project has developed a series of

lessons learned related to transitioning to the provider that is being
shared within the NMCI community; one example was that all personnel
should be available during scheduled testing and deployment.  NIMA has
experience in contracting techniques emphasizing a

partnering approach with providers to refine requirements and establish a
common understanding of costs. In addition, a departmental IT outsourcing
knowledge- sharing approach could include links to information about other
government agencies* IT

outsourcing projects. For example, our 2002 report on desktop outsourcing
includes an extensive discussion of lessons learned by agencies that have
implemented this type of IT services outsourcing. 39

Developing an effective lessons learned activity is not easy. For example,
NMCI officials said that for a lessons learned initiative to be effective,
a process must exist that is clearly understood by everyone and allows
capturing and sharing of knowledge to occur with minimum effort. Other
challenges in developing an effective lessons learned process were
outlined by the Share A- 76! analyst. The analyst stated that only a small
number of site users have contributed lessons learned to the Share A- 76!
Web site, which she attributed, in part, to the amount of time and effort
needed to

document and obtain agreement by all levels of the organization on the
lessons learned. In addition, the analyst stated that there is reluctance
to share negative lessons, and often the review and approval process
sanitizes best practice information so that it becomes too general to be
most helpful to users. Such challenges can be overcome by executive- level
support. Indeed, DOD acquisition and IT officials stated that for lessons
learning activities to be effective, senior management must devote support
and

resources to the effort. This is consistent with our prior work, which
showed that knowledge can be effectively shared only when employees are
given adequate time as well as established places where they can actually
transfer knowledge. 40

Last year, we outlined a generic lessons learned process that could be
used to guide the development of such a process for outsourcing IT
services. 41 Although the mechanism or processes used to collect, share,
and

disseminate lessons learned may vary, in general such a process comprises
four main elements: collection, verification, storage, and dissemination.
The collection process involves the capture of information through
structured and unstructured processes. Verification serves to verify the
correctness and applicability of lessons submitted. The storage aspect of
lessons learned usually involves incorporating the lessons into an
electronic database for the dissemination and sharing of information,

including the ability to conduct information searches. The final element,
and the most important, is the dissemination of lessons learned, since

39 GAO- 02- 329. 40 GAO- 02- 195. 41 GAO- 02- 195.

lessons are of little benefit unless they are distributed and used by
people who will benefit from them. Lessons can be *pushed,* or
automatically delivered to a user, or *pulled* in situations where a user
must manually search for them. Lessons can also be disseminated with an
assigned priority descriptor, which denotes the risk, immediacy, and
urgency of the

lessons learned content. Conclusions The projects in our review
substantially implemented leading commercial

practices for outsourcing IT services, which has increased each project*s
probability of success. Capturing how these projects operationalized
leading commercial practices could help other IT services outsourcing
projects succeed. Although currently there is no such DOD- wide mechanism,
such as an electronic tool, to easily share and leverage lessons

learned, DOD IT and acquisition officials agreed that a departmentwide
effort to identify, capture, and disseminate lessons learned could offer
valuable insights and new ideas that would benefit others. Lessons learned
that are pragmatic and easily accessible could give DOD managers a more
informed understanding of the important issues to be addressed when making
outsourcing decisions, as well as the factors to be considered to help
ensure the success of these endeavors. DOD managers can also benefit from
lessons learned on the basis of negative experiences. The projects in our
review were well into implementation, and therefore, at this late stage,
we see little advantage for them to revisit practices that were not
implemented. Nevertheless, an electronic tool for capturing and
disseminating lessons learned would allow the rest of DOD to benefit from
the negative consequences and increased risks associated with those
practices that the projects did not implement. Developing a lessons
learned mechanism is not easy; thus, senior

management support and resources are keys to success. Without such support
driving the capture and dissemination of lessons learned, DOD is losing an
opportunity for wider application of leading practices and thus better
ensuring that its IT outsourcing efforts are successful.

Recommendations To assist DOD organizations in planning and implementing
outsourcing projects for IT services, we recommend that the Secretary of
Defense

direct the Undersecretary of Defense for Acquisition, Technology, and
Logistics, working in conjunction with the Assistant Secretary of Defense
for Command, Control, Communications, and Intelligence (C3I), to provide

senior management support and adequate resources to develop and implement
an electronic tool to capture and disseminate examples and lessons learned
from actual IT outsourcing projects. These examples and lessons learned,
at a minimum, should include the results of our review of the five
projects discussed in this report.

We also recommend that the Secretary of Defense direct the Undersecretary
of Defense for Acquisition, Technology, and Logistics, working in
conjunction with the Assistant Secretary of Defense for Command, Control,
Communications, and Intelligence (C3I), to ensure that the method used to
gather information for this electronic tool incorporate the main elements
of a lessons learned process* namely, collection, verification, storage,
and dissemination.

Agency Comments and In written comments on a draft of our report, signed
by DOD*s Principal

Our Evaluation Director, Deputy Assistant Secretary of Defense (Deputy
Chief Information

Officer), the department partially concurred with our recommendations.
Specifically, DOD agreed that capturing lessons learned in the development
and implementation of its IT outsourcing initiatives is important to
continually improve its outsourcing methods and results achieved. The
department also stated that before deciding on a specific method to
achieve this aim, it intends to explore a variety of mechanisms that could

be used. In particular, the department stated that it currently has
several processes and communities of interest that collect and disseminate
lessons learned in other areas, which are logical starting points for
determining the best path forward. DOD*s written comments are reproduced
in appendix IX.

We agree that it is prudent to explore various alternatives to leveraging
lessons learned from DOD*s IT services outsourcing experiences. Our
recommendations are not prescriptive as to the electronic method to be
used to capture and disseminate lessons learned. Therefore, the
department*s plan to explore various alternatives is consistent with our
recommendations.

We are sending copies of this report to the appropriate congressional
committees, the Secretary of Defense, and the Director of the Office of
Management and Budget. We will also provide copies to others upon

request. In addition, the report will be available at no charge on the GAO
Web site at http:// www. gao. gov.

If you have any questions on matters discussed in this report, please
contact me at (202) 512- 3439 or Linda J. Lambert, Assistant Director, at
(202) 512- 9556. We can also be reached by E- mail at hiter@ gao. gov and
lambertl@ gao. gov, respectively. Other contacts and key contributors to
this report are listed in appendix X.

Randolph C. Hite Director, Information Technology Architecture

and Systems Issues

Appendi Appendi xes x I

Objectives, Scope, and Methodology Our objectives were to determine (1)
the extent to which selected Department of Defense (DOD) information
technology (IT) services outsourcing projects use leading commercial
practices as specified in our framework and (2) whether DOD is sharing
lessons learned from its IT outsourcing projects across the department.

To determine the extent to which selected DOD outsourcing projects for IT
services use leading commercial practices, we identified the practices in
our November 2001 report on leading commercial practices 1 that (1) are
typically applied at the project level and (2) were verifiable through
documentation and interviews. Because DOD did not centrally maintain a
list of outsourcing projects for IT services, we asked the department to
identify candidate projects for our evaluation. From this list, we
selected the following five projects for our review: (1) Air Force
Kirtland Air Force Base*s Command, Control, Communications, and Computer
(C4) Services project; (2) Army Network Enterprise Technology Command*s
(NETCOM) Total Army Communications * Southwest Asia (TAC- SWA) project;
(3) Military Health System/ Information Technology Organization (MHS/ ITO)
Help Desk project; (4) Department of the Navy*s Navy and Marine Corps
Intranet (NMCI) project; and (5) National Imagery and Mapping Agency
(NIMA) Information Technology/ Information Services (IT/ IS) project. We
chose each project on the basis of the following criteria: (1) no more
than

one project from each military service and two agencies, (2) illustrative
example of DOD IT outsourcing, (3) dollar value greater than $10 million,
and (4) enough time elapsed for services to have been delivered and
performance measured.

At our request, each project completed a self- assessment on whether and
how it implemented leading commercial practices. We reviewed the agency
self- assessments and accompanying documentation and interviewed the

appropriate agency project officials to verify whether the practices were
followed. In addition, we interviewed representatives from each of the
providers associated with these projects. We also researched additional
information on commercial practices in our November 2001 guide 2 and the
challenges the federal government faces in

1 U. S. General Accounting Office, Information Technology: Leading
Commercial Practices for Outsourcing of Services, GAO- 02- 214
(Washington, D. C.: Nov. 30, 2001). 2 GAO- 02- 214.

implementing them. Specifically, we performed a literature search, which
included reviewing reports issued by leading research firms, such as
Gartner, Inc., and Giga Information Group, Inc. In addition, we
interviewed

representatives from industry organizations that have an interest in
outsourcing IT services, including the Coalition for Government
Procurement; the Information Technology Association of America; the
Professional Services Council; and Acquisition Solutions, Inc.

To determine whether DOD is sharing lessons learned from its IT
outsourcing projects across the department, we identified and reviewed
various approaches that DOD currently uses to capture and disseminate such
information. This included identifying and reviewing various Web sites and
performing key word searches on these sites to identify lessons learned
for outsourcing IT services. We also interviewed applicable DOD
acquisition and IT officials. Finally, we identified and reviewed a
generic lessons learned process contained in our January 2002 report on
the National Aeronautics and Space Administration*s lessons learned
mechanisms. 3

We performed our work at the Army*s NETCOM in Ft. Huachuca, Arizona; the
MHS/ ITO Help Desk project office in Falls Church, Virginia; the NMCI
Director*s office in Crystal City, Virginia; and NIMA*s headquarters in
Bethesda, Maryland. We conducted our review between May 2002 and early
March 2003 in accordance with generally accepted government auditing
standards.

3 U. S. General Accounting Office, NASA: Better Mechanisms Needed for
Sharing Lessons Learned, GAO- 02- 195 (Washington, D. C.: Jan. 30, 2002).

Projects* Implementation of Phase I:

Appendi x II

Determine Sourcing Strategy In the first phase of our outsourcing
framework, the client organization determines whether internal capability
or external expertise can more effectively meet its IT needs. The purpose
of this sourcing strategy is to achieve the optimal balance between
internal and external capabilities, activities, processes, and services to
ensure that strategic business objectives are achieved at the lowest risk.
Among the factors that an organization should evaluate in crafting this
strategy are technology,

business, financial, and personnel requirements and whether it has skilled
business and IT managers. In addition, according to Gartner, Inc., sound
sourcing decisions depend on whether IT organizations (1) know and
understand their business priorities, (2) are prepared to invest in some
skills and divest others, and (3) identify and assess trade- offs. 1 The
six practices in this phase that we used to evaluate the five projects are

as follows:  Use third- party assistance with experience in a variety of
sourcing

arrangements when formulating a sourcing strategy.  Incorporate lessons
learned from peers who have engaged in similar

sourcing decisions.  Estimate impact of sourcing decision on internal
organization.  Benchmark and baseline productivity of internal services
before making

the final sourcing decision. 1 Gartner, Inc., Five Tough Questions About
Skill Sourcing, Research Note SPA- 13- 2537 (Mar. 28, 2001).

 Determine the business reasons for outsourcing IT.  Determine reasons
for outsourcing IT that can improve the

organization*s ability to use and manage technology. Figure 6 shows that
the implementation of the practices by the five projects in our review was
uneven.

Figure 6: Project Implementation of Phase I: Determine Sourcing Strategy

Note: Not applicable* The practice was not relevant to the project*s
particular circumstances. No* The agency did not implement the practice.
Limited* The agency fully implemented some but not all aspects of the
practice and did not take alternative actions that fully satisfied the
practice. Yes* The agency fully implemented the practice or took an
alternative action that fully satisfied its intent.

Table 4 provides detailed information on whether and how each project
implemented each of the six practices in this phase.

Table 4: Summary of Projects* Use of Phase I Practices Did the project use
the practice? Air Force C4 Practice Services Army TAC- SWA MHS/ ITO Help
Desk Navy NMCI NIMA IT/ IS

Use third- party No* According to Air No* According to a

Yes* For example, Yes* The Navy Yes* NIMA

assistance with Force project

TAC- SWA project MHS worked with the

employed a employed a

experience in a officials, they official, the Army Department of the
contractor with

contractor with variety of sourcing decided that they decided that it had

Interior*s GovWorks sourcing expertise sourcing expertise.

arrangements when had sufficient inhouse sufficient expertise

Program and the and contacted other formulating a

expertise, and in- house since the

Defense Acquisition government entities sourcing strategy.

the project had no TAC- SWA contract University staff to

about their funding available to

was a consolidation obtain expertise on

experiences. employ a contractor. of three existing

sourcing strategies. However, the former

contracts. Also, MHS employed functional area chief

a contractor with told us that using

sourcing expertise. third- party assistance would have been beneficial
because the requirements would

have been better written. According to this official, the

requirements had to be substantially rewritten 2 years after contract
award.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice Services Army TAC-
SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Incorporate lessons Yes* Primarily from Yes* According to a

Yes* Primarily from Yes* Navy officials Yes* A NIMA learned from peers

other Department of TAC- SWA project

industry peers and stated that there contractor provided who have engaged
in

Defense (DOD) official, it used

MHS*s prior helpdesk were no peers that

the agency with a similar sourcing

projects that lessons learned from

function. had engaged in

report on industry decisions. implemented Office

its prior contracts for similar sourcing

best practices. In of Management and

these services and decisions because addition, NIMA held Budget (OMB)

another similar Army no other outsourcing discussions with the

Circular A- 76 contract.

project was of as National Security

policies. large a scale as Agency on its

NMCI. However, the outsourcing effort. Navy did talk to members of private

industry and the National Aeronautics and Space Administration on their
more limited efforts.

Estimate impact of Yes* The Air Force No* The Army did Yes* MHS assessed

Yes* The Navy Yes* NIMA

sourcing decision on estimated that there

not analyze the the staff and financial

performed an performed an internal organization. would be substantial
impact on its internal impact of its sourcing assessment of

assessment of internal impact, such organization

decision. staffing and other

staffing and other as to its staff, due to because, according

impacts, such as impacts, such as

its decision to to the TAC- SWA cost, related to its

cost, related to its outsource.

program manager, sourcing decision. sourcing decision.

NETCOM was outsourcing a function that was already contracted out. a

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice Services Army TAC-
SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Benchmark and No* The Air Force

No* The Army did Limited* MHS Limited* The Navy

Yes* A NIMA baseline productivity did not implement

not perform a attempted to had a private- sector

contractor baselined of internal services this practice

benchmark or establish a baseline

firm benchmark its the existing NIMA

before making the because, according

baseline analysis of its prior

environment against environment and final sourcing

to project officials, an because, according environment but,

seven large public benchmarked it to decision.

executive decision to the TAC- SWA according to project and private

peers. was made to

program manager, officials, available organizations. In outsource the C4

NETCOM was documentation was

addition, the Navy services following the

outsourcing a limited to historical and the Marine

policies contained in function that was

trouble ticket Corps performed an

OMB Circular A- 76. already contracted workload data;

analysis at a sample Accordingly, Air

out. therefore, this of representative Force officials stated baseline was
a best

locations to obtain a that such analyses

estimate. In addition, baseline. However, would not have MHS project
officials

this baseline did not affected the final

stated that a include an sourcing decision.

contractor performed assessment of the a benchmark Department of the
analysis, but they did

Navy*s legacy not provide applications since supporting

project officials documentation.

decided to rely upon an inventory developed in addressing the Year 2000
challenge. The Navy subsequently found that it had substantially
underestimated its number of legacy applications. According to NMCI
program officials, this underestimation contributed to the transition
period slipping from 2- 1/2 years to 3- 1/2 years.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice Services Army TAC-
SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Determine the Yes* To achieve cost Yes* To address

Yes* To achieve Yes* To have private Yes* To obtain business reasons for

savings and to shift shortages in military

increased industry capitalize

improved customer outsourcing IT.

military personnel to personnel to perform

productivity and infrastructure

services and other work. its mission.

customer satisfaction improvements that

decreased costs. and to decrease were needed to Also, this function costs.

quickly and securely was determined to share knowledge be a commercial
around the globe.

function under the Federal Activities Inventory Reform Act. b

Determine reasons No* According to

No* According to Yes* To improve IT Yes* To improve, for

Yes* To achieve for outsourcing IT Kirtland Air Force

TAC- SWA project expertise and example, system

better IT that can improve the Base officials, the Air

officials, improving its knowledge.

security, management

organization*s ability Force decided to ability to use and

interoperability, performance.

to use and manage hold a public/ private

manage technology reliability, and technology.

competition following was not a factor in

network response. OMB Circular A- 76 determining its policies; therefore,
outsourcing strategy. improving its ability to use and manage technology
was not a factor in determining its outsourcing strategy. Source: GAO.

a Having had the activity previously performed by a contractor does not
obviate the need to estimate the impact of a sourcing decision on the
internal organization* there are still risks involved, such as the
potential disruption of services during the transfer to the new
contractor. In fact, the staff from the incumbent contractor did not
transition to the TAC- SWA provider, and the provider had difficulty
filling

these slots within the schedule outlined in the contract. b The Federal
Activities Inventory Reform Act of 1998 requires federal agencies to
prepare and submit

to OMB, by June 30 of each year, inventories of the commercial activities
performed by federal employees.

Project*s Implementation of Phase II: Define

Appendi x III

Operational Model Critical to the successful outsourcing relationship is
an operational model for guiding the structure of the contract and the
plans for transition. In defining the operational model, client
organizations formalize executive leadership, team composition, client
responsibilities, and operating relationships between the client and
provider. The operational model helps the organization to compare its
plans with the expectations that were set as the initial decision to
outsource was made and to ascertain whether these plans will enable the
organization to meet those objectives. An important aspect of the
operational model is an explicit understanding of how the client
organization plans to communicate its needs and provide feedback to the
provider. In addition, communication between the business and IT offices
within the client organization is always critical. This is particularly
true in the case of outsourcing because the IT service provider is outside
the client organization and disconnects are more likely to occur.
Therefore, organizational processes to facilitate good communication are
critical.

The 13 practices in this phase that we used to evaluate the five projects
are as follows:  Establish executive leadership for IT to facilitate the
outsourcing

initiative.  Continually communicate/ clarify outsourcing objectives,
while

correcting misinformation that affects the organization.  Establish a
core group of people who will be involved in all phases of

outsourcing.  Select a person involved in the negotiation of the contract
to manage the

outsourcing relationship.

 Create and define a contract management structure with operational
points of contact and managers.

 Define the role of internal IT managers and business leaders.  Ensure
that the right skills are in place to support the outsourcing

relationship.  Establish a point of contact high in the provider
management structure

for elevating provider performance concerns.  Have provider establish an
on- site support team to serve as liaison

between client and provider.  Train provider on client business
environment and goals.  Select or develop standard tools for managing the
relationship.  Use third- party assistance to take advantage of expertise
from a variety

of outsourcing arrangements in defining the operational model.  Ensure
that the provider management team has prior experience in the client*s
field of business.

Figure 7 illustrates that the five projects in our review largely
implemented the practices.

Figure 7: Project Implementation of Phase II: Define Operational Model

Note: Not applicable* The practice was not relevant to the project*s
particular circumstances. No* The agency did not implement the practice.
Limited* The agency fully implemented some but not all aspects of the
practice and did not take alternative actions that fully satisfied the
practice. Yes* The agency fully implemented the practice or took an
alternative action that fully satisfied its intent.

Table 5 provides detailed information on whether and how each project
implemented each of the 13 practices in this phase.

Table 5: Summary of Projects* Use of Phase II Practices Did the project
use the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Establish executive Yes* The initiative Yes* NETCOM used Yes* MHS used its

Yes* In 1999, the Yes* The NIMA

leadership for IT to was planned by a

the existing existing leadership Navy established a

enterprise facilitate the

steering group made leadership in its structure and

program executive transformation

outsourcing initiative. up of representatives

offices of operations processes. This office for IT primarily

directorate formed a from various major

and logistics to structure includes a

to support the NMCI strategic sourcing

offices at Kirtland Air provide executive

program executive outsourcing effort.

office to provide Force Base. leadership. office, steering

The Congress later executive oversight

Subsequent to committee, and directed the Navy to

of the effort. contract award, the program review identify a single Air
Force established board.

individual whose sole a functional area responsibility would chief to
manage the

be to oversee and initiative, and a direct the NMCI Lieutenant Colonel
program. As a result,

was appointed to be in February 2002, the responsible for this Navy
established the project.

NMCI Director*s Office to take over responsibility for NMCI.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Continually Yes* The Air Force

Yes* According to Yes* MHS provided

Yes* The Navy Yes* NIMA created

communicate/ clarify provided TAC- SWA project

updates to the Chief established an action

an internal Web site outsourcing communication

officials, the Information Officer

collaboration team to post information

objectives, while through, for example, applicable NETCOM

(CIO) and program structure to involve

on the outsourcing correcting briefings and

officials were review boards, the Navy community

project, distributed misinformation that meetings with

informed about the provided briefings to in the NMCI

periodic global affects organization. employees.

initiative and the deputy surgeon

communications electronic mails, and

provided comments general, and

process. Also, the held town hall

on the draft published questions

NMCI Information meetings. performance work and answers.

Bureau initiated statement. TAC- SWA

press conferences, project officials also briefings, site visits, brief
new informational commanders on the

pamphlets, and contract before they promotional material. are transferred
to the In addition, the NMCI military theater.

Director and other staff provide numerous briefings

and presentations to commands, industry gatherings, and government
officials. Finally, Web sites

and Web- based collaboration sites were established to facilitate
communications.

Establish a core Yes* The core group

Yes* According to Yes* According to

Yes* The program Yes* NIMA

group of people who included the TAC- SWA project

program officials, the executive office for IT

established the will be involved in all

contracting officer, officials, the core core group included

and the NMCI Strategic Sourcing phases of

functional area chief, group included representatives from Director*s

Office, which was outsourcing.

and manpower representatives from the MHS program

organization dedicated to specialist. logistics, operations, executive
office,

comprise the core managing and and resource

MHS IT program group responsible for

facilitating the management as well offices, military

managing the outsourcing initiative.

as the contracting department chief

outsourcing initiative. officer.

information officers* offices, and Interior*s GovWorks

organization, which provided contracting services.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Select a person Yes* The contracting Yes* The contracting Yes* According
to

Yes* The primary Yes* A lead

involved in the officer. a officer. MHS program contracting officer.
contracting officer.

negotiation of the officials, the chair of contract to manage the contract

the outsourcing evaluation committee relationship. and the individual in
charge of transition

are responsible for managing the outsourcing

relationship. Create and define Yes* The structure

Yes* The overall Yes* Responsibility Yes* The primary

Yes* NIMA has contract

was defined by the responsibility for the for contract

contracting officer appointed an management

roles and contract rests with management is

establishes operational point of

structure with responsibilities of the

the contracting jointly held by the

procedures and contact for each of operational points of contracting
officer

officer. The contracting officer, controls necessary the seven functional
contact and

and her staff. contracting officer*s who is part of

for effective areas being managers.

representative acts Interior*s GovWorks

contractual oversight outsourced, which as a liaison between

organization, and the of the NMCI initiative are documented in a

the government and MHS program office,

and has a matrix *rules of

the contractor. which provides dayto- relationship with the

engagement* day oversight of

NMCI Director. The agreement between

the provider. primary contracting

the government and officer certifies

the contractor. contracting officer representatives for the NMCI contract
to provide technical coordination efforts.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Define the role of Yes* Kirtland Air Limited* TAC- SWA

Yes* MHS*s help- Yes* The NMCI

Yes* NIMA internal IT managers

Force Base did not project officials

desk performance execution plan transformation teams

and business initially define the

explained the roles of assessment plan describes the roles of

for the activities leaders.

roles of its internal IT the principal staff

defines the roles of internal and external being outsourced

and business involved with this

various groups and organizations that

define the roles of managers. However, project but generally individuals
directly or indirectly their managers and subsequent to did not have
associated with the

affect the leaders. contract award,

supporting program. management of

Kirtland Air Force documentation NMCI and explains in Base established a

defining these roles. detail the duties and

CIO committee and responsibilities of the configuration control

program executive board comprising the

office for IT and the functional area chief

NMCI program (Kirtland*s CIO) and

management offices. representatives from various business areas. Although
not exclusively devoted to the C4 Services project, these groups help
provide direction to the project.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Ensure that the right Yes* According to

Yes* According to Yes* According to

Limited* The Yes* According to

skills are in place to Air Force officials, the TAC- SWA project MHS
program

Director, NMCI, has NIMA, its

support the the right skills are in

officials, the Army officials, the TriService staff that report
transformation teams

outsourcing place. has created several Management

directly or are provide the skills

relationship, positions to ensure

Program Office, matrixed with him

necessary to support including those that the right skills

GovWorks (the that are responsible

the outsourcing dealing with

are in place to Interior organization for performing all but relationship.

manage the that provides

two of these  contract

outsourcing contracting

functions. management,

relationship, assistance to MHS), Specifically, at this

 financial including a

and a contractor time, NMCI does not

management, contracting officer, collectively employ have staff assigned 
IT management,

legal advisor, the skill sets needed to support teaming

 negotiation operations and to support the and interpersonal strategies,

logistics personnel outsourcing

relationships and  teaming and

as well as a relationship.

relationship interpersonal NETCOM unit

management. The relationships, commander.

Navy recognizes the  project

According to Army need for these skills management, and

TAC- SWA officials, and is taking, or  relationship each position

plans to take, various management.

employs an actions to obtain individual with the

these skills. necessary skills to support the outsourcing relationship.
Establish a point of Yes* According to

Yes* The provider Yes* The provider Yes* The NMCI

Yes* The contact high in the the contracting

established a established an Director interacts

transformation team provider officer, concerns can

program manager as executive program

directly with the charters define

management be addressed to the the point of contact

manager as the point provider*s program

procedures for structure for

senior vice president for elevating

of contact for executive on a

addressing issues, elevating provider at the provider

concerns. elevating concerns.

regularly scheduled including possibly performance

headquarters. basis; such meetings elevating concerns to concerns.

would include any the provider*s performance issues.

general manager. Have provider

Yes* The provider Yes* Site managers

Yes* The provider*s Yes* The provider Yes* The rules of

establish an on- site has established an have been

transition plan establishes site leads engagement define

support team to on- site support team, established in

identifies the on- site that remain on- site

the on- site support serve as a liaison which is led by the

accordance with the support team.

as the location goes team for each major

between client and on- site manager.

contract. through the cutover

NIMA location. provider.

to NMCI.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Train provider on Limited* Prior to

Not applicable* The Yes* MHS officials

Limited* The Navy Not applicable*

client business contract award,

Army contract was a stated that they did not provide

According to the environment and

Kirtland Air Force consolidation of prior

provided some training on its

project*s contracting goals.

Base provided an contracts for this training to the

business officer, such training

orientation session in activity, and the provider on their

environment and was not necessary which they discussed

winning bidder was business

goals; instead it because the senior their operations,

one of the incumbent environment. In

relied on the NMCI officers of the

business contractors.

particular, MHS design reference

provider are all environment, and

trained and certified mission document,

recent NIMA goals. Subsequent to the provider*s staff on which was
included management contract award,

the agency*s with the request for

employees and were specific Kirtland Air

application systems. proposals (RFP). already

Force Base In addition, the This document

knowledgeable of government provider had

defined the NMCI these areas. For personnel were

previous MHS operational

example, among responsible for aiding

experience. environment. those who

the provider during However, a provider transitioned to the

the transition period. official stated that provider was a However,
provider

although the former director of the

officials stated that document was

hydrographic they did not receive useful, it did not

topographic center of adequate training, identify Navy

the Defense Mapping which made the enterprisewide

Agency, a transition period operations. predecessor agency more difficult.
According to NMCI

to NIMA. program officials, the provider somewhat mitigated this problem
by hiring several highly

knowledgeable staff from the Department of the Navy shortly

after contract award. Select or develop

Yes* For example, Yes* NETCOM uses Yes* According to

Yes* The Navy is Yes* NIMA used

standard tools for the revised contract

a monthly report MHS officials, the using a balanced various tools,
managing the outlines the use of from the contracting

program office and scorecard process to

including twice- a- day relationship (e. g., specific software to

officer*s the provider have provide Navy and

performance reports performance

help manage the representative to

selected, for Marine Corps for the operational scorecards, provider*s
evaluate the provider.

example, an leadership with help- desk function enterprise resource

performance. enterprise resource information to judge

and monthly management

management how well NMCI is performance reports.

system). system.

supporting the missions and strategies of the

department.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Use third- party Yes* Although thirdparty

No* The Army Yes* For example, No* The Navy Yes* NIMA

assistance to take assistance was believed that it had

MHS contacted the believed it had

employed advantage of not used at the onset sufficient expertise

Interior*s GovWorks sufficient internal

contractors to expertise from a

of the C4 Services in- house.

Program, Defense experience and develop contractual variety of outsourcing

project, a person with Acquisition expertise. roles and assist in

arrangements in prior experience in University, and evaluating the

defining the managing IT

others. Also, MHS contract.

operational model services contracts for employed a (i. e., defining roles

the Air Force was contractor with

and responsibilities). brought in to provide

sourcing expertise. advice. He was subsequently put in charge of the
project.

Ensure that the Yes* Provider

Yes* Vendor past Yes* The request for

Yes* According to Yes* It was expected provider management

performance was quote laid out MHS*s

NMCI program that most of the

management team capabilities and

part of the evaluation evaluation criteria for officials, the Navy

provider*s workforce has prior experience experience in

criteria during source selecting a provider,

required bidders to would be composed in the client*s field of providing
IT services

selection and the which included prior provide information

of former NIMA business.

were evaluation provider was an experience in health

on their prior employees. Among factors in the

incumbent contractor care.

experience in related those who technical evaluation with NETCOM.

business fields. For transitioned to the

process. example, the provider was a

solicitation required former director of the

bidders to hydrographic

demonstrate topographic center of experience the Defense Mapping managing
a similar

Agency. effort of 100, 000 or more seats at one time. b

Source: GAO. a The Air Force*s initial contract was not negotiated because
Kirtland Air Force Base used an OMB

Circular A- 76 2- step, sealed bid process. However, the contracting
officer was involved in the bid process as well as in managing the winning
contractor. b This threshold could be met through multiple efforts, but at
least one had to include at least 20,000

seats.

Project*s Implementation of Phase III: Develop

Appendi x IV

the Contract Phase III focuses on the development of the contract, which
defines the legal terms of the relationship between client and provider.
While other phases in the outsourcing process describe the need for mutual
trust and a close, flexible working relationship, this phase focuses on
the development of the contract, which is the foundation on which a
working relationship will be built. A well- written contract helps the
outsourcing organization meet its requirements while allowing the service
provider to make a fair profit. It sets the expectations for service
levels, delivery of essential services, and continuous improvement and is
intended to protect the interests of all parties.

The 16 practices in this phase that we used to evaluate the five projects
are as follows:  Base performance requirements on business outcomes. 
Include measures that reflect end- user satisfaction as well as technical

IT performance.  Review and update performance requirements periodically.
 Require the provider to meet the minimum performance in each

category of service.  Require the provider to achieve escalating
performance standards at

agreed- upon intervals.  Incorporate sufficient flexibility so that
minimum acceptable performance can be adjusted as conditions change, as
the provider

becomes more adept at satisfying customer demands, and as improvement
goals are achieved.

 Use service- level agreements (SLA) 1 to clearly articulate all aspects
of performance, including management, processes, and requirements.

 Client and provider work together to define the appropriate number of
SLAs and appropriate structure for each.  Specify circumstances under
which the provider is excused from

performance levels mandated by master service agreements.  Client and
provider work together to identify SLAs for which

compensation is based, while additional SLAs may be defined to manage
performance.

 The contract should include clauses for (1) determining pricing
structures; (2) performing customer satisfaction surveys and using the
results to redefine performance levels; (3) terminating the contract; (4)
resolving disputes in a timely manner; (5) taking work away, without
penalty, from provider for nonperformance; (6) declaring a significant
event that can lead to a change in the contract; (7) defining performance
requirements; and (8) conducting regularly scheduled meetings.

 Consider setting up a master services agreement under which all
arrangements between client and provider operate.

 Include the appropriate representation from each major organizational
unit on the contract negotiation team.

 Specify the use of volume purchases to obtain optimal discounts.  Use
third- party assistance in negotiating and developing the contract.  Sign
the contract after contract negotiations and final vendor selections. As
figure 8 shows, the five projects largely implemented the practices. 1
SLAs define the agency*s expectations and are used to track and measure a
contractor*s performance.

Figure 8: Project Implementation of Phase III: Develop the Contract

Note: Not applicable* The practice was not relevant to the project*s
particular circumstances. No* The agency did not implement the practice.
Limited* The agency fully implemented some but not all aspects of the
practice and did not take alternative actions that fully satisfied the
practice. Yes* The agency fully implemented the practice or took an
alternative action that fully satisfied its intent.

Table 6 provides detailed information on whether and how each project
implemented each of the 16 practices in this phase.

Table 6: Summary of Projects* Use of Phase III Practices Did the project
use the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Base performance Ye s *T h e Yes* The

Yes* Industry Yes* The

Yes* Performancebased requirements on performance performance

standards and performance requirements

business outcomes. requirements are requirements are

metrics were used to requirements are

are in the contract, contained in the

outcome based and base performance included in the including those
contract and are

stated in terms of the requirements on

contract and are related to customer based on business

level of operation and business outcomes.

based on business satisfaction, process

outcomes. For maintenance

outcomes. SLAs improvement, quality

example, the services required.

establish the control, and timely requirements define

According to TACSWA performance and accurate

the percentage of officials, standards and completion of

system availability multiple trips to

service quality for all requirements. needed to support

overseas locations types of NMCI seats.

users. were made to identify

the performance requirements.

Include measures Limited* The Air

Limited* The Army*s Yes* MHS measures Yes* The SLAs Yes* The SLAs

that reflect end- user Force measures

contract includes the service provider*s include measures for

contain technical IT satisfaction as well

include technical IT technical IT

technical IT technical IT

performance as technical IT

performance. performance performance and performance and

measures. In performance.

However, end- user measures but not

uses an electronic customer

addition, after a satisfaction end- user satisfaction customer
satisfaction

satisfaction. In function is measures are limited performance survey to
assess the

addition, the contract transitioned, the to the percentage of

measures. For quality of help- desk

requires the service contractor must

complaints received example, the contract services. Satisfaction

provider to measure provide a plan that

and do not measure includes a

rates and number of and report on overall

includes measures overall customer

requirement for helpdesk survey responses customer satisfaction

for customer satisfaction. Project

services, but are tracked and used

with NMCI services. satisfaction, which

officials did not know does not include for both evaluation the provider
is why overall end- user measures related to

and incentive contractually

satisfaction customer satisfaction

payments (positive obligated to meet.

measures were not associated with such and negative).

established. services. According

to the project official that developed the performance work statement in
the

contract, NETCOM did not include such customer satisfaction measures
because it did not think that it was necessary to have a performance
standard for that service.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Review and update Ye s*The Yes* The contract is Yes* According to

Yes* Performance Yes* According to

performance performance reviewed and

MHS program reviews and

NIMA, requirements requirements

requirements were modified as needed officials,performance adjustments are

are updated as periodically.

revised about 2 years on the basis of requirements are

ongoing. For needed or once a

after contract award. additions or changes

updated as required example, the Navy is year when the

to such and reviewed on an

in the process of government is

requirements. annual basis before

refining its SLAs to assessing whether to

According to TACSWA the decision is made

ensure that they exercise its annual and provider

to execute the option adequately reflect option.

officials, changes are year on the contract.

technical coordinated between

performance and the government and

customer satisfaction the contractor before needs. they are finalized.

Require the provider Yes* The minimum Yes* The contract

Yes* The minimal Yes* The SLAs Yes* The provider

to meet the minimum performance is

defines minimum acceptable provide the basic

must meet the performance in each

defined in the performance

performance criteria level of service the performance

category of service. contract.

requirements. the contractor must

contractor must standards as

meet are based upon deliver for every

specified in the commercial industry NMCI seat.

contract, SLAs, and standards and are

NIMA*s performance defined in the

assessment plan. contract.

Require the provider No* According to Not applicable*

Yes* The contract No* According to

Yes* The contract to achieve escalating Kirtland Air Force

According to a TAC defines negative,

NMCI program allows for the

performance Base officials, they SWA project official,

acceptable, and officials, the Navy redefinition of service

standards at agreed did not include

this practice was not positive incentive

expected vendors to levels. In addition,

upon intervals. escalating

applicable because ranges associated develop their pricing

according to NIMA performance NETCOM believed with escalating proposal
assuming 8

program officials, the standards because that the performance

performance years b of providing

need for escalating the focus of this standards in the

standards. service in performance

contract was on contract were

accordance with the standards will be meeting

already high. SLAs included in the

evaluated on an as requirements at the

Provider officials also RFP.

needed basis as well least cost. a stated that the as during the
performance

semiannual award requirements in the

fee analysis period contract were high. and during the The Army expected,

annual review on and the contractor whether to exercise agreed, to meet

the contract option. these standards However, such immediately. escalating
standards

have not yet been identified.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Incorporate sufficient No* The contracting

Not applicable* Yes* According to

Yes* The contract Yes* According to

flexibility so that officer said that the According to a TACSWA

MHS program includes a provision

NIMA, the minimum acceptable requirements should project official,

officials, the contract requiring the provider

performance performance can be

be built into the NETCOM believed incorporates

to submit an annual measures are

adjusted as contract, which can

that the performance sufficient flexibility to plan for technology

designed to first conditions change,

be modified if standards in the adjust minimal

refreshment and stabilize NIMA*s IT

as the provider necessary.

contract were acceptable deployment. This

environment and becomes more adept already high, and

performance as plan is to include

then to be adjusted at satisfying

provider officials conditions change.

proposed revisions to to bring minimum

customer demands, agreed. According to For example, the the contract and
an performance into and as improvement

the contracting contract includes

estimate of changes alignment with goals are achieved.

officer, the military incentives for the

in performance that industry best

theater where service provider to

would result. practices. Also, the

performance is introduce new

contract indicates delivered is not an

capabilities and new that the SLAs may be area where anything

services within the further defined

less than meeting the scope of the

throughout the defined standards is

performance work contract life. acceptable.

statement without further competition. Use SLAs to clearly

Yes* The SLA Yes* The TAC- SWA

Yes* The contract Yes* According to Yes* The contract

articulate all aspects defines the

project did not use defines the NMCI officials, SLAs

incorporates SLAs, of performance, requirements, SLAs, but the

requirements, cover all aspects of which articulate including

processes, and who contract addresses

service performance provider performance management,

is responsible for the management,

standards, performance. For requirements and

processes, and meeting the

processes, and thresholds, example, the E- mail take effect when the
requirements. requirements.

requirements objectives, and

services SLA function is associated with the

metrics as well as the includes location and transitioned to the project.
requirements for

frequency of the provider.

quarterly service, performance

management reviews categories, the

and quality performance assurance plans.

measures, and methods of measurement.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Client and provider Yes* Although the Yes* SLAs are not Yes* MHS did not

Yes* The SLAs were Yes* According to

work together to original SLA was used but, according

work with the service developed by the

NIMA and service define the prepared by the Air

to TAC- SWA and provider to determine

Navy with the help of provider officials, appropriate number Force, the
revised provider officials, the number and a third- party

they used a of SLAs and

SLA included they work together

structure of these contractor. According partnering

appropriate structure contractor input and on any contract

requirements. to the NMCI Deputy contracting approach for each.

concurrence. modifications dealing However, according

Director for to jointly define, with requirements

to MHS and provider Enterprise

develop, and and performance officials, they have Operations, it would

structure the SLAs. levels. In addition, worked together on have been the
partnering clause

revisions to these inappropriate to work in the contract

requirements. with individual

emphasizes a mutual competing commitment

contractors before between government the contract was and industry to
work awarded. However, as a team.

NMCI staff and the service provider are now working together on SLA
revisions.

Specify Yes* The contract

Yes* The contract Yes* For example, Yes* Contract terms

Yes* The contractor circumstances under

includes a clause includes clauses that

the provider*s and conditions can be excused from which the provider is

that the contractor is the provider is

proposal, which is specify the situations

meeting performance excused from excused in the event

excused from incorporated as part where the service

levels with the performance levels

of government delay performance levels if

of the contract, provider does not

permission of the mandated by master of work.

there are states that during

have to meet the contracting officer if service agreements.

government delays transition periods,

SLAs. For example, circumstances occur or factors beyond its

metrics will not be during transition, the beyond the service

control. reported for incentive

service provider provider*s control. and penalty

does not have to purposes. As a meet the result, according to performance
levels program officials, the

set by the SLAs. provider would be excused from required performance
levels when baseline requirements are being established for new
applications.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Client and provider Yes* According to Not applicable*

Yes* Before the Yes* Meetings were

Yes* NIMA and the work together to Kirtland Air Force

SLAs are not used. contract was held with potential service provider work
identify SLAs for

Base officials, the awarded, MHS did

bidders to obtain together in

which compensation service provider had not work with the their input in
defining transformation teams

is based, while input on the revised

contractor to identify the SLAs, including to develop SLAs, additional
SLAs may

SLAs, including SLAs for which discussions of

including those be defined to those for

compensation is compensation. In

affecting manage

compensation. based. However, addition, Navy and

compensation. performance.

according to MHS the provider are now and provider officials, refining the
SLAs, after award, there including those have been instances

affecting in which they have compensation. worked together on refining the
SLAs,

including those affecting compensation.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

The contract should Limited* All clauses

Yes* All contract Yes* All contract

Yes* All contract Yes* All contract

include clauses for are included in the

clauses are included. clauses are included. clauses are included. clauses
are included. contract, except a  determining pricing

clause pertaining to structures;

customer satisfaction  performing

surveys. customer satisfaction surveys and using the

results to redefine performance levels;  terminating the

contract, including early terminations;  resolving contract

disputes in a timely manner;  taking work away, without penalty, from the
provider for nonperformance;

 declaring a significant event that can lead to a change in the contract;
 defining

performance requirements; and  conducting

regularly scheduled meetings. Consider setting up

Yes* According to Yes* According to

Yes* According to Yes* According to Yes* According to

master services Kirtland Air Force

TAC- SWA project MHS program

NMCI program NIMA, the contract is

agreement under Base officials, the

officials, the contract officials, the contract

officials, the contract considered the which all

contract, which is considered the is considered the

is considered to be a master services

arrangements incorporates the master services master services

master services agreement.

between client and contractor proposal,

agreement. agreement.

agreement. provider operate.

governs all arrangements between the government and the contractor and is

considered the master services agreement.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Include appropriate Not applicable* The Ye s *

Yes* Staff from Yes* The contracting

Yes* The NIMA representation from C4 Services contract

Representatives from affected program

team was built with contract negotiating each major

was awarded under a the field and

management offices experts from each of

team included organizational unit on

sealed bid process headquarters

and the chief the major systems

representatives from contract negotiation and was not

commands and the information offices

commands. Also, the each major

team. negotiated.

contracting office were on the

source selection organizational unit.

participated in negotiation team. evaluation board

contract negotiation. consisted of more

than 50 people from various commands.

Specify the use of Not applicable*

Not applicable* Yes* Call bands c are Yes* The contract

Yes* NIMA has volume purchases to

According to Kirtland According to TACSWA

used to obtain includes volume authorized the

obtain optimal Air Force Base

project officials, optimal discounts on discounts. For

provider to use discounts.

officials, such volume such volume the number of calls

example, seat government sources, purchases were not purchases were not

being fielded to the moves, adds, and such as the Federal relevant to this
relevant to this

help desk. changes are

Supply Schedule, in contract.

contract. purchased in bulk procuring products, only, because prices

services, and for these actions are

supplies related to lower when

this contract. purchased in groups

According to the of 250.

contracting officer, this authorization was made so that the provider
could take advantage of discounts available to

the government. Use third- party

No* This was not No* NETCOM

Yes* Private- sector Yes* The Navy used

Yes* NIMA obtained assistance in done because the Air officials believed
that

and other private- sector firms

help from (1) private negotiating and

Force staff believed the Army*s government

to assist in contractors to help

developing the that it had adequate designated organizations

developing the develop the contract.

expertise available contracting agency provided assistance.

overall NMCI concept performance work in- house.

had adequate and negotiation statement, SLAs,

expertise and strategies as well as

and award fee plan; experience.

in drafting the (2) another agency

contract documents. on a particular

contract technique; and (3) a privatesector firm to compare the vendor*s
proposal with industry best practices and trends and to attend some
negotiation meetings to answer questions.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Sign contract after Yes* The contract

Yes* The contract Yes* The contract

Yes* The contract Yes* The contract

contract negotiations was signed after

was signed after was signed after

was signed after was signed after

and final vendor negotiations and final

negotiations and final negotiations and final negotiations and final

negotiations with the selections.

vendor selection. vendor selection.

vendor selection. vendor selection.

selected vendor. Source: GAO.

a Requiring a provider to meet escalating requirements and focusing on
achieving results at the least cost are not mutually exclusive goals. As
the provider becomes more familiar with the client organization, it may be
able to exceed the original performance requirements at the same, or
possibly lower, cost. b The NMCI contract was subsequently extended to 7
years, with an option for an additional 3 years.

c Call bands are call volume ranges used to determine contractor pricing.

Projects* Implementation of Phase IV: Select

Appendi x V

the Provider( s) Critical to the success of any project to outsource IT
services is the identification of potential providers and the ultimate
selection of a provider( s) that will best meet the needs of the
organization. Developing a strategy that will lead to the selection of the
*right contractor* is especially important in a performance- based
acquisition. 1 The overall success of the outsourcing project requires the
contractor to understand the

performance- based approach, know or develop an understanding of the
organization*s requirement, have a history of performing exceptionally in
the field, and have the processes and resources in place to support the
mission.

The seven practices in this phase that we used to evaluate the five
projects are as follows:

 Conduct research on the state of the market, vendors, and technology
before defining vendor selection criteria.

 Identify and evaluate various sourcing solutions (e. g., single vendor,
multivendor, and alliance).  Define a process for selecting vendors to be
providers.  Define vendor selection and evaluation (acceptance) criteria
at the

outset. 1 An Interagency- Industry Partnership in Performance, Seven Steps
to Performance- Based Services Acquisition, Benchmark Version (January
2002).

 When issuing an RFP, identify services with expected performance levels
and define client and provider roles and responsibilities.

 Use third- party assistance with expertise in a variety of outsourcing
arrangements when selecting provider( s), including developing the RFP.

 Conduct due diligence activities to verify vendor capabilities before
signing the contract.

As figure 9 illustrates, all five projects largely implemented the
practices.

Figure 9: Project Implementation of Phase IV: Select the Provider( s)

Note: Not applicable* The practice was not relevant to the project*s
particular circumstances. No* The agency did not implement the practice.
Limited* The agency fully implemented some but not

all aspects of the practice and did not take alternative actions that
fully satisfied the practice. Yes* The agency fully implemented the
practice or took an alternative action that fully satisfied its intent.

Table 7 provides detailed information on whether and how the projects in
our review implemented each of the seven practices in this phase.

Table 7: Summary of Projects* Use of Phase IV Practices Did the project
use the practice? Air Force C4 Practice Services Army TAC- SWA MHS/ ITO
Help Desk Navy NMCI NIMA IT/ IS

Conduct research on Yes* The Air Force Yes* A Commerce Yes* MHS had a

Yes* The Navy and a Limited* NIMA staff

state of the market, advertised in the

Business Daily notice third- party contractor private- sector firm and a
contractor

vendors, and Commerce Business was published perform market performed
market

researched technology before Daily and interested identifying TAC- SWA

research. research to, for

preferential providers defining vendor

parties submitted requirements to example, help define before defining

selection criteria. statements of interested parties. In

the market conditions vendor selection capabilities. This was

addition, a draft RFP and vendor selection criteria. NIMA

done before the Air was issued before

criteria. The Navy program officials

Force identified the the final solicitation to

also held meetings noted that they vendor selection solicit comments

with leading limited their analysis criteria.

from industry that companies that had

to such providers might affect the outsourced IT

because they had requirements. services on an

previously decided According to the

enterprisewide level. on a strategy to

contracting officer, Finally, more than directly convert their comments
received 200 companies

IT/ IS activities to a from industry were

participated in the preferential provider.

incorporated into the July 1999 NMCI

final solicitation, Industry Day where applicable.

Conference, which Additionally, a

informed companies preproposal about the NMCI conference was

vision, requirements, conducted in the

and procurement overseas operations

strategy. location to inform industry representatives

about issues and the procurement strategy.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice Services Army TAC-
SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Identify and evaluate Yes* As outlined in

Yes* A multivendor Yes* According to an

Yes* The Navy Yes* NIMA

various sourcing OMB Circular A- 76, approach has been

MHS program evaluated singlevendor

considered the solutions (e. g.,

the Air Force used in the past,

official, MHS and

sourcing solutions single- vendor,

evaluated privatesector which the Army evaluated continuing

multivendor allowed under OMB

multivendor, and and internal opted not to

with the status quo, approaches. A joint Circular A- 76,

alliance). government staff

continue. Instead, using another federal

Navy and Marine including the direct proposals to satisfy

NETCOM chose to agency*s help- desk

Corps team conversion to a

the C4 services consolidate its services as part of a

determined that preferential provider,

requirements. requirements and

cross- services single- vendor point of holding a contractor oversight

agreement, and contact for

public/ private by choosing a singlevendor contracting with a

accountability and competition, or

solution. commercial firm.

responsibility was obtaining an agency

According to the critical to mission

cost comparison MHS program office,

success. waiver. NIMA MHS chose a singlevendor

decided that directly solution to

converting the consolidate

selected IT functions requirements and to a preferential performance

procurement source a accountability. was the lowest risk to its mission
and NIMA employees.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice Services Army TAC-
SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Define a process for Yes* As explained in

Yes* The contract Yes* MHS used a

Yes* The NMCI RFP Limited* NIMA*s selecting vendors to

the invitation to bid, was awarded on

selection process used a negotiated decision to use a

be providers (e. g., the Air Force

*best value* that included the

commercial items phased direct

issuing an RFP and followed the two- step considerations of

prequalification of 13 evaluation process. conversion of its

prequalifying process outlined in technical and industry leaders in The
vendor

IT/ IS functions to a vendors).

OMB Circular A- 76 management both medical and evaluation criteria

statutory, preferential for public/ private capabilities, past commercial
helpdesk included technical procurement vendor

competitions. performance, and

operations. approach, limited its vendor

price, as defined in MHS also issued a

management plan, selection process to

the solicitation. request for comment

small business the identification of and a request for

utilization, past potential Alaska

quote, which defined performance, and Native Corporation

vendor evaluation price. companies capable

and selection criteria of performing the

on the basis of requirements. NIMA technical approach,

decided to take this past performance, approach because it key personnel
believed that it would qualifications,

avoid schedule organizational delays and mission experience, and risk that
it thought price.

would result from taking other approaches. In addition, NIMA

program officials believed that the preferential provider approach would
maximize the retention of institutional

knowledge whether employees stayed at NIMA or transitioned to the
provider.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice Services Army TAC-
SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Define vendor Yes* The criteria Yes* The solicitation

Yes* Vendor Yes* The final RFP

Yes* NIMA selection and

were included in the included vendor evaluation and

required bidders to evaluated the past

evaluation invitation to bid.

evaluation and selection were based

have implemented performance and

(acceptance) criteria selection criteria. The upon industrydefined and
provided service

financial capabilities at the outset.

source selection help- desk

to at least 100,000 of Alaska Native evaluation plan criteria. MHS used a
seats, of which Corporation describes the

multistep process 20, 000 were to be on companies. evaluation process.

that included an the same contract. In According to NIMA,

assessment of addition, bidders only two companies

minimum were evaluated on

had demonstrated qualifications, their technical

successful evaluation of written

approach, performance in technical and cost

management plan, service environments proposals, and oral

small business similar to NIMA*s presentations for utilization, past
requirements. qualified vendors. performance, and

However, to be able price. Finally, vendors to fully meet NIMA*s in the
competitive requirements, these

range had to provide vendors formed a

demonstrations of joint venture, which technical network was awarded the
operations centers to

contract. verify their ability to achieve required

service levels set forth in technical proposals.

When issuing an Yes* The services Yes* The RFP

Yes* The request for Yes* The NMCI

Yes* Because this RFP, identify services and performance

identified the quote identified

solicitation was a sole source

with expected levels were included government and

services and documents identified contract, an RFP was performance levels

in the original contractor roles and expected the required services

not issued. However, and define client and performance work
responsibilities, and

performance levels and expected according to NIMA provider roles and

statement included system operational and specified a performance levels.

and provider officials, responsibilities. in the invitation to and
availability

performance- based, These documents, they worked jointly to

bid. requirements. incentivized, sharedrisk along with the

refine the relationship with

contract, define the requirements and

the service provider. client and provider

expected It also defined client

roles and performance levels

and provider roles responsibilities.

and to define client and responsibilities. and provider roles and
responsibilities in the contract.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice Services Army TAC-
SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Use third- party No* This was not

No* The Army Yes* A privatesector Yes* A private- sector

Yes* NIMA used a assistance with done because the Air

believed that it had contractor, the

firm assisted with private- sector firm expertise in a variety

Force believed that adequate Defense Acquisition

market research and and the NIMA

of outsourcing adequate in- house government

University, and developing vendor

Acquisition Center to arrangements when expertise was

experience and GovWorks (an

pass/ fail criteria. support the

selecting provider( s), available.

expertise covering Interior organization

Another contractor evaluation of the

including developing technical, resource offering procurement

assisted the Navy vendor proposal.

the RFP. management, and

services to with developing SLAs

contracting areas. government

and the technical agencies) provided evaluation of assistance in this
vendors. area.

Conduct due Yes* The Air Force

Yes* NETCOM Yes* Minimum

Yes* As part of due Yes* NIMA

diligence activities to performed a evaluated the

vendor qualifications diligence, the Navy

researched both verify vendor technical evaluation

contractor*s financial were established and

ensured that all parent companies of

capabilities before of the vendor*s

and past evaluated early in the bidders had relevant

the new joint venture signing the contract. capabilities. Also, as
performance

process for selecting experience corporation to ensure

part of its due information before

the provider. implementing large

that they had diligence activities,

signing the contract. seat management

financial and the Air Force

contracts. In addition, technical backing.

reviewed the past performance,

debarred list and including reference DOD*s Central

checks, was a Contractor Registry.

source- selection evaluation factor. Source: GAO.

a Preferential procurement programs are special commercial source
programs, such as Federal Prison Industries and the workshops administered
by the Committee for the Purchase from the Blind and Other Severely
Handicapped under the Javis- Wagner O*Day Act.

Projects* Implementation of Phase V: Appendi x VI

Transition to Provider( s) In this phase, client organizations transfer
responsibility of IT functions to one or more providers. A clear
definition of responsibilities among the various parties and the careful
consideration of employees* needs matched against the organization*s needs
enable both the client and provider to focus on execution and give staff
confidence in their future employment. If

the contractor is assuming responsibility for functions previously
performed by federal employees, it is especially important that the
organization communicate a clear transition process. Without such
communication, an outsourcing project can be negatively affected if
misinformation and mistrust ensues.

The 11 practices in this phase that we used to evaluate the five projects
are as follows:  Communicate a clear transition process to all key
players from both

client and provider organizations.  Clearly communicate to employees what
is going to happen and when it

is going to happen.  Establish a client transition team with
representatives from across the

organization to facilitate the transition.  Place the transition under a
single program manager.  Create client/ provider transition teams to
address short- term transition

tasks as required.  Recognize that it takes time to effect transition and
plan accordingly.

 Encourage the transition of staff to the provider, where appropriate,
using bonuses, stock options, and other appropriate methods.

 Develop employee- retention programs and offer bonuses to keep key
people, where appropriate.

 When consistent with organizational objectives, assist employees who do
not want to transfer in finding other jobs, either within an organization
or at another organization.

 Document key information to preserve organizational knowledge in the
event that one or more providers change.

 Use change management strategies to help client employees deal with the
transition. Figure 10 demonstrates that the five projects in our review
were largely using the practices.

Figure 10: Project Implementation of Phase V: Transition to Provider( s)

Note: Not applicable* The practice was not relevant to the project*s
particular circumstances. No* The agency did not implement the practice.
Limited* The agency fully implemented some but not all aspects of the
practice and did not take alternative actions that fully satisfied the
practice. Yes* The agency fully implemented the practice or took an
alternative action that fully satisfied its intent.

Table 8 provides information on whether and how the projects implemented
each of the 11 practices in this phase.

Table 8: Summary of Projects* Use of Phase V Practices Did the project use
the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Communicate a clear No* There were two

Yes* The provider Yes* The contract

Yes* The Navy Yes* Several NIMA

transition process to transition plans developed a phase- in delivery
order and developed numerous

transformation teams all key players from prepared, one by the plan that
addressed

the provider*s guides and Web sites were formed,

both client and government and one

personnel issues, the transition plan laid to communicate with

consisting of both provider by the provider.

integration of added out a transition

the NMCI community client and provider

organizations. However, the

locations, updating process. In addition, about site, technical,

personnel, to develop government*s plan plans, and finalizing

a kickoff meeting was and personnel

a single transition contained conflicting new subcontract/ held between
the

transition processes, plan. The plan was direction. In addition,

sponsorship client and provider to

procedures, and developed to provider officials said arrangements. This

communicate the tasks.

formalize that the two separate

phase- in plan was transition process to

communications with transition plans were

discussed at a all key players.

all key players. not well coordinated

postaward and that the Air

conference, attended Force did not have a

by representatives strong advocate to from both the client ensure that the

and provider. transition process was well planned and executed. Both
provider and Kirtland Air Force Base officials acknowledged that there
were problems with the transition

process, including incorrect information about upgrades that would be made
before the transition

and staff morale problems that hampered knowledge transfer to the
provider.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Clearly communicate Yes* The Air Force

Limited* According Yes* The

Limited* The Navy Yes* With help from

to employees what is kept its employees

to TAC- SWA officials, government task has used its normal

a contractor, NIMA going to happen and informed of the

the Army manager for the TriService chain of command to developed a

when it is going to outsourcing project communicated with communicate

communications happen.

through town hall its employees, but

Infrastructure transition

plan. The plan meetings and they did not provide Management Office

information, but included town hall internal briefings. As

documentation held regular found that the

meetings, global part of this process, supporting any

meetings with MHS implementation was

E- mail messages, employees were

communication program officials to

uneven. As a result, and a Web site

informed about the efforts. However, the discuss transition some staff did
not

posting that provided reduction- in- force provider was

issues. The MHS know current

employees with process and the

required to submit a program office also information about

access to transition procedure for those

phase- in plan, which periodically sent how NMCI would

information. interested in being

included a section on global E- mails to

affect them until the transitioned or

communicating with MHS staff on

provider was ready to reassigned.

the incumbent transition activities

contact them contractors* staff. and MHS leadership regarding their

made on- site visits to possible transition to San Antonio, the

the contractor. location of the helpdesk However, according

function. to the NMCI

Director*s office, this problem was somewhat mitigated by the provider*s
Web site that provides transition

information to all NMCI customers/ users.

Establish client Ye s *T h e Yes* According to

Yes* MHS program Yes* The

Yes* NIMA*s transition team with government*s TAC- SWA officials,
officials for the TriService headquarters- level

transition team representatives from transition team the contracting

transition team included across the

included personnel officer*s Infrastructure

consists of representatives from organization to

from the functional representative Management Office representatives from

various offices within facilitate the

areas being assisted the provider

established transition the NMCI Director*s NIMA to help transition.

outsourced (i. e., with the transition.

teams with Office and the

facilitate the communications and The officials also representatives from
program

transition. IT services) as well stated that Army the former TriService
management office.

as personnel from technical points of Medical Also, every site has a

other offices. contact at each site

System Support transition team made

also helped facilitate Center contract.

up of customer and the transition to the provider personnel. new contract.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Place transition Yes* The Air Force

Yes* The Army Yes* The transition Yes*NMCI*s Yes* The transition

under single program established a single

designated the was assigned to the Director has overall

process is under a manager.

program manager, contracting officer*s

government task

responsibility for the single program who is referred to as

representative to manager in the TriService

transition.

manager. the functional area oversee the chief.

transition. Infrastructure

Management Office. Create

Yes* During the Yes* Personnel from Yes* Client/ provider

Yes* NMCI and the Yes* An integrated client/ provider

transition phase, both the Army and transition teams were provider
designated

NIMA transformation transition teams to management from

provider formed established to

staff to work together team comprising

address short- term both the government three transition address short-
term on short- term NIMA and provider

transition tasks as and the provider met

teams, one for each transition tasks. The transition issues.

staff, was required. weekly to track the

site location. These teams met weekly to responsible for

progress of the teams worked with

address any addressing short

transition. the contracting

transition issues. term tasks, including

officer*s (1) transition

representative and planning, (2)

the provider*s project resources and manager to deal with

recruitment, (3) transition issues.

program budget, and (4) contract development and costing.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Recognize that it Yes* The Air Force Yes* The contract

Yes* The contract Limited* The Navy

Yes* According to takes time to effect

allowed 60 days for allowed a 60- day

provided for a said that it had

NIMA, it has transition and plan

the transition. phase- in period.

transition period of initially

scheduled its seven accordingly.

30 to 90 days. underestimated the

functional areas to According to MHS

scope of the project be transitioned to the

program officials, the and the magnitude of

provider over a 3- to transition period the problems brought

5- year period. The lasted 90 days. on by legacy

provider will not applications and

assume associated responsibility for a information

given functional area assurance and until the NIMA cyber- security

transition review issues. According to board has given its NMCI program

approval to the officials, addressing provider*s turnover these problems
and

plan. completing the operational testing that was mandated by the Congress

subsequent to contract award led to the transition period being lengthened
from 2- 1/ 2 to 3- 1/ 2 years. NMCI officials also said the extension will
allow the Department of

the Navy to have time to operate NMCI as a fully transitioned enterprise
before having to decide whether to exercise

the contract option.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Encourage transition Yes* The Air Force Yes* The Army had

Yes* According to Yes* The NMCI

Yes* The contract of staff to provider,

staff were given the no personnel

provider officials, contract provides requires the provider where
appropriate, right to transfer to the expected to transition

they extended offers that displaced federal to actively recruit, using
bonuses, stock

contractor. It was left to the provider since

to employees from employees that hire, and make options, and other

up to the contractor the activity being the prior contractor

transition to the reasonable efforts to

appropriate methods. to decide whether to

outsourced had and some MHS staff. provider under retain NIMA staff. If

offer incentives or previously been employment the provider meets

not, which it chose contracted out. openings as a result certain target

not to do. However, the

of NMCI will be given thresholds for provider was

guaranteed 3 years recruiting NIMA staff, interested in

of employment with it receives a retaining existing the provider, a 15
monetary incentive. contractor staff and percent salary According to the
had a program to

increase, and a signon provider*s general recruit them.

bonus. manager, the company offered

signing bonuses to NIMA employees that wanted to transition.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Develop employeeretention No* According to

Not applicable* Yes* The

Yes* NMCI program Yes* According to

programs Kirtland Air Force

Since this activity government task

officials noted that NIMA program

and offer bonuses to Base officials, they was previously

manager persuaded the Navy has human

officials, NIMA keep key people, did not develop an contracted out,

key government resources programs

categorized affected where appropriate. employee- retention

employee retention employees from the that would allow its

employees into three program because

of Army civilian Tri- Service Medical organizations a

tiers, on the basis of they did not believe personnel was not System
Support

range of options, whether they would

that they could offer relevant.

Center to temporarily including bonuses, to be allowed to incentives.

work at the TriService help keep key staff. transition to the

Additionally, the Navy provider or whether

Infrastructure directed all Navy and

there would be Management Office

Marine Corps restrictions on their until positions commands affected

activities if they did became available by NMCI to develop

transition. This with a contractor that

civilian personnel distinction was made

provides support to transition plans to to comply with

this office. In mitigate the impact of government conflict

addition, according to NMCI

of interest rules. a program official, to implementation on According to
the keep expertise in employees.

program manager, particular

NIMA also sought applications, key

personnel who employees were wanted to remain at transferred to the the
agency to help MHS program

monitor the executive office and

provider*s Tr i - Ser v i c e performance and Infrastructure

considered the Management Office. qualifications of those who wished to
serve in this role prior to deciding who

to retain.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

When consistent with Yes* Kirtland Air

Not applicable* Yes* According to

Yes* NMCI program Yes* For staff that

organizational Force Base officials

According to the MHS program

officials said that did not want to

objectives, assist stated that they have TAC- SWA

officials, the existing Navy civilian

transition, NIMA employees who do reduction- in- force

contracting officer, no government task

personnel programs provided various

not want to transfer in procedures to help

Army personnel were manager assisted are in place to assist

types of assistance, finding other jobs, place civilian displaced by the

government employees. They such as retirement

either within an employees in other

contract. employees with the noted that two Navy

planning, resume organization or at

Air Force jobs. Tri- Service Medical organizations found

writing, and another organization. Employees also

System Support other jobs within their

interviewing skills. In receive priority status Center staff in finding

command for their addition, NIMA

for other DOD job new positions with small number of

reported that if an openings. Further, other MHS

employees affected employee did not the base education

organizations. For by NMCI. Other Navy

want to transition to office provided example, he located organizations
are the provider, that it assistance with open positions and

also reassigning would consider readjusting outplacement;

contacted the affected personnel to

the resume writing; and, organizations.

other government employee*s work for those nearing jobs, where

assignment or retirement, planning

applicable. provide training to

advice. support the

individual*s placement within other areas.

Document key Yes* Kirtland Air

Yes* Documentation Yes* The Tri- Service

Yes* NMCI Yes* The executing information to

Force Base*s work produced by the Medical System

maintains a Web NIMA transformation preserve functions and provider,
including Support Center

portal with team was charged

organizational workload size are

maintenance logs, processes and

organizational with documenting the

knowledge in the documented in the become the property

procedures were knowledge

activities for each event that one or

contract. of the government.

captured during information. In functional area that is

more providers transition. The addition, a new

being outsourced. change. provider*s processes

Intranet site is being This includes and procedures are

developed with up- todate developing and currently being

interactive capturing operational captured. The

organizational processes and contract also

knowledge relevant procedures for each

includes to the many varied

of the functional requirements on

NMCI communities. areas.

documenting key information.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Use change Yes* Kirtland Air

Not applicable* The Yes* According to

Yes* The Navy has Yes* NIMA*s change

management Force Base held activity had been

MHS program provided employees

management strategies to help

town hall meetings previously contracted

officials, the affected by NMCI

strategies included client employees and internal briefings

out. government task

with a range of training seminars

deal with the to communicate the manager for the TriService

employment options offered by its human transition.

objectives of the and incentives.

resources office on outsourcing project

Infrastructure resume writing,

and the changes that Management Office

interviewing skills, would take place assisted government career
transition internally. It also had

employees with workshops, and one- on- one meetings

finding new positions employment trends.

with each employee and kept them that would be informed of transition
displaced.

issues. Weekly E- mails were also sent out to all affected employees.
Source: GAO.

Projects* Implementation of Phase VI: Manage

Appendi x VII

Provider( s) Performance The effectiveness with which the performance of
the provider( s) is managed* the focus of this phase* is critical to the
successful implementation of an outsourcing project. Indeed, according to
Gartner, Inc., an outsourcing project can be thwarted by poorly designed,
funded, and delivered processes for managing the delivery of services. 1
This firm also points out that an enterprise needs to retain the resources
to oversee

the planning and implementation of the IT services being delivered by the
provider to ensure that the contractor meets the client*s business needs
throughout the life of the agreement. 2 Moreover, frequent and clear
communication between the client and provider ensures that potential
problems are resolved before they cause disruptions. In addition,
performance reviews should take place regularly to keep the project on

course, measure performance levels, and make adjustments as necessary. 3
The 11 practices in this phase that we used to evaluate the five projects
are as follows:

 Consider incentives to motivate provider( s) to exceed performance
requirements.

1 Gartner, Inc., Retain Enough Resources to Manage Outsourcing Deals,
Research Note COM- 16- 8425 (June 17, 2002). 2 Gartner, Inc., Successful
Outsourcing Means Retaining Some Staff, Research Note COM18- 9692 (Dec.
18, 2002). 3 An Interagency- Industry Partnership in Performance, Seven
Steps to Performance- Based Services Acquisition, Benchmark Version
(January 2002).

 Use penalties to motivate provider( s) to meet performance requirements.

 Periodically undertake studies to assess (1) how the provider*s
performance compares with the value being delivered to similar clients and
(2) the extent to which the provider*s performance is improving over time.

 Schedule periodic working- level meetings with both the end- user groups
and the provider to review the provider*s performance.

 Conduct executive- level oversight meetings with the provider*s senior
management to review provider*s performance.

 Distribute performance data to stakeholders.  Reserve audit rights on
performance data supplied by the provider.  Ensure that the provider
measures and reports on performance.  Work with the provider to redefine
service levels, as appropriate.  Sample performance data frequently
enough to perform trend analysis

and to permit extrapolation based on historical data.  Allow employees
and possibly stakeholders to rate the provider on a

regular basis. As shown by figure 11, the five projects in our review
generally implemented the practices.

Figure 11: Project Implementation of Phase VI: Managing Provider
Performance

Note: Not applicable* The practice was not relevant to the project*s
particular circumstances. No* The agency did not implement the practice.
Limited* The agency fully implemented some but not all aspects of the
practice and did not take alternative actions that fully satisfied the
practice. Yes* The agency fully implemented the practice or took an
alternative action that fully satisfied its intent.

Table 9 depicts whether and how the five projects in our review
implemented each of the 11 practices in this phase.

Table 9: Summary of Projects* Use of Phase VI Practices Did the project
use the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Consider incentives No* The Air Force

No* There are no Yes* The contract Yes* The NMCI

Yes* The contract to motivate did not include

monetary incentives, includes incentives contract includes a

includes monetary provider( s) to exceed

incentives in the but, according to linked to each

one- time payment of incentives in performance

contract because the TAC- SWA officials, performance

$10 million if the accordance with an requirements.

agency expects the the fact that the requirement. Each

provider successfully award fee plan. It

contractor to meet agency will prepare a requirement has a

completes full also includes sharein-

the contract*s performance positive, negative, operational

savings provisions requirements without

evaluation report can and acceptable capability. The

to encourage incentives.

help motivate a performance range

contract also has process

Specifically, provider to meet that provides a basis incentives for

improvements. according to the

requirements. for monetary customer

former functional However, the

incentives (as well as satisfaction,

area chief, because contracting officer

penalties). information

the contract was said that incentives assurance, and small

issued under the might have been

business and small rules established by useful to motivate the

disadvantaged OMB Circular A- 76,

provider to exceed business the focus of the

performance participation.

project was on requirements. achieving cost savings, and incentives were
not included in the contract. As a result, he noted that the provider did
not have any incentive to be innovative or

creative.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Use penalties to Yes* The contract

Limited* The Yes* The contract

Yes* The contract Limited* NIMA did

motivate provider( s) contains monetary contract does not

includes monetary includes monetary not include monetary to meet
performance

penalties that are have specific penalties linked to

penalties in the form penalties in its requirements, such

linked to each of the monetary penalties

each performance of credits to the

contract. However, if as

performance linked to requirement. Each

agency if the the provider does not

requirements. If performance

requirement has provider fails to meet certain

 assess penalties for imposed, they would

requirements. positive, negative, perform to the levels

minimum failure to perform at reduce the amount of

However, the and acceptable specified in the

performance required individual

the payment owed to contract states that performance ranges

SLAs. standards, only its

service as well as the provider. unacceptable work that provide a basis
costs would be aggregate service

must be redone at for monetary covered.

levels; the provider*s penalties as well as

 apply penalties in expense, and, if the

incentives. the form of credit to defects and services the client;

cannot be corrected,  increase penalty for

the government may recurring deficient

reduce the contract*s performance; price to reflect the  hold back a

reduced value of the percentage of

services performed. provider*s pay for a TAC- SWA project particular
service

officials stated that until performance

monetary penalties requirements are

were not included in met; the contract because  refund a penalty if

the Army was the provider returns

concerned that to agreed- upon

contractors might not performance levels

bid on the contract, within a designated

and administrating period of time; and this type of contract  ensure that
the is more difficult.

provider will cover costs, but not profit, when a particular performance

requirement is not met.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Periodically Limited*( 1) This has

No* TAC- SWA Yes* The provider*s Limited*( 1) NMCI Limited*( 1) At this

undertake studies to not been done officials stated that

performance is only has

time, NIMA has not assess: (1) how the

because, according they address

reviewed monthly by performance data undertaken such

provider*s to a project official, performance issues the government task

since October 2002. studies because it performance

the Air Force was not at the time that they manager and in The Navy
intends to only recently

compares with the aware of similar

occur and that they quarterly do such a study but

transitioned functions value being delivered clients. have not performed
management reviews

has not established a to the provider.

to similar clients and (2) This is done any studies.

of the provider*s schedule for it. However, NIMA

(2) the extent to through periodic

performance. (2) The provider*s program officials which the provider*s
monitoring of

According to MHS improvements over

stated that they performance is performance by program officials, the time
are being intend to use a improving over time.

quality assurance results are compared monitored monthly. private- sector
firm to evaluators and with peers and

periodically compare through monthly reviewed for how the

the provider*s review meetings held provider has performance with by the
functional area

improved over time. those of similar

chief to identify and organizations.

address any (2) The provider*s problems that are

improvements over starting to occur.

time are being monitored monthly. Schedule periodic

Yes* Monthly Yes* According to

Yes* Periodic Yes* At each

Yes* Quarterly working- level

meetings are held at the provider and

working- level implemented site, the

performance meetings with both which the contractor TAC- SWA officials,
meetings are held provider and

management review the end- user groups briefs the functional they hold
daily

with the government government meetings are held

and the provider to area chief and other meetings to discuss

task manager and managers meet on a

and are attended by review the provider*s Kirtland Air Force any issues.

the program frequent, as- needed

NIMA and the performance.

Base management office

basis to review provider. In addition, representatives.

representatives to performance and

the provider hired an obtain help- desk resolve any issues.

ombudsman to feedback. obtain anonymous comments from NIMA staff and
management on its performance.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Conduct executivelevel Ye s*Air F orce Yes* According to

Yes* Executive Yes* The NMCI Yes* This is done

oversight schedules executivelevel

provider and TACSWA oversight meetings

Director and the through the quarterly meetings with the

oversight officials, are held quarterly

provider executive review meetings.

provider*s senior meetings whenever

quarterly in- progress with the provider*s meet weekly to management to
they are necessary. reviews are held with

senior management review performance review provider*s A recent meeting

the provider*s to review and discuss other performance.

included the corporate

performance. NMCI

president of the management and the

implementation provider.

battalion issues. In addition,

commander. the Department of

the Navy recently established an operations advisory board consisting of

Department of the Navy leadership and the provider. The goal of this board
is to focus senior

leadership on issues affecting NMCI in order to establish priorities and
make decisions.

Distribute Yes* Performance Yes* Thecontracting Yes* The help- desk Yes*
The NMCI

Yes* Performance performance data to data were not officer*s

monthly reports, Navy and Marine data are distributed

stakeholders. distributed to

representative including results Corps program

at quarterly review stakeholders at the distributes monthly

against performance managers are meetings, which beginning of the

performance reports metrics, are e- mailed

responsible for according to the project, but began to Army stakeholders

to stakeholders providing SLA

contracting officer, after complaints from for review. monthly.
performance data to

are attended by stakeholders about Performance issues their commands.

stakeholders. the need for such

or problems information. Metrics, identified are such as system

discussed with the reliability or *uptime,*

contracting officer*s are now provided to representative and stakeholders
on line.

the contracting officer, and corrective actions are planned to prevent

reoccurrence. Reserve audit rights

Yes* Addressed in Yes* Addressed in Yes* Addressed in

Yes* Addressed in Yes* Addressed in on performance data

the contract. the contract. the contract. the contract. the contract.
supplied by the provider.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Ensure that provider Yes* As required by Yes* Thecontracting Yes* The
contract

Yes* The contract Yes* The contract

measures and the quality control

officer*s requires the provider

specifies that the requires the provider reports on

plan in the contract, representative to submit monthly

provider must to submit data,

performance. the provider makes

receives and quarterly

measure and report including

information available performance performance reports

on whether its SLA measurements of

on product or service information in daily that are then performance goals

service, quarterly. quality and any

meetings with the reviewed and

are being met. actions needed to

provider*s project verified.

correct decreases in manager.

quality. In addition, according to the former functional area chief, the
provider provides performance information during monthly meetings.

Work with provider to Yes* This is done on Yes* According to

Yes* MHS and the Yes* The Navy and

Yes* Theredefinition redefine service an as- needed basis. TAC- SWA and

provider have worked

service providerhave

of service levels is levels, as For example, the

provider officials, together to redefine and continue to

provided for in the appropriate.

service levels were they have worked

SLAs. For example, refine NMCI SLAs to

contract. According redefined in April together to redefine

the first call closure

ensure more precise

to NIMA, the 2002.

performance performance metric

performance

redefinition of service requirements in the

was redefined to be measures and to

levels is expected to contract.

more realistic.

more accurately

occur in the future,

capture user

generally at the time satisfaction with the

that the annual

system. For

contract option is

example, a contract

exercised.

modification standardized many of the performance categories that dealt
with service availability.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Sample performance Ye s *A i r F o r c e Yes* The provider*s Yes*
Performance

Yes* The NMCI Yes* The contract

data frequently quality assurance proposal and quality

data are sampled Director monitors

requires data to be enough to perform evaluators monitor

control plan state and reported monthly

provider performance provided periodically trend analysis and to
performance in that it will provide and quarterly. They to identify trends
by for NIMA to use for

permit extrapolation accordance with the

various trend are analyzed and assessing providersupplied

trend and other types based on historical quality assurance analyses to
the

verified, including information of analyses. data.

surveillance plan and government.

any supporting data. as well as

the performance According to TACSWA In addition, the information from

requirements project officials, provider*s knowledge

Navy independent summary. Any trends

these data are management system verification and identified are

submitted to the provides analysis validation testing and

addressed in monthly contracting officer*s

and trend data to customer satisfaction status meetings.

representative for MHS.

survey results. In analysis. addition, NMCI program officials said

that they have requested funding to develop a performance measurement data
repository to support trend analyses.

Allow employees and Limited* The

Yes* The contracting Yes* The customer Yes* User surveys

Yes* The provider possibly

provider conducts officer*s

satisfaction survey is are used to measure

surveys NIMA stakeholders to rate annual customer

representative and one method used to

satisfaction with employees on their

the provider on a satisfaction surveys, on- site points of rate the
provider. In specific services satisfaction with the regular basis (e. g.,

but it is not required contact obtain

addition, according to such as E- mail

help- desk function. scorecards and

to submit the results feedback from

MHS program access, help desk,

The results of these quarterly report

to the Air Force. employees on a officials, stakeholder

etc., and overall user surveys are reviewed

cards). Kirtland Air Force

regular basis. input is also obtained satisfaction with the by NIMA during
Base also obtains ad

from program service provider*s quarterly meetings.

hoc feedback from management reviews

performance. In addition,

employees who, after and meetings held by stakeholders are also they
report a

the government task responsible for

problem, are asked manager. providing monthly

to provide assessments of information about

contractor how well the problem

performance. was addressed and Moreover, customer their overall
satisfaction is a satisfaction level. factor in determining

the amount of the semiannual award fee earned by the provider. Source:
GAO.

Projects* Implementation of Phase VII: Ensure

Appendi x VIII

Services are Provided Although outsourcing focuses on the provider*s
ability to perform, the client organization is ultimately responsible for
ensuring that services are provided and that end- user needs are met. The
previous phases addressed the extensive preparation that must precede the
provider*s assuming responsibility for the client organization*s services.
This phase encompasses practices designed to ensure that an effective
oversight approach is established. In addition, over the course of the
outsourcing project, the client learns more about the capabilities of the
provider, and

market conditions may change. As a result, it is important to monitor
service levels internally as well as maintain an external view of the
performance of other providers in order to identify opportunities to
improve and ensure that the outsourcing arrangement maintains its value to
the client.

The six practices in this phase that we used to evaluate the five projects
are as follows:

 Monitor the provider*s work to anticipate issues for resolution.  Make
sure that the provider uses the standard tools and processes

defined as part of the operational model.  Use provider performance data
to continuously improve processes.  Pursue improvement based on customer
satisfaction surveys.  Ensure that an appropriately empowered individual
from the client organization oversees the work.

 Set realistic time frames that are agreed to by the provider.

As illustrated by figure 12, the five projects in our review largely
implemented the practices.

Figure 12: Project Implementation of Phase VII: Ensure Services Are
Provided

Note: Not applicable* The practice was not relevant to the project*s
particular circumstances. No* The agency did not implement the practice.
Limited* The agency fully implemented some but not all aspects of the
practice and did not take alternative actions that fully satisfied the
practice. Yes* The agency fully implemented the practice or took an
alternative action that fully satisfied its intent.

Table 10 provides details on whether and how each project implemented each
of the six practices associated with this phase.

Table 10: Summary of Projects* Use of Phase VII Practices Did the project
use the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Monitor the Yes* Quality

Yes* Thecontracting Yes* The provider*s Yes* Navy Yes* NIMA reviews

provider*s work to assurance evaluators

officer*s work is monitored

representatives at the provider*s work in

anticipate issues for monitor the quality of representative through
monthly

each implemented quarterly program resolution.

the provider*s work reports monthly on reports and meetings site
(including

managementreviews and identify problems

contractor as well as through headquarters)

and as needed in or trends. The results performance. In

on- site meetings and monitor the

response to issues and any problems

addition, he meets readiness

provider*s work and raised by NIMA*s are reported to the

daily with provider assessments. identify issues. performance

contracting officer officials to discuss

According to NMCI monitoring officials. and the functional

performance results. program officials, the

area chief. NMCI Enterprise

ManagementSystem enables the government to monitor the performance of the
network and associated service delivery.

Make sure the Yes* The provider

Yes* According to Yes* According to

Yes* The Navy has Yes* The tools are provider uses the uses software tools

TAC- SWA project MHS program

monitored the documented in the standard tools and

to help monitor officials, the provider

officials, the provider service provider*s quality management

processes defined as system performance uses various

uses the standard use of standard tools plan. For example, part of the

against performance standard tools

tools outlined in the and processes. Tools the provider is using

operational model. standards to monitor

outlined in its contract. are being used to

a specific quality and and manage the

proposal. support legacy

process help- desk function.

migration, client improvement

installation, and helpdesk methodology. procedures.

Use provider Yes* Although Yes* Thecontracting Yes* For example, Yes* The
Navy uses Yes* NIMA*s performance data to Kirtland Air Force

officer*s MHS used provider

the provider*s data to performance

continuously improve Base does not rely

representative performance data to help determine ways monitors use

processes. on data provided by monitors provider

improve the to improve

provider data to its provider, its

performance data, customer satisfaction

processes, such as assess progress and

quality assurance and any potential survey process. to improve NMCI

ways to improve evaluators areas for

implementation performance.

continually monitor improvement are

procedures and the provider

incorporated in the timeliness of help performance. monthly report.

desk problem resolution.

(Continued From Previous Page)

Did the project use the practice? Air Force C4 Practice

Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS

Pursue improvement Limited* The Air

Yes* Surveys are Yes* Negative

Yes* Surveys of Yes* To pursue

based on customer Force does not

received by the feedback that is user satisfaction with

improvement, NIMA satisfaction surveys.

perform, or require contracting officer*s

received based on specific services and uses both customer

the provider to technical the results of the

overall satisfaction satisfaction surveys perform, customer

representatives and customer satisfaction

with provider and interviews with surveys. However, provided as input for

surveys is reviewed performance are

senior- level the provider has the contracting

by the MHS helpdesk used to identify areas

customers. initiated such officer*s

manager. needing surveys but is not

representative improvement. required to distribute

report. the results to the Air Force. Ensure that an Yes* The functional

Yes* Thecontracting Yes* The Yes* The NMCI

Yes* Thecontracting appropriately

area chief oversees officer*s government task

Director oversees the officer*s

empowered the work, and the representative is the

manager is the work and discusses representative and a individual from the

quality assurance empowered individual performance with the

staff of technical client organization evaluators support

individual from empowered to

NMCI provider monitors oversee the

oversees the work. this oversight effort.

NETCOM. oversee the work of

executive during work. the provider. weekly meetings.

Set realistic time Yes* Kirtland Air

Yes* Time frames Yes* The monthly

Yes* According to Yes* NIMA and the frames that are Force Base and the

for new tasks are performance reports

NMCI program provider have agreed

agreed to by the provider agree that

established by and quarterly officials, Navy*s on dates associated
provider.

reasonable time contract

program reviews set practice is to work with specific actions.

frames for modifications that are time frames that with the provider in
performance have agreed to by the have been agreed to

establishing now been set. Some provider.

by the provider. schedules.

revisions were made to them in April 2002. Source: GAO

Appendi x IX

Comments from the Department of Defense Note: GAO comments supplementing
those in the report text appear at the end of this appendix.

See comment 1. See comment 2.

See comment 3.

The following are GAO*s comments on the Department of Defense*s letter
dated April 8, 2003.

GAO Comments 1. Addressed in the Agency Comments and Our Evaluation
section of this report.

2. On January 29, 2003, NIMA granted the provider authorization to use
government sources in performing the IT/ IS contract that was limited to
*products, services, and supplies that support the performance of the

specific functional areas and miscellaneous items required under this
contract.* Accordingly, we did not modify this report.

3. Addressed in the Agency Comments and Our Evaluation section of this
report.

Appendi x X

GAO Contact and Staff Acknowledgments GAO Contact Linda J. Lambert, (202)
512- 9556 Staff

Season Dietrich, James Houtz, Anjalique Lawrence, Patricia Slocum, and
Acknowledgments Thomas Wright made key contributions to this report.

(310239)

GAO*s Mission The General Accounting Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities

and to help improve the performance and accountability of the federal
government for the American people. GAO examines the use of public funds;
evaluates federal programs and policies; and provides analyses,
recommendations, and other assistance to help Congress make informed
oversight, policy, and funding decisions. GAO*s commitment to good
government is reflected in its core values of accountability, integrity,
and reliability.

Obtaining Copies of The fastest and easiest way to obtain copies of GAO
documents at no cost is

through the Internet. GAO*s Web site (www. gao. gov) contains abstracts
and fulltext GAO Reports and

files of current reports and testimony and an expanding archive of older
Testimony

products. The Web site features a search engine to help you locate
documents using key words and phrases. You can print these documents in
their entirety, including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as *Today*s Reports,* on its
Web site daily. The list contains links to the full- text document files.
To have GAO e- mail this

list to you every afternoon, go to www. gao. gov and select *Subscribe to
GAO Mailing Lists* under *Order GAO Products* heading.

Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO

also accepts VISA and Mastercard. Orders for 100 or more copies mailed to
a single address are discounted 25 percent. Orders should be sent to:

U. S. General Accounting Office 441 G Street NW, Room LM Washington, D. C.
20548

To order by Phone: Voice: (202) 512- 6000 TDD: (202) 512- 2537 Fax: (202)
512- 6061

To Report Fraud, Contact:

Waste, and Abuse in Web site: www. gao. gov/ fraudnet/ fraudnet. htm

E- mail: fraudnet@ gao. gov Federal Programs

Automated answering system: (800) 424- 5454 or (202) 512- 7470 Public
Affairs Jeff Nelligan, Managing Director, NelliganJ@ gao. gov (202) 512-
4800

U. S. General Accounting Office, 441 G Street NW, Room 7149 Washington, D.
C. 20548

Report to the Subcommittee on Readiness and Management Support, Committee
on Armed Services, U. S. Senate

April 2003 INFORMATION TECHNOLOGY DOD Needs to Leverage Lessons Learned
from Its Outsourcing Projects

GAO- 03- 371

Letter 1 Results in Brief 2 Background 5 Projects Substantially Used
Leading Commercial Practices 14 Leveraging Lessons Learned DOD- wide

Could Assist Other DOD Projects 23 Conclusions 27 Recommendations 27
Agency Comments and Our Evaluation 28

Appendixes

Appendix I: Objectives, Scope, and Methodology 30

Appendix II: Projects* Implementation of Phase I: Determine Sourcing
Strategy 32

Appendix III: Project*s Implementation of Phase II: Define Operational
Model 38

Appendix IV: Project*s Implementation of Phase III: Develop the Contract
48

Appendix V: Projects* Implementation of Phase IV: Select the Provider( s)
59

Appendix VI: Projects* Implementation of Phase V: Transition to Provider(
s) 66

Appendix VII: Projects* Implementation of Phase VI: Manage Provider( s)
Performance 77

Appendix VIII: Projects* Implementation of Phase VII: Ensure Services are
Provided 86

Appendix IX: Comments from the Department of Defense 90 GAO Comments 93

Appendix X: GAO Contact and Staff Acknowledgments 94 GAO Contact 94 Staff
Acknowledgments 94

Tables Table 1: Definition of Phases for IT Outsourcing 8 Table 2: Profile
of Outsourcing Projects Reviewed 13

Table 3: Percentage of Practices Implemented, by Project 17 Table 4:
Summary of Projects* Use of Phase I Practices 34

Table 5: Summary of Projects* Use of Phase II Practices 41 Table 6:
Summary of Projects* Use of Phase III Practices 51 Table 7: Summary of
Projects* Use of Phase IV Practices 61 Table 8: Summary of Projects* Use
of Phase V Practices 69 Table 9: Summary of Projects* Use of Phase VI
Practices 80 Table 10: Summary of Projects* Use of Phase VII Practices 88

Figures Figure 1: Percentage of Practices Implemented in Each Phase 3
Figure 2: Roles of the Client and Provider in an Outsourcing

Relationship 6 Figure 3: GAO*s Framework for Outsourcing IT Services 7
Figure 4: Use of Leading Commercial Practices for Outsourcing of

IT Services, by Project 15 Figure 5: Percentage of Practices Implemented
in Each Phase 16 Figure 6: Project Implementation of Phase I: Determine
Sourcing

Strategy 33 Figure 7: Project Implementation of Phase II: Define
Operational

Model 40 Figure 8: Project Implementation of Phase III: Develop the

Contract 50 Figure 9: Project Implementation of Phase IV: Select the

Provider( s) 60 Figure 10: Project Implementation of Phase V: Transition
to

Provider( s) 68 Figure 11: Project Implementation of Phase VI: Managing
Provider

Performance 79 Figure 12: Project Implementation of Phase VII: Ensure
Services Are

Provided 87

Abbreviations

C3I command, control, communications, and intelligence C4 command,
control, communications, and computer CIO chief information officer DOD
Department of Defense IT information technology IT/ IS Information
Technology/ Information Services MHS/ ITO Military Health System/
Information Technology Organization NETCOM Network Enterprise Technology
Command NIMA National Imagery and Mapping Agency NMCI Navy and Marine
Corps Intranet OMB Office of Management and Budget RFP request for
proposals SLA service- level agreement TAC- SWA Total Army Communications*
Southwest Asia

This is a work of the U. S. Government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. It may contain
copyrighted graphics, images or other materials. Permission from the
copyright holder may be necessary should you wish to reproduce copyrighted
materials separately from GAO*s product.

a

GAO United States General Accounting Office

The projects in GAO*s review substantially used leading commercial
practices as specified in GAO*s framework for outsourcing IT services.
Specifically, the agencies fully implemented 88 percent of the practices
(not including practices not applicable to a particular project). This
framework consists of practices organized into seven phases: (I) determine
sourcing strategy, (II) define operational model, (III) develop the
contract, (IV) select the provider( s), (V) transition to provider( s),
(VI) manage the performance of the provider( s), and (VII) ensure services
are provided. The figure below shows the percentage of practices that were
implemented in each phase.

Although DOD has acted on gathering and disseminating lessons learned and
commercial leading practices related to general acquisition issues, its
actions have generally not been focused on outsourcing or on sharing the
lessons learned from IT services outsourcing across the department. By not
systematically capturing and disseminating such information across the
department, DOD is losing the opportunity to leverage the knowledge gained
on IT services projects like those in GAO*s review. Lessons learned that
are pragmatic and easily accessible would give DOD managers a more
informed understanding of important issues to be addressed when making
outsourcing decisions, as well as the factors to be considered to help
ensure the success of these endeavors.

Percentage of Practices Implemented, by Phase Note: Not applicable* The
practice was not relevant to the project*s particular circumstances. No*
The agency did not implement the practice. Limited* The agency fully
implemented some but not all aspects of the practice and did not take
alternative actions that fully satisfied the practice. Yes* The agency
fully implemented the practice or took an alternative action that fully
satisfied its intent. INFORMATION TECHNOLOGY

DOD Needs to Leverage Lessons Learned from Its Outsourcing Projects

www. gao. gov/ cgi- bin/ getrpt? GAO- 03- 371. To view the full report,
including the scope and methodology, click on the link above. For more
information, contact Randolph C. Hite at (202) 512- 3439 or hiter@ gao.
gov. Highlights of GAO- 03- 371, a report to the

Subcommittee on Readiness and Management Support, Committee on Armed
Services, U. S. Senate

April 2003

Given the magnitude of its reported spending on information technology
(IT) services* more than $6. 2 billion in fiscal year 2001* it is critical
that the

Department of Defense (DOD) adopt effective practices for acquiring IT
services. GAO researched leading commercial practices for the

outsourcing of IT services, and, in November 2001, published a framework
consisting of seven phases that span the full range of activities that are
performed during the outsourcing of those services

(this is an acquisition in which a client organization transfers
responsibility for performing services to an external provider).

GAO was asked to determine (1) the extent to which selected DOD projects
for outsourcing IT services use leading commercial practices as specified
in GAO*s framework and (2) whether DOD is sharing lessons learned from its
IT outsourcing projects across the department.

GAO is making recommendations to the Secretary of Defense aimed at
leveraging lessons learned across the

department from its components* IT outsourcing experiences. DOD agreed
that capturing lessons learned related to IT outsourcing initiatives is
important and stated that it intends to explore a variety of mechanisms to
do so. DOD*s plans are consistent with our recommendations.

Page i GAO- 03- 371 DOD IT Services Outsourcing

Contents

Contents

Page ii GAO- 03- 371 DOD IT Services Outsourcing

Contents

Page iii GAO- 03- 371 DOD IT Services Outsourcing

Page 1 GAO- 03- 371 DOD IT Services Outsourcing United States General
Accounting Office

Washington, D. C. 20548 Page 1 GAO- 03- 371 DOD IT Services Outsourcing

A

Page 2 GAO- 03- 371 DOD IT Services Outsourcing

Page 3 GAO- 03- 371 DOD IT Services Outsourcing

Page 4 GAO- 03- 371 DOD IT Services Outsourcing

Page 5 GAO- 03- 371 DOD IT Services Outsourcing

Page 6 GAO- 03- 371 DOD IT Services Outsourcing

Page 7 GAO- 03- 371 DOD IT Services Outsourcing

Page 8 GAO- 03- 371 DOD IT Services Outsourcing

Page 9 GAO- 03- 371 DOD IT Services Outsourcing

Page 10 GAO- 03- 371 DOD IT Services Outsourcing

Page 11 GAO- 03- 371 DOD IT Services Outsourcing

Page 12 GAO- 03- 371 DOD IT Services Outsourcing

Page 13 GAO- 03- 371 DOD IT Services Outsourcing

Page 14 GAO- 03- 371 DOD IT Services Outsourcing

Page 15 GAO- 03- 371 DOD IT Services Outsourcing

Page 16 GAO- 03- 371 DOD IT Services Outsourcing

Page 17 GAO- 03- 371 DOD IT Services Outsourcing

Page 18 GAO- 03- 371 DOD IT Services Outsourcing

Page 19 GAO- 03- 371 DOD IT Services Outsourcing

Page 20 GAO- 03- 371 DOD IT Services Outsourcing

Page 21 GAO- 03- 371 DOD IT Services Outsourcing

Page 22 GAO- 03- 371 DOD IT Services Outsourcing

Page 23 GAO- 03- 371 DOD IT Services Outsourcing

Page 24 GAO- 03- 371 DOD IT Services Outsourcing

Page 25 GAO- 03- 371 DOD IT Services Outsourcing

Page 26 GAO- 03- 371 DOD IT Services Outsourcing

Page 27 GAO- 03- 371 DOD IT Services Outsourcing

Page 28 GAO- 03- 371 DOD IT Services Outsourcing

Page 29 GAO- 03- 371 DOD IT Services Outsourcing

Page 30 GAO- 03- 371 DOD IT Services Outsourcing

Appendix I

Appendix I Objectives, Scope, and Methodology

Page 31 GAO- 03- 371 DOD IT Services Outsourcing

Page 32 GAO- 03- 371 DOD IT Services Outsourcing

Appendix II

Appendix II Projects* Implementation of Phase I: Determine Sourcing
Strategy

Page 33 GAO- 03- 371 DOD IT Services Outsourcing

Appendix II Projects* Implementation of Phase I: Determine Sourcing
Strategy

Page 34 GAO- 03- 371 DOD IT Services Outsourcing

Appendix II Projects* Implementation of Phase I: Determine Sourcing
Strategy

Page 35 GAO- 03- 371 DOD IT Services Outsourcing

Appendix II Projects* Implementation of Phase I: Determine Sourcing
Strategy

Page 36 GAO- 03- 371 DOD IT Services Outsourcing

Appendix II Projects* Implementation of Phase I: Determine Sourcing
Strategy

Page 37 GAO- 03- 371 DOD IT Services Outsourcing

Page 38 GAO- 03- 371 DOD IT Services Outsourcing

Appendix III

Appendix III Project*s Implementation of Phase II: Define Operational
Model

Page 39 GAO- 03- 371 DOD IT Services Outsourcing

Appendix III Project*s Implementation of Phase II: Define Operational
Model

Page 40 GAO- 03- 371 DOD IT Services Outsourcing

Appendix III Project*s Implementation of Phase II: Define Operational
Model

Page 41 GAO- 03- 371 DOD IT Services Outsourcing

Appendix III Project*s Implementation of Phase II: Define Operational
Model

Page 42 GAO- 03- 371 DOD IT Services Outsourcing

Appendix III Project*s Implementation of Phase II: Define Operational
Model

Page 43 GAO- 03- 371 DOD IT Services Outsourcing

Appendix III Project*s Implementation of Phase II: Define Operational
Model

Page 44 GAO- 03- 371 DOD IT Services Outsourcing

Appendix III Project*s Implementation of Phase II: Define Operational
Model

Page 45 GAO- 03- 371 DOD IT Services Outsourcing

Appendix III Project*s Implementation of Phase II: Define Operational
Model

Page 46 GAO- 03- 371 DOD IT Services Outsourcing

Appendix III Project*s Implementation of Phase II: Define Operational
Model

Page 47 GAO- 03- 371 DOD IT Services Outsourcing

Page 48 GAO- 03- 371 DOD IT Services Outsourcing

Appendix IV

Appendix IV Project*s Implementation of Phase III: Develop the Contract

Page 49 GAO- 03- 371 DOD IT Services Outsourcing

Appendix IV Project*s Implementation of Phase III: Develop the Contract

Page 50 GAO- 03- 371 DOD IT Services Outsourcing

Appendix IV Project*s Implementation of Phase III: Develop the Contract

Page 51 GAO- 03- 371 DOD IT Services Outsourcing

Appendix IV Project*s Implementation of Phase III: Develop the Contract

Page 52 GAO- 03- 371 DOD IT Services Outsourcing

Appendix IV Project*s Implementation of Phase III: Develop the Contract

Page 53 GAO- 03- 371 DOD IT Services Outsourcing

Appendix IV Project*s Implementation of Phase III: Develop the Contract

Page 54 GAO- 03- 371 DOD IT Services Outsourcing

Appendix IV Project*s Implementation of Phase III: Develop the Contract

Page 55 GAO- 03- 371 DOD IT Services Outsourcing

Appendix IV Project*s Implementation of Phase III: Develop the Contract

Page 56 GAO- 03- 371 DOD IT Services Outsourcing

Appendix IV Project*s Implementation of Phase III: Develop the Contract

Page 57 GAO- 03- 371 DOD IT Services Outsourcing

Appendix IV Project*s Implementation of Phase III: Develop the Contract

Page 58 GAO- 03- 371 DOD IT Services Outsourcing

Page 59 GAO- 03- 371 DOD IT Services Outsourcing

Appendix V

Appendix V Projects* Implementation of Phase IV: Select the Provider( s)

Page 60 GAO- 03- 371 DOD IT Services Outsourcing

Appendix V Projects* Implementation of Phase IV: Select the Provider( s)

Page 61 GAO- 03- 371 DOD IT Services Outsourcing

Appendix V Projects* Implementation of Phase IV: Select the Provider( s)

Page 62 GAO- 03- 371 DOD IT Services Outsourcing

Appendix V Projects* Implementation of Phase IV: Select the Provider( s)

Page 63 GAO- 03- 371 DOD IT Services Outsourcing

Appendix V Projects* Implementation of Phase IV: Select the Provider( s)

Page 64 GAO- 03- 371 DOD IT Services Outsourcing

Appendix V Projects* Implementation of Phase IV: Select the Provider( s)

Page 65 GAO- 03- 371 DOD IT Services Outsourcing

Page 66 GAO- 03- 371 DOD IT Services Outsourcing

Appendix VI

Appendix VI Projects* Implementation of Phase V: Transition to Provider(
s)

Page 67 GAO- 03- 371 DOD IT Services Outsourcing

Appendix VI Projects* Implementation of Phase V: Transition to Provider(
s)

Page 68 GAO- 03- 371 DOD IT Services Outsourcing

Appendix VI Projects* Implementation of Phase V: Transition to Provider(
s)

Page 69 GAO- 03- 371 DOD IT Services Outsourcing

Appendix VI Projects* Implementation of Phase V: Transition to Provider(
s)

Page 70 GAO- 03- 371 DOD IT Services Outsourcing

Appendix VI Projects* Implementation of Phase V: Transition to Provider(
s)

Page 71 GAO- 03- 371 DOD IT Services Outsourcing

Appendix VI Projects* Implementation of Phase V: Transition to Provider(
s)

Page 72 GAO- 03- 371 DOD IT Services Outsourcing

Appendix VI Projects* Implementation of Phase V: Transition to Provider(
s)

Page 73 GAO- 03- 371 DOD IT Services Outsourcing

Appendix VI Projects* Implementation of Phase V: Transition to Provider(
s)

Page 74 GAO- 03- 371 DOD IT Services Outsourcing

Appendix VI Projects* Implementation of Phase V: Transition to Provider(
s)

Page 75 GAO- 03- 371 DOD IT Services Outsourcing

Appendix VI Projects* Implementation of Phase V: Transition to Provider(
s)

Page 76 GAO- 03- 371 DOD IT Services Outsourcing

Page 77 GAO- 03- 371 DOD IT Services Outsourcing

Appendix VII

Appendix VII Projects* Implementation of Phase VI: Manage Provider( s)
Performance

Page 78 GAO- 03- 371 DOD IT Services Outsourcing

Appendix VII Projects* Implementation of Phase VI: Manage Provider( s)
Performance

Page 79 GAO- 03- 371 DOD IT Services Outsourcing

Appendix VII Projects* Implementation of Phase VI: Manage Provider( s)
Performance

Page 80 GAO- 03- 371 DOD IT Services Outsourcing

Appendix VII Projects* Implementation of Phase VI: Manage Provider( s)
Performance

Page 81 GAO- 03- 371 DOD IT Services Outsourcing

Appendix VII Projects* Implementation of Phase VI: Manage Provider( s)
Performance

Page 82 GAO- 03- 371 DOD IT Services Outsourcing

Appendix VII Projects* Implementation of Phase VI: Manage Provider( s)
Performance

Page 83 GAO- 03- 371 DOD IT Services Outsourcing

Appendix VII Projects* Implementation of Phase VI: Manage Provider( s)
Performance

Page 84 GAO- 03- 371 DOD IT Services Outsourcing

Appendix VII Projects* Implementation of Phase VI: Manage Provider( s)
Performance

Page 85 GAO- 03- 371 DOD IT Services Outsourcing

Page 86 GAO- 03- 371 DOD IT Services Outsourcing

Appendix VIII

Appendix VIII Projects* Implementation of Phase VII: Ensure Services are
Provided

Page 87 GAO- 03- 371 DOD IT Services Outsourcing

Appendix VIII Projects* Implementation of Phase VII: Ensure Services are
Provided

Page 88 GAO- 03- 371 DOD IT Services Outsourcing

Appendix VIII Projects* Implementation of Phase VII: Ensure Services are
Provided

Page 89 GAO- 03- 371 DOD IT Services Outsourcing

Page 90 GAO- 03- 371 DOD IT Services Outsourcing

Appendix IX

Appendix IX Comments from the Department of Defense

Page 91 GAO- 03- 371 DOD IT Services Outsourcing

Appendix IX Comments from the Department of Defense

Page 92 GAO- 03- 371 DOD IT Services Outsourcing

Appendix IX Comments from the Department of Defense

Page 93 GAO- 03- 371 DOD IT Services Outsourcing

Page 94 GAO- 03- 371 DOD IT Services Outsourcing

Appendix X

United States General Accounting Office Washington, D. C. 20548- 0001
Official Business Penalty for Private Use $300 Address Service Requested

Presorted Standard Postage & Fees Paid

GAO Permit No. GI00
*** End of document. ***