Information Technology: DOD Needs to Leverage Lessons Learned
from Its Outsourcing Projects (25-APR-03, GAO-03-371).
Given the magnitude of its reported spending on information
technology (IT) services--more than $6.2 billion in fiscal year
2001--it is critical that the Department of Defense (DOD) adopt
effective practices for acquiring IT services. GAO researched
leading commercial practices for the outsourcing of IT services,
and, in November 2001, published a framework consisting of seven
phases that span the full range of activities that are performed
during the outsourcing of those services (this is an acquisition
in which a client organization transfers responsibility for
performing services to an external provider). GAO was asked to
determine (1) the extent to which selected DOD projects for
outsourcing IT services use leading commercial practices as
specified in GAO's framework and (2) whether DOD is sharing
lessons learned from its IT outsourcing projects across the
department.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-03-371
ACCNO: A06721
TITLE: Information Technology: DOD Needs to Leverage Lessons
Learned from Its Outsourcing Projects
DATE: 04/25/2003
SUBJECT: Information technology
Defense procurement
Internal controls
Best practices
Procurement practices
Procurement policy
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-03-371
A
Letter
April 25, 2003 The Honorable John Ensign Chairman The Honorable Daniel K.
Akaka Ranking Minority Member Subcommittee on Readiness and Management
Support Committee on Armed Services United States Senate
The Department of Defense (DOD) is the government*s largest purchaser of
information technology (IT) services, such as desktop support, network
operations, and software development services. In fiscal year 2001, DOD
reportedly obligated more than $6.2 billion on IT services, 1 and this
amount is expected to grow substantially. Given the magnitude of DOD*s
spending on such services, it is critical that the department adopt
effective practices for acquiring IT services.
Since 1996, we have conducted a series of studies for the Senate Committee
on Armed Services concerning how DOD can improve its acquisition processes
by adopting proven practices of leading commercial organizations. In this
vein, in November 2001, we issued a guide that
organized leading commercial practices for the outsourcing 2 of IT
services into a framework of seven phases that span the full range of
activities that are performed during IT services outsourcing. 3 This
report responds to your request that we determine (1) the extent to
which selected DOD IT services outsourcing projects use leading commercial
practices as specified in our framework and (2) whether DOD is sharing
lessons learned from its IT outsourcing projects across the department. To
address the first objective, we selected five projects from a group of
projects to outsource IT services that were
1 This figure is from the Federal Procurement Data System, which contains
detailed information on contract actions over $25,000. 2 IT services
outsourcing is a type of acquisition in which a client organization
transfers responsibility for performance of one or more IT services to one
or more external providers. 3 U. S. General Accounting Office, Information
Technology: Leading Commercial Practices for Outsourcing of Services, GAO-
02- 214 (Washington, D. C.: Nov. 30, 2001).
identified by the military services and other DOD components. 4 We then
asked the component responsible for each project to perform a
selfassessment against selected practices in our framework for outsourcing
IT services. 5 Next, we obtained and reviewed agencies* supporting
documentation and interviewed the appropriate agency and provider project
officials to independently determine whether a practice was met. To
address the second objective, we reviewed applicable DOD approaches for
capturing and disseminating lessons learned from IT services outsourcing
projects and interviewed the applicable acquisition and IT officials.
Details of our objectives, scope, and methodology are discussed in
appendix I.
Results in Brief The projects in our review substantially used leading
commercial practices as specified in our framework on outsourcing IT
services. Specifically, the
agencies fully implemented 88 percent of the practices. 6 This framework
consists of practices organized into seven phases that span the full range
of activities that are performed during IT outsourcing: (I) determine
sourcing strategy, (II) define operational model, (III) develop the
contract, (IV)
select the provider( s), (V) transition to provider( s), (VI) manage
provider( s) performance, and (VII) ensure services are provided. Figure 1
illustrates the percentage of practices that were followed in each phase.
Collectively, the projects fully implemented from 70 to 97 percent of the
practices in each phase.
4 We asked the military services and other DOD components to identify
candidate projects because DOD does not maintain a central list of IT
services outsourcing projects. We chose each project on the basis of the
following criteria: (1) no more than one project from each military
service and two agencies, (2) illustrative example of DOD IT outsourcing,
(3) dollar value greater than $10 million, and (4) enough time elapsed for
services to have been delivered and performance measured.
5 We identified 70 practices in our November 2001 report on leading
commercial practices that (1) are typically applied at the project level
and (2) were verifiable through documentation and interviews. 6 This
calculation does not include practices that were not applicable to a
particular project.
Figure 1: Percentage of Practices Implemented in Each Phase
Note: Not applicable* The practice was not relevant to the project*s
particular circumstances. No* The agency did not implement the practice.
Limited* The agency fully implemented some but not all aspects of the
practice and did not take alternative actions that fully satisfied the
practice. Yes* The agency fully implemented the practice or took an
alternative action that fully satisfied its intent.
Although implementing the leading commercial practices in our framework
does not guarantee the success of an outsourcing project, the consensus
view of the leading commercial activities that we studied is that these
practices are the most critical to success when acquiring IT services. 7
In addition, not implementing or only partially implementing particular
practices can produce negative consequences or add risk to a project. For
example, the Department of the Navy project*s baseline of its existing
environment (a phase I practice) was limited because it did not include an
assessment of its legacy applications. Instead, project officials decided
to rely on a preexisting inventory developed to address the Year 2000
challenge. The Navy subsequently found that it had substantially
underestimated the number of legacy applications, which, according to
program officials, later contributed to the transition period slipping
from 2- 1/2 years to 3- 1/2 years. As DOD gathers more experience in
implementing projects for outsourcing
IT services, it can benefit from leveraging the lessons derived from these
initiatives. For example, the projects in our review have identified
lessons learned in such areas as transitioning to the provider and
partnering with the provider. Although DOD has taken action to gather and
disseminate lessons learned and best practices on general acquisition
issues, these efforts generally do not focus on outsourcing or include
sharing the lessons learned from IT outsourcing projects across the
department. By not capturing and disseminating such information in a
systematic manner across the department, DOD is losing the opportunity to
leverage the knowledge gained on IT services projects like those in our
review.
We are making recommendations to the Secretary of Defense aimed at
leveraging lessons learned across the department from its components* IT
services outsourcing experiences. In written comments on a draft of this
report signed by the DOD Principal
Director, Deputy Assistant Secretary of Defense (Deputy Chief Information
Officer), DOD agreed that capturing lessons learned in the development and
implementation of its IT outsourcing initiatives is important to
continually improving its outsourcing methods and results achieved. The
7 This consensus view was based on interviews with managers in leading
commercial organizations, discussions with academic and professional
authorities, and extensive research on IT acquisition practices.
department also stated that it intends to explore a variety of mechanisms
for best exploiting lessons learned from its IT outsourcing initiatives.
We agree that it is prudent to consider alternative means to leveraging
these
lessons learned, and we believe that this is consistent with our
recommendations.
Background To protect the security of the United States, DOD relies on a
complex array of computer- dependent and mutually supportive
organizational components, including the military services, Commanders in
Chief, and Defense agencies. As such, it invests tens of billions of
dollars each year in a broad array of computer systems, which include
weapon systems, command and control systems, satellite systems, inventory
management
systems, transportation management systems, health systems, financial
systems, personnel systems, and payment systems. In addition, DOD spends
billions of dollars annually on IT services, which include database
management, help- desk operations, software maintenance, and network
services. In fiscal year 2001, DOD reportedly obligated more than $6. 2
billion on IT services alone. 8
Decisions regarding the purchasing of services are critical to ensuring
the effectiveness of DOD*s operations as well as those of the government
as a whole. Our November 2001 report recognizes the importance of such
sourcing decisions and provides a framework that spans the full range of
activities that are performed during IT services outsourcing. 9 At the
same time, governmentwide policies, initiatives, and challenges exist that
significantly influence the government*s sourcing decisions.
8 This figure is from the Federal Procurement Data System, which contains
detailed information on contract actions over $25,000. 9 GAO- 02- 214.
GAO*s Framework for Outsourcing of IT services has become increasingly
popular in both the
Outsourcing IT Services public and private sector. For example, according
to the Giga Information Group, Inc., a leading research firm, such
outsourcing is expected to grow
an average of 5 to 6 percent in 2003. 10 The federal sector*s outsourcing
is predicted to rise at an even greater rate. For example, INPUT, an IT
market research firm, forecasts that defense IT outsourcing will increase
about 143
percent between fiscal years 2002 and 2007. 11 IT outsourcing involves the
activities associated with acquiring services from one or more external
providers. During outsourcing, a client organization transfers
responsibilities for performing one or more IT
services to one or more external providers. This responsibility is
executed through control and management of the processes, people, and
technology associated with these services.
Figure 2 depicts the roles of the client and provider organizations in an
outsourcing relationship.
Figure 2: Roles of the Client and Provider in an Outsourcing Relationship
10 Giga Information Group, Inc., IT Trends 2003: IT Services (Dec. 19,
2002). We did not independently verify these data. 11 INPUT, The Federal
IT Outsourcing Market View (December 2002). We did not independently
verify these data.
Our November 2001 guide on leading commercial practices for outsourcing IT
services provides a generic framework of practices from leading commercial
organizations that can improve purchasing decisions and manage the
resulting government/ provider relationship. 12 The framework is
represented in figure 3 as a hierarchy of phases, practices, and critical
success factors.
Figure 3: GAO*s Framework for Outsourcing IT Services
Note: The arrow from phase VII to phase I represents the need to reflect
on lessons learned from previous phases. The arrows between phase III and
IV represent the iterative nature of developing the contract and selecting
the provider. Although there is a logical order to the sequence of the
common phases, the order of the practices within each phase does not imply
any priority or sequence. Table 1 provides a definition of each phase of
the framework. Each of the
phases has specific practices associated with it. Implementing these
practices does not guarantee the success of an outsourcing project.
However, our November 2001 study reflected a consensus view that these
practices were the most critical to success when IT services are being
acquired. 13 Restated, application of these practices increases the
probability of a successful outsourcing project.
12 GAO- 02- 214. 13 This consensus view was based on interviews with
managers in leading commercial organizations, discussions with academic
and professional authorities, and extensive research on IT acquisition
practices.
Table 1: Definition of Phases for IT Outsourcing Phase number Title
Definition
I Determine sourcing strategy Client organizations determine whether
internal capability or external expertise can more effectively meet IT
needs.
II Define operational model Client organizations formalize executive
leadership, team composition, client responsibilities, and operating
relationships between client and provider organizations.
III Develop the contract Client organizations establish the legal terms
for the IT outsourcing relationship. IV Select the provider( s) Client
organizations find one or more providers who can help them reach their
IT outsourcing goals. V Transition to provider( s) Client organizations
transfer responsibility of IT functions to one or more providers.
VI Manage provider( s) performance Client organizations make sure each
provider is meeting performance requirements.
VII Ensure services are provided Client organizations make sure that
services are provided and end- user needs are met.
Source: GAO.
The organizations that we studied also identified certain capabilities
(identified as critical success factors) that were essential for
implementing the practices identified in our framework. First, executive
leadership strengthens the interaction between executive management and
the
employees of the client organization. Second, partner alignment
strengthens the interaction between the client and provider organization
at the executive level, which ensures that the goals and objectives of
these organizations support each other. Third, relationship management
strengthens the interaction between the client and provider organization
at the operational level.
Influences on Government The federal government is one of the world*s
largest users of services.
Sourcing Decisions Because of the large dollar value and the number of
private- and publicsector
jobs involved, deciding whether the public or private sector would be the
most appropriate provider of the services the government needs (IT or
otherwise) is an important, and often highly charged, question. Among the
factors that agencies must consider as they determine how best to meet
their missions is whether the public or private sector would be the most
appropriate provider of the services the government needs. Phase I of our
framework, determine the sourcing strategy, addresses the client*s
assessment of whether expertise from within or outside of the organization
can more effectively meet the client*s needs.
The National Defense Authorization Act for Fiscal Year 2001 required the
Comptroller General of the United States to convene a panel of experts to
study the current process used by the government to make sourcing
decisions. The resulting Commercial Activities Panel conducted a year-
long
study and heard repeatedly about the importance of competition and its
central role in fostering economy, efficiency, and continuous performance
improvement. In particular, the panel reviewed the government*s
implementation of the Office of Management and Budget*s (OMB) Circular
A- 76, which sets forth federal policy for determining whether federal
employees or private contractors will perform commercial activities for
the government. 14 Circular A- 76 (1) outlines conditions under which
agencies are permitted to perform a commercial activity with government
employees or by contract and (2) provides guidance for whether, and if so,
how, agencies should conduct a cost comparison when they are considering
transferring the performance of commercial activities from the public to
the private sector (or vice versa). The panel reported that there
were positive elements to Circular A- 76 but that both federal employees
and private firms complained that it does not meet the standard of a
clear, transparent, and consistently applied process. For example, both
federal
employees and private firms criticized the Circular A- 76 process as
unequal and therefore unfair.
The Commercial Activities Panel strongly supported continued emphasis on
competition and concluded that whenever the government is considering
converting work from one sector to another, public/ private competitions
should be the norm. In addition, the panel made four recommendations,
including that all sourcing decisions be consistent with the principles
adopted unanimously by the panel, such as the principle that federal
policy provide for accountability in connection with all sourcing
decisions. 15 14 Office of Management and Budget, Circular A- 76,
Performance of Commercial Activities (Washington, D. C.: June 14, 1999).
In November 2002, OMB issued proposed revisions that
would substantially change this circular. As of April 8, 2003, these
revisions have not yet been made final.
15 Final report of the Commercial Activities Panel, Improving the Sourcing
Decisions of the Government (Washington, D. C.: Apr. 30, 2002).
As part of the administration*s efforts to implement the recommendations
of the Commercial Activities Panel, OMB has published proposed changes to
Circular A- 76. Key highlights of the proposed changes include presuming
that all functions are commercial in nature unless they are justified as
inherently governmental; 16 limiting the length of time for competitions;
and emphasizing awarding contracts on the basis of best value, not just
lowest cost. Best value allows the contracting official to consider
technical superiority, quality, innovation, and past performance as well
as price.
However, we reported that there are several areas in which the proposed
revisions to the circular are not consistent with the principles or
recommendations of the Commercial Activities Panel. 17 Specifically, the
proposed revision does not include a link between sourcing policy and
agency missions, has unnecessarily complicated source selection
procedures, contains certain unrealistic time frames, and includes
insufficient guidance on calculating savings.
Beyond the Commercial Activities Panel, other bodies have identified
challenges that the federal government faces in reaching and executing
effective sourcing decisions. For example, members of the Coalition for
Government Procurement, the Professional Services Council, and the
Information Technology Association of America told us that organizational
culture is one of the biggest differences between the commercial sector
and the federal government and one of the greatest barriers to the
government*s use of commercial practices. Also, as we have previously
reported, moving to outsourcing solutions can involve a cultural change
for government organizations because it may require a change to an
agency*s operating model, such as using a contractor to provide IT
services
previously performed by government staff or using a performance- based
contract. 18 This view was echoed by a 2001 study of DOD competitive
16 Section 5 of P. L. 105- 270, codified at 31 U. S. C. 501 note (1998),
defines an inherently governmental function as a *function that is so
intimately related to the public interest as to require performance by
Federal Government employees.* 17 U. S. General Accounting Office,
Proposed Revisions to OMB Circular A- 76, GAO- 03- 391R (Washington, D.
C.: Jan. 16, 2003).
18 U. S. General Accounting Office, Desktop Outsourcing: Positive Results
Reported, but Analyses Could Be Strengthened, GAO- 02- 329 (Washington, D.
C.: Mar. 29, 2002).
sourcing that found cultural, process, execution, and training barriers.
19 The study stated that these barriers need to be understood and
mitigated before the benefits of outsourcing can be fully realized.
Barriers such as these can be overcome by strong executive leadership,
which is a critical success factor in our framework. Another challenge is
creating a productive agency/ provider relationship* another critical
success factor in our framework. According to a report sponsored by the
PricewaterhouseCoopers Endowment for the Business of Government, such
public/ private partnerships are based on trust, commitment to problem or
conflict resolution, and the recognition that
flexibility is necessary and that the relationship will evolve and change
over time. 20 If deadlines are not met, or public agency goals change with
differing political climates, the partners need to discuss the basis of
the partnership and construct a different relationship. Our prior report
on desktop outsourcing found that developing a productive agency/
contractor relationship is not always easy. 21 Both sides must recognize
and understand
each other*s underlying motives and strive to achieve established
expectations.
Finally, human capital issues are another challenge facing federal
agencies that affect their ability to implement outsourcing. Our framework
recognizes the importance of having the right skills in place to support
the outsourcing relationship. However, as we have previously reported,
procurement reforms and technological changes have placed unprecedented
demands on the acquisition workforce. 22 Contracting personnel are now
expected to have a much greater knowledge of market conditions, industry
trends, and the technical details of the commodities and services they
procure. The Commercial Activities Panel report stated that developing and
maintaining a skilled acquisition workforce is the
critical first step in managing this more complex procurement environment.
The panel also reported that DOD bore the brunt of a 22
19 The Industrial College of the Armed Forces, Case Study: Complex
Business Management for Competitive Sourcing (2001). 20
PricewaterhouseCoopers Endowment for the Business of Government,
Contracting for the 21 st Century: A Partnership Model (January 2002).
21 GAO- 02- 329. 22 U. S. General Accounting Office, Contract Management:
Taking a Strategic Approach to Improving Service Acquisitions, GAO- 02-
499T (Washington, D. C.: Mar. 7, 2002).
percent downsizing of the federal acquisition workforce in the last
decade, going from 96,000 staff in 1991 to about 68,000 in fiscal year
2001.
Addressing human capital issues is not just a matter of the size of the
workforce; it is also a knowledge and skills issue. According to the
Commercial Activities Panel, it is critically important that federal
agencies adequately address human capital needs in meeting the current and
emerging needs of government and its citizens in the most effective,
efficient, and economical manner possible. This will require increased
emphasis on training and development, particularly in the area of
technology. Description of Five Projects
The five projects in our study varied in how they approached outsourcing
Reviewed
IT services, such as in using various solicitation methods, including
holding a public/ private competition under the policies outlined in OMB
Circular A76 or carrying out a negotiated competitive procurement. In
addition, the types of services being outsourced differed: services ranged
from the narrowly focused (e. g., help- desk services) to the very broad
(e. g., enterprisewide end- to- end information services); contract terms
ranged from 5 to 15 years (assuming all option years are exercised); and
estimated contract values ranged from $23 million to $8.8 billion. Table 2
provides information on the variety of IT services and outsourcing
approaches taken by the projects.
Table 2: Profile of Outsourcing Projects Reviewed Date of Solicitation
contract
Contract Contract Estimated total Agency/ Project method
award term type contract value a Project description
Air Force/ Kirtland Air Competitive, April 2000 1 year, with
Firm, fixedprice $23 million Management, operations, and Force Base*s
under OMB
4 option maintenance of command, Command, Control, Circular A- 76 years
control, communications, and Communications, and and small computer
systems, multimedia Computer (C4)
business setaside services, and information
Services rules
management for the Kirtland Air Force Base.
Army/ Network Negotiated March 2001 1 year, with
Firm, fixedprice $204 million Operation and maintenance
Enterprise Technology competitive 4 option services, including repair,
Command*s solicitation years
installation, and supply, for (NETCOM) Total Army communications equipment
in Communications*
Southwest Asia. Southwest Asia (TACSWA)
Military Health Negotiated
June 2001 1 year, with Firm, fixedprice
$71 million Call and help- desk services for System/ Information
competitive 7 option with all MHS software applications.
Technology solicitation
years incentive
Organization awards (MHS/ ITO) Help Desk
Department of the Negotiated October
7 years, Firm, fixedprice $8.8 billion Department of the Navy- wide
Navy/ Navy and Marine competitive 2000 with an
with end- to- end information services
Corps Intranet (NMCI) solicitation
option for 3 incentive through a common computing
additional awards
and communication years environment.
National Imagery and Sole- source,
December 1 year, with
Cost plus $2.1 billion NIMA- wide IT/ IS support
Mapping Agency*s using a 2001 14 option award fees services for printing,
digital
(NIMA) Information statutory
years replication, networks, distributed
Technology/ preferential
and centralized systems and Information Services
provider services, video and voice
(IT/ IS) (Alaska native communications, information
corporation) b research, and help- desk
operations. c Source: DOD.
a Estimated value if all option years are exercised. b NIMA performed an
OMB Circular A- 76 analysis and, on the basis of this analysis,
implemented a direct conversion to a preferential provider rather than
holding a public/ private competition or obtaining an agency cost-
comparison waiver. c As of February 24, 2003, NIMA had transitioned four
of these functions to the provider: printing, digital
replication, video and voice communications, and help- desk operations.
Projects Substantially As illustrated in figure 4, the five IT services
projects substantially used
leading commercial practices. Specifically, each project used at least 76
Used Leading percent of the practices. 23 Reasons for projects
implementing different Commercial Practices
percentages of the practices include differences in their individual
circumstances and objectives. For example, the Army*s Total Army
Communications* Southwest Asia (TAC- SWA) project, which used the
fewest number of practices and had the largest number of practices that
were not applicable, was largely a continuation of an existing approach
that already relied on the private sector but with fewer providers. In
contrast, the National Imagery and Mapping Agency*s (NIMA) Information
Technology/ Information Services (IT/ IS) project, which fully or
partially implemented all of the applicable practices, involved a
significant operational shift (e. g., functions previously performed by
NIMA staff are now performed by a contractor) and was intended to result
in substantial process improvements. In addition, the three projects that
implemented the largest percentage of practices also used third- party
assistance* including employing a contractor with sourcing expertise* to
help formulate their sourcing strategy, which could account for the extent
of their compliance.
23 This calculation does not include practices that were not applicable to
a particular project.
Figure 4: Use of Leading Commercial Practices for Outsourcing of IT
Services, by Project
Note: Not applicable* The practice was not relevant to the project*s
particular circumstances. No* The agency did not implement the practice.
Limited* The agency fully implemented some but not all aspects of the
practice and did not take alternative actions that fully satisfied the
practice. Yes* The agency fully implemented the practice or took an
alternative action that fully satisfied its intent.
In addition, figure 5 illustrates that project compliance extended to each
of the phases of our framework. Collectively, the projects fully
implemented from 70 to 97 percent of the practices in each phase. Phase I,
determine sourcing strategy, had the lowest percentage of practices
implemented by
the projects (70 percent). This result is not inconsistent with a recent
Giga Information Group, Inc., survey, which found that only half of the
respondent organizations had documented an IT sourcing strategy. 24 This
approach carries risk since phase I sets the tone for the outsourcing
initiative within the client organization.
Figure 5: Percentage of Practices Implemented in Each Phase
Note: Not applicable* The practice was not relevant to the project*s
particular circumstances. No* The agency did not implement the practice.
Limited* The agency fully implemented some but not all aspects of the
practice and did not take alternative actions that fully satisfied the
practice. Yes* The agency fully implemented the practice or took an
alternative action that fully satisfied its intent.
24 Giga Information Group, Inc., Optimizing IT Sourcing Strategy: Key
Stages and Phases of the IT Sourcing Process (Jan. 31, 2003).
In addition, collectively the projects fully implemented 88 percent of the
practices (see table 3). 25 Table 3: Percentage of Practices Implemented,
by Project
Percentage of practices implemented a Air Force
Army Phase
C4 Services TAC- SWA
MHS/ ITO Help Navy NMCI
NIMA IT/ IS project
project Desk project project project Overall
Phase I: Determine sourcing strategy (6 practices). 50 33 83 83 100 70
Phase II: Define operational model (13 practices). 92 83 100 77 100 90
Phase III: Develop the contract (16 practices). 64 83 100 94 100 89
Phase IV: Select provider( s) (7 practices). 86 86 100 100 71 89
Phase V: Transition to provider( s) (11 practices). 82 88 100 82 100 90
Phase VI: Manage provider( s) performance (11 practices). 73 73 100 91 82
84
Phase VII: Ensure services are provided (6 practices). 83 100 100 100 100
97
Overall 76 79 99 89 94 88
Source: GAO. a These calculations do not include practices that were not
applicable to a particular project.
The following provides additional information on the projects*
implementation of each phase of our framework.
25 This calculation does not include practices that were not applicable to
a particular project.
Phase I: Determine sourcing strategy. In the first phase of our
outsourcing framework, the client organization determines whether internal
capability or external expertise can more effectively meet its IT needs.
The purpose of a sourcing strategy is to achieve the optimal balance
between internal and external capabilities, activities, processes, and
services to ensure the achievement of strategic business objectives at the
lowest risk. 26 The five projects* implementation of this phase was
uneven. In particular, two of the six practices in this phase were fully
implemented by all five projects, but the other four practices were not.
Among the practices that were implemented by all of the projects was
determining the business reasons for outsourcing. In addition, the three
projects that implemented the largest percentage of practices in our
framework* the Military Health System/ Information Technology Organization
(MHS/ ITO) Help Desk, the Department of the Navy*s Navy and Marine Corps
Intranet (NMCI), and NIMA IT/ IS projects* used the third- party
assistance practice in this phase to help formulate their sourcing
strategy, which could account for the extent of their compliance. For
example, the MHS/ ITO Help Desk project, which implemented the largest
percentage of practices, worked with the Department of the Interior*s
GovWorks Program, 27 the Defense Acquisition University staff, and a
private- sector contractor to obtain expertise on sourcing strategies.
The practice in this phase that was the most unevenly implemented was the
benchmarking 28 and baselining of existing internal services. Of the five
projects in our review, (1) one fully benchmarked and baselined the
productivity of the activity being outsourced before making the final
sourcing decision, (2) two partially baselined their existing activities,
and (3) two did not perform benchmark and baseline analyses at all. The
agencies* reasons for not fully implementing this practice included that
an executive decision had been made to conduct a public/ private
competition following the OMB Circular A- 76 policy, so such an analysis
would not have affected the sourcing decision, or that
26 Gartner, Inc., Strategic Analysis Report, How to Build a Sourcing
Strategy, Research Note R- 18- 1099 (Sept. 23, 2002). 27 The Department of
the Interior*s GovWorks Program is a federal fee- for- service acquisition
center that helps other federal agencies acquire supplies and services for
their programs on a project- by- project basis.
28 Gartner, Inc., defines benchmarking as a method to compare the cost or
price of an IT environment to peer groups with the same workload
characteristics.
available documentation to perform such an analysis was limited. Leading
research firms suggest benchmarking and baselining the entity*s current
processes before outsourcing because only then would it be able to
determine whether the arrangement has been successful. 29 In addition, the
risk of not fully baselining the existing environment is
illustrated by the NMCI project. Specifically, the NMCI project*s baseline
of its existing environment was limited because it did not include an
assessment of its legacy applications since project officials decided to
rely on a preexisting inventory developed to address the Year 2000
challenge. The Navy subsequently found that it had substantially
underestimated its number of legacy applications, that,
according to program officials, later contributed to the transition period
slipping from 2- 1/2 years to 3- 1/2 years. Appendix II provides
additional information on projects* implementation of the practices
comprising this phase.
Phase II: Define operational model. The operational model is an
important mechanism for an organization to compare its plans with the
expectations that were set when the decision to outsource was made and to
ascertain whether these plans will enable the organization to meet
expectations. The five projects had largely implemented the 13 practices
contained in this phase. Specifically, about 90 percent 30 of the
practices were implemented. For example, all projects implemented the
practice that executive leadership be established to facilitate the
outsourcing effort. NIMA, for instance, formed a strategic sourcing office
to oversee the IT/ IS project. Another practice* training the provider on
the organization*s business environment and goals* was fully implemented
by one project (in two cases, the practice was not applicable). One
project that did not fully implement this practice was the Air Force*s
Kirtland Air Force Base*s Command, Control, Communications, and Computer
(C4) Services project. Although some training was provided (e. g.,
Kirtland held an orientation session for potential bidders), provider
officials stated that they did not receive adequate training, which made
the transition period more difficult. Appendix III provides additional
information on projects*
implementation of the practices comprising this phase. 29 Giga Information
Group, Inc., Payment and Incentives for Outsourcing Management (July 27,
2000) and Gartner, Inc., Benchmarking Helps Outsourcing Deals Stay
Competitive, Research Note COM- 16- 8055 (June 14, 2002).
30 This calculation does not include practices that were not applicable to
a particular project.
Phase III: Develop the contract. A well- written contract is necessary
for the outsourcing organization to meet its requirements while allowing
the service provider to make a fair profit. It sets the expectations for
service levels, delivery of essential services, and continuous improvement
and should protect the interests of all parties. The five projects largely
implemented the practices in this phase. Specifically, about 89 percent 31
of the practices were implemented in this phase, and two projects (the
MHS/ ITO Help Desk and NIMA IT/ IS projects)
implemented all of the practices. Several practices in this phase address
performance requirements. For example, all five projects implemented the
practices that called for basing performance requirements on business
requirements and reviewing and updating them periodically. One practice
that was not fully implemented by two projects was including performance
measures that address both technical and enduser satisfaction aspects of
performance. For example, the Army TACSWA project included technical
performance measures in its contract but not measures related to end- user
satisfaction, even though the contract included help- desk services.
According to the project official that developed the performance work
statement in the contract, the command did not include customer
satisfaction measures because it did not think that it was necessary to
have a performance standard for the help- desk service. However, without
such measures, the agency does not have a contractual standard with which
to judge the provider*s performance. Appendix IV provides additional
information on projects*
implementation of the practices comprising this phase. Phase IV: Select
the provider( s). Critical to the success of any
outsourcing project for IT services is identifying potential providers and
ultimately selecting a provider( s) that will best meet the needs of the
agency. The five projects had largely implemented the seven practices
contained in this phase. About 89 percent of the practices were
implemented in this phase and two projects (the MHS/ ITO Help Desk and
Navy NMCI projects) used each applicable practice. For example,
the five projects implemented the practice related to conducting due
diligence activities to verify provider capabilities before signing the
contract. In the case of the Army TAC- SWA project, the Network Enterprise
Technology Command (NETCOM) evaluated the provider*s financial and past
performance information. The Department of the
Navy*s NMCI project also evaluated bidders* past performance and 31 This
calculation does not include practices that were not applicable to a
particular project.
performed reference checks. Moreover, as part of its due diligence
activities, the Navy required all bidders to demonstrate that they had
experience in implementing large seat management contracts. 32 A practice
that was not implemented by two projects was using third- party assistance
when selecting the provider. Projects that did not implement this practice
believed that they had adequate in- house expertise with outsourcing,
making third- party assistance unnecessary. However, because third- party
assistance provides an independent resource that
can suggest options or processes that the client organization may not be
aware of, these projects may have missed an opportunity to implement their
outsourcing projects more effectively. Appendix V provides additional
information on projects* implementation of the practices comprising this
phase.
Phase V: Transition to provider( s). This phase focuses on the client
organization*s transfer of the IT function to one or more providers. As
part of this transition, the clear definition of responsibilities and the
careful consideration of employees* needs matched against the client
organization*s needs enable both the client and provider to focus on
execution and give staff confidence in their future employment. The five
projects largely implemented the 11 practices associated with this phase.
Specifically, about 90 percent 33 of the practices were implemented in
this phase, and two projects implemented all of the practices (the MHS/
ITO Help Desk and NIMA IT/ IS projects). Several practices in this phase
address dealing with employees affected by the outsourcing projects. For
example, in the four projects in which federal employees were affected,
the projects provided assistance to those who did not want to transfer to
the provider, including helping to place them
in other positions and helping with resume writing. A related practice is
to clearly communicate to all employees what is going to happen and when
it is going to happen. Two projects did not fully implement this practice.
For example, the Navy used its normal chain of command to communicate
transition information, but found that implementation of
this practice was uneven. As a result, some staff did not know current
information about how NMCI would affect them until the provider was ready
to contact them regarding their possible transition to the 32 Seat
management generally refers to service provision arrangements in which
contractorowned desktop and other computing hardware, software, and
related services are bundled and provided to a client organization at a
fixed price per unit (or seat).
33 This calculation does not include practices that were not applicable to
a particular project.
contractor. However, according to the NMCI*s Director*s office, this
problem was somewhat mitigated by the provider*s Web site that provides
transition information to all NMCI customers/ users. Appendix VI provides
additional information on projects* implementation of the practices
comprising this phase.
Phase VI: Manage provider( s) performance. The effectiveness with which
the provider( s) performance is managed is critical to the successful
implementation of an outsourcing project. Indeed, according to Gartner,
Inc., a leading research firm, an outsourcing project can be thwarted by
poorly designed, funded, and delivered processes for
managing the delivery of services. 34 The five projects generally
implemented the 11 practices contained in this phase, with about 84
percent of the practices being implemented in this phase. For example, the
practices related to obtaining feedback on provider performance
were largely implemented. This is important because different levels of an
agency can have different perceptions about the value of the outsourcing
project. For example, an outsourcing project may be considered successful
by the agency*s executive management if it is focused on controlling
costs, but be considered inadequate by business managers and users who may
be expecting higher levels of service. Each of these viewpoints is valid
and should be taken into account when the
provider*s performance is evaluated. Two other practices* including
incentives and penalties in contracts* were fully implemented by two
projects. Incentives and penalties are important because they can help
motivate the provider to exceed or meet performance requirements.
Nonetheless, two projects did not include monetary incentives and two
projects did not include monetary penalties in their contracts. For
example, the Army TAC- SWA project did not include monetary incentives,
although the contracting officer stated that incentives might have been
useful to motivate the provider to exceed performance requirements.
Incentives can also help control risks. According to a guide on
performance- based services acquisition, if the incentives in the
contract are right and if the provider and agency share the same goals,
risk is largely controlled and effective performance is *almost the 34
Gartner, Inc., Retain Enough Resources to Manage Outsourcing Deals,
Research Note COM- 16- 8425 (June 17, 2002).
inevitable outcome.* 35 Appendix VII provides additional information on
projects* implementation of the practices comprising this phase. Phase
VII: Ensure services are provided. Although outsourcing
transfers responsibility for performing the service to the provider( s),
the client organization is ultimately responsible for ensuring that
services are provided and that end- user needs are met. Accordingly, it is
critical that the agency ensure that services are provided. The projects
had implemented 97 percent of these practices, and four projects
implemented all of them. For example, every project monitored the
providers* work. In the case of the Air Force*s C4 Services project,
quality assurance evaluators monitored the provider*s work to identify
problems or trends in accordance with the project*s quality assurance
surveillance plan. The results were reported to the contracting officer
and to the functional area chief for resolution. Another practice, using
customer satisfaction surveys, was fully implemented by four of the five
projects. However, the Air Force project did not conduct, or require its
contractor to conduct, customer satisfaction surveys. Although the
provider surveys staff annually, the Air Force is nevertheless relying on
the provider to voluntarily implement an important practice for
determining how customers view the services being delivered and whether
changes need to be made. Appendix VIII provides additional
information on projects* implementation of the practices comprising this
phase.
Leveraging Lessons We have previously reported on the importance of
collecting and
disseminating lessons learned. 36 For example, a critical activity in IT
Learned DOD- wide
investment management is establishing a process for developing and Could
Assist Other
capturing lessons learned in a written product or knowledge base and DOD
Projects
disseminating them to decision- makers. 37 In addition, one of the
practices in our framework for outsourcing IT services addresses
incorporating
35 An Interagency- Industry Partnership in Performance, Seven Steps to
Performance- Based Services Acquisition, Benchmark Version (January 2002).
36 U. S. General Accounting Office, NASA: Better Mechanisms Needed for
Sharing Lessons Learned, GAO- 02- 195 (Washington, D. C.: Jan. 30, 2002).
37 U. S. General Accounting Office, Information Technology Investment
Management: A Framework for Assessing and Improving Process Maturity, GAO/
AIMD- 10- 1. 23, Exposure Draft (Washington, D. C.: May 2000).
lessons learned from peers who have engaged in similar sourcing decisions.
Use of lessons learned is a principal component of an organizational
culture committed to continuous improvement. Sharing such information
serves to communicate acquired knowledge more
effectively and to ensure that beneficial information is factored into
planning, work processes, and activities. Lessons learned can be based on
positive experiences or on negative experiences that result in undesirable
outcomes.
Although DOD has taken action to gather and disseminate lessons learned
and best practices on general acquisition issues, these efforts generally
do not focus on outsourcing or include sharing the lessons learned from IT
outsourcing projects across the department. Specifically, a number of DOD
Web sites provide guidance, lessons learned, and best practices related to
general acquisition issues. However, using these sites to locate specific
information on IT outsourcing best practices and lessons learned can be
time- consuming and difficult because so many topics and information
sources are provided. Specifically, MHS/ ITO Help Desk project officials
said that searching numerous Web sites to get relevant information to
address questions and concerns about outsourcing IT services can consume
hours. For example, when we entered the keywords *IT outsourcing* and
*best practices* into the search feature on the Office of the
Undersecretary for Defense of Acquisition, Technology, and Logistics site,
ACQWeb (www. acq. osd. mil) in early March, it provided us with links to
1,251 documents. Likewise, lessons learned covers so many topics that it
is difficult to search for an applicable IT lesson. For example, when we
used the phrase *lessons learned in IT outsourcing* no documents were
identified on the ACQWeb, but when we inserted *lessons learned* and *IT
outsourcing* links to more than 1,700 documents were produced.
One DOD Web site, Share A- 76!, was established to address one of our
previous recommendations, 38 that is, to establish a framework for
identifying and analyzing best practices and lessons learned from
competitive sourcing studies and disseminating them DOD- wide. Share A76!
promotes the sharing of best practices and lessons learned related to one
form of outsourcing that was conducted under the OMB Circular A- 76
competitive sourcing process. Among other things, the site contains
guidance, links to other relevant sites, sample documents, and a best
38 U. S. General Accounting Office, DOD Competitive Sourcing: Lessons
Learned System Could Enhance A- 76 Study Process, GAO/ NSIAD- 99- 152
(Washington, D. C.: July 21, 1999).
practices library that communicates field staff experiences and advice
about the Circular A- 76 process. A NIMA project official said that NIMA*s
staff routinely accesses the Web site because it contains a wealth of
information on policies, procedures, lessons learned, and links to other
outsourcing sites. The DOD analyst responsible for Share A- 76! estimated
that the site receives about 12,000 visits per month and said that on the
basis of E- mails and anecdotes, user satisfaction is favorable. However,
this Web site is specific to the OMB Circular A- 76 process, which may not
apply to other types of outsourcing. For example, Circular A- 76*s policy
pertains to public/ private competitions and requires that the final
evaluation
between the government and the private sector be based exclusively on
cost. DOD acquisition and IT officials acknowledged that there is no
mechanism in DOD to easily share and leverage lessons learned relating to
outsourcing IT services. However, these officials agreed that a
departmentwide effort to identify, capture, and disseminate lessons
learned and leading practices of projects with experience in carrying out
IT outsourcing could offer valuable insights and new ideas that would
benefit others. Moreover, officials from three of the projects in our
review told us that there is value
in collecting and disseminating the knowledge acquired from IT outsourcing
projects in a systematic manner across the department.
Each of the projects in our review identified knowledge and experience
gained from their approaches to outsourcing IT services that could offer
insights and practices for other ongoing and future projects to consider.
For example:
MHS developed specific guidance and lessons learned for implementing a
performance- based incentive contract for help- desk operations. The
Department of the Navy*s NMCI project has developed a series of
lessons learned related to transitioning to the provider that is being
shared within the NMCI community; one example was that all personnel
should be available during scheduled testing and deployment. NIMA has
experience in contracting techniques emphasizing a
partnering approach with providers to refine requirements and establish a
common understanding of costs. In addition, a departmental IT outsourcing
knowledge- sharing approach could include links to information about other
government agencies* IT
outsourcing projects. For example, our 2002 report on desktop outsourcing
includes an extensive discussion of lessons learned by agencies that have
implemented this type of IT services outsourcing. 39
Developing an effective lessons learned activity is not easy. For example,
NMCI officials said that for a lessons learned initiative to be effective,
a process must exist that is clearly understood by everyone and allows
capturing and sharing of knowledge to occur with minimum effort. Other
challenges in developing an effective lessons learned process were
outlined by the Share A- 76! analyst. The analyst stated that only a small
number of site users have contributed lessons learned to the Share A- 76!
Web site, which she attributed, in part, to the amount of time and effort
needed to
document and obtain agreement by all levels of the organization on the
lessons learned. In addition, the analyst stated that there is reluctance
to share negative lessons, and often the review and approval process
sanitizes best practice information so that it becomes too general to be
most helpful to users. Such challenges can be overcome by executive- level
support. Indeed, DOD acquisition and IT officials stated that for lessons
learning activities to be effective, senior management must devote support
and
resources to the effort. This is consistent with our prior work, which
showed that knowledge can be effectively shared only when employees are
given adequate time as well as established places where they can actually
transfer knowledge. 40
Last year, we outlined a generic lessons learned process that could be
used to guide the development of such a process for outsourcing IT
services. 41 Although the mechanism or processes used to collect, share,
and
disseminate lessons learned may vary, in general such a process comprises
four main elements: collection, verification, storage, and dissemination.
The collection process involves the capture of information through
structured and unstructured processes. Verification serves to verify the
correctness and applicability of lessons submitted. The storage aspect of
lessons learned usually involves incorporating the lessons into an
electronic database for the dissemination and sharing of information,
including the ability to conduct information searches. The final element,
and the most important, is the dissemination of lessons learned, since
39 GAO- 02- 329. 40 GAO- 02- 195. 41 GAO- 02- 195.
lessons are of little benefit unless they are distributed and used by
people who will benefit from them. Lessons can be *pushed,* or
automatically delivered to a user, or *pulled* in situations where a user
must manually search for them. Lessons can also be disseminated with an
assigned priority descriptor, which denotes the risk, immediacy, and
urgency of the
lessons learned content. Conclusions The projects in our review
substantially implemented leading commercial
practices for outsourcing IT services, which has increased each project*s
probability of success. Capturing how these projects operationalized
leading commercial practices could help other IT services outsourcing
projects succeed. Although currently there is no such DOD- wide mechanism,
such as an electronic tool, to easily share and leverage lessons
learned, DOD IT and acquisition officials agreed that a departmentwide
effort to identify, capture, and disseminate lessons learned could offer
valuable insights and new ideas that would benefit others. Lessons learned
that are pragmatic and easily accessible could give DOD managers a more
informed understanding of the important issues to be addressed when making
outsourcing decisions, as well as the factors to be considered to help
ensure the success of these endeavors. DOD managers can also benefit from
lessons learned on the basis of negative experiences. The projects in our
review were well into implementation, and therefore, at this late stage,
we see little advantage for them to revisit practices that were not
implemented. Nevertheless, an electronic tool for capturing and
disseminating lessons learned would allow the rest of DOD to benefit from
the negative consequences and increased risks associated with those
practices that the projects did not implement. Developing a lessons
learned mechanism is not easy; thus, senior
management support and resources are keys to success. Without such support
driving the capture and dissemination of lessons learned, DOD is losing an
opportunity for wider application of leading practices and thus better
ensuring that its IT outsourcing efforts are successful.
Recommendations To assist DOD organizations in planning and implementing
outsourcing projects for IT services, we recommend that the Secretary of
Defense
direct the Undersecretary of Defense for Acquisition, Technology, and
Logistics, working in conjunction with the Assistant Secretary of Defense
for Command, Control, Communications, and Intelligence (C3I), to provide
senior management support and adequate resources to develop and implement
an electronic tool to capture and disseminate examples and lessons learned
from actual IT outsourcing projects. These examples and lessons learned,
at a minimum, should include the results of our review of the five
projects discussed in this report.
We also recommend that the Secretary of Defense direct the Undersecretary
of Defense for Acquisition, Technology, and Logistics, working in
conjunction with the Assistant Secretary of Defense for Command, Control,
Communications, and Intelligence (C3I), to ensure that the method used to
gather information for this electronic tool incorporate the main elements
of a lessons learned process* namely, collection, verification, storage,
and dissemination.
Agency Comments and In written comments on a draft of our report, signed
by DOD*s Principal
Our Evaluation Director, Deputy Assistant Secretary of Defense (Deputy
Chief Information
Officer), the department partially concurred with our recommendations.
Specifically, DOD agreed that capturing lessons learned in the development
and implementation of its IT outsourcing initiatives is important to
continually improve its outsourcing methods and results achieved. The
department also stated that before deciding on a specific method to
achieve this aim, it intends to explore a variety of mechanisms that could
be used. In particular, the department stated that it currently has
several processes and communities of interest that collect and disseminate
lessons learned in other areas, which are logical starting points for
determining the best path forward. DOD*s written comments are reproduced
in appendix IX.
We agree that it is prudent to explore various alternatives to leveraging
lessons learned from DOD*s IT services outsourcing experiences. Our
recommendations are not prescriptive as to the electronic method to be
used to capture and disseminate lessons learned. Therefore, the
department*s plan to explore various alternatives is consistent with our
recommendations.
We are sending copies of this report to the appropriate congressional
committees, the Secretary of Defense, and the Director of the Office of
Management and Budget. We will also provide copies to others upon
request. In addition, the report will be available at no charge on the GAO
Web site at http:// www. gao. gov.
If you have any questions on matters discussed in this report, please
contact me at (202) 512- 3439 or Linda J. Lambert, Assistant Director, at
(202) 512- 9556. We can also be reached by E- mail at hiter@ gao. gov and
lambertl@ gao. gov, respectively. Other contacts and key contributors to
this report are listed in appendix X.
Randolph C. Hite Director, Information Technology Architecture
and Systems Issues
Appendi Appendi xes x I
Objectives, Scope, and Methodology Our objectives were to determine (1)
the extent to which selected Department of Defense (DOD) information
technology (IT) services outsourcing projects use leading commercial
practices as specified in our framework and (2) whether DOD is sharing
lessons learned from its IT outsourcing projects across the department.
To determine the extent to which selected DOD outsourcing projects for IT
services use leading commercial practices, we identified the practices in
our November 2001 report on leading commercial practices 1 that (1) are
typically applied at the project level and (2) were verifiable through
documentation and interviews. Because DOD did not centrally maintain a
list of outsourcing projects for IT services, we asked the department to
identify candidate projects for our evaluation. From this list, we
selected the following five projects for our review: (1) Air Force
Kirtland Air Force Base*s Command, Control, Communications, and Computer
(C4) Services project; (2) Army Network Enterprise Technology Command*s
(NETCOM) Total Army Communications * Southwest Asia (TAC- SWA) project;
(3) Military Health System/ Information Technology Organization (MHS/ ITO)
Help Desk project; (4) Department of the Navy*s Navy and Marine Corps
Intranet (NMCI) project; and (5) National Imagery and Mapping Agency
(NIMA) Information Technology/ Information Services (IT/ IS) project. We
chose each project on the basis of the following criteria: (1) no more
than
one project from each military service and two agencies, (2) illustrative
example of DOD IT outsourcing, (3) dollar value greater than $10 million,
and (4) enough time elapsed for services to have been delivered and
performance measured.
At our request, each project completed a self- assessment on whether and
how it implemented leading commercial practices. We reviewed the agency
self- assessments and accompanying documentation and interviewed the
appropriate agency project officials to verify whether the practices were
followed. In addition, we interviewed representatives from each of the
providers associated with these projects. We also researched additional
information on commercial practices in our November 2001 guide 2 and the
challenges the federal government faces in
1 U. S. General Accounting Office, Information Technology: Leading
Commercial Practices for Outsourcing of Services, GAO- 02- 214
(Washington, D. C.: Nov. 30, 2001). 2 GAO- 02- 214.
implementing them. Specifically, we performed a literature search, which
included reviewing reports issued by leading research firms, such as
Gartner, Inc., and Giga Information Group, Inc. In addition, we
interviewed
representatives from industry organizations that have an interest in
outsourcing IT services, including the Coalition for Government
Procurement; the Information Technology Association of America; the
Professional Services Council; and Acquisition Solutions, Inc.
To determine whether DOD is sharing lessons learned from its IT
outsourcing projects across the department, we identified and reviewed
various approaches that DOD currently uses to capture and disseminate such
information. This included identifying and reviewing various Web sites and
performing key word searches on these sites to identify lessons learned
for outsourcing IT services. We also interviewed applicable DOD
acquisition and IT officials. Finally, we identified and reviewed a
generic lessons learned process contained in our January 2002 report on
the National Aeronautics and Space Administration*s lessons learned
mechanisms. 3
We performed our work at the Army*s NETCOM in Ft. Huachuca, Arizona; the
MHS/ ITO Help Desk project office in Falls Church, Virginia; the NMCI
Director*s office in Crystal City, Virginia; and NIMA*s headquarters in
Bethesda, Maryland. We conducted our review between May 2002 and early
March 2003 in accordance with generally accepted government auditing
standards.
3 U. S. General Accounting Office, NASA: Better Mechanisms Needed for
Sharing Lessons Learned, GAO- 02- 195 (Washington, D. C.: Jan. 30, 2002).
Projects* Implementation of Phase I:
Appendi x II
Determine Sourcing Strategy In the first phase of our outsourcing
framework, the client organization determines whether internal capability
or external expertise can more effectively meet its IT needs. The purpose
of this sourcing strategy is to achieve the optimal balance between
internal and external capabilities, activities, processes, and services to
ensure that strategic business objectives are achieved at the lowest risk.
Among the factors that an organization should evaluate in crafting this
strategy are technology,
business, financial, and personnel requirements and whether it has skilled
business and IT managers. In addition, according to Gartner, Inc., sound
sourcing decisions depend on whether IT organizations (1) know and
understand their business priorities, (2) are prepared to invest in some
skills and divest others, and (3) identify and assess trade- offs. 1 The
six practices in this phase that we used to evaluate the five projects are
as follows: Use third- party assistance with experience in a variety of
sourcing
arrangements when formulating a sourcing strategy. Incorporate lessons
learned from peers who have engaged in similar
sourcing decisions. Estimate impact of sourcing decision on internal
organization. Benchmark and baseline productivity of internal services
before making
the final sourcing decision. 1 Gartner, Inc., Five Tough Questions About
Skill Sourcing, Research Note SPA- 13- 2537 (Mar. 28, 2001).
Determine the business reasons for outsourcing IT. Determine reasons
for outsourcing IT that can improve the
organization*s ability to use and manage technology. Figure 6 shows that
the implementation of the practices by the five projects in our review was
uneven.
Figure 6: Project Implementation of Phase I: Determine Sourcing Strategy
Note: Not applicable* The practice was not relevant to the project*s
particular circumstances. No* The agency did not implement the practice.
Limited* The agency fully implemented some but not all aspects of the
practice and did not take alternative actions that fully satisfied the
practice. Yes* The agency fully implemented the practice or took an
alternative action that fully satisfied its intent.
Table 4 provides detailed information on whether and how each project
implemented each of the six practices in this phase.
Table 4: Summary of Projects* Use of Phase I Practices Did the project use
the practice? Air Force C4 Practice Services Army TAC- SWA MHS/ ITO Help
Desk Navy NMCI NIMA IT/ IS
Use third- party No* According to Air No* According to a
Yes* For example, Yes* The Navy Yes* NIMA
assistance with Force project
TAC- SWA project MHS worked with the
employed a employed a
experience in a officials, they official, the Army Department of the
contractor with
contractor with variety of sourcing decided that they decided that it had
Interior*s GovWorks sourcing expertise sourcing expertise.
arrangements when had sufficient inhouse sufficient expertise
Program and the and contacted other formulating a
expertise, and in- house since the
Defense Acquisition government entities sourcing strategy.
the project had no TAC- SWA contract University staff to
about their funding available to
was a consolidation obtain expertise on
experiences. employ a contractor. of three existing
sourcing strategies. However, the former
contracts. Also, MHS employed functional area chief
a contractor with told us that using
sourcing expertise. third- party assistance would have been beneficial
because the requirements would
have been better written. According to this official, the
requirements had to be substantially rewritten 2 years after contract
award.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice Services Army TAC-
SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Incorporate lessons Yes* Primarily from Yes* According to a
Yes* Primarily from Yes* Navy officials Yes* A NIMA learned from peers
other Department of TAC- SWA project
industry peers and stated that there contractor provided who have engaged
in
Defense (DOD) official, it used
MHS*s prior helpdesk were no peers that
the agency with a similar sourcing
projects that lessons learned from
function. had engaged in
report on industry decisions. implemented Office
its prior contracts for similar sourcing
best practices. In of Management and
these services and decisions because addition, NIMA held Budget (OMB)
another similar Army no other outsourcing discussions with the
Circular A- 76 contract.
project was of as National Security
policies. large a scale as Agency on its
NMCI. However, the outsourcing effort. Navy did talk to members of private
industry and the National Aeronautics and Space Administration on their
more limited efforts.
Estimate impact of Yes* The Air Force No* The Army did Yes* MHS assessed
Yes* The Navy Yes* NIMA
sourcing decision on estimated that there
not analyze the the staff and financial
performed an performed an internal organization. would be substantial
impact on its internal impact of its sourcing assessment of
assessment of internal impact, such organization
decision. staffing and other
staffing and other as to its staff, due to because, according
impacts, such as impacts, such as
its decision to to the TAC- SWA cost, related to its
cost, related to its outsource.
program manager, sourcing decision. sourcing decision.
NETCOM was outsourcing a function that was already contracted out. a
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice Services Army TAC-
SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Benchmark and No* The Air Force
No* The Army did Limited* MHS Limited* The Navy
Yes* A NIMA baseline productivity did not implement
not perform a attempted to had a private- sector
contractor baselined of internal services this practice
benchmark or establish a baseline
firm benchmark its the existing NIMA
before making the because, according
baseline analysis of its prior
environment against environment and final sourcing
to project officials, an because, according environment but,
seven large public benchmarked it to decision.
executive decision to the TAC- SWA according to project and private
peers. was made to
program manager, officials, available organizations. In outsource the C4
NETCOM was documentation was
addition, the Navy services following the
outsourcing a limited to historical and the Marine
policies contained in function that was
trouble ticket Corps performed an
OMB Circular A- 76. already contracted workload data;
analysis at a sample Accordingly, Air
out. therefore, this of representative Force officials stated baseline was
a best
locations to obtain a that such analyses
estimate. In addition, baseline. However, would not have MHS project
officials
this baseline did not affected the final
stated that a include an sourcing decision.
contractor performed assessment of the a benchmark Department of the
analysis, but they did
Navy*s legacy not provide applications since supporting
project officials documentation.
decided to rely upon an inventory developed in addressing the Year 2000
challenge. The Navy subsequently found that it had substantially
underestimated its number of legacy applications. According to NMCI
program officials, this underestimation contributed to the transition
period slipping from 2- 1/2 years to 3- 1/2 years.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice Services Army TAC-
SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Determine the Yes* To achieve cost Yes* To address
Yes* To achieve Yes* To have private Yes* To obtain business reasons for
savings and to shift shortages in military
increased industry capitalize
improved customer outsourcing IT.
military personnel to personnel to perform
productivity and infrastructure
services and other work. its mission.
customer satisfaction improvements that
decreased costs. and to decrease were needed to Also, this function costs.
quickly and securely was determined to share knowledge be a commercial
around the globe.
function under the Federal Activities Inventory Reform Act. b
Determine reasons No* According to
No* According to Yes* To improve IT Yes* To improve, for
Yes* To achieve for outsourcing IT Kirtland Air Force
TAC- SWA project expertise and example, system
better IT that can improve the Base officials, the Air
officials, improving its knowledge.
security, management
organization*s ability Force decided to ability to use and
interoperability, performance.
to use and manage hold a public/ private
manage technology reliability, and technology.
competition following was not a factor in
network response. OMB Circular A- 76 determining its policies; therefore,
outsourcing strategy. improving its ability to use and manage technology
was not a factor in determining its outsourcing strategy. Source: GAO.
a Having had the activity previously performed by a contractor does not
obviate the need to estimate the impact of a sourcing decision on the
internal organization* there are still risks involved, such as the
potential disruption of services during the transfer to the new
contractor. In fact, the staff from the incumbent contractor did not
transition to the TAC- SWA provider, and the provider had difficulty
filling
these slots within the schedule outlined in the contract. b The Federal
Activities Inventory Reform Act of 1998 requires federal agencies to
prepare and submit
to OMB, by June 30 of each year, inventories of the commercial activities
performed by federal employees.
Project*s Implementation of Phase II: Define
Appendi x III
Operational Model Critical to the successful outsourcing relationship is
an operational model for guiding the structure of the contract and the
plans for transition. In defining the operational model, client
organizations formalize executive leadership, team composition, client
responsibilities, and operating relationships between the client and
provider. The operational model helps the organization to compare its
plans with the expectations that were set as the initial decision to
outsource was made and to ascertain whether these plans will enable the
organization to meet those objectives. An important aspect of the
operational model is an explicit understanding of how the client
organization plans to communicate its needs and provide feedback to the
provider. In addition, communication between the business and IT offices
within the client organization is always critical. This is particularly
true in the case of outsourcing because the IT service provider is outside
the client organization and disconnects are more likely to occur.
Therefore, organizational processes to facilitate good communication are
critical.
The 13 practices in this phase that we used to evaluate the five projects
are as follows: Establish executive leadership for IT to facilitate the
outsourcing
initiative. Continually communicate/ clarify outsourcing objectives,
while
correcting misinformation that affects the organization. Establish a
core group of people who will be involved in all phases of
outsourcing. Select a person involved in the negotiation of the contract
to manage the
outsourcing relationship.
Create and define a contract management structure with operational
points of contact and managers.
Define the role of internal IT managers and business leaders. Ensure
that the right skills are in place to support the outsourcing
relationship. Establish a point of contact high in the provider
management structure
for elevating provider performance concerns. Have provider establish an
on- site support team to serve as liaison
between client and provider. Train provider on client business
environment and goals. Select or develop standard tools for managing the
relationship. Use third- party assistance to take advantage of expertise
from a variety
of outsourcing arrangements in defining the operational model. Ensure
that the provider management team has prior experience in the client*s
field of business.
Figure 7 illustrates that the five projects in our review largely
implemented the practices.
Figure 7: Project Implementation of Phase II: Define Operational Model
Note: Not applicable* The practice was not relevant to the project*s
particular circumstances. No* The agency did not implement the practice.
Limited* The agency fully implemented some but not all aspects of the
practice and did not take alternative actions that fully satisfied the
practice. Yes* The agency fully implemented the practice or took an
alternative action that fully satisfied its intent.
Table 5 provides detailed information on whether and how each project
implemented each of the 13 practices in this phase.
Table 5: Summary of Projects* Use of Phase II Practices Did the project
use the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Establish executive Yes* The initiative Yes* NETCOM used Yes* MHS used its
Yes* In 1999, the Yes* The NIMA
leadership for IT to was planned by a
the existing existing leadership Navy established a
enterprise facilitate the
steering group made leadership in its structure and
program executive transformation
outsourcing initiative. up of representatives
offices of operations processes. This office for IT primarily
directorate formed a from various major
and logistics to structure includes a
to support the NMCI strategic sourcing
offices at Kirtland Air provide executive
program executive outsourcing effort.
office to provide Force Base. leadership. office, steering
The Congress later executive oversight
Subsequent to committee, and directed the Navy to
of the effort. contract award, the program review identify a single Air
Force established board.
individual whose sole a functional area responsibility would chief to
manage the
be to oversee and initiative, and a direct the NMCI Lieutenant Colonel
program. As a result,
was appointed to be in February 2002, the responsible for this Navy
established the project.
NMCI Director*s Office to take over responsibility for NMCI.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Continually Yes* The Air Force
Yes* According to Yes* MHS provided
Yes* The Navy Yes* NIMA created
communicate/ clarify provided TAC- SWA project
updates to the Chief established an action
an internal Web site outsourcing communication
officials, the Information Officer
collaboration team to post information
objectives, while through, for example, applicable NETCOM
(CIO) and program structure to involve
on the outsourcing correcting briefings and
officials were review boards, the Navy community
project, distributed misinformation that meetings with
informed about the provided briefings to in the NMCI
periodic global affects organization. employees.
initiative and the deputy surgeon
communications electronic mails, and
provided comments general, and
process. Also, the held town hall
on the draft published questions
NMCI Information meetings. performance work and answers.
Bureau initiated statement. TAC- SWA
press conferences, project officials also briefings, site visits, brief
new informational commanders on the
pamphlets, and contract before they promotional material. are transferred
to the In addition, the NMCI military theater.
Director and other staff provide numerous briefings
and presentations to commands, industry gatherings, and government
officials. Finally, Web sites
and Web- based collaboration sites were established to facilitate
communications.
Establish a core Yes* The core group
Yes* According to Yes* According to
Yes* The program Yes* NIMA
group of people who included the TAC- SWA project
program officials, the executive office for IT
established the will be involved in all
contracting officer, officials, the core core group included
and the NMCI Strategic Sourcing phases of
functional area chief, group included representatives from Director*s
Office, which was outsourcing.
and manpower representatives from the MHS program
organization dedicated to specialist. logistics, operations, executive
office,
comprise the core managing and and resource
MHS IT program group responsible for
facilitating the management as well offices, military
managing the outsourcing initiative.
as the contracting department chief
outsourcing initiative. officer.
information officers* offices, and Interior*s GovWorks
organization, which provided contracting services.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Select a person Yes* The contracting Yes* The contracting Yes* According
to
Yes* The primary Yes* A lead
involved in the officer. a officer. MHS program contracting officer.
contracting officer.
negotiation of the officials, the chair of contract to manage the contract
the outsourcing evaluation committee relationship. and the individual in
charge of transition
are responsible for managing the outsourcing
relationship. Create and define Yes* The structure
Yes* The overall Yes* Responsibility Yes* The primary
Yes* NIMA has contract
was defined by the responsibility for the for contract
contracting officer appointed an management
roles and contract rests with management is
establishes operational point of
structure with responsibilities of the
the contracting jointly held by the
procedures and contact for each of operational points of contracting
officer
officer. The contracting officer, controls necessary the seven functional
contact and
and her staff. contracting officer*s who is part of
for effective areas being managers.
representative acts Interior*s GovWorks
contractual oversight outsourced, which as a liaison between
organization, and the of the NMCI initiative are documented in a
the government and MHS program office,
and has a matrix *rules of
the contractor. which provides dayto- relationship with the
engagement* day oversight of
NMCI Director. The agreement between
the provider. primary contracting
the government and officer certifies
the contractor. contracting officer representatives for the NMCI contract
to provide technical coordination efforts.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Define the role of Yes* Kirtland Air Limited* TAC- SWA
Yes* MHS*s help- Yes* The NMCI
Yes* NIMA internal IT managers
Force Base did not project officials
desk performance execution plan transformation teams
and business initially define the
explained the roles of assessment plan describes the roles of
for the activities leaders.
roles of its internal IT the principal staff
defines the roles of internal and external being outsourced
and business involved with this
various groups and organizations that
define the roles of managers. However, project but generally individuals
directly or indirectly their managers and subsequent to did not have
associated with the
affect the leaders. contract award,
supporting program. management of
Kirtland Air Force documentation NMCI and explains in Base established a
defining these roles. detail the duties and
CIO committee and responsibilities of the configuration control
program executive board comprising the
office for IT and the functional area chief
NMCI program (Kirtland*s CIO) and
management offices. representatives from various business areas. Although
not exclusively devoted to the C4 Services project, these groups help
provide direction to the project.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Ensure that the right Yes* According to
Yes* According to Yes* According to
Limited* The Yes* According to
skills are in place to Air Force officials, the TAC- SWA project MHS
program
Director, NMCI, has NIMA, its
support the the right skills are in
officials, the Army officials, the TriService staff that report
transformation teams
outsourcing place. has created several Management
directly or are provide the skills
relationship, positions to ensure
Program Office, matrixed with him
necessary to support including those that the right skills
GovWorks (the that are responsible
the outsourcing dealing with
are in place to Interior organization for performing all but relationship.
manage the that provides
two of these contract
outsourcing contracting
functions. management,
relationship, assistance to MHS), Specifically, at this
financial including a
and a contractor time, NMCI does not
management, contracting officer, collectively employ have staff assigned
IT management,
legal advisor, the skill sets needed to support teaming
negotiation operations and to support the and interpersonal strategies,
logistics personnel outsourcing
relationships and teaming and
as well as a relationship.
relationship interpersonal NETCOM unit
management. The relationships, commander.
Navy recognizes the project
According to Army need for these skills management, and
TAC- SWA officials, and is taking, or relationship each position
plans to take, various management.
employs an actions to obtain individual with the
these skills. necessary skills to support the outsourcing relationship.
Establish a point of Yes* According to
Yes* The provider Yes* The provider Yes* The NMCI
Yes* The contact high in the the contracting
established a established an Director interacts
transformation team provider officer, concerns can
program manager as executive program
directly with the charters define
management be addressed to the the point of contact
manager as the point provider*s program
procedures for structure for
senior vice president for elevating
of contact for executive on a
addressing issues, elevating provider at the provider
concerns. elevating concerns.
regularly scheduled including possibly performance
headquarters. basis; such meetings elevating concerns to concerns.
would include any the provider*s performance issues.
general manager. Have provider
Yes* The provider Yes* Site managers
Yes* The provider*s Yes* The provider Yes* The rules of
establish an on- site has established an have been
transition plan establishes site leads engagement define
support team to on- site support team, established in
identifies the on- site that remain on- site
the on- site support serve as a liaison which is led by the
accordance with the support team.
as the location goes team for each major
between client and on- site manager.
contract. through the cutover
NIMA location. provider.
to NMCI.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Train provider on Limited* Prior to
Not applicable* The Yes* MHS officials
Limited* The Navy Not applicable*
client business contract award,
Army contract was a stated that they did not provide
According to the environment and
Kirtland Air Force consolidation of prior
provided some training on its
project*s contracting goals.
Base provided an contracts for this training to the
business officer, such training
orientation session in activity, and the provider on their
environment and was not necessary which they discussed
winning bidder was business
goals; instead it because the senior their operations,
one of the incumbent environment. In
relied on the NMCI officers of the
business contractors.
particular, MHS design reference
provider are all environment, and
trained and certified mission document,
recent NIMA goals. Subsequent to the provider*s staff on which was
included management contract award,
the agency*s with the request for
employees and were specific Kirtland Air
application systems. proposals (RFP). already
Force Base In addition, the This document
knowledgeable of government provider had
defined the NMCI these areas. For personnel were
previous MHS operational
example, among responsible for aiding
experience. environment. those who
the provider during However, a provider transitioned to the
the transition period. official stated that provider was a However,
provider
although the former director of the
officials stated that document was
hydrographic they did not receive useful, it did not
topographic center of adequate training, identify Navy
the Defense Mapping which made the enterprisewide
Agency, a transition period operations. predecessor agency more difficult.
According to NMCI
to NIMA. program officials, the provider somewhat mitigated this problem
by hiring several highly
knowledgeable staff from the Department of the Navy shortly
after contract award. Select or develop
Yes* For example, Yes* NETCOM uses Yes* According to
Yes* The Navy is Yes* NIMA used
standard tools for the revised contract
a monthly report MHS officials, the using a balanced various tools,
managing the outlines the use of from the contracting
program office and scorecard process to
including twice- a- day relationship (e. g., specific software to
officer*s the provider have provide Navy and
performance reports performance
help manage the representative to
selected, for Marine Corps for the operational scorecards, provider*s
evaluate the provider.
example, an leadership with help- desk function enterprise resource
performance. enterprise resource information to judge
and monthly management
management how well NMCI is performance reports.
system). system.
supporting the missions and strategies of the
department.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Use third- party Yes* Although thirdparty
No* The Army Yes* For example, No* The Navy Yes* NIMA
assistance to take assistance was believed that it had
MHS contacted the believed it had
employed advantage of not used at the onset sufficient expertise
Interior*s GovWorks sufficient internal
contractors to expertise from a
of the C4 Services in- house.
Program, Defense experience and develop contractual variety of outsourcing
project, a person with Acquisition expertise. roles and assist in
arrangements in prior experience in University, and evaluating the
defining the managing IT
others. Also, MHS contract.
operational model services contracts for employed a (i. e., defining roles
the Air Force was contractor with
and responsibilities). brought in to provide
sourcing expertise. advice. He was subsequently put in charge of the
project.
Ensure that the Yes* Provider
Yes* Vendor past Yes* The request for
Yes* According to Yes* It was expected provider management
performance was quote laid out MHS*s
NMCI program that most of the
management team capabilities and
part of the evaluation evaluation criteria for officials, the Navy
provider*s workforce has prior experience experience in
criteria during source selecting a provider,
required bidders to would be composed in the client*s field of providing
IT services
selection and the which included prior provide information
of former NIMA business.
were evaluation provider was an experience in health
on their prior employees. Among factors in the
incumbent contractor care.
experience in related those who technical evaluation with NETCOM.
business fields. For transitioned to the
process. example, the provider was a
solicitation required former director of the
bidders to hydrographic
demonstrate topographic center of experience the Defense Mapping managing
a similar
Agency. effort of 100, 000 or more seats at one time. b
Source: GAO. a The Air Force*s initial contract was not negotiated because
Kirtland Air Force Base used an OMB
Circular A- 76 2- step, sealed bid process. However, the contracting
officer was involved in the bid process as well as in managing the winning
contractor. b This threshold could be met through multiple efforts, but at
least one had to include at least 20,000
seats.
Project*s Implementation of Phase III: Develop
Appendi x IV
the Contract Phase III focuses on the development of the contract, which
defines the legal terms of the relationship between client and provider.
While other phases in the outsourcing process describe the need for mutual
trust and a close, flexible working relationship, this phase focuses on
the development of the contract, which is the foundation on which a
working relationship will be built. A well- written contract helps the
outsourcing organization meet its requirements while allowing the service
provider to make a fair profit. It sets the expectations for service
levels, delivery of essential services, and continuous improvement and is
intended to protect the interests of all parties.
The 16 practices in this phase that we used to evaluate the five projects
are as follows: Base performance requirements on business outcomes.
Include measures that reflect end- user satisfaction as well as technical
IT performance. Review and update performance requirements periodically.
Require the provider to meet the minimum performance in each
category of service. Require the provider to achieve escalating
performance standards at
agreed- upon intervals. Incorporate sufficient flexibility so that
minimum acceptable performance can be adjusted as conditions change, as
the provider
becomes more adept at satisfying customer demands, and as improvement
goals are achieved.
Use service- level agreements (SLA) 1 to clearly articulate all aspects
of performance, including management, processes, and requirements.
Client and provider work together to define the appropriate number of
SLAs and appropriate structure for each. Specify circumstances under
which the provider is excused from
performance levels mandated by master service agreements. Client and
provider work together to identify SLAs for which
compensation is based, while additional SLAs may be defined to manage
performance.
The contract should include clauses for (1) determining pricing
structures; (2) performing customer satisfaction surveys and using the
results to redefine performance levels; (3) terminating the contract; (4)
resolving disputes in a timely manner; (5) taking work away, without
penalty, from provider for nonperformance; (6) declaring a significant
event that can lead to a change in the contract; (7) defining performance
requirements; and (8) conducting regularly scheduled meetings.
Consider setting up a master services agreement under which all
arrangements between client and provider operate.
Include the appropriate representation from each major organizational
unit on the contract negotiation team.
Specify the use of volume purchases to obtain optimal discounts. Use
third- party assistance in negotiating and developing the contract. Sign
the contract after contract negotiations and final vendor selections. As
figure 8 shows, the five projects largely implemented the practices. 1
SLAs define the agency*s expectations and are used to track and measure a
contractor*s performance.
Figure 8: Project Implementation of Phase III: Develop the Contract
Note: Not applicable* The practice was not relevant to the project*s
particular circumstances. No* The agency did not implement the practice.
Limited* The agency fully implemented some but not all aspects of the
practice and did not take alternative actions that fully satisfied the
practice. Yes* The agency fully implemented the practice or took an
alternative action that fully satisfied its intent.
Table 6 provides detailed information on whether and how each project
implemented each of the 16 practices in this phase.
Table 6: Summary of Projects* Use of Phase III Practices Did the project
use the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Base performance Ye s *T h e Yes* The
Yes* Industry Yes* The
Yes* Performancebased requirements on performance performance
standards and performance requirements
business outcomes. requirements are requirements are
metrics were used to requirements are
are in the contract, contained in the
outcome based and base performance included in the including those
contract and are
stated in terms of the requirements on
contract and are related to customer based on business
level of operation and business outcomes.
based on business satisfaction, process
outcomes. For maintenance
outcomes. SLAs improvement, quality
example, the services required.
establish the control, and timely requirements define
According to TACSWA performance and accurate
the percentage of officials, standards and completion of
system availability multiple trips to
service quality for all requirements. needed to support
overseas locations types of NMCI seats.
users. were made to identify
the performance requirements.
Include measures Limited* The Air
Limited* The Army*s Yes* MHS measures Yes* The SLAs Yes* The SLAs
that reflect end- user Force measures
contract includes the service provider*s include measures for
contain technical IT satisfaction as well
include technical IT technical IT
technical IT technical IT
performance as technical IT
performance. performance performance and performance and
measures. In performance.
However, end- user measures but not
uses an electronic customer
addition, after a satisfaction end- user satisfaction customer
satisfaction
satisfaction. In function is measures are limited performance survey to
assess the
addition, the contract transitioned, the to the percentage of
measures. For quality of help- desk
requires the service contractor must
complaints received example, the contract services. Satisfaction
provider to measure provide a plan that
and do not measure includes a
rates and number of and report on overall
includes measures overall customer
requirement for helpdesk survey responses customer satisfaction
for customer satisfaction. Project
services, but are tracked and used
with NMCI services. satisfaction, which
officials did not know does not include for both evaluation the provider
is why overall end- user measures related to
and incentive contractually
satisfaction customer satisfaction
payments (positive obligated to meet.
measures were not associated with such and negative).
established. services. According
to the project official that developed the performance work statement in
the
contract, NETCOM did not include such customer satisfaction measures
because it did not think that it was necessary to have a performance
standard for that service.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Review and update Ye s*The Yes* The contract is Yes* According to
Yes* Performance Yes* According to
performance performance reviewed and
MHS program reviews and
NIMA, requirements requirements
requirements were modified as needed officials,performance adjustments are
are updated as periodically.
revised about 2 years on the basis of requirements are
ongoing. For needed or once a
after contract award. additions or changes
updated as required example, the Navy is year when the
to such and reviewed on an
in the process of government is
requirements. annual basis before
refining its SLAs to assessing whether to
According to TACSWA the decision is made
ensure that they exercise its annual and provider
to execute the option adequately reflect option.
officials, changes are year on the contract.
technical coordinated between
performance and the government and
customer satisfaction the contractor before needs. they are finalized.
Require the provider Yes* The minimum Yes* The contract
Yes* The minimal Yes* The SLAs Yes* The provider
to meet the minimum performance is
defines minimum acceptable provide the basic
must meet the performance in each
defined in the performance
performance criteria level of service the performance
category of service. contract.
requirements. the contractor must
contractor must standards as
meet are based upon deliver for every
specified in the commercial industry NMCI seat.
contract, SLAs, and standards and are
NIMA*s performance defined in the
assessment plan. contract.
Require the provider No* According to Not applicable*
Yes* The contract No* According to
Yes* The contract to achieve escalating Kirtland Air Force
According to a TAC defines negative,
NMCI program allows for the
performance Base officials, they SWA project official,
acceptable, and officials, the Navy redefinition of service
standards at agreed did not include
this practice was not positive incentive
expected vendors to levels. In addition,
upon intervals. escalating
applicable because ranges associated develop their pricing
according to NIMA performance NETCOM believed with escalating proposal
assuming 8
program officials, the standards because that the performance
performance years b of providing
need for escalating the focus of this standards in the
standards. service in performance
contract was on contract were
accordance with the standards will be meeting
already high. SLAs included in the
evaluated on an as requirements at the
Provider officials also RFP.
needed basis as well least cost. a stated that the as during the
performance
semiannual award requirements in the
fee analysis period contract were high. and during the The Army expected,
annual review on and the contractor whether to exercise agreed, to meet
the contract option. these standards However, such immediately. escalating
standards
have not yet been identified.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Incorporate sufficient No* The contracting
Not applicable* Yes* According to
Yes* The contract Yes* According to
flexibility so that officer said that the According to a TACSWA
MHS program includes a provision
NIMA, the minimum acceptable requirements should project official,
officials, the contract requiring the provider
performance performance can be
be built into the NETCOM believed incorporates
to submit an annual measures are
adjusted as contract, which can
that the performance sufficient flexibility to plan for technology
designed to first conditions change,
be modified if standards in the adjust minimal
refreshment and stabilize NIMA*s IT
as the provider necessary.
contract were acceptable deployment. This
environment and becomes more adept already high, and
performance as plan is to include
then to be adjusted at satisfying
provider officials conditions change.
proposed revisions to to bring minimum
customer demands, agreed. According to For example, the the contract and
an performance into and as improvement
the contracting contract includes
estimate of changes alignment with goals are achieved.
officer, the military incentives for the
in performance that industry best
theater where service provider to
would result. practices. Also, the
performance is introduce new
contract indicates delivered is not an
capabilities and new that the SLAs may be area where anything
services within the further defined
less than meeting the scope of the
throughout the defined standards is
performance work contract life. acceptable.
statement without further competition. Use SLAs to clearly
Yes* The SLA Yes* The TAC- SWA
Yes* The contract Yes* According to Yes* The contract
articulate all aspects defines the
project did not use defines the NMCI officials, SLAs
incorporates SLAs, of performance, requirements, SLAs, but the
requirements, cover all aspects of which articulate including
processes, and who contract addresses
service performance provider performance management,
is responsible for the management,
standards, performance. For requirements and
processes, and meeting the
processes, and thresholds, example, the E- mail take effect when the
requirements. requirements.
requirements objectives, and
services SLA function is associated with the
metrics as well as the includes location and transitioned to the project.
requirements for
frequency of the provider.
quarterly service, performance
management reviews categories, the
and quality performance assurance plans.
measures, and methods of measurement.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Client and provider Yes* Although the Yes* SLAs are not Yes* MHS did not
Yes* The SLAs were Yes* According to
work together to original SLA was used but, according
work with the service developed by the
NIMA and service define the prepared by the Air
to TAC- SWA and provider to determine
Navy with the help of provider officials, appropriate number Force, the
revised provider officials, the number and a third- party
they used a of SLAs and
SLA included they work together
structure of these contractor. According partnering
appropriate structure contractor input and on any contract
requirements. to the NMCI Deputy contracting approach for each.
concurrence. modifications dealing However, according
Director for to jointly define, with requirements
to MHS and provider Enterprise
develop, and and performance officials, they have Operations, it would
structure the SLAs. levels. In addition, worked together on have been the
partnering clause
revisions to these inappropriate to work in the contract
requirements. with individual
emphasizes a mutual competing commitment
contractors before between government the contract was and industry to
work awarded. However, as a team.
NMCI staff and the service provider are now working together on SLA
revisions.
Specify Yes* The contract
Yes* The contract Yes* For example, Yes* Contract terms
Yes* The contractor circumstances under
includes a clause includes clauses that
the provider*s and conditions can be excused from which the provider is
that the contractor is the provider is
proposal, which is specify the situations
meeting performance excused from excused in the event
excused from incorporated as part where the service
levels with the performance levels
of government delay performance levels if
of the contract, provider does not
permission of the mandated by master of work.
there are states that during
have to meet the contracting officer if service agreements.
government delays transition periods,
SLAs. For example, circumstances occur or factors beyond its
metrics will not be during transition, the beyond the service
control. reported for incentive
service provider provider*s control. and penalty
does not have to purposes. As a meet the result, according to performance
levels program officials, the
set by the SLAs. provider would be excused from required performance
levels when baseline requirements are being established for new
applications.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Client and provider Yes* According to Not applicable*
Yes* Before the Yes* Meetings were
Yes* NIMA and the work together to Kirtland Air Force
SLAs are not used. contract was held with potential service provider work
identify SLAs for
Base officials, the awarded, MHS did
bidders to obtain together in
which compensation service provider had not work with the their input in
defining transformation teams
is based, while input on the revised
contractor to identify the SLAs, including to develop SLAs, additional
SLAs may
SLAs, including SLAs for which discussions of
including those be defined to those for
compensation is compensation. In
affecting manage
compensation. based. However, addition, Navy and
compensation. performance.
according to MHS the provider are now and provider officials, refining the
SLAs, after award, there including those have been instances
affecting in which they have compensation. worked together on refining the
SLAs,
including those affecting compensation.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
The contract should Limited* All clauses
Yes* All contract Yes* All contract
Yes* All contract Yes* All contract
include clauses for are included in the
clauses are included. clauses are included. clauses are included. clauses
are included. contract, except a determining pricing
clause pertaining to structures;
customer satisfaction performing
surveys. customer satisfaction surveys and using the
results to redefine performance levels; terminating the
contract, including early terminations; resolving contract
disputes in a timely manner; taking work away, without penalty, from the
provider for nonperformance;
declaring a significant event that can lead to a change in the contract;
defining
performance requirements; and conducting
regularly scheduled meetings. Consider setting up
Yes* According to Yes* According to
Yes* According to Yes* According to Yes* According to
master services Kirtland Air Force
TAC- SWA project MHS program
NMCI program NIMA, the contract is
agreement under Base officials, the
officials, the contract officials, the contract
officials, the contract considered the which all
contract, which is considered the is considered the
is considered to be a master services
arrangements incorporates the master services master services
master services agreement.
between client and contractor proposal,
agreement. agreement.
agreement. provider operate.
governs all arrangements between the government and the contractor and is
considered the master services agreement.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Include appropriate Not applicable* The Ye s *
Yes* Staff from Yes* The contracting
Yes* The NIMA representation from C4 Services contract
Representatives from affected program
team was built with contract negotiating each major
was awarded under a the field and
management offices experts from each of
team included organizational unit on
sealed bid process headquarters
and the chief the major systems
representatives from contract negotiation and was not
commands and the information offices
commands. Also, the each major
team. negotiated.
contracting office were on the
source selection organizational unit.
participated in negotiation team. evaluation board
contract negotiation. consisted of more
than 50 people from various commands.
Specify the use of Not applicable*
Not applicable* Yes* Call bands c are Yes* The contract
Yes* NIMA has volume purchases to
According to Kirtland According to TACSWA
used to obtain includes volume authorized the
obtain optimal Air Force Base
project officials, optimal discounts on discounts. For
provider to use discounts.
officials, such volume such volume the number of calls
example, seat government sources, purchases were not purchases were not
being fielded to the moves, adds, and such as the Federal relevant to this
relevant to this
help desk. changes are
Supply Schedule, in contract.
contract. purchased in bulk procuring products, only, because prices
services, and for these actions are
supplies related to lower when
this contract. purchased in groups
According to the of 250.
contracting officer, this authorization was made so that the provider
could take advantage of discounts available to
the government. Use third- party
No* This was not No* NETCOM
Yes* Private- sector Yes* The Navy used
Yes* NIMA obtained assistance in done because the Air officials believed
that
and other private- sector firms
help from (1) private negotiating and
Force staff believed the Army*s government
to assist in contractors to help
developing the that it had adequate designated organizations
developing the develop the contract.
expertise available contracting agency provided assistance.
overall NMCI concept performance work in- house.
had adequate and negotiation statement, SLAs,
expertise and strategies as well as
and award fee plan; experience.
in drafting the (2) another agency
contract documents. on a particular
contract technique; and (3) a privatesector firm to compare the vendor*s
proposal with industry best practices and trends and to attend some
negotiation meetings to answer questions.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Sign contract after Yes* The contract
Yes* The contract Yes* The contract
Yes* The contract Yes* The contract
contract negotiations was signed after
was signed after was signed after
was signed after was signed after
and final vendor negotiations and final
negotiations and final negotiations and final negotiations and final
negotiations with the selections.
vendor selection. vendor selection.
vendor selection. vendor selection.
selected vendor. Source: GAO.
a Requiring a provider to meet escalating requirements and focusing on
achieving results at the least cost are not mutually exclusive goals. As
the provider becomes more familiar with the client organization, it may be
able to exceed the original performance requirements at the same, or
possibly lower, cost. b The NMCI contract was subsequently extended to 7
years, with an option for an additional 3 years.
c Call bands are call volume ranges used to determine contractor pricing.
Projects* Implementation of Phase IV: Select
Appendi x V
the Provider( s) Critical to the success of any project to outsource IT
services is the identification of potential providers and the ultimate
selection of a provider( s) that will best meet the needs of the
organization. Developing a strategy that will lead to the selection of the
*right contractor* is especially important in a performance- based
acquisition. 1 The overall success of the outsourcing project requires the
contractor to understand the
performance- based approach, know or develop an understanding of the
organization*s requirement, have a history of performing exceptionally in
the field, and have the processes and resources in place to support the
mission.
The seven practices in this phase that we used to evaluate the five
projects are as follows:
Conduct research on the state of the market, vendors, and technology
before defining vendor selection criteria.
Identify and evaluate various sourcing solutions (e. g., single vendor,
multivendor, and alliance). Define a process for selecting vendors to be
providers. Define vendor selection and evaluation (acceptance) criteria
at the
outset. 1 An Interagency- Industry Partnership in Performance, Seven Steps
to Performance- Based Services Acquisition, Benchmark Version (January
2002).
When issuing an RFP, identify services with expected performance levels
and define client and provider roles and responsibilities.
Use third- party assistance with expertise in a variety of outsourcing
arrangements when selecting provider( s), including developing the RFP.
Conduct due diligence activities to verify vendor capabilities before
signing the contract.
As figure 9 illustrates, all five projects largely implemented the
practices.
Figure 9: Project Implementation of Phase IV: Select the Provider( s)
Note: Not applicable* The practice was not relevant to the project*s
particular circumstances. No* The agency did not implement the practice.
Limited* The agency fully implemented some but not
all aspects of the practice and did not take alternative actions that
fully satisfied the practice. Yes* The agency fully implemented the
practice or took an alternative action that fully satisfied its intent.
Table 7 provides detailed information on whether and how the projects in
our review implemented each of the seven practices in this phase.
Table 7: Summary of Projects* Use of Phase IV Practices Did the project
use the practice? Air Force C4 Practice Services Army TAC- SWA MHS/ ITO
Help Desk Navy NMCI NIMA IT/ IS
Conduct research on Yes* The Air Force Yes* A Commerce Yes* MHS had a
Yes* The Navy and a Limited* NIMA staff
state of the market, advertised in the
Business Daily notice third- party contractor private- sector firm and a
contractor
vendors, and Commerce Business was published perform market performed
market
researched technology before Daily and interested identifying TAC- SWA
research. research to, for
preferential providers defining vendor
parties submitted requirements to example, help define before defining
selection criteria. statements of interested parties. In
the market conditions vendor selection capabilities. This was
addition, a draft RFP and vendor selection criteria. NIMA
done before the Air was issued before
criteria. The Navy program officials
Force identified the the final solicitation to
also held meetings noted that they vendor selection solicit comments
with leading limited their analysis criteria.
from industry that companies that had
to such providers might affect the outsourced IT
because they had requirements. services on an
previously decided According to the
enterprisewide level. on a strategy to
contracting officer, Finally, more than directly convert their comments
received 200 companies
IT/ IS activities to a from industry were
participated in the preferential provider.
incorporated into the July 1999 NMCI
final solicitation, Industry Day where applicable.
Conference, which Additionally, a
informed companies preproposal about the NMCI conference was
vision, requirements, conducted in the
and procurement overseas operations
strategy. location to inform industry representatives
about issues and the procurement strategy.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice Services Army TAC-
SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Identify and evaluate Yes* As outlined in
Yes* A multivendor Yes* According to an
Yes* The Navy Yes* NIMA
various sourcing OMB Circular A- 76, approach has been
MHS program evaluated singlevendor
considered the solutions (e. g.,
the Air Force used in the past,
official, MHS and
sourcing solutions single- vendor,
evaluated privatesector which the Army evaluated continuing
multivendor allowed under OMB
multivendor, and and internal opted not to
with the status quo, approaches. A joint Circular A- 76,
alliance). government staff
continue. Instead, using another federal
Navy and Marine including the direct proposals to satisfy
NETCOM chose to agency*s help- desk
Corps team conversion to a
the C4 services consolidate its services as part of a
determined that preferential provider,
requirements. requirements and
cross- services single- vendor point of holding a contractor oversight
agreement, and contact for
public/ private by choosing a singlevendor contracting with a
accountability and competition, or
solution. commercial firm.
responsibility was obtaining an agency
According to the critical to mission
cost comparison MHS program office,
success. waiver. NIMA MHS chose a singlevendor
decided that directly solution to
converting the consolidate
selected IT functions requirements and to a preferential performance
procurement source a accountability. was the lowest risk to its mission
and NIMA employees.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice Services Army TAC-
SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Define a process for Yes* As explained in
Yes* The contract Yes* MHS used a
Yes* The NMCI RFP Limited* NIMA*s selecting vendors to
the invitation to bid, was awarded on
selection process used a negotiated decision to use a
be providers (e. g., the Air Force
*best value* that included the
commercial items phased direct
issuing an RFP and followed the two- step considerations of
prequalification of 13 evaluation process. conversion of its
prequalifying process outlined in technical and industry leaders in The
vendor
IT/ IS functions to a vendors).
OMB Circular A- 76 management both medical and evaluation criteria
statutory, preferential for public/ private capabilities, past commercial
helpdesk included technical procurement vendor
competitions. performance, and
operations. approach, limited its vendor
price, as defined in MHS also issued a
management plan, selection process to
the solicitation. request for comment
small business the identification of and a request for
utilization, past potential Alaska
quote, which defined performance, and Native Corporation
vendor evaluation price. companies capable
and selection criteria of performing the
on the basis of requirements. NIMA technical approach,
decided to take this past performance, approach because it key personnel
believed that it would qualifications,
avoid schedule organizational delays and mission experience, and risk that
it thought price.
would result from taking other approaches. In addition, NIMA
program officials believed that the preferential provider approach would
maximize the retention of institutional
knowledge whether employees stayed at NIMA or transitioned to the
provider.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice Services Army TAC-
SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Define vendor Yes* The criteria Yes* The solicitation
Yes* Vendor Yes* The final RFP
Yes* NIMA selection and
were included in the included vendor evaluation and
required bidders to evaluated the past
evaluation invitation to bid.
evaluation and selection were based
have implemented performance and
(acceptance) criteria selection criteria. The upon industrydefined and
provided service
financial capabilities at the outset.
source selection help- desk
to at least 100,000 of Alaska Native evaluation plan criteria. MHS used a
seats, of which Corporation describes the
multistep process 20, 000 were to be on companies. evaluation process.
that included an the same contract. In According to NIMA,
assessment of addition, bidders only two companies
minimum were evaluated on
had demonstrated qualifications, their technical
successful evaluation of written
approach, performance in technical and cost
management plan, service environments proposals, and oral
small business similar to NIMA*s presentations for utilization, past
requirements. qualified vendors. performance, and
However, to be able price. Finally, vendors to fully meet NIMA*s in the
competitive requirements, these
range had to provide vendors formed a
demonstrations of joint venture, which technical network was awarded the
operations centers to
contract. verify their ability to achieve required
service levels set forth in technical proposals.
When issuing an Yes* The services Yes* The RFP
Yes* The request for Yes* The NMCI
Yes* Because this RFP, identify services and performance
identified the quote identified
solicitation was a sole source
with expected levels were included government and
services and documents identified contract, an RFP was performance levels
in the original contractor roles and expected the required services
not issued. However, and define client and performance work
responsibilities, and
performance levels and expected according to NIMA provider roles and
statement included system operational and specified a performance levels.
and provider officials, responsibilities. in the invitation to and
availability
performance- based, These documents, they worked jointly to
bid. requirements. incentivized, sharedrisk along with the
refine the relationship with
contract, define the requirements and
the service provider. client and provider
expected It also defined client
roles and performance levels
and provider roles responsibilities.
and to define client and responsibilities. and provider roles and
responsibilities in the contract.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice Services Army TAC-
SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Use third- party No* This was not
No* The Army Yes* A privatesector Yes* A private- sector
Yes* NIMA used a assistance with done because the Air
believed that it had contractor, the
firm assisted with private- sector firm expertise in a variety
Force believed that adequate Defense Acquisition
market research and and the NIMA
of outsourcing adequate in- house government
University, and developing vendor
Acquisition Center to arrangements when expertise was
experience and GovWorks (an
pass/ fail criteria. support the
selecting provider( s), available.
expertise covering Interior organization
Another contractor evaluation of the
including developing technical, resource offering procurement
assisted the Navy vendor proposal.
the RFP. management, and
services to with developing SLAs
contracting areas. government
and the technical agencies) provided evaluation of assistance in this
vendors. area.
Conduct due Yes* The Air Force
Yes* NETCOM Yes* Minimum
Yes* As part of due Yes* NIMA
diligence activities to performed a evaluated the
vendor qualifications diligence, the Navy
researched both verify vendor technical evaluation
contractor*s financial were established and
ensured that all parent companies of
capabilities before of the vendor*s
and past evaluated early in the bidders had relevant
the new joint venture signing the contract. capabilities. Also, as
performance
process for selecting experience corporation to ensure
part of its due information before
the provider. implementing large
that they had diligence activities,
signing the contract. seat management
financial and the Air Force
contracts. In addition, technical backing.
reviewed the past performance,
debarred list and including reference DOD*s Central
checks, was a Contractor Registry.
source- selection evaluation factor. Source: GAO.
a Preferential procurement programs are special commercial source
programs, such as Federal Prison Industries and the workshops administered
by the Committee for the Purchase from the Blind and Other Severely
Handicapped under the Javis- Wagner O*Day Act.
Projects* Implementation of Phase V: Appendi x VI
Transition to Provider( s) In this phase, client organizations transfer
responsibility of IT functions to one or more providers. A clear
definition of responsibilities among the various parties and the careful
consideration of employees* needs matched against the organization*s needs
enable both the client and provider to focus on execution and give staff
confidence in their future employment. If
the contractor is assuming responsibility for functions previously
performed by federal employees, it is especially important that the
organization communicate a clear transition process. Without such
communication, an outsourcing project can be negatively affected if
misinformation and mistrust ensues.
The 11 practices in this phase that we used to evaluate the five projects
are as follows: Communicate a clear transition process to all key
players from both
client and provider organizations. Clearly communicate to employees what
is going to happen and when it
is going to happen. Establish a client transition team with
representatives from across the
organization to facilitate the transition. Place the transition under a
single program manager. Create client/ provider transition teams to
address short- term transition
tasks as required. Recognize that it takes time to effect transition and
plan accordingly.
Encourage the transition of staff to the provider, where appropriate,
using bonuses, stock options, and other appropriate methods.
Develop employee- retention programs and offer bonuses to keep key
people, where appropriate.
When consistent with organizational objectives, assist employees who do
not want to transfer in finding other jobs, either within an organization
or at another organization.
Document key information to preserve organizational knowledge in the
event that one or more providers change.
Use change management strategies to help client employees deal with the
transition. Figure 10 demonstrates that the five projects in our review
were largely using the practices.
Figure 10: Project Implementation of Phase V: Transition to Provider( s)
Note: Not applicable* The practice was not relevant to the project*s
particular circumstances. No* The agency did not implement the practice.
Limited* The agency fully implemented some but not all aspects of the
practice and did not take alternative actions that fully satisfied the
practice. Yes* The agency fully implemented the practice or took an
alternative action that fully satisfied its intent.
Table 8 provides information on whether and how the projects implemented
each of the 11 practices in this phase.
Table 8: Summary of Projects* Use of Phase V Practices Did the project use
the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Communicate a clear No* There were two
Yes* The provider Yes* The contract
Yes* The Navy Yes* Several NIMA
transition process to transition plans developed a phase- in delivery
order and developed numerous
transformation teams all key players from prepared, one by the plan that
addressed
the provider*s guides and Web sites were formed,
both client and government and one
personnel issues, the transition plan laid to communicate with
consisting of both provider by the provider.
integration of added out a transition
the NMCI community client and provider
organizations. However, the
locations, updating process. In addition, about site, technical,
personnel, to develop government*s plan plans, and finalizing
a kickoff meeting was and personnel
a single transition contained conflicting new subcontract/ held between
the
transition processes, plan. The plan was direction. In addition,
sponsorship client and provider to
procedures, and developed to provider officials said arrangements. This
communicate the tasks.
formalize that the two separate
phase- in plan was transition process to
communications with transition plans were
discussed at a all key players.
all key players. not well coordinated
postaward and that the Air
conference, attended Force did not have a
by representatives strong advocate to from both the client ensure that the
and provider. transition process was well planned and executed. Both
provider and Kirtland Air Force Base officials acknowledged that there
were problems with the transition
process, including incorrect information about upgrades that would be made
before the transition
and staff morale problems that hampered knowledge transfer to the
provider.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Clearly communicate Yes* The Air Force
Limited* According Yes* The
Limited* The Navy Yes* With help from
to employees what is kept its employees
to TAC- SWA officials, government task has used its normal
a contractor, NIMA going to happen and informed of the
the Army manager for the TriService chain of command to developed a
when it is going to outsourcing project communicated with communicate
communications happen.
through town hall its employees, but
Infrastructure transition
plan. The plan meetings and they did not provide Management Office
information, but included town hall internal briefings. As
documentation held regular found that the
meetings, global part of this process, supporting any
meetings with MHS implementation was
E- mail messages, employees were
communication program officials to
uneven. As a result, and a Web site
informed about the efforts. However, the discuss transition some staff did
not
posting that provided reduction- in- force provider was
issues. The MHS know current
employees with process and the
required to submit a program office also information about
access to transition procedure for those
phase- in plan, which periodically sent how NMCI would
information. interested in being
included a section on global E- mails to
affect them until the transitioned or
communicating with MHS staff on
provider was ready to reassigned.
the incumbent transition activities
contact them contractors* staff. and MHS leadership regarding their
made on- site visits to possible transition to San Antonio, the
the contractor. location of the helpdesk However, according
function. to the NMCI
Director*s office, this problem was somewhat mitigated by the provider*s
Web site that provides transition
information to all NMCI customers/ users.
Establish client Ye s *T h e Yes* According to
Yes* MHS program Yes* The
Yes* NIMA*s transition team with government*s TAC- SWA officials,
officials for the TriService headquarters- level
transition team representatives from transition team the contracting
transition team included across the
included personnel officer*s Infrastructure
consists of representatives from organization to
from the functional representative Management Office representatives from
various offices within facilitate the
areas being assisted the provider
established transition the NMCI Director*s NIMA to help transition.
outsourced (i. e., with the transition.
teams with Office and the
facilitate the communications and The officials also representatives from
program
transition. IT services) as well stated that Army the former TriService
management office.
as personnel from technical points of Medical Also, every site has a
other offices. contact at each site
System Support transition team made
also helped facilitate Center contract.
up of customer and the transition to the provider personnel. new contract.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Place transition Yes* The Air Force
Yes* The Army Yes* The transition Yes*NMCI*s Yes* The transition
under single program established a single
designated the was assigned to the Director has overall
process is under a manager.
program manager, contracting officer*s
government task
responsibility for the single program who is referred to as
representative to manager in the TriService
transition.
manager. the functional area oversee the chief.
transition. Infrastructure
Management Office. Create
Yes* During the Yes* Personnel from Yes* Client/ provider
Yes* NMCI and the Yes* An integrated client/ provider
transition phase, both the Army and transition teams were provider
designated
NIMA transformation transition teams to management from
provider formed established to
staff to work together team comprising
address short- term both the government three transition address short-
term on short- term NIMA and provider
transition tasks as and the provider met
teams, one for each transition tasks. The transition issues.
staff, was required. weekly to track the
site location. These teams met weekly to responsible for
progress of the teams worked with
address any addressing short
transition. the contracting
transition issues. term tasks, including
officer*s (1) transition
representative and planning, (2)
the provider*s project resources and manager to deal with
recruitment, (3) transition issues.
program budget, and (4) contract development and costing.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Recognize that it Yes* The Air Force Yes* The contract
Yes* The contract Limited* The Navy
Yes* According to takes time to effect
allowed 60 days for allowed a 60- day
provided for a said that it had
NIMA, it has transition and plan
the transition. phase- in period.
transition period of initially
scheduled its seven accordingly.
30 to 90 days. underestimated the
functional areas to According to MHS
scope of the project be transitioned to the
program officials, the and the magnitude of
provider over a 3- to transition period the problems brought
5- year period. The lasted 90 days. on by legacy
provider will not applications and
assume associated responsibility for a information
given functional area assurance and until the NIMA cyber- security
transition review issues. According to board has given its NMCI program
approval to the officials, addressing provider*s turnover these problems
and
plan. completing the operational testing that was mandated by the Congress
subsequent to contract award led to the transition period being lengthened
from 2- 1/ 2 to 3- 1/ 2 years. NMCI officials also said the extension will
allow the Department of
the Navy to have time to operate NMCI as a fully transitioned enterprise
before having to decide whether to exercise
the contract option.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Encourage transition Yes* The Air Force Yes* The Army had
Yes* According to Yes* The NMCI
Yes* The contract of staff to provider,
staff were given the no personnel
provider officials, contract provides requires the provider where
appropriate, right to transfer to the expected to transition
they extended offers that displaced federal to actively recruit, using
bonuses, stock
contractor. It was left to the provider since
to employees from employees that hire, and make options, and other
up to the contractor the activity being the prior contractor
transition to the reasonable efforts to
appropriate methods. to decide whether to
outsourced had and some MHS staff. provider under retain NIMA staff. If
offer incentives or previously been employment the provider meets
not, which it chose contracted out. openings as a result certain target
not to do. However, the
of NMCI will be given thresholds for provider was
guaranteed 3 years recruiting NIMA staff, interested in
of employment with it receives a retaining existing the provider, a 15
monetary incentive. contractor staff and percent salary According to the
had a program to
increase, and a signon provider*s general recruit them.
bonus. manager, the company offered
signing bonuses to NIMA employees that wanted to transition.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Develop employeeretention No* According to
Not applicable* Yes* The
Yes* NMCI program Yes* According to
programs Kirtland Air Force
Since this activity government task
officials noted that NIMA program
and offer bonuses to Base officials, they was previously
manager persuaded the Navy has human
officials, NIMA keep key people, did not develop an contracted out,
key government resources programs
categorized affected where appropriate. employee- retention
employee retention employees from the that would allow its
employees into three program because
of Army civilian Tri- Service Medical organizations a
tiers, on the basis of they did not believe personnel was not System
Support
range of options, whether they would
that they could offer relevant.
Center to temporarily including bonuses, to be allowed to incentives.
work at the TriService help keep key staff. transition to the
Additionally, the Navy provider or whether
Infrastructure directed all Navy and
there would be Management Office
Marine Corps restrictions on their until positions commands affected
activities if they did became available by NMCI to develop
transition. This with a contractor that
civilian personnel distinction was made
provides support to transition plans to to comply with
this office. In mitigate the impact of government conflict
addition, according to NMCI
of interest rules. a program official, to implementation on According to
the keep expertise in employees.
program manager, particular
NIMA also sought applications, key
personnel who employees were wanted to remain at transferred to the the
agency to help MHS program
monitor the executive office and
provider*s Tr i - Ser v i c e performance and Infrastructure
considered the Management Office. qualifications of those who wished to
serve in this role prior to deciding who
to retain.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
When consistent with Yes* Kirtland Air
Not applicable* Yes* According to
Yes* NMCI program Yes* For staff that
organizational Force Base officials
According to the MHS program
officials said that did not want to
objectives, assist stated that they have TAC- SWA
officials, the existing Navy civilian
transition, NIMA employees who do reduction- in- force
contracting officer, no government task
personnel programs provided various
not want to transfer in procedures to help
Army personnel were manager assisted are in place to assist
types of assistance, finding other jobs, place civilian displaced by the
government employees. They such as retirement
either within an employees in other
contract. employees with the noted that two Navy
planning, resume organization or at
Air Force jobs. Tri- Service Medical organizations found
writing, and another organization. Employees also
System Support other jobs within their
interviewing skills. In receive priority status Center staff in finding
command for their addition, NIMA
for other DOD job new positions with small number of
reported that if an openings. Further, other MHS
employees affected employee did not the base education
organizations. For by NMCI. Other Navy
want to transition to office provided example, he located organizations
are the provider, that it assistance with open positions and
also reassigning would consider readjusting outplacement;
contacted the affected personnel to
the resume writing; and, organizations.
other government employee*s work for those nearing jobs, where
assignment or retirement, planning
applicable. provide training to
advice. support the
individual*s placement within other areas.
Document key Yes* Kirtland Air
Yes* Documentation Yes* The Tri- Service
Yes* NMCI Yes* The executing information to
Force Base*s work produced by the Medical System
maintains a Web NIMA transformation preserve functions and provider,
including Support Center
portal with team was charged
organizational workload size are
maintenance logs, processes and
organizational with documenting the
knowledge in the documented in the become the property
procedures were knowledge
activities for each event that one or
contract. of the government.
captured during information. In functional area that is
more providers transition. The addition, a new
being outsourced. change. provider*s processes
Intranet site is being This includes and procedures are
developed with up- todate developing and currently being
interactive capturing operational captured. The
organizational processes and contract also
knowledge relevant procedures for each
includes to the many varied
of the functional requirements on
NMCI communities. areas.
documenting key information.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Use change Yes* Kirtland Air
Not applicable* The Yes* According to
Yes* The Navy has Yes* NIMA*s change
management Force Base held activity had been
MHS program provided employees
management strategies to help
town hall meetings previously contracted
officials, the affected by NMCI
strategies included client employees and internal briefings
out. government task
with a range of training seminars
deal with the to communicate the manager for the TriService
employment options offered by its human transition.
objectives of the and incentives.
resources office on outsourcing project
Infrastructure resume writing,
and the changes that Management Office
interviewing skills, would take place assisted government career
transition internally. It also had
employees with workshops, and one- on- one meetings
finding new positions employment trends.
with each employee and kept them that would be informed of transition
displaced.
issues. Weekly E- mails were also sent out to all affected employees.
Source: GAO.
Projects* Implementation of Phase VI: Manage
Appendi x VII
Provider( s) Performance The effectiveness with which the performance of
the provider( s) is managed* the focus of this phase* is critical to the
successful implementation of an outsourcing project. Indeed, according to
Gartner, Inc., an outsourcing project can be thwarted by poorly designed,
funded, and delivered processes for managing the delivery of services. 1
This firm also points out that an enterprise needs to retain the resources
to oversee
the planning and implementation of the IT services being delivered by the
provider to ensure that the contractor meets the client*s business needs
throughout the life of the agreement. 2 Moreover, frequent and clear
communication between the client and provider ensures that potential
problems are resolved before they cause disruptions. In addition,
performance reviews should take place regularly to keep the project on
course, measure performance levels, and make adjustments as necessary. 3
The 11 practices in this phase that we used to evaluate the five projects
are as follows:
Consider incentives to motivate provider( s) to exceed performance
requirements.
1 Gartner, Inc., Retain Enough Resources to Manage Outsourcing Deals,
Research Note COM- 16- 8425 (June 17, 2002). 2 Gartner, Inc., Successful
Outsourcing Means Retaining Some Staff, Research Note COM18- 9692 (Dec.
18, 2002). 3 An Interagency- Industry Partnership in Performance, Seven
Steps to Performance- Based Services Acquisition, Benchmark Version
(January 2002).
Use penalties to motivate provider( s) to meet performance requirements.
Periodically undertake studies to assess (1) how the provider*s
performance compares with the value being delivered to similar clients and
(2) the extent to which the provider*s performance is improving over time.
Schedule periodic working- level meetings with both the end- user groups
and the provider to review the provider*s performance.
Conduct executive- level oversight meetings with the provider*s senior
management to review provider*s performance.
Distribute performance data to stakeholders. Reserve audit rights on
performance data supplied by the provider. Ensure that the provider
measures and reports on performance. Work with the provider to redefine
service levels, as appropriate. Sample performance data frequently
enough to perform trend analysis
and to permit extrapolation based on historical data. Allow employees
and possibly stakeholders to rate the provider on a
regular basis. As shown by figure 11, the five projects in our review
generally implemented the practices.
Figure 11: Project Implementation of Phase VI: Managing Provider
Performance
Note: Not applicable* The practice was not relevant to the project*s
particular circumstances. No* The agency did not implement the practice.
Limited* The agency fully implemented some but not all aspects of the
practice and did not take alternative actions that fully satisfied the
practice. Yes* The agency fully implemented the practice or took an
alternative action that fully satisfied its intent.
Table 9 depicts whether and how the five projects in our review
implemented each of the 11 practices in this phase.
Table 9: Summary of Projects* Use of Phase VI Practices Did the project
use the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Consider incentives No* The Air Force
No* There are no Yes* The contract Yes* The NMCI
Yes* The contract to motivate did not include
monetary incentives, includes incentives contract includes a
includes monetary provider( s) to exceed
incentives in the but, according to linked to each
one- time payment of incentives in performance
contract because the TAC- SWA officials, performance
$10 million if the accordance with an requirements.
agency expects the the fact that the requirement. Each
provider successfully award fee plan. It
contractor to meet agency will prepare a requirement has a
completes full also includes sharein-
the contract*s performance positive, negative, operational
savings provisions requirements without
evaluation report can and acceptable capability. The
to encourage incentives.
help motivate a performance range
contract also has process
Specifically, provider to meet that provides a basis incentives for
improvements. according to the
requirements. for monetary customer
former functional However, the
incentives (as well as satisfaction,
area chief, because contracting officer
penalties). information
the contract was said that incentives assurance, and small
issued under the might have been
business and small rules established by useful to motivate the
disadvantaged OMB Circular A- 76,
provider to exceed business the focus of the
performance participation.
project was on requirements. achieving cost savings, and incentives were
not included in the contract. As a result, he noted that the provider did
not have any incentive to be innovative or
creative.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Use penalties to Yes* The contract
Limited* The Yes* The contract
Yes* The contract Limited* NIMA did
motivate provider( s) contains monetary contract does not
includes monetary includes monetary not include monetary to meet
performance
penalties that are have specific penalties linked to
penalties in the form penalties in its requirements, such
linked to each of the monetary penalties
each performance of credits to the
contract. However, if as
performance linked to requirement. Each
agency if the the provider does not
requirements. If performance
requirement has provider fails to meet certain
assess penalties for imposed, they would
requirements. positive, negative, perform to the levels
minimum failure to perform at reduce the amount of
However, the and acceptable specified in the
performance required individual
the payment owed to contract states that performance ranges
SLAs. standards, only its
service as well as the provider. unacceptable work that provide a basis
costs would be aggregate service
must be redone at for monetary covered.
levels; the provider*s penalties as well as
apply penalties in expense, and, if the
incentives. the form of credit to defects and services the client;
cannot be corrected, increase penalty for
the government may recurring deficient
reduce the contract*s performance; price to reflect the hold back a
reduced value of the percentage of
services performed. provider*s pay for a TAC- SWA project particular
service
officials stated that until performance
monetary penalties requirements are
were not included in met; the contract because refund a penalty if
the Army was the provider returns
concerned that to agreed- upon
contractors might not performance levels
bid on the contract, within a designated
and administrating period of time; and this type of contract ensure that
the is more difficult.
provider will cover costs, but not profit, when a particular performance
requirement is not met.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Periodically Limited*( 1) This has
No* TAC- SWA Yes* The provider*s Limited*( 1) NMCI Limited*( 1) At this
undertake studies to not been done officials stated that
performance is only has
time, NIMA has not assess: (1) how the
because, according they address
reviewed monthly by performance data undertaken such
provider*s to a project official, performance issues the government task
since October 2002. studies because it performance
the Air Force was not at the time that they manager and in The Navy
intends to only recently
compares with the aware of similar
occur and that they quarterly do such a study but
transitioned functions value being delivered clients. have not performed
management reviews
has not established a to the provider.
to similar clients and (2) This is done any studies.
of the provider*s schedule for it. However, NIMA
(2) the extent to through periodic
performance. (2) The provider*s program officials which the provider*s
monitoring of
According to MHS improvements over
stated that they performance is performance by program officials, the time
are being intend to use a improving over time.
quality assurance results are compared monitored monthly. private- sector
firm to evaluators and with peers and
periodically compare through monthly reviewed for how the
the provider*s review meetings held provider has performance with by the
functional area
improved over time. those of similar
chief to identify and organizations.
address any (2) The provider*s problems that are
improvements over starting to occur.
time are being monitored monthly. Schedule periodic
Yes* Monthly Yes* According to
Yes* Periodic Yes* At each
Yes* Quarterly working- level
meetings are held at the provider and
working- level implemented site, the
performance meetings with both which the contractor TAC- SWA officials,
meetings are held provider and
management review the end- user groups briefs the functional they hold
daily
with the government government meetings are held
and the provider to area chief and other meetings to discuss
task manager and managers meet on a
and are attended by review the provider*s Kirtland Air Force any issues.
the program frequent, as- needed
NIMA and the performance.
Base management office
basis to review provider. In addition, representatives.
representatives to performance and
the provider hired an obtain help- desk resolve any issues.
ombudsman to feedback. obtain anonymous comments from NIMA staff and
management on its performance.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Conduct executivelevel Ye s*Air F orce Yes* According to
Yes* Executive Yes* The NMCI Yes* This is done
oversight schedules executivelevel
provider and TACSWA oversight meetings
Director and the through the quarterly meetings with the
oversight officials, are held quarterly
provider executive review meetings.
provider*s senior meetings whenever
quarterly in- progress with the provider*s meet weekly to management to
they are necessary. reviews are held with
senior management review performance review provider*s A recent meeting
the provider*s to review and discuss other performance.
included the corporate
performance. NMCI
president of the management and the
implementation provider.
battalion issues. In addition,
commander. the Department of
the Navy recently established an operations advisory board consisting of
Department of the Navy leadership and the provider. The goal of this board
is to focus senior
leadership on issues affecting NMCI in order to establish priorities and
make decisions.
Distribute Yes* Performance Yes* Thecontracting Yes* The help- desk Yes*
The NMCI
Yes* Performance performance data to data were not officer*s
monthly reports, Navy and Marine data are distributed
stakeholders. distributed to
representative including results Corps program
at quarterly review stakeholders at the distributes monthly
against performance managers are meetings, which beginning of the
performance reports metrics, are e- mailed
responsible for according to the project, but began to Army stakeholders
to stakeholders providing SLA
contracting officer, after complaints from for review. monthly.
performance data to
are attended by stakeholders about Performance issues their commands.
stakeholders. the need for such
or problems information. Metrics, identified are such as system
discussed with the reliability or *uptime,*
contracting officer*s are now provided to representative and stakeholders
on line.
the contracting officer, and corrective actions are planned to prevent
reoccurrence. Reserve audit rights
Yes* Addressed in Yes* Addressed in Yes* Addressed in
Yes* Addressed in Yes* Addressed in on performance data
the contract. the contract. the contract. the contract. the contract.
supplied by the provider.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Ensure that provider Yes* As required by Yes* Thecontracting Yes* The
contract
Yes* The contract Yes* The contract
measures and the quality control
officer*s requires the provider
specifies that the requires the provider reports on
plan in the contract, representative to submit monthly
provider must to submit data,
performance. the provider makes
receives and quarterly
measure and report including
information available performance performance reports
on whether its SLA measurements of
on product or service information in daily that are then performance goals
service, quarterly. quality and any
meetings with the reviewed and
are being met. actions needed to
provider*s project verified.
correct decreases in manager.
quality. In addition, according to the former functional area chief, the
provider provides performance information during monthly meetings.
Work with provider to Yes* This is done on Yes* According to
Yes* MHS and the Yes* The Navy and
Yes* Theredefinition redefine service an as- needed basis. TAC- SWA and
provider have worked
service providerhave
of service levels is levels, as For example, the
provider officials, together to redefine and continue to
provided for in the appropriate.
service levels were they have worked
SLAs. For example, refine NMCI SLAs to
contract. According redefined in April together to redefine
the first call closure
ensure more precise
to NIMA, the 2002.
performance performance metric
performance
redefinition of service requirements in the
was redefined to be measures and to
levels is expected to contract.
more realistic.
more accurately
occur in the future,
capture user
generally at the time satisfaction with the
that the annual
system. For
contract option is
example, a contract
exercised.
modification standardized many of the performance categories that dealt
with service availability.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Sample performance Ye s *A i r F o r c e Yes* The provider*s Yes*
Performance
Yes* The NMCI Yes* The contract
data frequently quality assurance proposal and quality
data are sampled Director monitors
requires data to be enough to perform evaluators monitor
control plan state and reported monthly
provider performance provided periodically trend analysis and to
performance in that it will provide and quarterly. They to identify trends
by for NIMA to use for
permit extrapolation accordance with the
various trend are analyzed and assessing providersupplied
trend and other types based on historical quality assurance analyses to
the
verified, including information of analyses. data.
surveillance plan and government.
any supporting data. as well as
the performance According to TACSWA In addition, the information from
requirements project officials, provider*s knowledge
Navy independent summary. Any trends
these data are management system verification and identified are
submitted to the provides analysis validation testing and
addressed in monthly contracting officer*s
and trend data to customer satisfaction status meetings.
representative for MHS.
survey results. In analysis. addition, NMCI program officials said
that they have requested funding to develop a performance measurement data
repository to support trend analyses.
Allow employees and Limited* The
Yes* The contracting Yes* The customer Yes* User surveys
Yes* The provider possibly
provider conducts officer*s
satisfaction survey is are used to measure
surveys NIMA stakeholders to rate annual customer
representative and one method used to
satisfaction with employees on their
the provider on a satisfaction surveys, on- site points of rate the
provider. In specific services satisfaction with the regular basis (e. g.,
but it is not required contact obtain
addition, according to such as E- mail
help- desk function. scorecards and
to submit the results feedback from
MHS program access, help desk,
The results of these quarterly report
to the Air Force. employees on a officials, stakeholder
etc., and overall user surveys are reviewed
cards). Kirtland Air Force
regular basis. input is also obtained satisfaction with the by NIMA during
Base also obtains ad
from program service provider*s quarterly meetings.
hoc feedback from management reviews
performance. In addition,
employees who, after and meetings held by stakeholders are also they
report a
the government task responsible for
problem, are asked manager. providing monthly
to provide assessments of information about
contractor how well the problem
performance. was addressed and Moreover, customer their overall
satisfaction is a satisfaction level. factor in determining
the amount of the semiannual award fee earned by the provider. Source:
GAO.
Projects* Implementation of Phase VII: Ensure
Appendi x VIII
Services are Provided Although outsourcing focuses on the provider*s
ability to perform, the client organization is ultimately responsible for
ensuring that services are provided and that end- user needs are met. The
previous phases addressed the extensive preparation that must precede the
provider*s assuming responsibility for the client organization*s services.
This phase encompasses practices designed to ensure that an effective
oversight approach is established. In addition, over the course of the
outsourcing project, the client learns more about the capabilities of the
provider, and
market conditions may change. As a result, it is important to monitor
service levels internally as well as maintain an external view of the
performance of other providers in order to identify opportunities to
improve and ensure that the outsourcing arrangement maintains its value to
the client.
The six practices in this phase that we used to evaluate the five projects
are as follows:
Monitor the provider*s work to anticipate issues for resolution. Make
sure that the provider uses the standard tools and processes
defined as part of the operational model. Use provider performance data
to continuously improve processes. Pursue improvement based on customer
satisfaction surveys. Ensure that an appropriately empowered individual
from the client organization oversees the work.
Set realistic time frames that are agreed to by the provider.
As illustrated by figure 12, the five projects in our review largely
implemented the practices.
Figure 12: Project Implementation of Phase VII: Ensure Services Are
Provided
Note: Not applicable* The practice was not relevant to the project*s
particular circumstances. No* The agency did not implement the practice.
Limited* The agency fully implemented some but not all aspects of the
practice and did not take alternative actions that fully satisfied the
practice. Yes* The agency fully implemented the practice or took an
alternative action that fully satisfied its intent.
Table 10 provides details on whether and how each project implemented each
of the six practices associated with this phase.
Table 10: Summary of Projects* Use of Phase VII Practices Did the project
use the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Monitor the Yes* Quality
Yes* Thecontracting Yes* The provider*s Yes* Navy Yes* NIMA reviews
provider*s work to assurance evaluators
officer*s work is monitored
representatives at the provider*s work in
anticipate issues for monitor the quality of representative through
monthly
each implemented quarterly program resolution.
the provider*s work reports monthly on reports and meetings site
(including
managementreviews and identify problems
contractor as well as through headquarters)
and as needed in or trends. The results performance. In
on- site meetings and monitor the
response to issues and any problems
addition, he meets readiness
provider*s work and raised by NIMA*s are reported to the
daily with provider assessments. identify issues. performance
contracting officer officials to discuss
According to NMCI monitoring officials. and the functional
performance results. program officials, the
area chief. NMCI Enterprise
ManagementSystem enables the government to monitor the performance of the
network and associated service delivery.
Make sure the Yes* The provider
Yes* According to Yes* According to
Yes* The Navy has Yes* The tools are provider uses the uses software tools
TAC- SWA project MHS program
monitored the documented in the standard tools and
to help monitor officials, the provider
officials, the provider service provider*s quality management
processes defined as system performance uses various
uses the standard use of standard tools plan. For example, part of the
against performance standard tools
tools outlined in the and processes. Tools the provider is using
operational model. standards to monitor
outlined in its contract. are being used to
a specific quality and and manage the
proposal. support legacy
process help- desk function.
migration, client improvement
installation, and helpdesk methodology. procedures.
Use provider Yes* Although Yes* Thecontracting Yes* For example, Yes* The
Navy uses Yes* NIMA*s performance data to Kirtland Air Force
officer*s MHS used provider
the provider*s data to performance
continuously improve Base does not rely
representative performance data to help determine ways monitors use
processes. on data provided by monitors provider
improve the to improve
provider data to its provider, its
performance data, customer satisfaction
processes, such as assess progress and
quality assurance and any potential survey process. to improve NMCI
ways to improve evaluators areas for
implementation performance.
continually monitor improvement are
procedures and the provider
incorporated in the timeliness of help performance. monthly report.
desk problem resolution.
(Continued From Previous Page)
Did the project use the practice? Air Force C4 Practice
Services Army TAC- SWA MHS/ ITO Help Desk Navy NMCI NIMA IT/ IS
Pursue improvement Limited* The Air
Yes* Surveys are Yes* Negative
Yes* Surveys of Yes* To pursue
based on customer Force does not
received by the feedback that is user satisfaction with
improvement, NIMA satisfaction surveys.
perform, or require contracting officer*s
received based on specific services and uses both customer
the provider to technical the results of the
overall satisfaction satisfaction surveys perform, customer
representatives and customer satisfaction
with provider and interviews with surveys. However, provided as input for
surveys is reviewed performance are
senior- level the provider has the contracting
by the MHS helpdesk used to identify areas
customers. initiated such officer*s
manager. needing surveys but is not
representative improvement. required to distribute
report. the results to the Air Force. Ensure that an Yes* The functional
Yes* Thecontracting Yes* The Yes* The NMCI
Yes* Thecontracting appropriately
area chief oversees officer*s government task
Director oversees the officer*s
empowered the work, and the representative is the
manager is the work and discusses representative and a individual from the
quality assurance empowered individual performance with the
staff of technical client organization evaluators support
individual from empowered to
NMCI provider monitors oversee the
oversees the work. this oversight effort.
NETCOM. oversee the work of
executive during work. the provider. weekly meetings.
Set realistic time Yes* Kirtland Air
Yes* Time frames Yes* The monthly
Yes* According to Yes* NIMA and the frames that are Force Base and the
for new tasks are performance reports
NMCI program provider have agreed
agreed to by the provider agree that
established by and quarterly officials, Navy*s on dates associated
provider.
reasonable time contract
program reviews set practice is to work with specific actions.
frames for modifications that are time frames that with the provider in
performance have agreed to by the have been agreed to
establishing now been set. Some provider.
by the provider. schedules.
revisions were made to them in April 2002. Source: GAO
Appendi x IX
Comments from the Department of Defense Note: GAO comments supplementing
those in the report text appear at the end of this appendix.
See comment 1. See comment 2.
See comment 3.
The following are GAO*s comments on the Department of Defense*s letter
dated April 8, 2003.
GAO Comments 1. Addressed in the Agency Comments and Our Evaluation
section of this report.
2. On January 29, 2003, NIMA granted the provider authorization to use
government sources in performing the IT/ IS contract that was limited to
*products, services, and supplies that support the performance of the
specific functional areas and miscellaneous items required under this
contract.* Accordingly, we did not modify this report.
3. Addressed in the Agency Comments and Our Evaluation section of this
report.
Appendi x X
GAO Contact and Staff Acknowledgments GAO Contact Linda J. Lambert, (202)
512- 9556 Staff
Season Dietrich, James Houtz, Anjalique Lawrence, Patricia Slocum, and
Acknowledgments Thomas Wright made key contributions to this report.
(310239)
GAO*s Mission The General Accounting Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities
and to help improve the performance and accountability of the federal
government for the American people. GAO examines the use of public funds;
evaluates federal programs and policies; and provides analyses,
recommendations, and other assistance to help Congress make informed
oversight, policy, and funding decisions. GAO*s commitment to good
government is reflected in its core values of accountability, integrity,
and reliability.
Obtaining Copies of The fastest and easiest way to obtain copies of GAO
documents at no cost is
through the Internet. GAO*s Web site (www. gao. gov) contains abstracts
and fulltext GAO Reports and
files of current reports and testimony and an expanding archive of older
Testimony
products. The Web site features a search engine to help you locate
documents using key words and phrases. You can print these documents in
their entirety, including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as *Today*s Reports,* on its
Web site daily. The list contains links to the full- text document files.
To have GAO e- mail this
list to you every afternoon, go to www. gao. gov and select *Subscribe to
GAO Mailing Lists* under *Order GAO Products* heading.
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO
also accepts VISA and Mastercard. Orders for 100 or more copies mailed to
a single address are discounted 25 percent. Orders should be sent to:
U. S. General Accounting Office 441 G Street NW, Room LM Washington, D. C.
20548
To order by Phone: Voice: (202) 512- 6000 TDD: (202) 512- 2537 Fax: (202)
512- 6061
To Report Fraud, Contact:
Waste, and Abuse in Web site: www. gao. gov/ fraudnet/ fraudnet. htm
E- mail: fraudnet@ gao. gov Federal Programs
Automated answering system: (800) 424- 5454 or (202) 512- 7470 Public
Affairs Jeff Nelligan, Managing Director, NelliganJ@ gao. gov (202) 512-
4800
U. S. General Accounting Office, 441 G Street NW, Room 7149 Washington, D.
C. 20548
Report to the Subcommittee on Readiness and Management Support, Committee
on Armed Services, U. S. Senate
April 2003 INFORMATION TECHNOLOGY DOD Needs to Leverage Lessons Learned
from Its Outsourcing Projects
GAO- 03- 371
Letter 1 Results in Brief 2 Background 5 Projects Substantially Used
Leading Commercial Practices 14 Leveraging Lessons Learned DOD- wide
Could Assist Other DOD Projects 23 Conclusions 27 Recommendations 27
Agency Comments and Our Evaluation 28
Appendixes
Appendix I: Objectives, Scope, and Methodology 30
Appendix II: Projects* Implementation of Phase I: Determine Sourcing
Strategy 32
Appendix III: Project*s Implementation of Phase II: Define Operational
Model 38
Appendix IV: Project*s Implementation of Phase III: Develop the Contract
48
Appendix V: Projects* Implementation of Phase IV: Select the Provider( s)
59
Appendix VI: Projects* Implementation of Phase V: Transition to Provider(
s) 66
Appendix VII: Projects* Implementation of Phase VI: Manage Provider( s)
Performance 77
Appendix VIII: Projects* Implementation of Phase VII: Ensure Services are
Provided 86
Appendix IX: Comments from the Department of Defense 90 GAO Comments 93
Appendix X: GAO Contact and Staff Acknowledgments 94 GAO Contact 94 Staff
Acknowledgments 94
Tables Table 1: Definition of Phases for IT Outsourcing 8 Table 2: Profile
of Outsourcing Projects Reviewed 13
Table 3: Percentage of Practices Implemented, by Project 17 Table 4:
Summary of Projects* Use of Phase I Practices 34
Table 5: Summary of Projects* Use of Phase II Practices 41 Table 6:
Summary of Projects* Use of Phase III Practices 51 Table 7: Summary of
Projects* Use of Phase IV Practices 61 Table 8: Summary of Projects* Use
of Phase V Practices 69 Table 9: Summary of Projects* Use of Phase VI
Practices 80 Table 10: Summary of Projects* Use of Phase VII Practices 88
Figures Figure 1: Percentage of Practices Implemented in Each Phase 3
Figure 2: Roles of the Client and Provider in an Outsourcing
Relationship 6 Figure 3: GAO*s Framework for Outsourcing IT Services 7
Figure 4: Use of Leading Commercial Practices for Outsourcing of
IT Services, by Project 15 Figure 5: Percentage of Practices Implemented
in Each Phase 16 Figure 6: Project Implementation of Phase I: Determine
Sourcing
Strategy 33 Figure 7: Project Implementation of Phase II: Define
Operational
Model 40 Figure 8: Project Implementation of Phase III: Develop the
Contract 50 Figure 9: Project Implementation of Phase IV: Select the
Provider( s) 60 Figure 10: Project Implementation of Phase V: Transition
to
Provider( s) 68 Figure 11: Project Implementation of Phase VI: Managing
Provider
Performance 79 Figure 12: Project Implementation of Phase VII: Ensure
Services Are
Provided 87
Abbreviations
C3I command, control, communications, and intelligence C4 command,
control, communications, and computer CIO chief information officer DOD
Department of Defense IT information technology IT/ IS Information
Technology/ Information Services MHS/ ITO Military Health System/
Information Technology Organization NETCOM Network Enterprise Technology
Command NIMA National Imagery and Mapping Agency NMCI Navy and Marine
Corps Intranet OMB Office of Management and Budget RFP request for
proposals SLA service- level agreement TAC- SWA Total Army Communications*
Southwest Asia
This is a work of the U. S. Government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. It may contain
copyrighted graphics, images or other materials. Permission from the
copyright holder may be necessary should you wish to reproduce copyrighted
materials separately from GAO*s product.
a
GAO United States General Accounting Office
The projects in GAO*s review substantially used leading commercial
practices as specified in GAO*s framework for outsourcing IT services.
Specifically, the agencies fully implemented 88 percent of the practices
(not including practices not applicable to a particular project). This
framework consists of practices organized into seven phases: (I) determine
sourcing strategy, (II) define operational model, (III) develop the
contract, (IV) select the provider( s), (V) transition to provider( s),
(VI) manage the performance of the provider( s), and (VII) ensure services
are provided. The figure below shows the percentage of practices that were
implemented in each phase.
Although DOD has acted on gathering and disseminating lessons learned and
commercial leading practices related to general acquisition issues, its
actions have generally not been focused on outsourcing or on sharing the
lessons learned from IT services outsourcing across the department. By not
systematically capturing and disseminating such information across the
department, DOD is losing the opportunity to leverage the knowledge gained
on IT services projects like those in GAO*s review. Lessons learned that
are pragmatic and easily accessible would give DOD managers a more
informed understanding of important issues to be addressed when making
outsourcing decisions, as well as the factors to be considered to help
ensure the success of these endeavors.
Percentage of Practices Implemented, by Phase Note: Not applicable* The
practice was not relevant to the project*s particular circumstances. No*
The agency did not implement the practice. Limited* The agency fully
implemented some but not all aspects of the practice and did not take
alternative actions that fully satisfied the practice. Yes* The agency
fully implemented the practice or took an alternative action that fully
satisfied its intent. INFORMATION TECHNOLOGY
DOD Needs to Leverage Lessons Learned from Its Outsourcing Projects
www. gao. gov/ cgi- bin/ getrpt? GAO- 03- 371. To view the full report,
including the scope and methodology, click on the link above. For more
information, contact Randolph C. Hite at (202) 512- 3439 or hiter@ gao.
gov. Highlights of GAO- 03- 371, a report to the
Subcommittee on Readiness and Management Support, Committee on Armed
Services, U. S. Senate
April 2003
Given the magnitude of its reported spending on information technology
(IT) services* more than $6. 2 billion in fiscal year 2001* it is critical
that the
Department of Defense (DOD) adopt effective practices for acquiring IT
services. GAO researched leading commercial practices for the
outsourcing of IT services, and, in November 2001, published a framework
consisting of seven phases that span the full range of activities that are
performed during the outsourcing of those services
(this is an acquisition in which a client organization transfers
responsibility for performing services to an external provider).
GAO was asked to determine (1) the extent to which selected DOD projects
for outsourcing IT services use leading commercial practices as specified
in GAO*s framework and (2) whether DOD is sharing lessons learned from its
IT outsourcing projects across the department.
GAO is making recommendations to the Secretary of Defense aimed at
leveraging lessons learned across the
department from its components* IT outsourcing experiences. DOD agreed
that capturing lessons learned related to IT outsourcing initiatives is
important and stated that it intends to explore a variety of mechanisms to
do so. DOD*s plans are consistent with our recommendations.
Page i GAO- 03- 371 DOD IT Services Outsourcing
Contents
Contents
Page ii GAO- 03- 371 DOD IT Services Outsourcing
Contents
Page iii GAO- 03- 371 DOD IT Services Outsourcing
Page 1 GAO- 03- 371 DOD IT Services Outsourcing United States General
Accounting Office
Washington, D. C. 20548 Page 1 GAO- 03- 371 DOD IT Services Outsourcing
A
Page 2 GAO- 03- 371 DOD IT Services Outsourcing
Page 3 GAO- 03- 371 DOD IT Services Outsourcing
Page 4 GAO- 03- 371 DOD IT Services Outsourcing
Page 5 GAO- 03- 371 DOD IT Services Outsourcing
Page 6 GAO- 03- 371 DOD IT Services Outsourcing
Page 7 GAO- 03- 371 DOD IT Services Outsourcing
Page 8 GAO- 03- 371 DOD IT Services Outsourcing
Page 9 GAO- 03- 371 DOD IT Services Outsourcing
Page 10 GAO- 03- 371 DOD IT Services Outsourcing
Page 11 GAO- 03- 371 DOD IT Services Outsourcing
Page 12 GAO- 03- 371 DOD IT Services Outsourcing
Page 13 GAO- 03- 371 DOD IT Services Outsourcing
Page 14 GAO- 03- 371 DOD IT Services Outsourcing
Page 15 GAO- 03- 371 DOD IT Services Outsourcing
Page 16 GAO- 03- 371 DOD IT Services Outsourcing
Page 17 GAO- 03- 371 DOD IT Services Outsourcing
Page 18 GAO- 03- 371 DOD IT Services Outsourcing
Page 19 GAO- 03- 371 DOD IT Services Outsourcing
Page 20 GAO- 03- 371 DOD IT Services Outsourcing
Page 21 GAO- 03- 371 DOD IT Services Outsourcing
Page 22 GAO- 03- 371 DOD IT Services Outsourcing
Page 23 GAO- 03- 371 DOD IT Services Outsourcing
Page 24 GAO- 03- 371 DOD IT Services Outsourcing
Page 25 GAO- 03- 371 DOD IT Services Outsourcing
Page 26 GAO- 03- 371 DOD IT Services Outsourcing
Page 27 GAO- 03- 371 DOD IT Services Outsourcing
Page 28 GAO- 03- 371 DOD IT Services Outsourcing
Page 29 GAO- 03- 371 DOD IT Services Outsourcing
Page 30 GAO- 03- 371 DOD IT Services Outsourcing
Appendix I
Appendix I Objectives, Scope, and Methodology
Page 31 GAO- 03- 371 DOD IT Services Outsourcing
Page 32 GAO- 03- 371 DOD IT Services Outsourcing
Appendix II
Appendix II Projects* Implementation of Phase I: Determine Sourcing
Strategy
Page 33 GAO- 03- 371 DOD IT Services Outsourcing
Appendix II Projects* Implementation of Phase I: Determine Sourcing
Strategy
Page 34 GAO- 03- 371 DOD IT Services Outsourcing
Appendix II Projects* Implementation of Phase I: Determine Sourcing
Strategy
Page 35 GAO- 03- 371 DOD IT Services Outsourcing
Appendix II Projects* Implementation of Phase I: Determine Sourcing
Strategy
Page 36 GAO- 03- 371 DOD IT Services Outsourcing
Appendix II Projects* Implementation of Phase I: Determine Sourcing
Strategy
Page 37 GAO- 03- 371 DOD IT Services Outsourcing
Page 38 GAO- 03- 371 DOD IT Services Outsourcing
Appendix III
Appendix III Project*s Implementation of Phase II: Define Operational
Model
Page 39 GAO- 03- 371 DOD IT Services Outsourcing
Appendix III Project*s Implementation of Phase II: Define Operational
Model
Page 40 GAO- 03- 371 DOD IT Services Outsourcing
Appendix III Project*s Implementation of Phase II: Define Operational
Model
Page 41 GAO- 03- 371 DOD IT Services Outsourcing
Appendix III Project*s Implementation of Phase II: Define Operational
Model
Page 42 GAO- 03- 371 DOD IT Services Outsourcing
Appendix III Project*s Implementation of Phase II: Define Operational
Model
Page 43 GAO- 03- 371 DOD IT Services Outsourcing
Appendix III Project*s Implementation of Phase II: Define Operational
Model
Page 44 GAO- 03- 371 DOD IT Services Outsourcing
Appendix III Project*s Implementation of Phase II: Define Operational
Model
Page 45 GAO- 03- 371 DOD IT Services Outsourcing
Appendix III Project*s Implementation of Phase II: Define Operational
Model
Page 46 GAO- 03- 371 DOD IT Services Outsourcing
Appendix III Project*s Implementation of Phase II: Define Operational
Model
Page 47 GAO- 03- 371 DOD IT Services Outsourcing
Page 48 GAO- 03- 371 DOD IT Services Outsourcing
Appendix IV
Appendix IV Project*s Implementation of Phase III: Develop the Contract
Page 49 GAO- 03- 371 DOD IT Services Outsourcing
Appendix IV Project*s Implementation of Phase III: Develop the Contract
Page 50 GAO- 03- 371 DOD IT Services Outsourcing
Appendix IV Project*s Implementation of Phase III: Develop the Contract
Page 51 GAO- 03- 371 DOD IT Services Outsourcing
Appendix IV Project*s Implementation of Phase III: Develop the Contract
Page 52 GAO- 03- 371 DOD IT Services Outsourcing
Appendix IV Project*s Implementation of Phase III: Develop the Contract
Page 53 GAO- 03- 371 DOD IT Services Outsourcing
Appendix IV Project*s Implementation of Phase III: Develop the Contract
Page 54 GAO- 03- 371 DOD IT Services Outsourcing
Appendix IV Project*s Implementation of Phase III: Develop the Contract
Page 55 GAO- 03- 371 DOD IT Services Outsourcing
Appendix IV Project*s Implementation of Phase III: Develop the Contract
Page 56 GAO- 03- 371 DOD IT Services Outsourcing
Appendix IV Project*s Implementation of Phase III: Develop the Contract
Page 57 GAO- 03- 371 DOD IT Services Outsourcing
Appendix IV Project*s Implementation of Phase III: Develop the Contract
Page 58 GAO- 03- 371 DOD IT Services Outsourcing
Page 59 GAO- 03- 371 DOD IT Services Outsourcing
Appendix V
Appendix V Projects* Implementation of Phase IV: Select the Provider( s)
Page 60 GAO- 03- 371 DOD IT Services Outsourcing
Appendix V Projects* Implementation of Phase IV: Select the Provider( s)
Page 61 GAO- 03- 371 DOD IT Services Outsourcing
Appendix V Projects* Implementation of Phase IV: Select the Provider( s)
Page 62 GAO- 03- 371 DOD IT Services Outsourcing
Appendix V Projects* Implementation of Phase IV: Select the Provider( s)
Page 63 GAO- 03- 371 DOD IT Services Outsourcing
Appendix V Projects* Implementation of Phase IV: Select the Provider( s)
Page 64 GAO- 03- 371 DOD IT Services Outsourcing
Appendix V Projects* Implementation of Phase IV: Select the Provider( s)
Page 65 GAO- 03- 371 DOD IT Services Outsourcing
Page 66 GAO- 03- 371 DOD IT Services Outsourcing
Appendix VI
Appendix VI Projects* Implementation of Phase V: Transition to Provider(
s)
Page 67 GAO- 03- 371 DOD IT Services Outsourcing
Appendix VI Projects* Implementation of Phase V: Transition to Provider(
s)
Page 68 GAO- 03- 371 DOD IT Services Outsourcing
Appendix VI Projects* Implementation of Phase V: Transition to Provider(
s)
Page 69 GAO- 03- 371 DOD IT Services Outsourcing
Appendix VI Projects* Implementation of Phase V: Transition to Provider(
s)
Page 70 GAO- 03- 371 DOD IT Services Outsourcing
Appendix VI Projects* Implementation of Phase V: Transition to Provider(
s)
Page 71 GAO- 03- 371 DOD IT Services Outsourcing
Appendix VI Projects* Implementation of Phase V: Transition to Provider(
s)
Page 72 GAO- 03- 371 DOD IT Services Outsourcing
Appendix VI Projects* Implementation of Phase V: Transition to Provider(
s)
Page 73 GAO- 03- 371 DOD IT Services Outsourcing
Appendix VI Projects* Implementation of Phase V: Transition to Provider(
s)
Page 74 GAO- 03- 371 DOD IT Services Outsourcing
Appendix VI Projects* Implementation of Phase V: Transition to Provider(
s)
Page 75 GAO- 03- 371 DOD IT Services Outsourcing
Appendix VI Projects* Implementation of Phase V: Transition to Provider(
s)
Page 76 GAO- 03- 371 DOD IT Services Outsourcing
Page 77 GAO- 03- 371 DOD IT Services Outsourcing
Appendix VII
Appendix VII Projects* Implementation of Phase VI: Manage Provider( s)
Performance
Page 78 GAO- 03- 371 DOD IT Services Outsourcing
Appendix VII Projects* Implementation of Phase VI: Manage Provider( s)
Performance
Page 79 GAO- 03- 371 DOD IT Services Outsourcing
Appendix VII Projects* Implementation of Phase VI: Manage Provider( s)
Performance
Page 80 GAO- 03- 371 DOD IT Services Outsourcing
Appendix VII Projects* Implementation of Phase VI: Manage Provider( s)
Performance
Page 81 GAO- 03- 371 DOD IT Services Outsourcing
Appendix VII Projects* Implementation of Phase VI: Manage Provider( s)
Performance
Page 82 GAO- 03- 371 DOD IT Services Outsourcing
Appendix VII Projects* Implementation of Phase VI: Manage Provider( s)
Performance
Page 83 GAO- 03- 371 DOD IT Services Outsourcing
Appendix VII Projects* Implementation of Phase VI: Manage Provider( s)
Performance
Page 84 GAO- 03- 371 DOD IT Services Outsourcing
Appendix VII Projects* Implementation of Phase VI: Manage Provider( s)
Performance
Page 85 GAO- 03- 371 DOD IT Services Outsourcing
Page 86 GAO- 03- 371 DOD IT Services Outsourcing
Appendix VIII
Appendix VIII Projects* Implementation of Phase VII: Ensure Services are
Provided
Page 87 GAO- 03- 371 DOD IT Services Outsourcing
Appendix VIII Projects* Implementation of Phase VII: Ensure Services are
Provided
Page 88 GAO- 03- 371 DOD IT Services Outsourcing
Appendix VIII Projects* Implementation of Phase VII: Ensure Services are
Provided
Page 89 GAO- 03- 371 DOD IT Services Outsourcing
Page 90 GAO- 03- 371 DOD IT Services Outsourcing
Appendix IX
Appendix IX Comments from the Department of Defense
Page 91 GAO- 03- 371 DOD IT Services Outsourcing
Appendix IX Comments from the Department of Defense
Page 92 GAO- 03- 371 DOD IT Services Outsourcing
Appendix IX Comments from the Department of Defense
Page 93 GAO- 03- 371 DOD IT Services Outsourcing
Page 94 GAO- 03- 371 DOD IT Services Outsourcing
Appendix X
United States General Accounting Office Washington, D. C. 20548- 0001
Official Business Penalty for Private Use $300 Address Service Requested
Presorted Standard Postage & Fees Paid
GAO Permit No. GI00
*** End of document. ***