Desktop Outsourcing: Positive Results Reported, but Analyses
Could Be Strengthened (29-MAR-02, GAO-02-329).
GAO reviewed federal agencies' use of "seat management"--an
information technology (IT) outsourcing alternative for
distributed computing services. Under "seat management,"
contractor-owned desktop and other computing hardware, software,
and related services are bundled and provided on the basis of a
fixed price per unit. No single overarching reason emerged for
agencies adoption of seat management. The most common rationales
were to improve IT management, improve user support and
productivity, and upgrade agency IT. All six agencies GAO
reviewed reported that their seat management approaches had
yielded positive results, such as better IT management and
desk-help support. However, GAO could not determine whether any
of the agencies were achieving expected costs benefits because
they did not perform sufficient up-front analyses of their
baseline and projected costs and benefits or routinely monitor
all actual seat management costs and benefits. Four of the six
agencies identified risks associated with seat management, such
as possible cost overruns, schedule delays, or contractor
performance problems. However, none of the agencies prioritized
their risks, and only one identified actions to mitigate risks
before implementing seat management. Agencies and seat management
contractors have identified the following lessons from their
implementation experiences: (1) agency commitment is crucial,
especially by top management; (2) thorough up-front preparation
and planning activities must be completed; (3) solicitation and
contract award activities should be carefully managed; (4)
program and contract management activities, including monitoring
contractor performance, are key; (5) partnerships between
agencies and the seat management contractors, in which they have
common goals, are critical; and (6) effective and continual
communication within the agency, with the seat management
contractor, and among contractors working on related activities,
is important.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-02-329
ACCNO: A02933
TITLE: Desktop Outsourcing: Positive Results Reported, but
Analyses Could Be Strengthened
DATE: 03/29/2002
SUBJECT: Best practices
Fixed price contracts
Information technology
Computer services contracts
Risk management
Performance measures
NASA Integrated Financial Management
Program
NASA Outsourcing Desktop Initiative
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-02-329
A
Report to the Chairman, Subcommittee on Technology and Procurement Policy,
Committee on Government Reform, House of Representatives
March 2002 DESKTOP OUTSOURCING Positive Results Reported, but Analyses Could
Be Strengthened
GAO- 02- 329
Letter 1 Results in Brief 2 Background 5 Rationales for Adopting Seat
Management Varied 9 Agencies Reported Positive Seat Management Results,
but Costs and Benefits Were Not Adequately Assessed and Monitored 11 Risks
Were Identified, but Analyses Were Incomplete
or Not Timely 19 Lessons Learned Can Help Other Agencies More Effectively
Implement Seat Management 22 Conclusions 35 Recommendations 35 Agency
Comments and Our Evaluation 36
Appendixes
Appendix I: Information on Seat Management Contracts as of December 31,
2001, Awarded under the NASA ODIN Master Contract 39
Appendix II: Information on Seat Management Contracts as of December 31,
2001, Awarded under the GSA Seat Management Master Contract 42
Appendix III: Scope and Methodology 45
Appendix IV: Comments from the Department of the Treasury?s Departmental
Offices 47
Appendix V: Comments from the Bureau of Alcohol, Tobacco and Firearms 50
Appendix VI: Comments from the Peace Corps 51
Appendix VII: Comments from the Department of Defense 54
Appendix VIII: Comments from the General Services Administration 57
Appendix IX: Comments from the National Aeronautics and Space Administration
58 GAO Comments 60
Appendix X: GAO Contact and Staff Acknowledgments 62 GAO Contact 62
Acknowledgments 62
Table Table 1: Types of Seat Management Services and Cost Data for Agencies
We Reviewed 7
Abbreviations
ATF Bureau of Alcohol, Tobacco and Firearms CIO chief information officer
CMS Centers for Medicare and Medicaid Services DLA Defense Logistics Agency
FSS Federal Supply Service GSA General Services Administration IT
information technology NASA National Aeronautics and Space Administration
ODIN Outsourcing Desktop Initiative for NASA OMB Office of Management and
Budget
Lett er
March 29, 2002 The Honorable Tom Davis Chairman, Subcommittee on Technology
and Procurement Policy Committee on Government Reform House of
Representatives
Dear Mr. Chairman: This letter responds to your request that we review
federal agencies? efforts to engage in an information technology (IT)
outsourcing 1 alternative for acquiring distributed computing services
(typically those pertaining to desktop management) known as ?seat
management.? Generally speaking, under seat management, contractor- owned
desktop and other computing
hardware, software, and related services are bundled and provided on the
basis of a fixed price per unit (or seat). Since the Department of the
Treasury?s Bureau of Alcohol, Tobacco and Firearms (ATF) pioneered the use
of seat management in the federal government in 1997, more than a dozen
other federal entities have contracted for seat management services, and
still others are considering this approach.
Because more agencies have begun to use the seat management concept, you
expressed interest in identifying how well this approach has worked at
different entities. Accordingly, our objectives were to (1) determine
agencies? rationales for awarding seat management contracts, 2 (2) assess
whether estimated costs and benefits have been achieved, (3) ascertain how
well agencies have managed the risks associated with seat management, and
(4) identify lessons learned.
1 IT outsourcing describes the activities associated with acquiring IT
services from one or more external providers. During outsourcing, a client
organization transfers responsibility for one or more IT services to one or
more external providers. This responsibility is executed by controlling and
managing the processes, people, and technology associated with these
services. 2 The agencies in our review used various governmentwide contracts
to implement seat management. As such, they generally did not enter into
contracts with the seat management vendors; instead, they used task orders,
delivery orders, or blanket purchase agreements, which were placed against
existing contracts. For the purposes of this report, however, we will refer
to such acquisition documents as contracts.
In addition to the primary objectives, you also asked that we identify the
agencies using the National Aeronautics and Space Administration?s (NASA)
and the General Services Administration?s (GSA) seat- management
governmentwide acquisition contracts. For the agencies using these
contracts, you asked that we identify (1) the types of services for which
the agencies contracted and (2) the expected costs and benefits associated
with these services. We are providing this information in appendixes I and
II for the NASA and GSA governmentwide seat management contracts,
respectively.
To determine agencies? rationales for using seat management, estimated and
actual costs and benefits, and risk management analyses, we reviewed how
seat management was implemented at six agencies: NASA, the Centers
for Medicare and Medicaid Services (CMS), Treasury?s Departmental Offices
and ATF, the Peace Corps, and the Defense Logistics Agency (DLA). We chose
these agencies because they (1) used a variety of contracting
vehicles, (2) had seat management contracts that had been awarded more than
1 year before our review, and (3) had at least 500 seats. We reviewed the
agencies? seat management contracts, analyses of costs
and benefits, processes to monitor contractor performance, risk analyses,
and other related documents. To obtain information on lessons learned, we
interviewed officials from the six agencies and reviewed applicable
documentation. We also interviewed officials from private- sector
organizations, including the seat management contractors for the six
agencies in our review, and reviewed research on seat management and
distributed computer outsourcing published by private research firms. In
addition, we interviewed and obtained applicable lessons learned and other
documentation from GSA because it (1) manages one of the governmentwide seat
management contracts and (2) abandoned an attempt to implement seat
management internally in late 2001. Appendix III
provides more information on our scope and methodology. Results in Brief No
single overarching reason emerged regarding why agencies decided to adopt
seat management to address their distributed computing needs. Instead, the
six agencies in our review adopted seat management for a variety of reasons.
The most common rationales pertained to improving IT management, improving
user support and productivity, and obtaining or upgrading agency IT. While
acknowledging the importance of cost considerations in making the seat
management decisions, agency officials
generally stated that lowering the cost of their existing services was not
their primary reason for choosing this approach. All agencies in our review
reported that their seat management approaches had achieved positive
results, such as improving IT management (e. g., using one standard
technology environment to eliminate incompatible hardware and software and
improve information sharing across an agency) and improving end- user
support, such as help- desk support. However, we could not determine whether
any of the agencies were achieving expected
costs and benefits because they did not (1) perform sufficient up- front
analyses of their baseline and projected costs and benefits and (2)
routinely monitor all actual seat management costs and benefits. To their
credit, all six agencies tracked contractor performance against specific
performance metrics in their contracts in areas such as service delivery and
availability. While this type of tracking can be an indicator of whether
certain types of goals are being achieved, such as improved user
satisfaction, such metrics
do not fully address whether the overall cost and benefit goals of seat
management programs are being met. Like any important IT effort, it is
critical that agencies consider the risks associated with seat management
before adopting this approach, as noted by the Office of Management and
Budget (OMB) and our guidance. Four of the agencies in our review identified
risks associated with seat management, such as possible cost overruns,
schedule delays, or contractor performance problems. However, of these four
agencies, none ranked their risks in order of priority, and only one
identified actions to mitigate risks before implementing seat management. In
addition, two
agencies did not perform an analysis of risks at all. One area in which
agencies effectively addressed a critical risk was by executing contracts
that contained adequate clauses to protect the government in areas such as
quality assurance and termination rights. By applying the lessons learned by
agencies that have implemented seat management initiatives, agencies
considering this approach could more effectively plan their activities and
reduce the risk of encountering problems experienced by others. Agencies and
seat management contractors have identified important lessons learned from
their implementation experiences, namely that,
agency commitment is crucial, especially by top management;
thorough up- front preparation and planning activities must be completed;
solicitation and contract award activities should be carefully managed;
program and contract management activities, including monitoring contractor
performance, are key;
partnerships between agencies and the seat management contractors, in
which they work toward establishing and achieving common goals, are
critical; and effective and continual communication within the agency,
with the seat management contractor, and among contractors working on
related activities, is important.
So that agencies can determine the extent to which their current seat
management programs have achieved positive results, we are recommending that
each of the six agencies in our review monitor all actual seat management
costs and benefits. Also, to ensure that the future
seat management investments of these agencies are justified, we are
recommending that they establish a baseline for current costs; perform an
analysis of expected costs, benefits, and risks; and consider implementing
the lessons learned identified in this report.
In providing written comments on a draft of this report, three agencies
agreed with our findings or recommendations; two did not indicate whether
they agreed or disagreed; and NASA, while supporting many of the findings,
disagreed with portions of the report. Specifically, NASA did not agree with
our assessment that its up- front cost analysis was not sufficient,
that it did not track its internal seat management costs, and that it did
not adequately track benefits. As discussed in this report, (1) an analysis
conducted by a consulting firm concluded that NASA lacked a
comprehensive baseline of its pre- seat- management costs; (2) NASA tracked
contractor costs and the number of internal staff involved with seat
management; however, the agency did not fully track other internal costs,
such as overhead; and (3) NASA tracked contractor performance, which
addresses certain expected benefits, but it did not fully monitor others,
such as potential increases in user efficiency and productivity. As such,
NASA lacks a full picture and analysis of its seat management program.
Background Although there is no generally accepted definition of seat
management, at its core, it involves using a performance- based contract 3
to obtain equipment, software, and services from a private- sector firm to
meet an agency?s distributed computing requirements (typically pertaining to
desktop equipment). Agencies are charged on a per- seat basis for the
services provided, 4 but what constitutes a ?seat? can vary substantially in
terms of the type of equipment and type and level of service provided. For
example, a seat can consist of a single type of service sold at a fixed
price per unit or as a bundle of different services which, taken together,
are also sold at a fixed price. These different views of seat management
account, in
part, for the different contracting vehicles for seat management in the
government. Nevertheless, in its purest form, seat management turns personal
computer resources into a utility or commodity in which the customer
purchases the right to use the vendor?s equipment and resources. The vendor
remains the owner of the equipment 5 and is ultimately responsible for its
upkeep.
According to Gartner, Inc., a leading private research firm, 1996 to 1998
saw a growth trend in publicly reported desktop- management outsourcing
contracts in the private sector. 6 Leading organizations have identified
both
business- and technology- based reasons for outsourcing IT services.
Business- based reasons included being able to (1) focus resources on core
business competencies by transferring responsibility for IT services to an
external provider and (2) respond more quickly to business and industry
changes by leveraging the experience of an external provider.
Technologybased reasons for outsourcing included (1) gaining quicker access
to technology skills that are in high demand, (2) acquiring the flexibility
to grow and shrink high- technology skills as needed, (3) gaining access to
enhanced hardware and software, (4) acquiring the ability to refresh
hardware and software as needed, (5) aggregating the demand for IT 3
Performance- based service contracting is a process in which the customer
agency specifies the outcome or result it desires and leaves it to the
vendor to decide how best to achieve the desired outcome. 4 In some cases,
agencies used their seat management contract to obtain other IT services
that were not charged on a per- seat basis. 5 In at least one instance, an
agency chose to retain ownership of the assets obtained through
its seat management contract. 6 Gartner, Inc., Trends in Desktop Management
Outsourcing Contracts, 1996- 1998 (March 8, 1999).
resources from across the organization, and (6) achieving a more
standardized IT environment. A similar growth in IT service contracts
occurred in the federal government. Specifically, the government?s purchases
of IT services have increased from $3.7 billion in fiscal year 1990 to about
$13. 4 billion in fiscal year 2000. In the case of distributed computing
services outsourcing, ATF is generally acknowledged as the pioneering
federal agency in implementing
seat management since it initially outsourced its desktop and network
services in late 1997. In mid- 1998, NASA and GSA awarded governmentwide
seat- management indefinite- delivery, indefinite- quantity contracts.
First, in June 1998, NASA awarded contracts to seven prime contractors under
its Outsourcing Desktop Initiative for NASA (ODIN)
governmentwide contract. The next month, GSA awarded contracts to eight
prime contractors under its GSA seat management contract. As of December 31,
2001, two agencies were using the NASA ODIN contract, and eight agencies
were using the GSA seat management contract (see apps. I and II for
additional information about these agencies). In addition, some agencies are
using GSA?s Federal Supply Service (FSS) Schedule 70 contracts 7 to obtain
seat management services or have developed and managed their own seat
management contracts.
Table 1 illustrates the different seat management approaches taken by the
six agencies in our review, categorized by the contract vehicle used by
each.
7 Under the schedule program, GSA enters into indefinite- delivery,
indefinite- quantity contracts with commercial firms to provide commercial
goods and services governmentwide. Schedule 70 contains contracts related to
general- purpose commercial IT equipment, software, and services.
Table 1: Types of Seat Management Services and Cost Data for Agencies We
Reviewed Reported actual Date of
Number and types of seat contract costs,
Contract/ contract Contract
management services acquired, as of as of Agency
award a term July 31, 2001 July 31, 2001 Comments
GSA?s seat management master contract Peace Corps April 2000 3- year base
683 general- purpose desktop seats $7, 564, 396 Under the terms of the
period, with 171 portable computer seats
master GSA seat two 1- year 31 server seats
management contract, option 127 network printer seats
infrastructure periods 39 communication device seats management (e. g.,
network Additional services include off- site tape management); user storage
and application program support (e. g., help development support.
desks); and asset management services are bundled as a part of a seat.
Treasury?s
June 1999 1- year base 1, 642 desktop seats 26, 678, 415 Same as above.
Departmental period, with 700 portable computer seats Offices
nine 1- year (other equipment, such as servers, option printers, and
routers, are included in the periods price of these seats) First two
Additional services include options
voice/ telephone administration, Web- site exercised support, training, fax
support, personal digital assistant support, and the establishment of a
classified network.
(Continued From Previous Page)
Reported actual Date of
Number and types of seat contract costs,
Contract/ contract Contract
management services acquired, as of as of Agency
award a term July 31, 2001 July 31, 2001 Comments
NASA?s ODIN master contract
NASA?s November 3 years 3, 013 general- purpose desktop
30, 151, 512 Under the terms of the Kennedy Space 1998 computing seats NASA
ODIN master Center b 3- year followon 421 scientific and engineering
computing contract, agencies contract seats may acquire one or also awarded
364 maintenance- only seats more categories of
to incumbent 293 network- attached device seats (a service (such as
contractor c desktop unit acquired from non- ODIN desktop hardware and
sources attached to the network)
software) without other 57 World Wide Web seats types of services (such 40
application database seats
as local- or wide- areanetwork 109 file storage seats
services). 17, 615 telephone service seats Some associated 48 remote
communications seats services, such as user support and
Additional services include asset infrastructure management; help- desk
services;
management, are catalog orders (e. g., printers); special provided as a part
of orders (e. g., toner for network printers); these seats. and
infrastructure upgrades (e. g., upgraded telephone system).
CMS June 1999 3 years 5, 100 general- purpose desktop computer 20, 462, 675
Same as above. seats 325 portable computer seats Additional services include
asset
management; catalog orders; special orders (e. g., engineering services);
and infrastructure upgrades (e. g., gateway servers). GSA?s FSS Schedule 70
contracts
ATF April 2001 3 years 6, 000 desktop/ laptop computer seats 62, 070, 575
ATF awarded its first
300 server seats seat management
1, 143 printer/ scanner seats contract on October 9,
1, 275 nonseat annual server, printer, and 1997. After this scanner
maintenance support
contract expired, ATF awarded a second 3year Additional services include
centralized contract to the management, help- desk support, desk incumbent
contractor side support for key personnel, order
in April 2001. The tracking and support, network contract costs are from
infrastructure, and installation services the date of the original (which
includes asset management and contract. order tracking).
(Continued From Previous Page)
Reported actual Date of
Number and types of seat contract costs,
Contract/ contract Contract
management services acquired, as of as of Agency
award a term July 31, 2001 July 31, 2001 Comments
DLA d April 2000 1- year base 2, 102 desktop computer seats 6, 781,493 DLA
has maintained
period, with 251 laptop computer seats ownership of the
four 1- year 1, 523 printer seats assets acquired options e 68 server seats
through its seat 4 plotter seats, management contract. First option
64 personal digital assistant seats exercised Additional services include
local- areanetwork administration and operations; technical support; desktop
and server maintenance; asset and configuration management; installations,
moves, additions, and changes; network monitoring; monthly service- level
reporting; and help- desk support.
a This is the date of the resulting delivery order, task order, or blanket
purchase agreement signed by the agency. b NASA implemented seat management
at 10 different sites. Each site had a different mix of seats and services.
Since this table is for illustrative purposes, we used the Kennedy Space
Center as an example of a NASA seat implementation site because we visited
this site. See appendix I for further
information on the other 9 NASA seat implementation sites. c Instead of
optional years, NASA?s contract authorizes the award of sole- source follow-
on contracts in
which a new delivery order is negotiated and signed with the incumbent
contractor. d At the time of our review, DLA had implemented seat management
only at its headquarters complex.
e DLA?s base period was from October 1, 2000, through September 30, 2001,
following a transition period from the date of contract award through
September 30, 2000. Source: GAO, based on documents from each of the
agencies. We did not verify this information.
Rationales for Each agency in our review cited unique reasons for
implementing seat Adopting Seat
management. However, the most common ones fit into three categories: (1)
improving IT management, (2) improving end- user support and Management
Varied
productivity, and (3) obtaining new technology or upgrading current
technology. In addition, although lower cost was not the primary reason that
agencies chose the seat management approach, cost issues were a
consideration.
Improving IT management. Five of the six agencies reported that part of
their rationale for implementing a seat management approach was to enhance
management of IT resources, which included improvements to areas such as
standardization, asset management, and IT human capital. For example, the
Peace Corps implemented seat management, in part,
to improve and standardize its IT environment- which consisted of diverse,
old, and incompatible computing equipment- and to obtain a reliable
inventory of its IT assets. As another example, NASA wanted to shift IT
asset management responsibilities for its diverse enterprises to contractors
and use government personnel that were performing
desktop support for other mission- related work. Improving end- user
support and productivity. All six agencies reported that at least part of
their rationale for implementing seat management was to improve desktop
management from a user
perspective. Agency officials stated that they wanted to improve user
satisfaction and service levels or optimize service delivery by using
commercial best practices. For example, DLA implemented seat management to
address the operational impairment of the users at its headquarters complex
due to problems with its distributed computing
environment, including frequent and extended local- area- network downtime
and remote access problems. Treasury?s Departmental Offices implemented seat
management, in part, to combine several services, such as help- desk support
and maintenance, so that users
could have a single point of contact rather than having to deal with
multiple vendors. Obtaining new technology or upgrading current
technology. Four agencies reported that part of their rationale for
implementing seat management was to obtain technology, upgrade current
software and
hardware, and/ or provide for periodic technology refreshment. For example,
ATF wanted to provide its special agents, field inspectors, and support
staff with personal computers and laptops because they could not communicate
vital information with each other unless they used unsecured cellular or
public telephones. Furthermore, ATF wanted to institute a technology
refreshment program to upgrade or replace its equipment every 3 years.
Another example is CMS, which implemented
seat management in part to update its systems and to ensure that its
equipment was compliant with Year 2000 requirements.
Reducing Costs Was Not the Although half the agencies said that reducing
costs was one of their seat Primary Objective
management objectives and acknowledged its importance, agency officials
noted that it was not the primary objective. The other agencies cited
reasons other than costs for implementing seat management. For example, DLA
officials indicated that the critical issue for their agency was improving
its headquarters computing environment, which, among other
problems, had frequent downtimes. In addition, Peace Corps officials stated
that the agency did not choose seat management to reduce the agency?s budget
requests associated with distributed computing and did not expect that it
would. Although cost reduction may not have been a primary objective, other
cost- related considerations, such as the ability to better predict and
manage their costs, were cited by all of the agencies as reasons for
implementing seat management. The organizations in our November 2001 IT
services outsourcing study also
did not identify reducing the overall costs of the existing IT service as a
reason for their decision to implement an outsourcing solution. 8 However,
their reasons did include some cost considerations, namely, to reduce
capital investments and to better predict operating costs by contracting for
IT services using a standard unit of measure. For example, under seat
management, services are priced on a per- seat basis.
Agencies Reported Although every agency in our review reported positive
results from
Positive Seat implementing seat management, we could not determine whether
they were achieving expected costs and benefits because the agencies had not
Management Results, (1) conducted adequate pre- seat- management analyses
and (2) tracked all but Costs and Benefits
actual costs and benefits. Specifically, agencies generally did not Were Not
Adequately sufficiently analyze their baseline and projected costs and
benefits up front and monitor actual implementation results. Without such
critical Assessed and information, an agency is not positioned to make well-
informed decisions Monitored about seat management options or able to
convincingly demonstrate real results. Although the six agencies were not
monitoring overall seat
management benefits, all were tracking elements of contractor performance.
While it can be an important indicator of results, tracking contractor
performance is not a substitute for tracking whether the overall cost and
benefit goals of the seat management program are being achieved.
Agencies Reported a Variety The six agencies in our review reported that
their seat management
of Seat Management Results initiatives had achieved positive results. These
results relate to a wide
variety of areas that can be categorized into four general areas: improving
IT management, improving end- user support, enhancing mission support, 8 U.
S. General Accounting Office, Information Technology: Leading Commercial
Practices for Outsourcing of Services, GAO- 02- 214 (Washington, D. C.:
November 30, 2001).
and upgrading technology. Many of these reported accomplishments directly
correlate with the reasons that agencies chose to implement seat management.
Improving IT management. Five agencies noted improvements in areas such as
asset management, security, standardization and
interoperability, and/ or planning for periodic technology refreshments. For
example, the Peace Corps reported that before implementing seat management,
the agency had a wide variety of incompatible hardware and software that
made it difficult for employees to easily share information, and that
managing its assets was very difficult. According to Peace Corps officials,
seat management allowed the agency to implement a standard technology
environment and, as a result,
information is much more easily shared across the agency. Various NASA
organizations also reported a myriad of IT management improvements, such as
improved consistency and currency of operating
systems and applications; the automated distribution of software, including
computer virus protection; a better understanding of the entity?s IT
inventory, which resulted in the removal of obsolete equipment; and improved
software license management.
Improving end- user support. All six agencies reported that end- user
support has improved since they implemented seat management. In particular,
they generally reported improvement in the quality and timeliness of help-
desk support. At DLA, for example, before seat management, customer
dissatisfaction with its headquarters IT help- desk services was widespread.
DLA seat management officials believe
that, on the basis of the results of customer surveys, the quality and
timeliness of the help- desk support improved substantially after seat
management was implemented. At NASA, officials in the Office of Space Flight
reported that implementing seat management has resulted in more consistent
high- quality service to all users. Enhancing mission support. All six
agencies stated that seat management has enhanced their mission support and
productivity because, for example, the staff were assigned to other duties
or were freed from desktop management duties. In one case, the Treasury
assistant director responsible for seat management said that after
implementing seat management, fewer agency employees were needed to support
IT operations for the Departmental Offices. Therefore,
Treasury was able to reassign six staff to other organizations within the
department to help develop new applications software and work on
security issues. Similarly, NASA?s Office of Space Flight reported that seat
management helped improve the agency?s asset management services by reducing
the amount of work that agency staff have to perform to maintain annual
inventories of government- owned property
and auditing of contractors? property control procedures. Upgrading of
technology. Four agencies reported that they upgraded
their technology under seat management in a timely manner. For example,
using the seat management contract, ATF provided over 4,000 personal and
laptop computers to field agents, among others, in about 6 months. CMS
reported that seat management enabled it to deploy Year 2000 compliant
equipment and software throughout the agency in a little more than 3 months.
Agencies Lacked Facts on We could not determine whether the agencies were
achieving expected Whether Expected Costs
costs and benefits because they generally did not (1) sufficiently analyze
and Benefits Were Being their baseline and projected costs and benefits up
front and (2) routinely Achieved monitor all actual seat management costs
and benefits. Like any other major IT investment, seat management
initiatives should be supported by a
well- developed business case that evaluates the expected returns against
the costs. An explicit understanding of the expected costs and benefits up
front provides the basis for a sound financial and strategic decision and
creates a baseline against which managers and executives can measure
progress. According to OMB guidance, an analysis of the expected costs and
benefits of IT investments, as well as the costs and benefits of
alternatives, should be included in the justification for major investments.
9 Similarly, our guidance on IT investment management calls for agencies to
identify the expected costs and benefits of proposed investments. 10 In
addition, our research on leading commercial practices for acquiring IT 9
Office of Management and Budget (OMB), Circular A- 130, Management of
Federal Information Resources (November 30, 2000) and Circular A- 11, Part
3, Planning, Budgeting, and Acquisition of Capital Assets (July 2001). These
circulars define capital assets as including IT that is used by the federal
government and has an estimated useful life of 2 years or more. Such assets
may be acquired in different ways, including through lease/ purchase or
other capital leases regardless of whether title has passed to the federal
government or through an operating lease for an asset with an estimated
useful life of 2 years or more.
10 U. S. General Accounting Office, Information Technology Investment
Management: A Framework for Assessing and Improving Process Maturity, GAO/
AIMD- 10. 1. 23, Exposure Draft (Washington, D. C.: May 2000).
services found that the optimizing and baselining of existing internal IT
processes provide an organization with the information to make a sourcing
decision. 11
Although all six agencies analyzed their expected seat management costs and/
or benefits or baselined their existing environment, these analyses were
missing critical elements. Specifically, NASA and CMS estimated the costs
and benefits that they expected to achieve with seat management, including
an analysis of alternatives. However, internal agency costs, such as those
associated with program
and contract management, were not included in their estimates. Moreover,
NASA and CMS?s analyses did not include a thorough or reliable baseline of
the costs associated with their pre- seat- management computing
environments. For example, in a November 2001 postimplementation report, a
private- sector firm concluded that it was not possible to determine with
any degree of confidence whether NASA had saved money because of the lack of
a comprehensive baseline. 12
While Treasury?s Departmental Offices estimated the costs of seat
management and an alternative, it did not estimate the benefits for either
approach. In addition, the cost estimate was developed about a week before
the contract award and was based on the winning bidder?s proposal.
Although DLA and the Peace Corps performed analyses of their preseat-
management costs, they did not estimate the expected costs or benefits of
seat management or any alternatives. In addition, the Peace Corps? analysis
was not a true baseline for the seat management program that the agency
implemented because it included (and did not
distinguish between) the costs for both domestic and foreign operations,
whereas it implemented seat management only for its domestic operations.
ATF did not estimate the costs and benefits of seat management or
alternatives before implementing this approach in 1997. However, in 11 GAO-
02- 214. 12 Kelly, Anderson & Associates, ODIN Program, The Outsourcing
Desktop Initiative for NASA: Post Implementation Business Case Assessment
(November 2001).
2000, when the contract was under consideration for renewal and the agency
was reviewing its possible choices, a cost study was performed that
identified the agency?s total cost of ownership and service levels under
seat management and compared them with peer organizations. However, this
study did not include an analysis of non- seatmanagement 13 alternatives or
expected benefits.
The agencies cited various reasons for not completing a thorough analysis of
their existing environment and expected costs and benefits of seat
management and alternatives, including that they lacked time or resources to
complete the analysis, that upper- level management had mandated that
outsourcing and a performance- based contract be used, and that budgetary
savings was not a goal of the seat management program. However, such an
approach is risky. Together with OMB, we have long held that the analysis of
costs and benefits is critical to IT investment decision- making. As we
noted in our 1997 IT investment guide, an IT investment process cannot
operate effectively without accurate, reliable, and up- to- date data on
project costs and benefits. 14 In addition, it is important that agencies
consider the criticality of distributed computing to their mission, not just
the size of the investment, when determining the necessary
comprehensiveness of the analyses of costs and benefits. Once an IT
investment is implemented, OMB Circular A- 130 and our IT investment
management guide 15 recommend that agencies validate whether estimated costs
and benefits are being achieved. In an outsourcing
arrangement, to obtain a true picture of the cost of the investment, it is
important to consider both the cost of the contract as well as the agency?s
internal costs to administer the program and contract. Each of the six
agencies tracked contractor costs. However, while they provided us with
estimates of their internal costs for managing the seat management
contract and program (in some cases just providing salaries and benefits for
a given month), agencies in our review did not routinely monitor all
13 The analysis included an assessment of using the GSA seat- management
contract alternative. 14 U. S. General Accounting Office, Assessing Risks
and Returns: A Guide for Evaluating Federal Agencies? IT Investment
Decision- making, GAO/ AIMD- 10. 1. 13 (Washington D. C.: February 1997). 15
GAO/ AIMD- 10. 1. 23.
internal agency costs associated with seat management. 16 Such costs can be
substantial. For example, NASA?s Kennedy Space Center reported that its
salaries and benefits to manage the Office of Space Flight?s four seat
management implementations in July 2001 were about $146,000. If projected
over the life of the original 3- year contract, 17 the center?s salaries and
benefit costs to manage this initiative would be about $5. 3 million.
None of the agencies routinely monitored the actual overall benefits of
their seat management programs. This includes the two agencies (NASA and
CMS) that had estimated the quantifiable benefits that they had expected to
achieve. For example, CMS estimated that it would have achieved quantifiable
benefits of about $22 million between fiscal years 1999 and 2001 in
increased staff productivity and the avoidance of costs associated with, for
example, asset theft. However, CMS has not validated
whether these benefits have been achieved. The agencies had a variety of
reasons for not monitoring actual benefits, including that they lacked
resources or that, because they did not baseline their costs before seat
management, they did not have a basis to perform a comparison. Since
agencies? rationales for adopting seat management were generally to meet
certain programmatic objectives and not to achieve cost savings, it is
especially important that they monitor benefits to ensure that their seat
management programs are meeting these objectives. Although the six agencies
were not tracking overall seat management benefits, all of them were
measuring and tracking some aspects of contractor performance, typically by
reviewing contractor- provided data periodically (e. g., monthly or
quarterly). The agencies have specific
performance metrics within their contracts to measure contractor
performance, generally including measures related to service delivery and
availability and user satisfaction or help- desk performance. The agencies
16 During exit conferences with the agencies, most acknowledged that they
did not monitor all internal agency costs but stated that they could
identify the number of staff associated with the management of their seat
management programs. In addition, DLA provided us with several months? worth
of costs, which included categories such as salaries and
benefits, travel, rent, communication, and utilities. However, these costs
were for the entire Technology Services and Infrastructure Support
Directorate, of which the seat management office is only a part. A DLA
official stated that the agency could extract the seat management costs from
these amounts, but these data are not presented to management at that level
of detail. 17 In June 2001, NASA signed a 3- year follow- on contract with
the incumbent Kennedy Space Center seat management contractor, with a period
of performance beginning December 1, 2001.
use these measures as the basis for approving vendor payments, awarding
bonuses or other incentives, and/ or applying penalties. 18 For example, one
agency assessed its contractor?s performance against 24 service- level
agreements, 19 such as server availability and customer satisfaction, to
determine whether penalties were warranted (this agency?s contract does
not provide for incentives or bonuses). For fiscal year 2001, this agency
assessed its contractor penalties totaling $30,295. In another example, NASA
uses ?retainage? pools in which a certain percentage (the amount of which is
determined by each center?s contract) of the contractor?s monthly seat price
is retained by the government and disbursed periodically on the basis of a
review of the contractor?s performance. One center retained about $893,000
between December 1998 and June 2001, of which it disbursed about $509, 000.
Agency seat management contracts sometimes did not include contractor
performance measures pertaining to all business goals of the seat management
program. For example, one of the Peace Corps? performance goals in its
September 2000 strategic plan is to pursue efforts to cut costs and improve
agency productivity. Although the implementation of seat management is a
part of the evaluation criteria associated with this goal,
the agency?s seat- management performance metrics plan does not include a
performance measure for cost reductions or improvements to agency
productivity. In another case, a private- sector firm?s review of NASA?s
ODIN
program noted that the agency?s contracts address only portions of its
program objectives. The firm recommended that NASA initiate a full review of
its metrics.
Although critical to ensuring that the agency is obtaining contractually
required services and an indicator that certain types of benefits are, or
are not, being achieved (e. g., improved user satisfaction), measuring and
tracking contractor performance does not take the place of tracking the
overall benefits of seat management programs. For example, contractor
performance metrics do not measure many of the types of benefits that
18 Three of the agencies in our review have awarded incentives or applied
penalties to their contractors. In addition, ATF has not finalized its
service- level agreements with its seat management contractor. Agency
officials stated that they are monitoring contractor performance against
these interim agreements but have not finalized the penalties and incentives
that will be associated with the final service- level agreements. 19
Service- level agreements define the agency?s expectations and are used to
track and measure a contractor?s performance.
agencies believe they are achieving or wanted to achieve, such as increased
end- user productivity, improved mission support through the transfer of
agency IT staff to more critical tasks, and improved standardization. Some
agencies have recognized the need to review their actual seat management
costs and benefits and have obtained, or plan to obtain, independent
assessments of at least some aspects of their seat management program. For
example, NASA contracted with a private- sector firm to perform a
postimplementation business case assessment. In November 2001, this firm
reported that (1) over 90 percent of NASA management officials
surveyed indicated that ODIN provided some benefit in areas such as improved
service and standardization and (2) it was impossible to determine whether
NASA is saving money because of the lack of a comprehensive pre- seat-
management baseline, although there was some evidence that the agency had
achieved savings due, at least in part, to seat management. In late 2000,
CMS conducted a pilot total- cost- of- ownership study of its
post- seat- management distributed computing environment using a private-
sector methodology and software tool. CMS limited its data collection to
readily available information and did not attempt to conduct a full- scale
study.
DLA plans to have a contractor complete a postimplementation totalcost-
of- ownership study, which evaluates the current distributed computing
environment of an organization, and user satisfaction benchmark survey in
the spring.
In January, Treasury signed a contract for the completion of a total-
costof- ownership seat management study that is expected to be completed in
June 2002. In responding to a draft of this report, Treasury?s Departmental
Offices stated that this study is expected to (1) review current costs and
associated benefits and (2) allow the department to predict the expected
costs of services for the future years of the
contract and identify specific benefits expected to be derived from these
costs.
Peace Corps officials stated that the agency intends to issue a contract
for a total- cost- of- ownership review by the end of this fiscal year to
help it identify seat management results. In addition, the Peace Corps
contracted for an independent review of user satisfaction, and it expects to
receive the final report pertaining to this review by the end of March. In
commenting on a draft of this report, the Peace Corps stated that it expects
that these analyses will address agency productivity and costreduction
performance indicators. For postimplementation reviews to be most effective,
it is critical that agencies have an established baseline before
implementation so that there could be a basis of comparison for determining
progress. However, because most of the six agencies did not have a full or
reliable baseline of their costs before implementing seat management, it
would be difficult to validate whether any cost savings had been achieved.
For example, the CMS seat management program manager told us that it would
be difficult to determine whether expected cost savings are being achieved
because
the agency lacks comparable information on its costs before implementing
seat management.
Risks Were Identified, Although four agencies identified a variety of risks,
these analyses were but Analyses Were
incomplete or not performed in a timely manner. OMB and our guidance note
that agencies should address potential risks, such as investment, Incomplete
or Not
organizational, funding, and technical risks, when considering new IT Timely
investments. 20 Not identifying risks or developing strategies to resolve
them can be problematic since most IT investments require a constant focus
on interim results and successful risk management strategies to effectively
address existing or emerging factors that influence implementation results.
Nevertheless, of the four agencies that identified seat management risks,
none ranked their risks and only one identified actions to mitigate risks
before implementing seat management. In addition, two agencies did not
assess risks at all. One important risk area associated with outsourcing-
ensuring that contracts contain clauses that protect the government?s
interests- was adequately addressed by the six agencies for (1) quality
assurance, (2) termination rights, and (3) the government?s rights to
supplied hardware and software at the end of contract performance.
Of the six agencies we reviewed, four identified risks associated with their
seat management implementations. In particular, consistent with OMB and 20
Office of Management and Budget, Circular A- 11, Part 3 Supplement, Capital
Programming Guide, and GAO?s IT investment management guidance, GAO/ AIMD-
10. 1. 23, and GAO/ AIMD- 10. 1. 13.
our guidance, which specifies a variety of risks for agencies to consider,
21 these four agencies identified risks relating to costs, time frames for
implementation, contractor performance, and technical issues as well as
other risks. Examples of risks identified in these categories are discussed
below:
Costs. All four agencies identified cost as a risk. For example, Treasury
determined that its cost risk was low because multiple vendors would be
bidding. Time frames for implementation. Three agencies identified
possible delays as a risk. For example, CMS identified implementation as a
risk that it would mitigate by developing milestones and providing contract
incentives.
Contractor performance. Three agencies identified contractor performance
issues as a risk area. For example, NASA identified contractor performance
as a manageable risk, but was concerned that if it were to become a problem,
mission effectiveness might be
compromised because the agency would have limited staff available to correct
the deficiencies. The agency planned to address this risk by, for example,
limiting the contract to 3 years and having a pool of vendors from which to
choose. Technical issues. Three agencies identified technical issues as a
risk
area. For example, CMS was concerned about the suitability of the hardware
and software to be provided by its seat management contractor. CMS addressed
this risk by requiring testing of equipment proposed by firms competing for
the seat- management contract award,
requiring postaward testing of initial equipment installations, and using an
independent contractor to test the capabilities of systems proposed for the
refreshment cycle. Other risks. Two agencies identified other risks. For
example, NASA
identified the need for coordination between ODIN- related desktop and
intracenter communication architectures and other major agency initiatives,
including its Integrated Financial Management Program, and vendors providing
non- ODIN computing and communication services.
21 See footnote 20.
NASA?s plans for addressing these risks included extensive internal
communication and the development of program interface agreements. Even
though four agencies identified the risks associated with seat management,
these analyses were incomplete and/ or not timely. Specifically, None of
the analyses ranked risks in order of their potential impacts on
agency operations. OMB?s guidance provides examples of how risks can be
characterized by the likelihood of their occurrence and the severity of
potential consequences and then ranked according to their importance. 22
Only CMS identified actions, before implementing seat management, to
mitigate risks. NASA also identified actions to mitigate risks, but this
analysis was conducted over a year after award of the ODIN master contract.
The other two agencies did not identify planned mitigation actions. OMB?s
Capital Programming Guide states that agencies should determine how best to
mitigate the impact of each risk that they identify. 23 Developing
preventive measures and countermeasures to successfully deal with problems
as they develop is critical.
Two of the agencies in our review, ATF and the Peace Corps, did not assess
their seat management risks at all. According to ATF officials, the agency
did not assess the risks of its seat management initiative because it was
ordering equipment and services from the FSS schedule and chose to rely on
maintaining a close relationship with its contractor to ensure quality
services. However, this approach is not a substitute for a well- thought-
out assessment of risks that includes mechanisms to mitigate identified
risks. As for the Peace Corps, agency officials stated that they lacked the
resources to conduct a risk assessment.
22 Office of Management and Budget, Evaluating Information Technology
Investments: A Practical Guide (November 1, 1995) and Circular A- 11. One
ranking process included assigning a score ranging from 1 to 10 to each risk
and then multiplying each score by a percentage weight reflecting the
relative importance of each factor in the agency?s decision processes.
23 Mitigation approaches include transferring, avoiding, reducing, assuming,
and sharing the potential for an event that could produce an adverse
consequence, or for the adverse consequence itself.
Because seat management relies on the use of contractors to perform critical
IT services, it is essential that the contract contain certain clauses that
reduce the government?s risks. In particular, the Federal Acquisition
Regulation requires that contracting officers include in contracts,
including seat management contracts, appropriate quality requirements. The
type and extent of contract quality requirements needed depend on the
particular acquisition and may range from inspection at the time of
acceptance to requiring that the contractor implement a comprehensive
program for controlling quality. In addition, seat management contracts, as
government contracts, should include provisions for termination for the
convenience of the government and termination for default. Finally, seat
management contracts should include some provision for the disposition of
contractor- supplied property (hardware and software) at the end of contract
performance to ensure agency operations continue uninterrupted. All of the
agencies? contracts adequately addressed the issues of quality
assurance, termination rights, and rights to supplied hardware and software
at the end of contract performance. 24 In general, the contracts making use
of the GSA and NASA master contracts- the Peace Corps, Treasury?s
Departmental Offices, NASA, and CMS- tended to be more
comprehensive in dealing with these issues than the contracts making use of
GSA schedule contracts- DLA and ATF. The GSA and NASA master contracts were
more likely to include specific provisions, especially regarding the
disposition of hardware and software, than the GSA schedule contracts, which
more typically relied on standard government contract clauses.
Lessons Learned Can By incorporating lessons learned from those who have
implemented seat Help Other Agencies management, agencies considering this
approach could more effectively plan their activities and reduce the risk
that they will encounter problems
More Effectively experienced by others. Moreover, incorporating lessons
learned from peers Implement Seat
who have engaged in similar sourcing decisions is a leading commercial
Management practice for acquiring IT services. The numerous lessons
identified by agencies, their contractors, and private- sector research
firms generally fall into six categories: (1) agency commitment, (2)
preparation and planning, (3) solicitation and contract award, (4) program
and contract management,
24 The DLA seat management contract did not specifically address disposition
of contractorsupplied hardware and software. DLA advised us that it bought
and maintained ownership of all contract assets.
(5) agency/ contractor partnership, and (6) communication. These lessons are
generally consistent with our report issued last year on the leading
commercial practices for acquiring IT services. 25 In addition, these
lessons are often interrelated and build on one another. For example, if an
agency?s
preparation and planning are lacking, developing an effective solicitation
and contract award process could be difficult.
Agency Commitment Is Moving to outsourcing solutions such as seat management
can involve a
Crucial cultural change for government organizations because it may require
a
change to an agency?s operating model, such as using a contractor to provide
IT services using a performance- based contract. In the past, an agency?s IT
services may have been provided by government staff, in which case it is
important that the agency consider the impact of outsourcing on these
employees. In addition, the agency may not be familiar with outsourcing or
the use of performance- based contracts, in which the customer agency
specifies the outcome or result it desires and allows the vendor to decide
how best to achieve the desired outcome. As a result, it is crucial that
agencies demonstrate a commitment to change. Without such a
commitment, success is more difficult to ensure. For example, according to
the GSA Federal Technology Service?s chief information officer (CIO), GSA?s
decision to implement seat management was driven, in part, by the desire to
show the agency?s support for its seat- management governmentwide
acquisition contract. 26 He asserted that GSA?s seat management
implementation did not succeed because the agency?s culture
did not support the change, and seat management was not implemented in a
consistent manner throughout the agency. GSA?s FSS CIO provided a similar
assessment of the agency?s seat management implementation.
One way that an agency can demonstrate a commitment to seat management is
through the involvement of top agency officials. Our wideranging work on
federal management issues has shown that perhaps the single most important
element of successful management improvement
25 GAO- 02- 214. 26 In the exit conference with GSA, agency officials noted
that while the timing of the decision to implement seat management may have
been driven by the desire to show the agency?s support for the concept, the
agency also implemented seat management to achieve other goals, such as
increased standardization and decreased costs per seat.
initiatives is the demonstrated commitment of top leaders to change. 27
Moreover, our research has shown that executive leadership is a critical
success factor for outsourcing IT services in the commercial world. The
applicability of this principle to seat management was echoed by federal
agencies and private- sector organizations alike. For example, (1)
Treasury?s seat management program manager stated that for seat management
to
succeed, agency executives need to understand and support it; (2) NASA?s
ODIN delivery order contracting officer representative at the Goddard Space
Flight Center noted that outsourcing is extremely difficult unless there is
a clear mandate from the top of the organization; (3) a NASA
consultant?s report stressed that unequivocal senior management support of
ODIN is vital to its success; and (4) a member of the Industry Advisory
Council?s Managed Services Shared Interest Group 28 stated that top agency
officials, including the CIO, chief operating officer, and chief financial
officer must ?buy in? to seat management and be committed and involved.
To address the culture change sometimes imposed on an agency implementing
seat management, the Managed Services Shared Interest Group and GSA?s
Federal Technology Service noted that change
management processes can help address this issue. In addition, our research
of leading practices associated with the outsourcing of IT services provided
examples of actions that entities can take to demonstrate
executive leadership, namely the following: keep the entire organization
informed throughout the outsourcing initiative;
conduct regular peer- to- peer meetings at each level in the organization;
secure key executive support before eliciting organizational support; and
establish a communications team to promote the idea of outsourcing. 27 U.
S. General Accounting Office, Management Reform: Elements of Successful
Improvement Initiatives, GAO/ T- GGD- 00- 26 (Washington, D. C.: October 15,
1999). 28 The Industry Advisory Council is an organization of IT
professionals representing more than 270 companies nationwide that provide
products and services to the public. The mission of its Managed Services
Shared Interest Group is to facilitate information flow;
foster dialog about best practices; and influence the future of managed
services (which includes seat management) to the mutual benefit of
government, industry, and end users.
Preparation and Planning Success in implementing seat management rests in
large part on thorough
Are Critical preparation and planning both before and after the contract is
awarded.
For preaward planning, it is vital that an organization optimize and
baseline its internal IT processes to provide it with essential information
to make a sourcing decision. 29 Another important preaward planning process
is developing a business case that evaluates the expected returns against
the costs and focuses on strategic objectives that are agreed on by key
stakeholders. Moreover, it is critical that agencies gather as much data as
possible regarding, for example, inventories, transaction volumes, and
service requirements. Finally, according to Gartner, the most positive
desktop outsourcing outcomes should be expected when the outsourcing
organization?s desktop environment is already under control (including
having good standards, established processes, and predictable service
levels) and effort is expended to define and maintain an effective
relationship between the affected parties. 30 Without such preaward
preparation and planning, agencies could
encounter significant problems during implementation. For example, Peace
Corps officials reported that the agency had file conversion problems during
its seat management transition because it did not know the size and number
of files involved and greatly underestimated the time that it would take to
complete the conversion. Peace Corps seatmanagement contractor officials
estimated that solving the file conversion problem required the vendor to
add five staff members over a 2- month period. One of the contractor
officials added that the vendor did not pass these costs on to the Peace
Corps because the vendor wanted to demonstrate its commitment to the
contract.
29 GAO- 02- 214. 30 Gartner, Inc., Desktop Management Outsourcing: Do?s and
Don?ts, Research Note DF- 140931 (October 2, 2001).
Postcontract award planning is equally important. In particular, agencies in
our review, as well as private- sector organizations, 31 repeatedly cited
the importance of the transition period between awarding the contract and
reaching a ?steady state.? In addition, GSA?s seat- management program
office noted that one of the most difficult aspects for both the agency and
the contractor is learning the existing infrastructure during the transition
period. Officials from two agencies stated that they should have allowed
for more time to complete the transition period. One of these officials
asserted that the agency should have doubled the amount of time for
transition because the 3- month period that it had used did not allow
adequate time to set up help- desk procedures or define how to monitor the
network. The seat management contractor for this agency also acknowledged
that the transition period was ill- defined and did not adequately lay out
the parallel operations with the incumbent contractor. Continuous planning
after the transition period is also important. For example, officials at
Treasury and NASA noted the importance of planning ahead for technology
refreshments and future IT requirements,
respectively. Examples of practices that agencies and/ or private- sector
organizations implemented, or suggested should be implemented, in the
planning area are as follows: Solidly track and measure the existing IT
organization. A leading research firm, the Giga Information Group, Inc.,
suggested that an entity
employ solid tracking and measurement of its IT organization before
outsourcing because only then would it be able to determine whether the
arrangement has been successful. 32 In addition, this firm noted that
constructing payments and incentives accurately is extremely difficult, if
not impossible, unless the entity has already established measurements for
the functions being outsourced.
Consider using a total- cost- of- ownership study. Officials from the
Managed Services Shared Interest Group, the Peace Corps, DLA, and GSA cited
the value of total- cost- of- ownership studies, which evaluate 31 The
private- sector organizations referred to in this section include the seat
management contractors for the six agencies and GSA; Gartner, and Giga
Information Group, Inc. (private research firms); and members of the Managed
Services Shared Interest Group. 32 Giga Information Group, Inc., Payment and
Incentives for Outsourcing Management (July 27, 2000).
the current distributed computing environment of the organization. For
example, a member of the shared interest group stated that meaningful data
must be collected to put the agency?s current environment and its associated
costs in perspective. Officials from DLA and the Peace Corps also suggested
outsourcing the total- cost- of- ownership studies. For example, DLA
officials believed that the agency?s use of an objective
consulting firm to help develop a total- cost- of- ownership study and other
analyses worked well because the agency was able to take advantage of the
firm?s seat management expertise.
Develop a transition plan. Among the most important lessons learned
identified by a GSA- supported, seat- management lessons- learned workshop
was to develop a detailed project/ transition plan. In addition,
a leading commercial practice for the outsourcing of IT services is
developing a transition plan, which can help make the move from internal to
external IT services a smooth one. The clear definition of responsibilities
and the careful consideration of employees? needs matched against the
organization?s needs enable both the client and the provider to focus on
responsibilities that they can execute successfully. 33 In addition,
officials from DLA suggested that agencies establish a governance structure
for monitoring contractor performance
before the transition period begins. Implement the program in phases. ATF
and NASA?s Office of Space
Flight suggested that seat management be implemented in phases. For example,
ATF?s CIO touted the agency?s development of an installation ?cookbook? that
was used as a repeatable process for other locations and refined as seat
management was implemented at these other locations. NASA?s Office of Space
Flight suggested that an organization
new to seat management begin with core requirements and add new services
after the contractor has stabilized operations. 33 In cases in which the
seat management contractor is assuming responsibility for functions
previously performed by federal employees, it is especially important that
the organization address internal staff issues. This can include, when
appropriate and consistent with organizational objectives, (1) encouraging
the transition of staff to the seat management contractor; (2) assisting
employees who do not want to transfer in finding other jobs, either within
the organization or at another organization; and (3) developing employee
retention programs to keep key people.
Solicitation and Contract Agencies emphasized the importance of carefully
managing the contract Award Should Be Carefully solicitation and award
process. This involves using a solicitation and award Managed process that
(1) includes a solicitation document that is based on the goals and
requirements of the agency; (2) encourages contractors to bid; (3)
effectively evaluates the suitability of bidding contractors; (4) results in
a winning vendor that best suits the organization?s needs; and (5) produces
a
contract with fair pricing and with specific, relevant, and measurable
performance requirements geared toward outcomes. In particular, at the
beginning of the process, it is important that the agency understand its
needs and convey this in the solicitation document. For example, according
to Gartner, to develop valid pricing estimates, the contractor needs as much
information as possible regarding hardware and software inventories, current
transaction volumes, expected levels of service, and customization
requirements. 34 Gartner also encourages organizations to use the vendor
selection process to gain an understanding of the experience of the bidding
contractors and their relationships with subcontractors and to match the
contractors? services to user requirements and the specific account team
assigned. 35 If this is not done, an agency may overestimate the
contractor?s experience and capabilities. For example, officials from one
agency told us that because the winning bidder was a ?world class
organization,? it assumed
that the contractor had ?world class? expertise in areas such as testing.
The agency later found that this was not always the case.
Once the winning bidder is chosen, it is important to negotiate a contract
that provides a solid foundation for the working relationship between the
agency and its contractor by setting the expectations of service levels,
delivery of essential services, and continuous improvement. This includes
establishing appropriate performance metrics to measure and reward or
penalize contractor performance. Defining appropriate metrics and related
penalties and incentives can be challenging. For example, NASA and Treasury
have reassessed or are in the process of reassessing the performance metrics
in their seat management contracts because of concerns regarding their
adequacy. In particular, four NASA organizations
34 Gartner, Inc., Is Your Organization Ready for Seat Management?, Research
Note #DF- 092729 (October 4, 1999). 35 Gartner, Inc., Seat Management: Look
Beyond the Prime Vendors, Research Note #DF- 079093 (May 10, 1999).
reported that the metrics in the ODIN contract were inadequate, incomplete,
and/ or did not provide sufficient incentive to the contractor. One
organization stated that the metrics were complex and cumbersome, whereas
another noted that metrics applicable to e- mail and other enterprise
services were not adequate to ensure acceptable service performance and
adequate capacity for service growth. NASA has recognized these problems and
has made changes in its follow- on seat management contracts pertaining to
the payment provisions associated
with achieving the performance metrics. However, the private- sector firm
that performed a postimplementation review of the NASA ODIN program
recommended that the agency take additional action, including initiating a
full review of the metrics in the contract.
The following are examples of practices that agencies and/ or private-
sector organizations implemented, or suggested should be implemented, in the
solicitation and contract award area: Provide potential bidders with
critical requirements information.
Agencies and private- sector organizations made various suggestions related
to providing information to prospective bidders. For example, the Peace
Corps, DLA, and a contractor stated that issuing draft requirements to
obtain comments and questions proved useful. Moreover, this contractor
stated that the agency?s decision to provide the service levels that
specified the required levels and types of service to the bidders before
awarding the contract allowed them to structure and price their offers to
satisfy these levels. In another case, the Peace Corps and its contractor
both agreed that the preaward site visits that the agency sponsored helped
potential vendors better understand the
environment in which they would be working. Consider requiring oral
presentations. DLA and the Peace Corps noted
that requiring competing contractors to provide oral presentations clarified
the bidders? written material and allowed the agency to better understand
the bidders? management approach, respectively. DLA also
videotaped these sessions, which it later used to confirm contractor
representations. Use relevant performance metrics and related incentives
and penalties in contracts. Agencies and private- sector organizations alike
reported
that the structuring of the performance metrics and related incentives and
penalties in the contract was critical. For example, the Managed Services
Shared Interest Group stated that the performance
requirements should define the work in measurable, mission- related terms
and that the performance standards should be tied to the requirements. The
Giga Information Group provided the following examples of areas in which
bonus payments can be structured in desktop outsourcing: (1) procurement, in
which the contractor is provided incentives to reduce the purchase price of
equipment that are
tied to its overall repair history to avoid situations in which inferior
equipment is chosen; (2) uptime or meantime between failure, in which
incentive payments are tied directly to service costs or service quality;
and (3) service- level improvement, in which incentives are used to motivate
the contractor to invest in methodologies, technologies, and processes that
improve the services provided to the end user without substantially
increasing the associated costs. 36 In addition, the Giga
Information Group noted that companies should consider penalties structured
around these same areas. Our November 2001 report on the outsourcing of IT
services also provides specific examples of leading commercial practices
related to performance requirements. 37
Consider incentives related to the transition period. The master ODIN
contract provides for the payment of bonuses related to contractor
performance during the transition period. Officials from another agency that
did not include transition bonuses in its seat management contract stated
that they wished they had included an incentive related to the transition
period in its contract.
Strong Program and Since the agency is ultimately responsible for ensuring
that services are
Contract Management Is provided and users? needs are met, a key to the
successful implementation
Key to Success of seat management is program and contract management. While
much of
this responsibility is defined by the terms of the contract, market
conditions may change, and an arrangement that was once advantageous may
become less so over time. Therefore, it is important for an organization to
monitor service levels internally as well as to maintain an external view of
the performance of other providers in its peer group.
Gartner recommends having a person or group manage the contractor in a 36
See footnote 32. 37 GAO- 02- 214.
way that fits seamlessly into the overall IT service model. 38 Specifically,
according to Gartner, designated staff members, including representatives
from IT operations, business units, and procurement, need to be assigned to
manage the contractor and establish service- level metrics. The need for
dedicated federal personnel to manage the program and contractor was also
cited as important by NASA, GSA, ATF, and DLA officials. Also important is
that an agency implement a process to validate the information being
provided by the contractor to ensure that it is accurate. For example, the
contracting officer for the Department of State?s seat management contract
stated that she rejected an inventory report issued by
the agency?s contractor because it contained names of nonexistent users and
duplicate records.
The necessity for strong program and contract management was demonstrated by
GSA?s decision not to continue with its seat management implementation. In
this case, according to the Federal Technology Service?s
CIO, GSA attempted to manage its seat management program centrally while
also allowing each of its services and field locations to develop different
service- level agreements and define drastically different desktop
requirements. As a result, according to GSA and contractor officials, the
contract terms did not match the actual agency implementation. In addition,
GSA did not adhere to the implementation schedule or the
service- level mix it had agreed upon with its contractor. Moreover, even
though the contractor was responsible for asset management under the GSA
seat management contract, the agency?s Federal Technology Service?s CIO
stated that GSA had difficulty obtaining accurate information from the
contractor in terms of the number, type, and costs of these assets when
arranging to transfer ownership of these assets from the contractor to GSA.
According to the project manager for the GSA contractor, this was due, at
least in part, to actions by users, such as moving equipment without
authorization. Examples of practices that agencies and/ or private- sector
organizations implemented, or suggested should be implemented, in the
program and contract management area are as follows: Benchmark contractor
performance. NASA and CMS used an
independent contractor to perform market surveys quarterly or biannually and
to test original equipment manufacturers? products in 38 See footnote 30.
order to benchmark the IT equipment proposed by the seat management
contractor. In addition, DLA plans to annually benchmark its users?
satisfaction level through the use of surveys. Agency use of benchmarking is
consistent with our research of leading commercial practices for outsourcing
of IT services, which found that periodic benchmarking allows an
organization to ascertain whether it is still obtaining good value from its
provider. Use service- level agreements. The use of service- level
agreements was
cited as a critical practice by both agencies and contractors. For example,
a contractor noted that its service- level agreements with Treasury?s
Departmental Offices were a critical success factor because they established
a specific understanding between the contractor and the agency in which
customer expectations were realistic and in concert with the IT and support
services the contractor must deliver. In addition,
Treasury, two contractors, and the Managed Services Shared Interest Group
noted that it is important to periodically review the service- level
agreements to ensure that they are still appropriate. Ensure that services
are provided. In January 2001, NASA?s Office of Space Flight issued a plan
outlining a strategy for managing its four ODIN contracts, including (1)
observing contractor processes and
procedures; (2) sampling items for review (such as desktop hardware and
software and trouble tickets); (3) conducting audits using checklists; and
(4) assessing contractor- generated data (such as asset tracking and
performance- related data). Agency/ Contractor
Agencies and contractors often noted that it is vital that seat management
Partnerships Are Vital to be approached as a partnership, which can be
realized by working to Success
establish and achieve common goals. In addition, they cited the importance
of establishing an environment of mutual trust so that issues and potential
conflicts can be resolved more easily. For example, CMS emphasized the
importance of an agency?s developing a strong working relationship with its
seat management contractor that involves mutual trust, noting that
flexibility and shared goals are keys to success. The need for agency/
contractor partnerships is consistent with our research of commercial
practices, which found that aligning client and provider objectives in a
partnership is key to building consensus and is imperative to establishing
early trust among all stakeholders. For this alignment to occur, the client
and provider must work together to establish common project goals beyond
objectives stated in the request for proposal. Both sides must
recognize and understand each other?s underlying motives and strive to
achieve established expectations. Developing a productive agency/ contractor
relationship is not always easy. For example, even though the ODIN master
contract has been in place since mid- 1998, in November 2001, a NASA
consultant noted that a true partnership between the agency and its
contractors had not been realized, although some progress had been made.
NASA has since established a working group that comprises both agency and
contractor representatives to address this matter.
The following are examples of practices that agencies and/ or private-
sector organizations implemented, or suggested should be implemented, to
develop and nurture agency/ contractor partnerships: Develop a trusting
relationship with the seat management contractor.
A NASA consultant?s recommendation to the Goddard Space Flight Center was
that the center and its vendor build institutional and personal
relationships and develop a formal conflict- resolution process using
government and contractor staff. In responding to a draft of this report,
NASA noted that the Goddard Space Flight Center had established a partnering
arrangement with its seat management
contractor and developed a conflict resolution process. Consider including
the contractor in agency IT planning. One agency
suggested including the contractor in strategic and tactical planning, such
as enabling the contractor to provide input into future standards and
policies. Hold agency/ contractor meetings. A seat management contractor
recommended the approach taken by one agency of holding periodic off- site
meetings to discuss issues and establish mutually agreed- upon priorities.
Establishing Effective When the IT service provider is outside of the
entity, as it is in the case of Communication Among
seat management, disconnects between organizations are more likely; thus,
Various Entities Is Critical
processes to facilitate good communication are critical. Indeed, most of the
entities we reviewed cited communication among the seat management
contractor, the agency program office, users, and/ or other agency
contractors as critical to the success of seat management. For example, (1)
the Managed Services Shared Interest Group stated that stakeholders and
executives should be kept informed of progress ?early and often,? (2)
various NASA centers and two contractors emphasized the need to provide
proactive user outreach and manage user expectations, and (3) three agencies
and one contractor noted the importance of communication among the various
contractors that an agency might use for related
services. Continuous communication throughout the seat- management life
cycle is also important. For example, according to Gartner, communication is
a critical part of both the evaluation and implementation processes of seat
management. 39 Without effective communication, an agency?s seat management
initiative can encounter problems. For example, GSA?s Federal Technology
Service?s CIO noted that the agency did not take enough steps to market the
seat management program throughout the agency; consequently, users of
desktop services neither understood nor liked the changes that this approach
entailed. Inadequate user outreach was also cited as a problem at NASA. For
example, a consultant to the Goddard Space Flight Center noted that customer
satisfaction requires a high degree of openness and cooperation among ODIN
representatives, the ODIN project office, and the vendor, which was not
occurring. Indeed, the consultant found that among users there was a
perception that ?no one listens or cares.?
Examples of practices that agencies and/ or private- sector organizations
implemented, or suggested should be implemented, in the communications area
are as follows: Market seat management within the agency. Various
approaches to
marketing seat management to users were cited, including the sponsorship of
?town hall meetings? and technology days or the distribution of written
materials. In terms of the type of information to be provided to users,
GSA?s seat management program office noted that it is important that users
be educated about the benefits of seat management, the changes they may
expect, and the procedures for
using the new service. Consider the seat management contractor?s
relationship to other service providers. A NASA consultant recommended that
the Goddard
Space Flight Center identify all service providers that interface with or
whose activities or responsibilities overlap with the seat management
contractor and define how they will work together. Similarly, four 39 See
footnote 34.
agencies, a contractor, and the participants in a GSA- supported
seatmanagement lessons learned workshop suggested that the seat management
contractor establish formal agreements with other agency
contractors. The contractor noted that such an agreement is especially
important when the other agency contractor can affect the seat management
vendor?s ability to meet the service- level agreements. In addition, within
about 3 months of contract award, CMS signed a memorandum of understanding
with its seat management contractor and facilities contractor that set forth
the responsibilities of all three organizations.
Conclusions The agencies we reviewed implemented seat management for a
variety of reasons, including to (1) improve their IT management, (2)
improve enduser
support and productivity, and (3) obtain new or upgrade current technology.
In addition, these agencies reported a variety of accomplishments resulting
from implementing seat management, such as improved asset management and
end- user support. However, they have not performed the analyses necessary
to validate the overall results of this approach. Specifically, the agencies
performed limited or, in some cases, no
analyses of costs and benefits before implementing seat management and have
not routinely monitored all actual costs or benefits. As a result, these
agencies lack vital data to demonstrate actual investment results. Without
these data, it is difficult to determine whether the benefits of seat
management outweigh its costs and risks. Moreover, this lack of monitoring
could impair the agencies? ability to justify and implement future seat
management investments. These agencies, and others considering future seat
management investments, could benefit from the
myriad lessons learned by organizations that have implemented seat
management, such as the need for thorough preparation and planning, agency
commitment, program and contract management, and continual communication. By
applying the lessons learned in these critical areas and others, agencies
considering seat management could more effectively plan their activities and
reduce the risks associated with implementing such a relatively new concept.
Recommendations To determine to what extent their current seat management
programs have achieved positive results, we recommend that the secretary of
the treasury; administrators for the National Aeronautics and Space
Administration and Centers for Medicare and Medicaid Services; and directors
of the Peace
Corps, Bureau of Alcohol, Tobacco and Firearms, and Defense Logistics Agency
each routinely monitor all actual seat management costs and benefits.
To provide for adequate justification of any future seat management
investments, we recommend that the secretary of the treasury; administrators
for the National Aeronautics and Space Administration and Centers for
Medicare and Medicaid Services; and directors of the Peace Corps, Bureau of
Alcohol, Tobacco and Firearms, and Defense Logistics
Agency each ensure that existing federal policy and guidance for information
technology investments be followed when considering investments in
information- technology- service outsourcing. Specifically, for future seat
management investments, we recommend that these agencies
baseline the current costs of the service being outsourced, including the
cost of internal agency operations;
perform an analysis of expected costs and benefits; perform an analysis
of risks, including developing plans to mitigate risks identified;
monitor actual costs and benefits as a basis for results accountability;
and implement, to the extent feasible, the lessons learned that were
identified in this report.
Agency Comments and We received written comments on a draft of this report
from Treasury?s
Our Evaluation Departmental Offices and ATF, the Peace Corps, the Department
of Defense, GSA, and NASA. Three agencies agreed with the findings or
recommendations in the report, two did not indicate whether they agreed or
disagreed, and NASA supported many of the findings but disagreed with
portions of the report. We also requested comments from the Centers for
Medicare and Medicaid Services, but none were provided.
The comments provided by Treasury?s Departmental Offices and ATF, the Peace
Corps, Defense, and GSA varied in scope and detail. Specifically,
Treasury?s acting director, Customer Service Infrastructure and
Operations, Office of the Chief Information Officer, stated that Treasury?s
Departmental Offices has implemented, or is in the process of implementing,
our recommendations. Treasury also offered clarifying comments that we
incorporated into the report, as appropriate. The
comments from Treasury are reproduced in appendix IV. The director of ATF
stated that its Office of Science and Technology had
reviewed the report and had no comments at this time. ATF?s written response
is reproduced in appendix V.
The director of the Peace Corps did not address whether the agency agreed
or disagreed with the findings or recommendations in the report, but offered
clarifying comments that we incorporated into the report, as appropriate.
The comments from the Peace Corps are reproduced in
appendix VI. Defense?s deputy assistant secretary of defense (deputy CIO)
stated that the department generally concurs with the recommendations in the
report. Defense also included technical corrections that we incorporated
into the report, as appropriate. The comments from Defense are reproduced in
appendix VII. GSA?s chief information officer concurred with the findings
in the report. The comments from GSA are reproduced in appendix VIII.
Although NASA supported many of the findings, it disagreed with portions of
the report. Specifically, NASA did not agree with our assessment that (1)
its up- front cost analysis was not sufficient; (2) it did not track its
internal seat management costs, citing its tracking of the full- time-
equivalents associated with seat management; and (3) it did not adequately
track
benefits. We disagree with these NASA comments. First, the problems with
NASA?s up- front cost analysis were cited by the contractor the agency
employed to conduct a postimplementation review. The report prepared by the
contractor asserted that it was impossible to determine whether the agency
is saving money with seat management because of the lack of a comprehensive
pre- seat- management baseline. Second, while important,
monitoring of full- time- equivalents does not provide the agency with a
complete picture of internal costs associated with the implementation of
seat management. As NASA officials acknowledged during the exit
conference, the agency does not track the full costs of seat management,
which would include internal cost items such as overhead and salaries and
benefits. Finally, NASA?s efforts to track program benefits are not
complete. Specifically, while NASA?s quarterly reports and
postimplementation review address some of the agency?s expected seat
management benefits, other expected benefits, such as potential improved
staff productivity and efficiency, were not addressed. NASA also provided
technical comments that we have incorporated in this report, as appropriate.
NASA?s written
comments, along with our responses, are reproduced in appendix IX. As agreed
with your office, unless you publicly announce its contents earlier, we plan
no further distribution of this report for 30 days from the date of this
letter. At that time, we will send copies to the chairman and ranking
minority member, Senate Committee on Governmental Affairs; chairman and
ranking minority member, House Committee on Government Reform, ranking
minority member, Subcommittee on Technology and Procurement Policy, House
Committee on Government Reform; and other interested congressional
committees. We are also sending copies to the secretary of the treasury;
administrators for the National Aeronautics and
Space Administration and Centers for Medicare and Medicaid Services; and
directors of the Peace Corps, Bureau of Alcohol, Tobacco and Firearms,
Defense Logistics Agency, and the Office of Management and Budget; and other
interested parties. We will also make copies available to others upon
request. If you have any questions on matters discussed in this report,
please contact me at (202) 512- 6257 or by e- mail at mcclured@ gao. gov.
Other contacts and key contributors to this report are listed in appendix X.
Sincerely yours,
David L. McClure Director, Information Technology Management
Appendi xes Information on Seat Management Contracts as of December 31,
2001, Awarded under the
Appendi x I
NASA ODIN Master Contract Dollars in thousands
Reported estimated Agency/
contract Location Services identified in the contracts a
amount b Estimated benefits
National Aeronautics Application/ Database, file storage, $16,014 Estimated
benefits are the same for all contracts and Space
general- purpose desktop, maintenance awarded by NASA?s centers and are
described
Administration?s only, scientific and engineering desktop, in the original
business case for Outsourcing
(NASA)/ and World Wide Web seats
Desktop Initiative for NASA (ODIN), as follows: Ames Research Center
divesting the day- to- day management of noncore information technology
(IT) functions;
improved management of assets and their configuration;
improved technology refreshment;
improved interoperability;
improved standardization;
a consistent agencywide desktop strategy;
simplified procurement, contractor, and budget management processes;
transfer risk from the government to the commercial sector;
potential increases in user efficiency and productivity;
better use of civil service personnel previously employed in desktop
support; and
access to greater information technology expertise. NASA/
Application/ Database, file storage, 21, 216 Same as above. Dryden Research
general- purpose desktop, maintenance Center
only, scientific and engineering desktop, and World Wide Web seats NASA/
Application/ Database, file storage, 44, 074 Same as above. Glenn Research
general- purpose desktop, maintenance Center
only, scientific and engineering desktop, and World Wide Web seats NASA/
Application/ Database, cellular phone, 51, 083 Same as above. Langley
Research facsimile, file storage, general- purpose Center
desktop, local- area network, local video, maintenance only, network-
attached device, remote communications, scientific and engineering desktop,
telephone, and
World Wide Web seats
(Continued From Previous Page) Dollars in thousands
Reported estimated Agency/
contract Location Services identified in the contracts a
amount b Estimated benefits
NASA/ Application/ Database, facsimile, file
20, 005 c Same as above. Goddard Space Flight storage, general- purpose
desktop, localarea Center
network, scientific and engineering computer, maintenance only, network
attached device, and World Wide Web and remote communications seats NASA/
Administrative radio, 20, 190 Same as above. Headquarters application/
database, cellular phone, computational server, facsimile, file storage,
general- purpose desktop, localarea
network, local video, maintenance only, meeting place conferencing, public
address, remote communications, scientific and engineering desktop,
telephone, and World Wide Web seats NASA/
Application/ Database, cellular phone, 180, 721 Same as above. Johnson Space
facsimile, general- purpose desktop, local Center
video, remote communications, scientific and engineering desktop, telephone,
and World Wide Web seats
NASA/ Application/ Database, cellular phone, 65, 170 Same as above. Kennedy
Space
facsimile, general- purpose desktop, local Center
video, remote communications, scientific and engineering desktop, telephone,
and World Wide Web seats
NASA/ Marshall Application/ Database, cellular phone, 100, 187 Same as
above. Space Flight Center facsimile, general- purpose desktop, local
video, remote communications, scientific and engineering desktop, telephone,
and World Wide Web seats
NASA/ Application/ Database, cellular phone, 30, 396 Same as above. Stennis
Space Center facsimile, general- purpose desktop, local
video, remote communications, scientific and engineering desktop, telephone,
and World Wide Web seats
(Continued From Previous Page) Dollars in thousands
Reported estimated Agency/
contract Location Services identified in the contracts a
amount b Estimated benefits
Centers for Medicare General- purpose desktop seats, asset
25, 647 Avoidance of costs incurred from theft or loss and Medicaid
management, and engineering services of desktop computing assets. Services
(CMS)
Reduced costs of license compliance.
Avoidance of costs for duplicate computer equipment.
Cost savings over purchased equipment.
Cost savings from consolidation of contracts.
Improved user productivity.
Improved responsiveness at the user level.
Increased end- user satisfaction.
Compliance with Year 2000 requirements.
Improved mission effectiveness.
Reduced risks associated with Year 2000 readiness.
Improved morale.
Redirected staffing responsibilities. a In addition to the individual
categories of seat services acquired by each of the NASA implementing
entities and CMS, the NASA ODIN contract allows for acquiring other
hardware, software, and services through a catalog service offered by the
seat management contractors as well as a special order process. b We asked
the agencies to provide us with the total estimated costs through the
completion date of the
current delivery order. c The reported estimated contract amount for the
Goddard Space Flight Center reflects the amount estimated through the end of
its original delivery order. As of January 30, Goddard had not signed a
follow- on delivery order. According to the Goddard delivery- order
contracting officer, until this follow- on delivery order is signed, Goddard
has agreed to pay the contractor the amount of the monthly invoice
(generally about $600,000) plus 30 percent. Source: GAO, based on
information provided by the agencies. We did not verify this information.
Information on Seat Management Contracts as of December 31, 2001, Awarded
under the
Appendi x II
GSA Seat Management Master Contract Dollars in thousands
Services identified in Reported estimated
Agency the contracts a contract amount b Estimated benefits
Department of the Asset management,
$114,732 Ensure the continued reliability and availability of existing and
Treasury?s
commercial information planned IT systems, equipment, services, and data.
Departmental services management,
Provide for access to private- sector technical expertise, staff Offices
desktop/ portable support, management experience, and corporate
capabilities. computers, help desk, Acquire an operational IT platform
that results in a positive local- and wide- areanetwork return on
investment.
servers, Provide for a single point of contact for IT operations. network
printers, and
Increase user satisfaction with IT solutions. voice/ telephone
administration
U. S. Air Force Asset management, 7,266 Ability to establish and sustain
effective and efficient life- cycle
Special desktop/ portable support for IT operations. Operations computer,
help desk, Single point of contact. Forces/
local- and wide- areanetwork
Access to private- sector expertise regarding the latest Aeronautical
servers, available technologies. System Center
network printing,
Migration to a self- contained network. System Program peripheral device,
and
Paperless acquisition office environment. Office at WrightPatterson
software (including Cost savings from potential future expansion of seat
Air custom applications)
management. Force Base
services Ability of staff to focus efforts on mission needs. Department of
Asset management,
48, 139 Ability to establish and sustain effective and efficient managed
State/ Office of classified network, life- cycle support of the office?s
distributed computing Foreign Building
desktop/ portable environment with a single point of contact. Operations
computers, local- areanetwork servers, and
network printers Federal Highway Asset management,
150,000 Ability to establish and sustain effective and efficient managed
Administration desktop/ portable life- cycle support of a pilot project on
distributed computing computer, help desk, with a single point of contact.
local- area- network Achieving high- quality, reliable, and responsive
levels of server, network printing, service. programming, More easily
achieving agencywide technology transfers software/ software
through an integrated and uniform processing environment. maintenance, and
Obtaining technology upgrades at regular intervals. infrastructure Reduced
asset maintenance and management management services
responsibilities.
Achieving greater contracting efficiency.
Consolidation of diverse product and support service requirements.
More efficient use of resources resulting from a consolidated internal
infrastructure and support environment.
(Continued From Previous Page) Dollars in thousands
Services identified in Reported estimated
Agency the contracts a contract amount b Estimated benefits
U. S. Army/ Center Asset management,
3,058 Implementation of comprehensive and integrated desktop and for
Substance
desktop/ portable local- area- network management services. Abuse Programs
computer, help desk, Freeing agency personnel from the burden of
maintaining and Internet, local- areanetwork updating desktop resources,
services, and training so that they servers, can refocus their efforts on
the core mission.
network printing, and
Having a single point of contact. World Wide Web
Achieving the goal of acquiring distributed computing services management
services seamlessly and efficiently.
Department of Asset management, 48, 300 Obtain a secure network.
Housing and desktop/ portable Acquire applications to support the agency?s
audit and Urban
computers, help desk, investigation missions. Development/ local- and wide-
areanetwork
Easier technology transfers through an integrated and uniform Office of
Inspector servers, processing environment. General
network printers, and
Technology upgrades at regular intervals so the agency can virtual private
network benefit from new and improved technological improvements. services
Freedom from maintaining and managing capital assets and refocusing staff
on the office?s mission.
Greater contracting efficiency through consolidation of diverse
requirements.
More efficient use of resources resulting from a consolidated internal
infrastructure and support environment. Peace Corps Asset management,
30, 045 Migration to a different technology environment. desktop/ portable
Easier technology transfers through an integrated and uniform computer,
help desk, processing environment.
local- and wide- areanetwork
Technology upgrades at regular intervals. servers, Freedom from having
to maintain and manage capital assets. network printing, off- site Greater
contracting efficiency and a decrease in contract storage, and training
management functions by consolidating diverse product and
services support service requirements.
More efficient use of resources through a consolidated internal
infrastructure and support environment. Nuclear
Integrated infrastructure 80, 288 Desktop and network refreshment to
reduce maintenance Regulatory management, asset
costs and maintain a high level of service availability. Commission
management, help Streamlined integrated desktop refreshment strategy.
desk, maintenance, A single focal point for distributed computing support.
development/ Obtain a single operating system. integration, and Enhanced
help- desk support through access to subject- matter contingency operations
experts.
Defined and managed level of service delivery.
High level of customer satisfaction.
Quicker and more efficient access for the purchase of peripheral equipment
and software through the use of an online catalog.
Cost avoidance. Note: Two other organizations- the General Services
Administration (GSA) and the District of Columbia Housing Authority- issued
task orders under the GSA seat- management master contract, which they let
expire.
a In addition to the bundled seat management services shown, the GSA seat-
management master contract allows agencies to purchase catalog orders. b We
asked the agencies to provide us with the total estimated costs through the
completion date of the
current task order. Accordingly, the amounts provided assumed that all
option years would be exercised.
Source: GAO, based on information provided by the agencies. We did not
verify this information.
Appendi x I II
Scope and Methodology To evaluate agencies? approaches to the seat-
management outsourcing concept, we selected six agencies for review.
Specifically, as agreed with your office, we chose two agencies each that
were using the General Services Administration?s (GSA) seat management
contract, NASA?s ODIN contract, and GSA?s Federal Supply Service (FSS)
Schedule 70 contracts. In selecting the agencies, we chose those whose seat
management contract had been awarded more than 1 year before our review to
ensure that the
agencies had time to adjust to the use of seat management and whose contract
included at least 500 seats. Accordingly, we selected the Department of the
Treasury?s Departmental Offices and the Peace Corps, which use the GSA seat-
management master contract; NASA and CMS, which use the NASA ODIN master
contract; and the Defense Logistics Agency (DLA) and Treasury?s Bureau of
Alcohol, Tobacco and Firearms (ATF), which use GSA?s FSS Schedule 70
contracts.
To determine these agencies? rationales for using the seat management
alternative, whether they achieved estimated costs and benefits, and how
well they managed associated risks, we reviewed and analyzed documents
related to each agency?s decision to implement seat management and how
it managed the program. For example, we reviewed business cases; program and
implementation plans; and analyses of costs, benefits, and risks. We also
assessed whether these documents followed Office of Management and Budget
(OMB) 40 and our 41 guidelines. In addition, to determine whether the six
agencies had implemented contracts that protect the government, we evaluated
their seat management contracts to determine whether they adequately
addressed the issues of (1) quality
assurance, (2) termination rights, and (3) rights to supplied hardware and
software at the end of contract performance. We also interviewed agency
officials, including chief information officers (CIO) and seat- management
program and contract officials.
40 Office of Management and Budget, Circular A- 130, Management of Federal
Information Resources (November 30, 2000); Circular A- 11, Part 3, Planning,
Budgeting, and Acquisition of Capital Assets (July 2001); and Evaluating
Information Technology Investments: A Practical Guide (November 1, 1995). 41
U. S. General Accounting Office, Information Technology Investment
Management: A Framework for Assessing and Improving Process Maturity, GAO/
AIMD- 10. 1. 23, Exposure Draft (Washington, D. C.: May 2000) and Assessing
Risks and Returns: A Guide for
Evaluating Federal Agencies? IT Investment Decision- making, GAO/ AIMD- 10.
1. 13 (Washington, D. C.: February 1997).
To identify lessons learned, we reviewed applicable documentation and
interviewed officials from the six agencies and their contractors, and
reviewed applicable external and internal studies. We also conducted
interviews with GSA officials, including the agency CIO and the CIOs for FSS
and the Federal Technology Service, and the program manager for the
governmentwide seat management contract. We also reviewed documents relating
to GSA?s decision not to exercise the option to extend its seat management
contract. In addition, we obtained lessons learned from other
organizations, such as the Industry Advisory Council?s Managed Services
Shared Interest Group and leading private research firms Gartner, Inc., and
the Giga Information Group, Inc. To identify the agencies using the GSA and
NASA governmentwide seat management contracts, we obtained a list of these
agencies from GSA and
NASA. We contacted each of these agencies to obtain the services being
provided, the estimated contract amounts, and estimated benefits. We also
reviewed contracts and other documents provided by each of the agencies but
did not verify the data provided.
We performed our work at the headquarters offices of ATF, GSA, NASA, the
Peace Corps, and the Department of the Treasury located in Washington, D.
C.; at DLA headquarters in Ft. Belvoir, Va.; at CMS headquarters in
Baltimore, Md.; at the GSA offices located in Falls Church, Va.; and at
NASA?s Kennedy Space Center in Cape Canaveral, Fla. We conducted our review
between April 2001 and January 2002 in accordance with generally
accepted government auditing standards.
Comments from the Department of the
Appendi x V I Treasury?s Departmental Offices
Comments from the Bureau of Alcohol,
Appendi x V Tobacco and Firearms
Appendi x VI Comments from the Peace Corps
Appendi x VII Comments from the Department of Defense
Comments from the General Services
Appendi x VI II Administration
Comments from the National Aeronautics and
Appendi x IX
Space Administration Note: GAO comments supplementing those in the report
text appear at the end of this appendix.
See comment 1. See comment 2. See comment 3.
See comment 4. See comment 5. See comment 6.
The following are GAO?s comments on the National Aeronautics and Space
Administration?s letter dated February 28, 2002. GAO Comments 1. We disagree
that our characterization of NASA?s up- front cost analysis is not accurate.
For reasons discussed in this report, we do not agree
that NASA performed sufficient up- front cost analysis. A thorough
understanding of existing agency IT assets and functions is critical to an
agency?s ability to select appropriately among alternative ways to meet its
distributed computing needs, and to ensure the effective
management of contracts awarded to acquire these services. While we
acknowledge that performing such an analysis can be challenging, the
consequences of not doing so are nontrivial. This is clearly illustrated by
the inability of a contractor employed by NASA to determine whether the
agency is saving money with seat management because of the lack of a
comprehensive pre- seat- management baseline. Also, as discussed in this
report and our IT investment management guidance to agencies, 42 we agree
that cost represents only one element of a
complete preinvestment analysis. Other considerations include an evaluation
of expected benefits and risks. 2. We disagree that our characterization of
NASA?s tracking of internal seat costs is incorrect. We note in this report
that NASA tracks contractor costs and the number of staff associated with
the management of its seat management program. While important, the
monitoring of full- time- equivalents does not provide the agency with a
complete picture of internal costs associated with the implementation of
seat management. Moreover, as NASA officials acknowledged during the exit
conference, the agency does not track the full internal costs of seat
management, which would include cost items such as overhead and salaries and
benefits. This is critical since the internal costs to manage seat
management can be substantial. For example, NASA?s Kennedy Space Center
reported that its salaries and benefits to manage the Office of Space
Flight?s four seat- management implementations in July 2001 were about
$146,000.
42 U. S. General Accounting Office, Information Technology Investment
Management: A Framework for Assessing and Improving Process Maturity, GAO/
AIMD- 10. 1. 23, Exposure Draft (Washington, D. C.: May 2000) and Assessing
Risks and Returns: A Guide for Evaluating Federal Agencies? IT Investment
Decision- making, GAO/ AIMD- 10. 1. 13 (Washington, D. C.: February 1997).
3. We do not agree that NASA routinely monitors the actual overall benefits
of its seat management program. Tracking of actual benefits is a critical
aspect of accountability in that it answers the question of whether expected
cost savings and other qualitative and quantitative benefits are being
achieved. NASA?s seat management business case documented 11 program goals,
but neither the agency?s quarterly reporting process nor the ODIN
postimplementation review fully addresses whether these goals have been
achieved. In particular, the performance measures included in the quarterly
reports focus on certain contractor performance metrics (e. g., service
delivery and
availability) and the average seat costs for general- purpose desktop
computing seats. As we noted in this report, while this type of tracking can
be an indicator of whether certain types of benefits are being
achieved, such metrics do not fully address whether the overall costs and
benefits of the seat management program are being met. NASA?s quarterly
reports do not contain performance measures that address estimated benefits
such as potential increases in user efficiency and productivity and improved
staff focus on mission- related activities. With respect to NASA?s
postimplementation review, the contractor that performed the review
addressed whether NASA achieved certain types of benefits, such as cost
savings and standardization, but did not
address other expected program benefits, such as potential increases in user
productivity and efficiency. Nevertheless, we agree that this
postimplementation review was a good first step, which, if followed up
by subsequent reviews that address all expected benefits of the NASA seat
management program, would address our concerns in this area. 4. We modified
our report to clarify the terms of the NASA master
contract. 5. See comment 3. 6. We modified our report to recognize NASA?s
initiatives.
Appendi x X
GAO Contact and Staff Acknowledgments GAO Contact Linda J. Lambert, (202)
512- 9556 Acknowledgments Harold Brumm, Peggy A. Hegg, James C. Houtz,
Barbarol J. James, and
Frank Maguire made key contributions to this report.
(310414)
GAO?s Mission The General Accounting Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability of
the federal government for the American people. GAO examines the use of
public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance to
help Congress make informed oversight, policy, and funding decisions. GAO?s
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of The fastest and easiest way to obtain copies of GAO
documents is through the Internet. GAO?s Web site (www. gao. gov) contains
abstracts and full- text files of GAO Reports and current reports and
testimony and an expanding archive of older products. The Testimony Web site
features a search engine to help you locate documents using key words and
phrases. You can print these documents in their entirety, including charts
and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as ?Today?s Reports,? on its Web
site daily. The list contains links to the full- text document files. To
have GAO E- mail this list to you every afternoon, go to www. gao. gov and
select ?Subscribe to daily e- mail alert for newly released products? under
the GAO Reports heading.
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out to
the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:
U. S. General Accounting Office 441 G Street NW, Room LM Washington, D. C.
20548
To order by Phone: Voice: (202) 512- 6000 TDD: (202) 512- 2537 Fax: (202)
512- 6061
Visit GAO?s Document GAO Building Distribution Center Room 1100, 700 4th
Street, NW (corner of 4th and G Streets, NW)
Washington, D. C. 20013 To Report Fraud, Contact: Waste, and Abuse in
Web site: www. gao. gov/ fraudnet/ fraudnet. htm, E- mail: fraudnet@ gao.
gov, or
Federal Programs 1- 800- 424- 5454 or (202) 512- 7470 (automated answering
system).
Public Affairs Jeff Nelligan, Managing Director, NelliganJ@ gao. gov (202)
512- 4800 U. S. General Accounting Office, 441 G. Street NW, Room 7149,
Washington, D. C. 20548
a
GAO United States General Accounting Office
Why GAO Did This Study
Since 1997, federal agencies have been engaging in an informationtechnology
outsourcing approach for acquiring services in support of their desktop
computing environment from a single source, generally known as ?seat
management.? Among its objectives, GAO was asked to determine whether six
agencies involved in seat management have realized expected costs and
benefits and to identify lessons learned. GAO selected agencies that used
various contract vehicles and had at least 1 year of experience with the
contract.
March 2002 DESKTOP OUTSOURCING Positive Results Reported, but Analyses Could
Be Strengthened
This is a test for developing highlights for a GAO report. This full report,
including GAO?s objectives, scope, methodology, and analysis is available at
www. gao. gov/ cgi- bin/ getrpt? GAO- 02- 329. For additional information
about this report, contact David McClure (202- 512- 6257). To provide
comments on this test highlights, contact Keith Fultz (202- 512- 3200) or e-
mail HighlightsTest@ gao. gov.
Highlights of GAO- 02- 329, a report to the Chairman, Subcommittee on
Technology and Procurement Policy, Committee on Government Reform, House of
Representatives.
What GAO Recommends
So that the six agencies reviewed can determine the extent to which their
current seat management programs are achieving positive results, GAO
recommends that they monitor actual seat management costs and benefits.
Also, to help ensure that future seat management investments of these six
agencies are justified, GAO recommends that when considering such
investments, these agencies analyze expected costs and benefits and, to the
extent feasible, implement the lessons learned in this report.
Of the six agencies that commented on a draft of this report, three agreed
with its findings or recommendations, two did not indicate whether they
agreed or disagreed, and one supported many of the findings, but disagreed
with portions of the report.
United States General Accounting Office
What GAO Found
Although the six agencies reviewed reported various positive results from
implementing seat management, GAO could not determine whether they were
achieving expected costs and benefits because they did not perform
sufficient up- front analyses or routinely monitor actual results. All were
tracking elements of contractor performance, such as user satisfaction.
Although not a substitute for determining whether the benefits of seat
management outweigh its costs and risks, this tracking provides an
indication of whether expected services are being provided.
Positive results that these agencies reported can be categorized into four
general areas: improving information technology management (e. g., using a
standard technology environment to eliminate incompatible hardware and
software and improve information sharing across an agency); improving end-
user support, such as help- desk support; enhancing mission support (e. g.,
staff were freed from desktop management duties to perform other mission-
related duties); and more timely upgrading of technology.
However, these agencies performed limited or, in some cases, no analyses of
expected costs and benefits before implementing seat management and did not
routinely monitor all actual costs or benefits. These agencies and other
organizations consider up- front analyses and subsequent program management
critical practices for a successful implementation and have identified
various lessons learned from their experiences that would benefit other
agencies considering future seat management investments (see list below).
Lessons Learned Agencies can reduce the risk of an unsuccessful
implementation by
! obtaining agency commitment, especially by top management;
! completing thorough up- front preparation and planning activities;
! carefully managing solicitation and contract award activities;
! developing strong program and contract management activities, including
monitoring contractor performance;
! developing partnerships between agencies and the seat management
contractors in which they work toward establishing and achieving common
goals; and
! establishing effective and continual communication within the agency, with
the seat management contractor, and among contractors working on related
activities.
G A O Accountability Integrity Reliability
Highlights
Page i GAO- 02- 329 Desktop Outsourcing
Contents
Contents
Page ii GAO- 02- 329 Desktop Outsourcing
Page 1 GAO- 02- 329 Desktop Outsourcing United States General Accounting
Office
Washington, D. C. 20548 Page 1 GAO- 02- 329 Desktop Outsourcing
A
Page 2 GAO- 02- 329 Desktop Outsourcing
Page 3 GAO- 02- 329 Desktop Outsourcing
Page 4 GAO- 02- 329 Desktop Outsourcing
Page 5 GAO- 02- 329 Desktop Outsourcing
Page 6 GAO- 02- 329 Desktop Outsourcing
Page 7 GAO- 02- 329 Desktop Outsourcing
Page 8 GAO- 02- 329 Desktop Outsourcing
Page 9 GAO- 02- 329 Desktop Outsourcing
Page 10 GAO- 02- 329 Desktop Outsourcing
Page 11 GAO- 02- 329 Desktop Outsourcing
Page 12 GAO- 02- 329 Desktop Outsourcing
Page 13 GAO- 02- 329 Desktop Outsourcing
Page 14 GAO- 02- 329 Desktop Outsourcing
Page 15 GAO- 02- 329 Desktop Outsourcing
Page 16 GAO- 02- 329 Desktop Outsourcing
Page 17 GAO- 02- 329 Desktop Outsourcing
Page 18 GAO- 02- 329 Desktop Outsourcing
Page 19 GAO- 02- 329 Desktop Outsourcing
Page 20 GAO- 02- 329 Desktop Outsourcing
Page 21 GAO- 02- 329 Desktop Outsourcing
Page 22 GAO- 02- 329 Desktop Outsourcing
Page 23 GAO- 02- 329 Desktop Outsourcing
Page 24 GAO- 02- 329 Desktop Outsourcing
Page 25 GAO- 02- 329 Desktop Outsourcing
Page 26 GAO- 02- 329 Desktop Outsourcing
Page 27 GAO- 02- 329 Desktop Outsourcing
Page 28 GAO- 02- 329 Desktop Outsourcing
Page 29 GAO- 02- 329 Desktop Outsourcing
Page 30 GAO- 02- 329 Desktop Outsourcing
Page 31 GAO- 02- 329 Desktop Outsourcing
Page 32 GAO- 02- 329 Desktop Outsourcing
Page 33 GAO- 02- 329 Desktop Outsourcing
Page 34 GAO- 02- 329 Desktop Outsourcing
Page 35 GAO- 02- 329 Desktop Outsourcing
Page 36 GAO- 02- 329 Desktop Outsourcing
Page 37 GAO- 02- 329 Desktop Outsourcing
Page 38 GAO- 02- 329 Desktop Outsourcing
Page 39 GAO- 02- 329 Desktop Outsourcing
Appendix I
Appendix I Information on Seat Management Contracts as of December 31, 2001,
Awarded under the
NASA ODIN Master Contract Page 40 GAO- 02- 329 Desktop Outsourcing
Appendix I Information on Seat Management Contracts as of December 31, 2001,
Awarded under the
NASA ODIN Master Contract Page 41 GAO- 02- 329 Desktop Outsourcing
Page 42 GAO- 02- 329 Desktop Outsourcing
Appendix II
Appendix II Information on Seat Management Contracts as of December 31,
2001, Awarded under the
GSA Seat Management Master Contract Page 43 GAO- 02- 329 Desktop Outsourcing
Appendix II Information on Seat Management Contracts as of December 31,
2001, Awarded under the
GSA Seat Management Master Contract Page 44 GAO- 02- 329 Desktop Outsourcing
Page 45 GAO- 02- 329 Desktop Outsourcing
Appendix III
Appendix III Scope and Methodology
Page 46 GAO- 02- 329 Desktop Outsourcing
Page 47 GAO- 02- 329 Desktop Outsourcing
Appendix IV
Appendix IV Comments from the Department of the Treasury?s Departmental
Offices
Page 48 GAO- 02- 329 Desktop Outsourcing
Appendix IV Comments from the Department of the Treasury?s Departmental
Offices
Page 49 GAO- 02- 329 Desktop Outsourcing
Page 50 GAO- 02- 329 Desktop Outsourcing
Appendix V
Page 51 GAO- 02- 329 Desktop Outsourcing
Appendix VI
Appendix VI Comments from the Peace Corps
Page 52 GAO- 02- 329 Desktop Outsourcing
Appendix VI Comments from the Peace Corps
Page 53 GAO- 02- 329 Desktop Outsourcing
Page 54 GAO- 02- 329 Desktop Outsourcing
Appendix VII
Appendix VII Comments from the Department of Defense
Page 55 GAO- 02- 329 Desktop Outsourcing
Appendix VII Comments from the Department of Defense
Page 56 GAO- 02- 329 Desktop Outsourcing
Page 57 GAO- 02- 329 Desktop Outsourcing
Appendix VIII
Page 58 GAO- 02- 329 Desktop Outsourcing
Appendix IX
Appendix IX Comments from the National Aeronautics and Space Administration
Page 59 GAO- 02- 329 Desktop Outsourcing
Appendix IX Comments from the National Aeronautics and Space Administration
Page 60 GAO- 02- 329 Desktop Outsourcing
Appendix IX Comments from the National Aeronautics and Space Administration
Page 61 GAO- 02- 329 Desktop Outsourcing
Page 62 GAO- 02- 329 Desktop Outsourcing
Appendix X
United States General Accounting Office Washington, D. C. 20548- 0001
Official Business Penalty for Private Use $300
Address Correction Requested Presorted Standard
Postage & Fees Paid GAO Permit No. GI00
*** End of document. ***