Telecommunications: GSA Has Accumulated Adequate Funding for
Transition to New Contracts but Needs Cost Estimation Policy
(23-FEB-07, GAO-07-268).
The General Services Administration (GSA) and its customer
agencies are preparing to transition new governmentwide
telecommunications contracts known as the Networx program. GSA
estimated the costs for which it is responsible to be $151.5
million. This report addresses (1) the soundness of the analysis
GSA used to derive the estimate of funding that would be required
for the transition and (2) whether GSA will have accumulated
adequate funding to pay for transition costs. In performing this
work, GAO reviewed cost estimation best practices, analyzed
relevant GSA documents, and performed an uncertainty analysis on
GSA's estimate.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-07-268
ACCNO: A66186
TITLE: Telecommunications: GSA Has Accumulated Adequate Funding
for Transition to New Contracts but Needs Cost Estimation Policy
DATE: 02/23/2007
SUBJECT: Cost analysis
Documentation
Federal funds
Federal procurement
Funds management
Government contracts
Indefinite delivery contracts
Inflation
IT acquisitions
Program management
Service contracts
Strategic planning
Telecommunications
Telecommunications systems
Cost estimates
Indefinite quantity contracts
Program goals or objectives
FTS Networx
GSA Acquisition Services Fund
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-07-268
* [1]Results in Brief
* [2]Background
* [3]GSA's Analysis Was Not Sound
* [4]GSA's Analysis Was Not Sufficiently Accurate
* [5]GSA's Analysis May Not Be Comprehensive
* [6]GSA's Analysis Was Not Adequately Documented
* [7]GSA's Analysis Was Not Validated
* [8]Quantifiable Weaknesses Are Small, but Effect of Estimate's
* [9]GSA Does Not Have a Policy to Ensure That Cost Estimates Are
* [10]GSA Has Accumulated Adequate Funding for the Transition
* [11]Conclusions
* [12]Recommendations
* [13]Agency Comments and Our Evaluation
* [14]GAO Contact
* [15]Staff Acknowledgments
* [16]GAO's Mission
* [17]Obtaining Copies of GAO Reports and Testimony
* [18]Order by Mail or Phone
* [19]To Report Fraud, Waste, and Abuse in Federal Programs
* [20]Congressional Relations
* [21]Public Affairs
Report to the Ranking Member, Committee on Oversight and Government
Reform, House of Representatives
United States Government Accountability Office
GAO
February 2007
TELECOMMUNICATIONS
GSA Has Accumulated Adequate Funding for Transition to New Contracts but
Needs Cost Estimation Policy
GAO-07-268
Contents
Letter 1
Results in Brief 2
Background 3
GSA's Analysis Was Not Sound 7
GSA Has Accumulated Adequate Funding for the Transition 13
Conclusions 14
Recommendations 15
Agency Comments and Our Evaluation 16
Appendix I Objectives, Scope, and Methodology 18
Appendix II Uncertainty Analysis for GSA's Transition Estimate 20
Appendix III Comments from the GSA Administrator 21
Appendix IV GAO Contact and Staff Acknowledgments 23
Tables
Table 1: GSA Transition Cost Estimate 6
Table 2: Cumulative Effect of Quantifiable Weaknesses 11
Table 3: Risk Adjusted Factors 20
Figures
Figure 1: Estimated Cost of Transition Using Various Percents of
Transition Traffic 12
Figure 2: Confidence That Certain Amounts of Funding Will Account for
Actual Costs 13
Abbreviations
GSA General Services Administration
IMC Interagency Management Council
IT Information Technology
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
United States Government Accountability Office
Washington, DC 20548
February 23, 2007
The Honorable Tom Davis
Ranking Member
Committee on Oversight and Government Reform
House of Representatives
Dear Mr. Davis:
As innovations in telecommunications services continue to transform the
way the federal government conducts business, the General Services
Administration's (GSA) governmentwide telecommunications acquisition
programs offer federal agencies the opportunity to apply innovative
services and solutions to their operations. With the current set of
governmentwide telecommunications contracts approaching expiration, GSA
and its customer agencies will have to transition the services acquired
under these contracts to their replacements, known collectively as
Networx.
GSA will incur program management costs associated with planning and
executing this transition. It has also made a commitment to absorb certain
agency transition costs. To ensure it would have the funds necessary to
pay for these costs, GSA estimated that it would need to set aside
approximately $151.5 million.
This report responds to your request that we determine (1) the soundness
of the analysis GSA used to derive the estimate of funding that would be
required for the transition and (2) whether GSA will have accumulated
adequate funding to pay for its transition management costs. To accomplish
the first objective, we conducted a search of over 250 documents from both
government and industry1 for examples of best practices in the field of
cost estimation and identified common characteristics among them. We
determined that high-quality, reliable estimates should be accurate,
comprehensive, well-documented, and validated. Using these
characteristics, we analyzed transition estimate documentation developed
by GSA, documentation provided by GSA on the previous transition, the
Networx Request for Proposals, and other relevant documents. To accomplish
the second objective, we analyzed financial and operational documents from
related GSA programs. In addition, we interviewed GSA program officials
about both objectives and conducted an analysis of GSA's estimate to
examine the effects of varying the main cost driver in the estimate. We
conducted our work at GSA's Washington, D.C., area headquarters between
June 2006 and January 2007 in accordance with generally accepted
government auditing standards. A detailed discussion of our objectives,
scope, and methodology can be found in appendix I.
1These documents included published literature on cost estimation from the
Society of Cost Estimating and Analysis, the Department of the U.S. Army,
the Office of the Secretary of Defense, the National Aeronautics and Space
Administration, and the Department of Energy.
Results in Brief
GSA did not use sound analysis when estimating the amount of funding
needed to meet its transition-related commitments. Specifically, its
analysis was not sufficiently accurate, comprehensive, documented, or
validated. A primary weakness is that the estimate is largely based on the
assumption that 76 percent of the services provided under the current
contracts will transition to a different provider under the Networx
contracts. However, according to program officials, this assumption is
intentionally conservative and represents a worst-case scenario that is
unlikely to occur. Additionally, GSA may have double-counted a cost, did
not update its analysis to reflect a nearly 2-year delay, and did not
adequately document or validate its analysis. The weaknesses we identified
can be attributed in part to the lack of a policy requiring cost estimates
to be developed using best practices. Without such a policy, GSA's future
cost estimates could exhibit similar weaknesses, increasing the risk that
it will retain excess funds that could be reallocated for other purposes.
GSA has accumulated adequate funding to support its anticipated transition
costs. As of fiscal year-end 2006, GSA had approximately $142 million in a
transition reserve to be used to pay for its costs associated with the
transition. Our analysis of the estimate indicates that it is unlikely
that GSA will need to accumulate the entire $151.5 million it estimated
and that the funds it has already accumulated should be sufficient to fund
the transition. Specifically, the $142 million already retained will be
adequate to cover anticipated costs 96 percent of the time. Further, the
recent merger of two GSA funds increases the agency's flexibility and
could provide additional money, if needed. With Networx contracts
scheduled to be awarded starting in March 2007, GSA will soon have the
information necessary to reassess the main cost driver underlying its
estimate and address the weaknesses we identified. Once it has a current,
accurate estimate, GSA can reevaluate the funding needed to meet
anticipated commitments.
To ensure that future cost estimates by GSA are sound and can be used as a
reliable basis for decisions, we recommend that the GSA Administrator
establish an agencywide policy requiring that cost estimates be developed
using best practices. In addition, we recommend that the Administrator
revise the transition cost estimate using best practices after the award
of contracts under the Networx program and, if feasible, reallocate any
excess funds for other purposes or return them to the Treasury.
In written comments on a draft of this report, the GSA Administrator
concurred with our recommendations and emphasized the importance of
supporting a successful governmentwide telecommunications transition. Her
comments also state that GSA's transition cost estimate was within 2
percent of our independent analysis using the same assumptions. However,
while we identified only $3 million in quantifiable errors (roughly 2
percent of GSA's total estimate), we also determined that if the extent of
transitioning services is significantly lower than GSA's intentionally
conservative assumption, actual costs could be more than $110 million less
than GSA's estimate.
In addition, the Administrator presented two main objections to our draft.
First, she questioned whether our reported findings were balanced given
the facts and results presented. We clarified our report to ensure that
our findings better reflect the information we discuss. Second, she stated
that we incorrectly suggested comparability between the pending Networx
transition and the prior transition. We maintain that the two transitions
are comparable, particularly because GSA's analysis is based, in part, on
the results of the previous transition. Appendix III provides the full
text of GSA's comments. GSA also provided technical comments that have
been incorporated in our report as appropriate.
Background
As part of its mission of providing federal agencies with acquisition
services and solutions at best value, GSA's technology programs offer
agencies options to acquire needed telecommunications services. An option
chosen by more than 135 agencies is the FTS2001 program, which consists of
two large governmentwide telecommunications contracts--one awarded to
Sprint2 in December 1998 and the other to MCI3 in January 1999--and
FTS2001 crossover contracts.4
GSA is planning to replace the FTS2001 contracts, FTS2001 crossover
contracts, and separate wireless contracts with a new set of contracts.
Collectively known as the Networx program, these new contracts are to
provide governmentwide telecommunications services through two
indefinite-delivery/indefinite-quantity acquisitions--Networx Universal
and Networx Enterprise. The Universal acquisition is expected to satisfy
the requirements for a full range of national and international network
services and, according to GSA, to ensure the continuity of broad-ranging
services with global geographic coverage rendered under expiring
contracts. The Enterprise acquisition is expected to offer agencies
leading-edge services and solutions with less extensive geographic and
service requirements than Universal. The services required in these
contracts focus on Internet-based offerings and related security and
management services.
GSA expects the transition to begin when Networx Universal awards are made
in March 2007 and to continue until fiscal year 2010. Because the FTS2001
contracts with Sprint Nextel Corporation and Verizon Business expired in
December 2006 and January 2007, respectively, GSA and the incumbent
vendors negotiated separate sole-source contracts that essentially extend
the terms of the FTS2001 contracts for 42 months.5 These sole-source
contracts were awarded to ensure uninterrupted service and allow agencies
adequate time to complete the transition to Networx.
GSA is working with representatives of federal agencies to prepare for the
upcoming transition to Networx, both directly and through the Interagency
Management Council (IMC), a group of senior federal information resource
officials who advise GSA on issues related to telecommunications
contracts. The IMC has worked with GSA to document lessons learned from
the transition to FTS2001 that began in the late 1990s, GSA's most recent
governmentwide telecommunications transition. One important lesson learned
was that GSA and agency plans for funding transition expenses should be
determined early to allow agencies to gauge the impact of transition
expenses on their budgets. Specifically, the lessons-learned document
recommended that guidelines be established to allow agencies to complete
the financial planning required to ensure that the resources needed for
transition were available. This led the IMC and GSA to develop a Taxonomy
and Allocation of Transition Costs document that identified which Networx
transition costs would be borne by GSA and which would be borne by
transitioning agencies.
2Sprint Corporation merged with Nextel Communications, Inc., to form
Sprint Nextel Corporation in August 2005.
3MCI merged with Verizon to form Verizon Business in January 2006.
4In August 2001, GSA allowed contractors that had been awarded local
telecommunications contracts in selected metropolitan areas, through GSA's
Metropolitan Area Acquisition program, to offer long-distance services on
the FTS2001 contracts. This process is termed "crossover."
5GSA negotiated sole-source contracts for a 24-month base period with
three 6-month optional periods, for a total of 42 months.
This taxonomy document indicated that GSA will incur or reimburse
agencies, including:
o GSA contractor support costs to, for example, aid in planning
for the transition and oversight of Networx contractors. GSA
officials also indicated that contractor support will be used to
develop a methodology for tracking transition progress and the
establishment of a transition coordination center; and
o certain costs incurred by agencies during transition.
In 2004, following the development of the taxonomy document, GSA
generated an estimate of its costs for the transition. GSA's
methodology for its estimate was to:
o develop assumptions for the estimate,
o define and develop a baseline for the estimate based on
experiences and lessons learned from the previous transition,
o determine the network growth as well as the total business
volume projections for fiscal year 2006 based on historical
traffic and cost trends,
o define the transition cost elements,
o define and develop a formula for calculating an estimate for
each cost element, and
o design and develop the estimated transition cost model for
sensitivity analysis.
Using this methodology, GSA estimated that it would need a total
of $151.5 million for a 30-month transition, most of which would
be used to reimburse agencies' transition costs. Table 1 below
details the transition costs GSA expects to pay.
Table 1: GSA Transition Cost Estimate
Source: GSA.
GSA planned to pay for its costs and the reimbursement of certain
agency costs using its Information Technology (IT) Fund.6 The IT
Fund was a full-cost recovery revolving fund7 whereby GSA fully
recovered all costs of its technology programs and its operations
via estimated fee rates. The fees, which are charged to agencies
for the use of GSA contracts cover the direct costs of its
operations--such as the development and management of contract
vehicles--and indirect costs associated with its headquarters,
such as support for the Offices of the Chief Information Officer
and the Chief Financial Officer. The IT Fund allowed GSA to
stabilize rates for its services when expenses varied and was used
to provide funding for the previous transition to FTS2001. The IT
Fund also contained a working capital reserve that was used to
offset losses due to fluctuations in business volumes and other
unexpected contingencies. At each fiscal year-end, the uncommitted
balance of funds remaining was to be transferred to the general
fund of the treasury as miscellaneous receipts.
Recently, a new law changed the structure of the IT Fund and GSA's
organization. On October 6, 2006, the General Services
Administration Modernization Act8 combined the IT Fund with
another GSA revolving fund--the General Supply Fund--to create the
Acquisition Services Fund. This legislation, which also merged
GSA's technology and supply programs,9 made all capital assets and
balances remaining in the IT and General Supply Funds available
for the purposes of the Acquisition Services Fund. According to
GSA, the Federal Acquisition Service will increase agency savings,
enhance GSA's capability to meet customer requirements for
excellence, and improve internal efficiencies.
GSA�s Analysis Was Not Sound
The analysis GSA used to derive its estimate of $151.5 million was
not sound because it was not sufficiently accurate, comprehensive,
documented, or validated. The analysis was not sufficiently
accurate because it is largely based on the assumption that
agencies will transition 76 percent of the services acquired under
the current FTS2001 contracts to a different provider under
Networx--an intentionally conservative scenario that GSA program
officials believe is unlikely to occur. Further, the analysis has
not been updated after a nearly 2-year delay in the contract
award. While GSA appears to have included all pertinent costs, it
may have double-counted a cost, calling into question the
comprehensiveness of its analysis. In addition, GSA did not
document significant assumptions and data. Finally, GSA's analysis
was not validated by an independent cost estimate nor was an
uncertainty analysis performed that would allow GSA to quantify
the level of confidence it has in its estimate. These weaknesses
can be attributed in part to the lack of a cost estimation policy
that would help to ensure that such estimates are developed using
best practices. While an intentionally conservative approach
minimizes the risk that GSA would have inadequate funds to pay for
committed transition costs, it increases the risk that GSA will
retain excess funds that could be used for other purposes.
GSA�s Analysis Was Not Sufficiently Accurate
Estimates are accurate when they are not overly conservative,
based on an assessment of the most likely costs, and adjusted
properly for inflation. Best practices further dictate that as
schedules change, cost estimates should be revised to provide
management with insight into the current program status, effective
control of the program, and the ability to balance resources and
the budget.
The analysis GSA used to derive the estimate, however, was not
sufficiently accurate. First, the estimate's main cost driver is
based on the assumption that agencies will transition 76 percent
of the services acquired under the current FTS2001 contracts to a
different provider under Networx--60 percent due to agencies being
forced to change providers and 16 percent due to voluntary
changes. However, according to program officials, the 76 percent
"transition traffic factor" is intentionally conservative and
represents a worst-case scenario that is unlikely to occur.
The 76 percent transition traffic factor is also overly
conservative when compared with the previous transition. Then,
approximately 60 percent of services were shifted from an
incumbent to a different provider under FTS2001, the bulk being
forced to change providers when an incumbent providing more than
half of the services (AT&T) was not awarded an FTS2001 contract.
This forced shift happened in part because GSA limited the FTS2001
awards to only two vendors. In contrast, GSA has placed no such
limit on the number of Networx vendors, and therefore, all
incumbent FTS2001 vendors could potentially be awarded a Networx
contract. Further, because the most-used FTS2001 incumbent
(Verizon Business) provides approximately 50 percent of the
services under the current FTS2001 contracts, the assumption of a
60 percent forced shift would only be realized if no awards were
made to Verizon Business and at least one other incumbent.
Finally, GSA's assumption of a 16 percent voluntary shift is
significantly higher than the 3 percent that voluntarily changed
providers during the previous transition.
Program officials could not identify any basis for the assumption
that voluntary changes will reach 16 percent. However, the
officials stated that the percentage of voluntary changes could be
higher because agencies will likely have more options when
selecting vendors and because of improved guidance on federal
requirements regarding the fair opportunity process, which is
intended to give each awardee an equal opportunity to compete for
agencies' telecommunications services requirements based on
agency-established selection criteria.
Second, the analysis has not been updated to reflect schedule
changes. When the estimate was developed in 2004, GSA expected to
award the first Networx Universal contracts in mid-2005. Since
that time, delays have pushed the time frame back almost 2 years.
Universal awards are now expected in March 2007. GSA's analysis
has not been adjusted to reflect additional inflation during this
period, which could increase the estimates total by as much as $9
million.
Finally, GSA has not revised its analysis to reflect an assessment
of most likely costs using currently available information. The
analysis assumes that reimbursable agency transition costs would
be greater than during the previous transition due to growth in
the volume of services ordered. While GSA estimated that service
levels would grow 60 percent over the previous transition by the
time of contract award, as of August 2006, service levels had
actually grown by 55.9 percent and are now expected to remain
stable. The overestimation of service growth added approximately
$1.7 million to the estimate.
GSA�s Analysis May Not Be Comprehensive
Estimates are comprehensive when their level of detail ensures
that all pertinent costs are included and no costs are
double-counted. It is important to ensure the completeness,
consistency, and realism of the information contained in the cost
estimate.
GSA appears to have included all pertinent costs in its analysis;
however, according to officials, a specific agency cost valued at
$4.3 million may have been double-counted. For several of the
transition costs estimated, GSA based its calculations on the
actual charges incurred during the previous transition. However,
officials indicated that a specific agency cost that is estimated
separately in its current estimate may have already been included
in a more general category of its previous costs. Despite this
uncertainty, GSA calculated the separate total for this specific
agency cost based on current service levels and added it to its
estimate. As a result, it is likely this specific agency cost is
double-counted and therefore overstated.
GSA�s Analysis Was Not Adequately Documented
Cost estimates are well-documented when they can be easily
repeated or updated and can be traced to original sources through
auditing. Rigorous documentation increases the credibility of an
estimate and helps support an organization's decision-making
process. The documentation should explicitly identify the primary
methods, calculations, results, rationales or assumptions, and
sources of the data used to generate each cost element.
GSA provided us documentation of its methodology, the calculations
it used to derive each cost element, results, and many of the
previous transition costs.10 However, it did not document
significant assumptions. Specifically, GSA did not document the
rationale behind its 76 percent transition traffic factor or why
it used a 30-month time period for the transition--two key
assumptions of its analysis.
GSA also did not provide documentation of certain data sources.
Specifically, program officials could not provide supporting data
used to estimate an agency transition cost valued at $4.7 million.
In addition, GSA could not document the data sources used to
estimate costs for contractor support in planning and implementing
the transition. While many costs in its estimate are based on the
charges incurred during the previous transition, GSA officials
stated that it was not appropriate to use previous costs as a
basis for the contractor cost element. These officials explained
that unlike the previous transition, GSA would not provide
agencies with on-site contractor support. Officials made this
decision, in part, because the 2 1/2 years of transition planning
that has occurred to date is expected to result in better
preparation by agencies and the ability for them to facilitate
their transitions without direct assistance from GSA or its
contractors. Instead of basing their projection of contractor
costs on prior charges, program officials told us that GSA
management decided that contractor support costs should not exceed
$35 million. Program officials could not provide any data or
analysis to support this decision.
GSA�s Analysis Was Not Validated
Estimates are adequately validated when they have been
cross-checked with an independent cost estimate, and when a level
of uncertainty associated with the estimate is identified. An
independent cost estimate provides the estimator with an unbiased
test of the reasonableness of the estimate and reduces the cost
risk associated with the project by demonstrating that alternate
methods generate similar results. In performing an uncertainty
analysis, an entity examines the effects of varying multiple
elements and, as a result, is able to express a level of
confidence in its estimate.
GSA did not validate its analysis against an independent cost
estimate or perform an uncertainty analysis. GSA program officials
could not provide a rationale of why these activities were not
performed.
Quantifiable Weaknesses Are Small, but Effect of Estimate�s
Intentionally Conservative Assumption Could Be Significant
The cumulative effect of the quantifiable weaknesses we identified
in GSA's analysis is relatively small, resulting in an
underestimation of $3 million, or roughly 2 percent of the total
$151.5 million estimate (as shown in table 2). The underestimation
of costs, related to inflation during the extended delay in making
award, was offset by the overestimation of service growth and the
possible double-counting of a specific agency transition cost.
Table 2: Cumulative Effect of Quantifiable Weaknesses
Source: GAO analysis of GSA information.
In contrast, if the actual level of services transitioning to a
different vendor is lower than the 76 percent transition traffic
factor assumed by GSA, the effect will be greater. Because this
factor is the primary cost driver in the estimate, significant
changes in transitioning traffic result in similarly significant
changes in total costs. This effect can be illustrated using GSA's
transition cost estimate model (see fig. 1). For example, if a
transition traffic factor of 66 percent is used (GSA's assumption
of 16 percent for a voluntary transition plus a forced shift of 50
percent that would occur if Verizon Business is not awarded a
Networx contract), the total estimated cost of the transition
falls to about $136 million. Similarly, if all incumbents are
awarded Networx contracts and only GSA's assumption of a 16
percent voluntary transition occurs, the total estimated cost of
the transition is reduced to approximately $40 million. GSA
officials explained that they used a risk-averse transition
traffic factor to minimize the possibility of underestimating
costs.
Figure 1: Estimated Cost of Transition Using Various Percents of
Transition Traffic
GSA Does Not Have a Policy to Ensure That Cost Estimates Are Sound
The weaknesses in GSA's analysis can be attributed in part to the
lack of a policy requiring cost estimates to be developed using
best practices. Officials from the Offices of the Chief
Acquisition Officer, the Chief Information Officer, and the
program office's Controller confirmed that GSA does not have
centralized policy or guidance on cost estimation. Instead, the
officials that prepared the estimate stated that they based their
work loosely on best practices learned as a result of past
experiences and were comfortable with the estimate. Officials
believe there is no reason to revise their estimate to address the
issues we raised in this report because their purpose in producing
it was to minimize the risk of underestimating costs. While GSA's
approach minimized the risk of having inadequate funds to fulfill
its commitments, it increased the risk that GSA will retain excess
funds that could be used for other purposes.
GSA Has Accumulated Adequate Funding for the Transition
GSA has accumulated adequate funding to support its anticipated
commitments related to the Networx transition. As of fiscal
year-end 2006, GSA accumulated approximately $142 million in a
reserve dedicated to costs associated with the transition from
FTS2001 to Networx. The uncertainty analysis we performed on GSA's
cost estimate indicates that the funding GSA has already
accumulated will most likely be adequate to pay for expected
transition costs. By varying the transition traffic factor, our
analysis indicates that the $142 million already retained should
be adequate to cover expected expenses 96 percent of the time.
Figure 2 illustrates the probability that a particular level of
funding will be adequate to account for the total actual costs
incurred. For example, the $151.5 million estimate represents a
confidence level of 100 percent: there is almost no chance that
costs will exceed the estimate. The full detail of our methodology
is detailed in appendix II.
Figure 2: Confidence That Certain Amounts of Funding Will Account
for Actual Costs
The merger of the IT and General Supply Funds provides GSA with
additional flexibility, further reducing the need to retain the
entire amount of the estimate. As discussed, legislation11
combined the two funds into an Acquisition Services Fund, making
their capital assets and balances available for the purposes of
the new Federal Acquisition Service. The legislation also allows
the Federal Acquisition Service to establish a reserve to retain
surplus revenues from GSA's technology and service programs
specifically for the purpose of offsetting losses and other
unexpected contingencies. Further, at the end of each year, the
statute requires GSA to return to the Treasury any funds not
expended or held in a working capital reserve. During fiscal year
2006, GSA's technology programs experienced losses, but its supply
programs reported overall positive earnings of approximately $126
million. If operations continue in this fashion, excess revenue
will be available in the combined fund to offset losses or account
for contingencies, such as the Networx transition, be retained
within the Acquisition Services Fund, or be returned to the
Treasury.
With Networx contracts scheduled to be awarded starting in March
2007, GSA will soon be in a position to reassess its main
assumption, the transition traffic factor, and the resulting level
of funding needed to meet anticipated commitments. Unless GSA
revises its estimate, it risks unnecessarily retaining funds that
could be reallocated to other agency priorities or returned to the
Treasury.
Conclusions
While GSA achieved its goal of minimizing the risk that it would
have inadequate funds to pay its transition commitments, the
analysis used to develop the transition cost estimate was not
sound because it was not sufficiently accurate, comprehensive,
documented, or validated. The weaknesses can be attributed in part
to a lack of a policy at GSA to ensure that such estimates are
developed using best practices and make the best use of agency
resources. While the quantifiable effect of the weaknesses in
accuracy and comprehensiveness is small, the effect of potential
inaccuracies resulting from the intentional use of an overly
conservative assumption about a main cost driver could be more
significant. Without the use of a cost estimation policy that
reflects best practices, GSA could continue to produce similarly
unsound estimates, increasing the risk that it will unnecessarily
retain funds that could be reallocated for other purposes.
Despite the weaknesses in its analysis, GSA has accumulated
adequate funding to support its anticipated commitments related to
the Networx transition. Our analysis indicates that it is highly
unlikely that GSA will need more than the $142 million it has
already accumulated. In addition, the merger of two revolving
funds gives it increased flexibility in meeting costs. Once
Networx contracts are awarded beginning in March 2007, GSA will be
able to forecast the number of forced transitions more accurately,
and, if necessary, reduce the amount of funding it plans to
accumulate in the future or free already accumulated funds for
other purposes. This reassessment will also be an opportunity for
GSA to address the other weaknesses in its analysis that resulted
from its deviation from best practices.
Recommendations
To improve GSA's program management, we are making two
recommendations. First, to ensure that future cost estimates are
sound and can be used as a reliable basis for decisions, we
recommend that the GSA Administrator establish a policy for cost
estimation efforts at GSA. Specifically, this policy should
reflect best practices by requiring that estimates are:
o accurate (not overly conservative, based on an assessment of the
most likely costs, and adjusted properly for inflation);
o comprehensive (their level of detail ensures that all pertinent
costs are included and no costs are double-counted);
o well-documented (can be easily repeated or updated and can be
traced to original sources through auditing); and
o validated (they have been cross-checked with an independent cost
estimate and a level of uncertainty associated with the estimate
has been identified.)
Second, to ensure the most efficient use of federal funds, we
recommend that the Administrator revise the transition cost
estimate following the award of contracts under the Networx
program. Specifically, this revision should reflect best
practices, include a more precise transition traffic factor, and
address the overestimation of service growth, the possible
double-counting of a nonrecurring charge, and the effects of
inflation during the extended delay in making awards. If the
results of this new estimate indicate that the full $151.5 million
is not needed to reasonably support the transition effort, GSA
should reallocate any excess funds for other purposes allowable
within the Acquisition Services Fund or return them to the
Treasury.
Agency Comments and Our Evaluation
In written comments on a draft of this report, the GSA
Administrator concurred with our recommendations and emphasized
the importance of supporting a successful governmentwide
telecommunications transition. To address our recommendations, she
stated that GSA will issue improved policy guidance on cost
estimating and review and adjust the cost estimate as additional
information becomes available.
The Administrator also commented that GSA's transition cost
estimate was within 2 percent of our analysis using the same
assumptions. However, while we identified only $3 million in
quantifiable errors (roughly 2 percent of GSA's total estimate),
our report also states that if the extent of transitioning
services is significantly lower than GSA's intentionally
conservative assumption, the actual costs of the transition could
be considerably less than GSA's estimate. Specifically, if all
incumbents are awarded Networx contracts and only GSA's assumption
of a 16 percent voluntary transition occurs, the total cost of the
transition could be reduced to approximately $40 million--over
$110 million less than GSA's estimate.
In addition, the Administrator stated that GSA does not concur
with the entirety of the draft report. She presented two main
objections. First, she questioned whether our reported findings
were balanced given the facts and results presented. We clarified
our report to ensure that our findings better reflect the
information we discussed. Second, she raised a concern that we
incorrectly suggested comparability between the pending Networx
transition and the prior transition. We maintain that the two
transitions are comparable, particularly because GSA's analysis is
based, in part, on the results of the previous transition.
Specifically, GSA's analysis relied on experiences and lessons
learned from the previous transition to establish costs for the
current estimate.
1This included documentation available on a Transition Manager Web site
established by GSA, such as presentations, meeting minutes, and FTS2001
lessons learned.
2A Monte Carlo simulation allows the model's parameters to vary according
to their associated probability distribution. The result is a set of
estimated probabilities of achieving alternative outcomes, given the
uncertainty in the underlying parameters.
Appendix III provides the full text of GSA's comments. GSA also
provided technical comments that have been incorporated in this
report, as appropriate.
6The IT Fund was authorized by the Paperwork Reduction Reauthorization Act
of 1986, Public Law 99-500 and 99-691; 40 U.S.C. 322.
7A revolving fund is established by Congress to finance a cycle of
businesslike operations through amounts received by the fund. A revolving
fund charges for the sale of products or services and uses the proceeds to
finance its spending, usually on a self-sustaining basis. Instead of
recording the collections in receipt accounts, the budget records the
collections and the outlays of revolving funds in the same account. A
revolving fund is a form of permanent appropriation.
8General Services Administration Modernization Act, Pub. L. No. 109-313,
120 Stat. 1734 (2006).
9GSA's supply programs provide agencies a source for commercial products
and services such as office supplies, vehicle purchasing, travel, and
furniture.
10For example, GSA provided data on contractor support costs incurred
during the previous transition.
11Public Law No. 109-313.
As agreed with your office, unless you publicly announce the
contents of this report earlier, we plan no further distribution
until 30 days from the report date. At that time, we will send
copies of this report to the GSA Administrator and interested
congressional committees. We will also make copies available to
others upon request. In addition, the report will be available at
no charge on the GAO Web site at http://www.gao.gov .
Should you or your offices have any questions about matters
discussed in this report, please contact me at (202) 512-6240 or
by e-mail at [email protected]. Contact points for our Offices of
Congressional Relations and Public Affairs may be found on the
last page of this report. GAO staff who made major contributions
to this report are listed in appendix IV.
Sincerely,
Linda D. Koontz
Director, Information Management Issues
Appendix I: Objectives, Scope, and Methodology
Our objectives were to determine (1) the soundness of the analysis
the General Services Administration (GSA) used to derive the
estimate of funding that would be required for the transition and
(2) whether GSA will have accumulated adequate funding to pay for
its transition management costs.
To determine the soundness of GSA's analysis, we conducted an
intensive search of over 250 source documents of both government
and industry literature for examples of best practices in the
field of cost estimation. This included literature on cost
estimation from the Society of Cost Estimating and Analysis, the
Department of the U.S. Army, the Office of the Secretary of
Defense, the National Aeronautics and Space Administration, and
the Department of Energy. This indicated that high quality,
reliable cost estimates are:
o accurate (not overly conservative, based on an assessment of the
most likely costs, and adjusted properly for inflation);
o comprehensive (their level of detail ensures that all pertinent
costs are included and no costs are double-counted);
o well-documented (can be easily repeated or updated and can be
traced to original sources through auditing); and
o validated (they have been cross-checked with an independent cost
estimate and a level of uncertainty associated with the estimate
has been identified).
To determine the extent to which GSA followed these practices, we
analyzed documentation supporting the transition estimate,
documentation provided by GSA on the previous transition estimate,
and the Networx Request for Proposals. We also interviewed GSA
Networx program managers and attended a GSA sponsored transition
conference and meetings of the Interagency Management Council
Transition Working Group.1
To address whether GSA has accumulated adequate funding to pay for
its transition management costs, we obtained and analyzed Cost and
Capital Requirements Plans for the Information Technology Fund
submitted to the Office of Management and Budget and legislation
for GSA's Information Technology, General Supply, and Acquisition
Services Funds. In addition, to determine the extent of funding
held in GSA's related accounts, we analyzed financial statements
for GSA's technology and service programs. To verify the
reliability of these records, we obtained and analyzed the results
of the most recent GSA financial audits and audit reports from
GSA's Inspector General, and we interviewed GSA's independent
financial auditor regarding the quality control procedures in
place. The independent auditor did not specifically review the
dollar amounts in the Information Technology or General Supply
Funds for accuracy but did test the controls in place for
compliance with laws and regulations. This auditor stated that
there were no reportable findings associated with either fund, and
it was reasonable to assume that these accounts were fairly
stated. As a result, we determined the data were sufficiently
reliable for the purposes of this report. We also interviewed
officials from the Office of Management and Budget and officials
from GSA's technology and service programs, Office of the Chief
Acquisition Officer, Office of the Chief Financial Officer, and
Office of the Chief Information Officer.
To assess the adequacy of the level of funding already accumulated
by GSA, we performed an uncertainty analysis for GSA's estimate
using a Monte Carlo simulation.2 A Monte Carlo simulation provides
a perspective on the potential variability of the cost estimate
should the facts, circumstances, or assumptions change. We chose
to vary only the transition traffic factor because it is the main
driver of the costs in GSA's estimate. To carry out this
simulation, we identified a minimum, maximum, and median value for
the transition traffic factor based on information received from
GSA.
We conducted our work between May 2006 and January 2007 in
accordance with generally accepted government auditing standards.
Appendix II: Uncertainty Analysis for GSA�s Transition Estimate
An uncertainty analysis provides decision makers with a
perspective on the potential variability of the estimate should
the facts, circumstances, and assumptions change. By examining the
effects of varying the estimates elements, a degree of uncertainty
about the estimate can be expressed, possibly as an estimated
range or qualified by some factor of confidence. For example, an
estimate that produces a 100 percent confidence level indicates
that, based on the methodology used to create that estimate, there
is almost no chance that costs will exceed the estimate.
We performed our uncertainty analysis on GSA's estimate using a
Monte Carlo simulation.1 We chose to vary only the transition
traffic factor because it is the main driver of the costs in GSA's
estimate. To carry out this analysis, we identified a minimum,
maximum, and median value for the transition traffic factor, based
on information received from GSA.
The transition traffic factor represents the possible percentage
of services under FTS2001 that may transition to a different
provider under the Networx contracts. For this factor, we chose a
minimum of 3 percent, which represents the voluntary shift to a
different vendor that occurred during GSA's previous transition to
FTS2001. For the maximum, we used 76 percent, as this value was
chosen by GSA officials as a worst-case scenario in an effort to
mitigate the risks of underestimating costs. The median of these
two numbers is 39.5 percent. Table 3 shows the variations of each
factor used in our uncertainty analysis.
Table 3: Risk Adjusted Factors
Source: GAO analysis of GSA data.
1See appendix I, footnote 2, for a description of Monte Carlo simulation.
Appendix III: Comments from the GSA Administrator
Appendix IV: GAO Contact and Staff Acknowledgments
GAO Contact
Linda D. Koontz (202) 512-6240 or [email protected]
Staff Acknowledgments
In addition to the contact named above, James R. Sweetman, Jr.,
Assistant Director; Jamey A. Collins; Neil J. Doherty; Jennifer K.
Echard; Wilfred B. Holloway; Ethan J. Iczkovitz; Frank Maguire;
Karen A. Richey; Glenn D. Slocum; and Amos A. Tevelow made key
contributions to this report.
GAO�s Mission
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in
meeting its constitutional responsibilities and to help improve
the performance and accountability of the federal government for
the American people. GAO examines the use of public funds;
evaluates federal programs and policies; and provides analyses,
recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony
The fastest and easiest way to obtain copies of GAO documents at
no cost is through GAO's Web site ( www.gao.gov ). Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of
newly posted products every afternoon, go to www.gao.gov and
select "Subscribe to Updates."
Order by Mail or Phone
The first copy of each printed report is free. Additional copies
are $2 each. A check or money order should be made out to the
Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are
discounted 25 percent. Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM
Washington, D.C. 20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax:
(202) 512-6061
To Report Fraud, Waste, and Abuse in Federal Programs
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail:
[email protected] Automated answering system: (800) 424-5454 or
(202) 512-7470
Congressional Relations
Gloria Jarmon, Managing Director, [email protected] (202)
512-4400 U.S. Government Accountability Office, 441 G Street NW,
Room 7125 Washington, D.C. 20548
Public Affairs
Paul Anderson, Managing Director, [email protected] (202)
512-4800 U.S. Government Accountability Office, 441 G Street NW,
Room 7149 Washington, D.C. 20548
(310761)
www.gao.gov/cgi-bin/getrpt?GAO-07-268 .
To view the full product, including the scope
and methodology, click on the link above.
For more information, contact Linda Koontz at (202) 512-6240 or
[email protected].
Highlights of [30]GAO-07-268 , a report to the Ranking Member, Committee
on Oversight and Government Reform, House of Representatives
February 2007
TELECOMMUNICATIONS
GSA Has Accumulated Adequate Funding for Transition to New Contracts but
Needs Cost Estimation Policy
The General Services Administration (GSA) and its customer agencies are
preparing to transition new governmentwide telecommunications contracts
known as the Networx program. GSA estimated the costs for which it is
responsible to be $151.5 million. This report addresses (1) the soundness
of the analysis GSA used to derive the estimate of funding that would be
required for the transition and (2) whether GSA will have accumulated
adequate funding to pay for transition costs. In performing this work, GAO
reviewed cost estimation best practices, analyzed relevant GSA documents,
and performed an uncertainty analysis on GSA's estimate.
[31]What GAO Recommends
GAO recommends that the GSA Administrator establish a cost estimation
policy that reflects best practices. In addition, GAO recommends that the
Administrator revise the transition cost estimate using best practices
after the award of the Networx contracts and reassess the funding needed
to meet its commitments. GSA concurred with these recommendations but
questioned the way GAO characterized the soundness of GSA's analysis and
whether this transition was comparable with the previous one. GAO
clarified its characterization of GSA's analysis.
GSA did not use sound analysis when estimating the amount of funding
needed to meet its transition-related commitments. Specifically, its
analysis was not sufficiently accurate, comprehensive, documented, or
validated. A primary weakness is that the estimate is largely based an
assumption--known as the transition traffic factor--that 76 percent of the
services provided under the current contracts would be moved to a
different provider under the Networx contracts. However, according to
program officials, this assumption is intentionally conservative and
represents a worst-case scenario that is unlikely to occur. Additionally,
GSA may have double-counted a cost and did not update its analysis to
reflect a nearly 2-year delay. Finally, GSA did not document significant
assumptions and data sources used in its analysis, or validate it. These
weaknesses can be attributed in part to the lack of a cost estimation
policy that reflects best practices. While GSA's intentionally
conservative approach minimizes the risk that it would have inadequate
funds to pay for committed transition costs, it increases the risk that
GSA will retain excess funds that could be used for other purposes.
GSA has accumulated adequate funding to support its anticipated transition
costs. As of fiscal year-end 2006, GSA had approximately $142 million in a
transition reserve. GAO analysis of the estimate indicates it is unlikely
that GSA will need more than it has already accumulated to fund the
transition. Specifically, the $142 million already retained will be
adequate to cover anticipated costs 96 percent of the time. The recent
merger of two GSA funds gives the agency additional flexibility that
reduces its need to accumulate the entire $151.5 million it estimated
would be needed (see table). With Networx contracts scheduled to be
awarded starting in March 2007, GSA will soon have the information
necessary to reassess the main assumption underlying its estimate--the
transition traffic factor--and address the weaknesses GAO identified. Once
this has been accomplished, GSA can reevaluate the funding needed to meet
anticipated commitments.
GSA's Transition Cost Estimate
Source: GSA.
References
Visible links
30. http://www.gao.gov/cgi-bin/getrpt?GAO-07-268
*** End of document. ***