Telecommunications: GSA Has Accumulated Adequate Funding for	 
Transition to New Contracts but Needs Cost Estimation Policy	 
(23-FEB-07, GAO-07-268).					 
                                                                 
The General Services Administration (GSA) and its customer	 
agencies are preparing to transition new governmentwide 	 
telecommunications contracts known as the Networx program. GSA	 
estimated the costs for which it is responsible to be $151.5	 
million. This report addresses (1) the soundness of the analysis 
GSA used to derive the estimate of funding that would be required
for the transition and (2) whether GSA will have accumulated	 
adequate funding to pay for transition costs. In performing this 
work, GAO reviewed cost estimation best practices, analyzed	 
relevant GSA documents, and performed an uncertainty analysis on 
GSA's estimate. 						 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-07-268 					        
    ACCNO:   A66186						        
  TITLE:     Telecommunications: GSA Has Accumulated Adequate Funding 
for Transition to New Contracts but Needs Cost Estimation Policy 
     DATE:   02/23/2007 
  SUBJECT:   Cost analysis					 
	     Documentation					 
	     Federal funds					 
	     Federal procurement				 
	     Funds management					 
	     Government contracts				 
	     Indefinite delivery contracts			 
	     Inflation						 
	     IT acquisitions					 
	     Program management 				 
	     Service contracts					 
	     Strategic planning 				 
	     Telecommunications 				 
	     Telecommunications systems 			 
	     Cost estimates					 
	     Indefinite quantity contracts			 
	     Program goals or objectives			 
	     FTS Networx					 
	     GSA Acquisition Services Fund			 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-07-268

   

     * [1]Results in Brief
     * [2]Background
     * [3]GSA's Analysis Was Not Sound

          * [4]GSA's Analysis Was Not Sufficiently Accurate
          * [5]GSA's Analysis May Not Be Comprehensive
          * [6]GSA's Analysis Was Not Adequately Documented
          * [7]GSA's Analysis Was Not Validated
          * [8]Quantifiable Weaknesses Are Small, but Effect of Estimate's
          * [9]GSA Does Not Have a Policy to Ensure That Cost Estimates Are

     * [10]GSA Has Accumulated Adequate Funding for the Transition
     * [11]Conclusions
     * [12]Recommendations
     * [13]Agency Comments and Our Evaluation
     * [14]GAO Contact
     * [15]Staff Acknowledgments
     * [16]GAO's Mission
     * [17]Obtaining Copies of GAO Reports and Testimony

          * [18]Order by Mail or Phone

     * [19]To Report Fraud, Waste, and Abuse in Federal Programs
     * [20]Congressional Relations
     * [21]Public Affairs

Report to the Ranking Member, Committee on Oversight and Government
Reform, House of Representatives

United States Government Accountability Office

GAO

February 2007

TELECOMMUNICATIONS

GSA Has Accumulated Adequate Funding for Transition to New Contracts but
Needs Cost Estimation Policy

GAO-07-268

Contents

Letter 1

Results in Brief 2
Background 3
GSA's Analysis Was Not Sound 7
GSA Has Accumulated Adequate Funding for the Transition 13
Conclusions 14
Recommendations 15
Agency Comments and Our Evaluation 16
Appendix I Objectives, Scope, and Methodology 18
Appendix II Uncertainty Analysis for GSA's Transition Estimate 20
Appendix III Comments from the GSA Administrator 21
Appendix IV GAO Contact and Staff Acknowledgments 23

Tables

Table 1: GSA Transition Cost Estimate 6
Table 2: Cumulative Effect of Quantifiable Weaknesses 11
Table 3: Risk Adjusted Factors 20

Figures

Figure 1: Estimated Cost of Transition Using Various Percents of
Transition Traffic 12
Figure 2: Confidence That Certain Amounts of Funding Will Account for
Actual Costs 13

Abbreviations

GSA General Services Administration
IMC Interagency Management Council
IT Information Technology

This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.

United States Government Accountability Office
Washington, DC 20548

February 23, 2007

The Honorable Tom Davis
Ranking Member
Committee on Oversight and Government Reform
House of Representatives

Dear Mr. Davis:

As innovations in telecommunications services continue to transform the
way the federal government conducts business, the General Services
Administration's (GSA) governmentwide telecommunications acquisition
programs offer federal agencies the opportunity to apply innovative
services and solutions to their operations. With the current set of
governmentwide telecommunications contracts approaching expiration, GSA
and its customer agencies will have to transition the services acquired
under these contracts to their replacements, known collectively as
Networx.

GSA will incur program management costs associated with planning and
executing this transition. It has also made a commitment to absorb certain
agency transition costs. To ensure it would have the funds necessary to
pay for these costs, GSA estimated that it would need to set aside
approximately $151.5 million.

This report responds to your request that we determine (1) the soundness
of the analysis GSA used to derive the estimate of funding that would be
required for the transition and (2) whether GSA will have accumulated
adequate funding to pay for its transition management costs. To accomplish
the first objective, we conducted a search of over 250 documents from both
government and industry1 for examples of best practices in the field of
cost estimation and identified common characteristics among them. We
determined that high-quality, reliable estimates should be accurate,
comprehensive, well-documented, and validated. Using these
characteristics, we analyzed transition estimate documentation developed
by GSA, documentation provided by GSA on the previous transition, the
Networx Request for Proposals, and other relevant documents. To accomplish
the second objective, we analyzed financial and operational documents from
related GSA programs. In addition, we interviewed GSA program officials
about both objectives and conducted an analysis of GSA's estimate to
examine the effects of varying the main cost driver in the estimate. We
conducted our work at GSA's Washington, D.C., area headquarters between
June 2006 and January 2007 in accordance with generally accepted
government auditing standards. A detailed discussion of our objectives,
scope, and methodology can be found in appendix I.

1These documents included published literature on cost estimation from the
Society of Cost Estimating and Analysis, the Department of the U.S. Army,
the Office of the Secretary of Defense, the National Aeronautics and Space
Administration, and the Department of Energy.

Results in Brief

GSA did not use sound analysis when estimating the amount of funding
needed to meet its transition-related commitments. Specifically, its
analysis was not sufficiently accurate, comprehensive, documented, or
validated. A primary weakness is that the estimate is largely based on the
assumption that 76 percent of the services provided under the current
contracts will transition to a different provider under the Networx
contracts. However, according to program officials, this assumption is
intentionally conservative and represents a worst-case scenario that is
unlikely to occur. Additionally, GSA may have double-counted a cost, did
not update its analysis to reflect a nearly 2-year delay, and did not
adequately document or validate its analysis. The weaknesses we identified
can be attributed in part to the lack of a policy requiring cost estimates
to be developed using best practices. Without such a policy, GSA's future
cost estimates could exhibit similar weaknesses, increasing the risk that
it will retain excess funds that could be reallocated for other purposes.

GSA has accumulated adequate funding to support its anticipated transition
costs. As of fiscal year-end 2006, GSA had approximately $142 million in a
transition reserve to be used to pay for its costs associated with the
transition. Our analysis of the estimate indicates that it is unlikely
that GSA will need to accumulate the entire $151.5 million it estimated
and that the funds it has already accumulated should be sufficient to fund
the transition. Specifically, the $142 million already retained will be
adequate to cover anticipated costs 96 percent of the time. Further, the
recent merger of two GSA funds increases the agency's flexibility and
could provide additional money, if needed. With Networx contracts
scheduled to be awarded starting in March 2007, GSA will soon have the
information necessary to reassess the main cost driver underlying its
estimate and address the weaknesses we identified. Once it has a current,
accurate estimate, GSA can reevaluate the funding needed to meet
anticipated commitments.

To ensure that future cost estimates by GSA are sound and can be used as a
reliable basis for decisions, we recommend that the GSA Administrator
establish an agencywide policy requiring that cost estimates be developed
using best practices. In addition, we recommend that the Administrator
revise the transition cost estimate using best practices after the award
of contracts under the Networx program and, if feasible, reallocate any
excess funds for other purposes or return them to the Treasury.

In written comments on a draft of this report, the GSA Administrator
concurred with our recommendations and emphasized the importance of
supporting a successful governmentwide telecommunications transition. Her
comments also state that GSA's transition cost estimate was within 2
percent of our independent analysis using the same assumptions. However,
while we identified only $3 million in quantifiable errors (roughly 2
percent of GSA's total estimate), we also determined that if the extent of
transitioning services is significantly lower than GSA's intentionally
conservative assumption, actual costs could be more than $110 million less
than GSA's estimate.

In addition, the Administrator presented two main objections to our draft.
First, she questioned whether our reported findings were balanced given
the facts and results presented. We clarified our report to ensure that
our findings better reflect the information we discuss. Second, she stated
that we incorrectly suggested comparability between the pending Networx
transition and the prior transition. We maintain that the two transitions
are comparable, particularly because GSA's analysis is based, in part, on
the results of the previous transition. Appendix III provides the full
text of GSA's comments. GSA also provided technical comments that have
been incorporated in our report as appropriate.

Background

As part of its mission of providing federal agencies with acquisition
services and solutions at best value, GSA's technology programs offer
agencies options to acquire needed telecommunications services. An option
chosen by more than 135 agencies is the FTS2001 program, which consists of
two large governmentwide telecommunications contracts--one awarded to
Sprint2 in December 1998 and the other to MCI3 in January 1999--and
FTS2001 crossover contracts.4

GSA is planning to replace the FTS2001 contracts, FTS2001 crossover
contracts, and separate wireless contracts with a new set of contracts.
Collectively known as the Networx program, these new contracts are to
provide governmentwide telecommunications services through two
indefinite-delivery/indefinite-quantity acquisitions--Networx Universal
and Networx Enterprise. The Universal acquisition is expected to satisfy
the requirements for a full range of national and international network
services and, according to GSA, to ensure the continuity of broad-ranging
services with global geographic coverage rendered under expiring
contracts. The Enterprise acquisition is expected to offer agencies
leading-edge services and solutions with less extensive geographic and
service requirements than Universal. The services required in these
contracts focus on Internet-based offerings and related security and
management services.

GSA expects the transition to begin when Networx Universal awards are made
in March 2007 and to continue until fiscal year 2010. Because the FTS2001
contracts with Sprint Nextel Corporation and Verizon Business expired in
December 2006 and January 2007, respectively, GSA and the incumbent
vendors negotiated separate sole-source contracts that essentially extend
the terms of the FTS2001 contracts for 42 months.5 These sole-source
contracts were awarded to ensure uninterrupted service and allow agencies
adequate time to complete the transition to Networx.

GSA is working with representatives of federal agencies to prepare for the
upcoming transition to Networx, both directly and through the Interagency
Management Council (IMC), a group of senior federal information resource
officials who advise GSA on issues related to telecommunications
contracts. The IMC has worked with GSA to document lessons learned from
the transition to FTS2001 that began in the late 1990s, GSA's most recent
governmentwide telecommunications transition. One important lesson learned
was that GSA and agency plans for funding transition expenses should be
determined early to allow agencies to gauge the impact of transition
expenses on their budgets. Specifically, the lessons-learned document
recommended that guidelines be established to allow agencies to complete
the financial planning required to ensure that the resources needed for
transition were available. This led the IMC and GSA to develop a Taxonomy
and Allocation of Transition Costs document that identified which Networx
transition costs would be borne by GSA and which would be borne by
transitioning agencies.

2Sprint Corporation merged with Nextel Communications, Inc., to form
Sprint Nextel Corporation in August 2005.

3MCI merged with Verizon to form Verizon Business in January 2006.

4In August 2001, GSA allowed contractors that had been awarded local
telecommunications contracts in selected metropolitan areas, through GSA's
Metropolitan Area Acquisition program, to offer long-distance services on
the FTS2001 contracts. This process is termed "crossover."

5GSA negotiated sole-source contracts for a 24-month base period with
three 6-month optional periods, for a total of 42 months.

This taxonomy document indicated that GSA will incur or reimburse
agencies, including:

           o GSA contractor support costs to, for example, aid in planning
           for the transition and oversight of Networx contractors. GSA
           officials also indicated that contractor support will be used to
           develop a methodology for tracking transition progress and the
           establishment of a transition coordination center; and
           o certain costs incurred by agencies during transition.

           In 2004, following the development of the taxonomy document, GSA
           generated an estimate of its costs for the transition. GSA's
           methodology for its estimate was to:

           o develop assumptions for the estimate,
           o define and develop a baseline for the estimate based on
           experiences and lessons learned from the previous transition,
           o determine the network growth as well as the total business
           volume projections for fiscal year 2006 based on historical
           traffic and cost trends,
           o define the transition cost elements,
           o define and develop a formula for calculating an estimate for
           each cost element, and
           o design and develop the estimated transition cost model for
           sensitivity analysis.

           Using this methodology, GSA estimated that it would need a total
           of $151.5 million for a 30-month transition, most of which would
           be used to reimburse agencies' transition costs. Table 1 below
           details the transition costs GSA expects to pay.

           Table 1: GSA Transition Cost Estimate

           Source: GSA.

           GSA planned to pay for its costs and the reimbursement of certain
           agency costs using its Information Technology (IT) Fund.6 The IT
           Fund was a full-cost recovery revolving fund7 whereby GSA fully
           recovered all costs of its technology programs and its operations
           via estimated fee rates. The fees, which are charged to agencies
           for the use of GSA contracts cover the direct costs of its
           operations--such as the development and management of contract
           vehicles--and indirect costs associated with its headquarters,
           such as support for the Offices of the Chief Information Officer
           and the Chief Financial Officer. The IT Fund allowed GSA to
           stabilize rates for its services when expenses varied and was used
           to provide funding for the previous transition to FTS2001. The IT
           Fund also contained a working capital reserve that was used to
           offset losses due to fluctuations in business volumes and other
           unexpected contingencies. At each fiscal year-end, the uncommitted
           balance of funds remaining was to be transferred to the general
           fund of the treasury as miscellaneous receipts.

           Recently, a new law changed the structure of the IT Fund and GSA's
           organization. On October 6, 2006, the General Services
           Administration Modernization Act8 combined the IT Fund with
           another GSA revolving fund--the General Supply Fund--to create the
           Acquisition Services Fund. This legislation, which also merged
           GSA's technology and supply programs,9 made all capital assets and
           balances remaining in the IT and General Supply Funds available
           for the purposes of the Acquisition Services Fund. According to
           GSA, the Federal Acquisition Service will increase agency savings,
           enhance GSA's capability to meet customer requirements for
           excellence, and improve internal efficiencies.
			  
			  GSAï¿½s Analysis Was Not Sound

           The analysis GSA used to derive its estimate of $151.5 million was
           not sound because it was not sufficiently accurate, comprehensive,
           documented, or validated. The analysis was not sufficiently
           accurate because it is largely based on the assumption that
           agencies will transition 76 percent of the services acquired under
           the current FTS2001 contracts to a different provider under
           Networx--an intentionally conservative scenario that GSA program
           officials believe is unlikely to occur. Further, the analysis has
           not been updated after a nearly 2-year delay in the contract
           award. While GSA appears to have included all pertinent costs, it
           may have double-counted a cost, calling into question the
           comprehensiveness of its analysis. In addition, GSA did not
           document significant assumptions and data. Finally, GSA's analysis
           was not validated by an independent cost estimate nor was an
           uncertainty analysis performed that would allow GSA to quantify
           the level of confidence it has in its estimate. These weaknesses
           can be attributed in part to the lack of a cost estimation policy
           that would help to ensure that such estimates are developed using
           best practices. While an intentionally conservative approach
           minimizes the risk that GSA would have inadequate funds to pay for
           committed transition costs, it increases the risk that GSA will
           retain excess funds that could be used for other purposes.
			  
			  GSAï¿½s Analysis Was Not Sufficiently Accurate

           Estimates are accurate when they are not overly conservative,
           based on an assessment of the most likely costs, and adjusted
           properly for inflation. Best practices further dictate that as
           schedules change, cost estimates should be revised to provide
           management with insight into the current program status, effective
           control of the program, and the ability to balance resources and
           the budget.

           The analysis GSA used to derive the estimate, however, was not
           sufficiently accurate. First, the estimate's main cost driver is
           based on the assumption that agencies will transition 76 percent
           of the services acquired under the current FTS2001 contracts to a
           different provider under Networx--60 percent due to agencies being
           forced to change providers and 16 percent due to voluntary
           changes. However, according to program officials, the 76 percent
           "transition traffic factor" is intentionally conservative and
           represents a worst-case scenario that is unlikely to occur.

           The 76 percent transition traffic factor is also overly
           conservative when compared with the previous transition. Then,
           approximately 60 percent of services were shifted from an
           incumbent to a different provider under FTS2001, the bulk being
           forced to change providers when an incumbent providing more than
           half of the services (AT&T) was not awarded an FTS2001 contract.
           This forced shift happened in part because GSA limited the FTS2001
           awards to only two vendors. In contrast, GSA has placed no such
           limit on the number of Networx vendors, and therefore, all
           incumbent FTS2001 vendors could potentially be awarded a Networx
           contract. Further, because the most-used FTS2001 incumbent
           (Verizon Business) provides approximately 50 percent of the
           services under the current FTS2001 contracts, the assumption of a
           60 percent forced shift would only be realized if no awards were
           made to Verizon Business and at least one other incumbent.
           Finally, GSA's assumption of a 16 percent voluntary shift is
           significantly higher than the 3 percent that voluntarily changed
           providers during the previous transition.

           Program officials could not identify any basis for the assumption
           that voluntary changes will reach 16 percent. However, the
           officials stated that the percentage of voluntary changes could be
           higher because agencies will likely have more options when
           selecting vendors and because of improved guidance on federal
           requirements regarding the fair opportunity process, which is
           intended to give each awardee an equal opportunity to compete for
           agencies' telecommunications services requirements based on
           agency-established selection criteria.

           Second, the analysis has not been updated to reflect schedule
           changes. When the estimate was developed in 2004, GSA expected to
           award the first Networx Universal contracts in mid-2005. Since
           that time, delays have pushed the time frame back almost 2 years.
           Universal awards are now expected in March 2007. GSA's analysis
           has not been adjusted to reflect additional inflation during this
           period, which could increase the estimates total by as much as $9
           million.

           Finally, GSA has not revised its analysis to reflect an assessment
           of most likely costs using currently available information. The
           analysis assumes that reimbursable agency transition costs would
           be greater than during the previous transition due to growth in
           the volume of services ordered. While GSA estimated that service
           levels would grow 60 percent over the previous transition by the
           time of contract award, as of August 2006, service levels had
           actually grown by 55.9 percent and are now expected to remain
           stable. The overestimation of service growth added approximately
           $1.7 million to the estimate.
			  
			  GSAï¿½s Analysis May Not Be Comprehensive

           Estimates are comprehensive when their level of detail ensures
           that all pertinent costs are included and no costs are
           double-counted. It is important to ensure the completeness,
           consistency, and realism of the information contained in the cost
           estimate.

           GSA appears to have included all pertinent costs in its analysis;
           however, according to officials, a specific agency cost valued at
           $4.3 million may have been double-counted. For several of the
           transition costs estimated, GSA based its calculations on the
           actual charges incurred during the previous transition. However,
           officials indicated that a specific agency cost that is estimated
           separately in its current estimate may have already been included
           in a more general category of its previous costs. Despite this
           uncertainty, GSA calculated the separate total for this specific
           agency cost based on current service levels and added it to its
           estimate. As a result, it is likely this specific agency cost is
           double-counted and therefore overstated.
			  
			  GSAï¿½s Analysis Was Not Adequately Documented

           Cost estimates are well-documented when they can be easily
           repeated or updated and can be traced to original sources through
           auditing. Rigorous documentation increases the credibility of an
           estimate and helps support an organization's decision-making
           process. The documentation should explicitly identify the primary
           methods, calculations, results, rationales or assumptions, and
           sources of the data used to generate each cost element.

           GSA provided us documentation of its methodology, the calculations
           it used to derive each cost element, results, and many of the
           previous transition costs.10 However, it did not document
           significant assumptions. Specifically, GSA did not document the
           rationale behind its 76 percent transition traffic factor or why
           it used a 30-month time period for the transition--two key
           assumptions of its analysis.

           GSA also did not provide documentation of certain data sources.
           Specifically, program officials could not provide supporting data
           used to estimate an agency transition cost valued at $4.7 million.
           In addition, GSA could not document the data sources used to
           estimate costs for contractor support in planning and implementing
           the transition. While many costs in its estimate are based on the
           charges incurred during the previous transition, GSA officials
           stated that it was not appropriate to use previous costs as a
           basis for the contractor cost element. These officials explained
           that unlike the previous transition, GSA would not provide
           agencies with on-site contractor support. Officials made this
           decision, in part, because the 2 1/2 years of transition planning
           that has occurred to date is expected to result in better
           preparation by agencies and the ability for them to facilitate
           their transitions without direct assistance from GSA or its
           contractors. Instead of basing their projection of contractor
           costs on prior charges, program officials told us that GSA
           management decided that contractor support costs should not exceed
           $35 million. Program officials could not provide any data or
           analysis to support this decision.
			  
			  GSAï¿½s Analysis Was Not Validated

           Estimates are adequately validated when they have been
           cross-checked with an independent cost estimate, and when a level
           of uncertainty associated with the estimate is identified. An
           independent cost estimate provides the estimator with an unbiased
           test of the reasonableness of the estimate and reduces the cost
           risk associated with the project by demonstrating that alternate
           methods generate similar results. In performing an uncertainty
           analysis, an entity examines the effects of varying multiple
           elements and, as a result, is able to express a level of
           confidence in its estimate.

           GSA did not validate its analysis against an independent cost
           estimate or perform an uncertainty analysis. GSA program officials
           could not provide a rationale of why these activities were not
           performed.
			  
			  Quantifiable Weaknesses Are Small, but Effect of Estimateï¿½s
			  Intentionally Conservative Assumption Could Be Significant

           The cumulative effect of the quantifiable weaknesses we identified
           in GSA's analysis is relatively small, resulting in an
           underestimation of $3 million, or roughly 2 percent of the total
           $151.5 million estimate (as shown in table 2). The underestimation
           of costs, related to inflation during the extended delay in making
           award, was offset by the overestimation of service growth and the
           possible double-counting of a specific agency transition cost.

           Table 2: Cumulative Effect of Quantifiable Weaknesses

           Source: GAO analysis of GSA information.

           In contrast, if the actual level of services transitioning to a
           different vendor is lower than the 76 percent transition traffic
           factor assumed by GSA, the effect will be greater. Because this
           factor is the primary cost driver in the estimate, significant
           changes in transitioning traffic result in similarly significant
           changes in total costs. This effect can be illustrated using GSA's
           transition cost estimate model (see fig. 1). For example, if a
           transition traffic factor of 66 percent is used (GSA's assumption
           of 16 percent for a voluntary transition plus a forced shift of 50
           percent that would occur if Verizon Business is not awarded a
           Networx contract), the total estimated cost of the transition
           falls to about $136 million. Similarly, if all incumbents are
           awarded Networx contracts and only GSA's assumption of a 16
           percent voluntary transition occurs, the total estimated cost of
           the transition is reduced to approximately $40 million. GSA
           officials explained that they used a risk-averse transition
           traffic factor to minimize the possibility of underestimating
           costs.

           Figure 1: Estimated Cost of Transition Using Various Percents of
           Transition Traffic
			  
			  GSA Does Not Have a Policy to Ensure That Cost Estimates Are Sound

           The weaknesses in GSA's analysis can be attributed in part to the
           lack of a policy requiring cost estimates to be developed using
           best practices. Officials from the Offices of the Chief
           Acquisition Officer, the Chief Information Officer, and the
           program office's Controller confirmed that GSA does not have
           centralized policy or guidance on cost estimation. Instead, the
           officials that prepared the estimate stated that they based their
           work loosely on best practices learned as a result of past
           experiences and were comfortable with the estimate. Officials
           believe there is no reason to revise their estimate to address the
           issues we raised in this report because their purpose in producing
           it was to minimize the risk of underestimating costs. While GSA's
           approach minimized the risk of having inadequate funds to fulfill
           its commitments, it increased the risk that GSA will retain excess
           funds that could be used for other purposes.
			  
			  GSA Has Accumulated Adequate Funding for the Transition

           GSA has accumulated adequate funding to support its anticipated
           commitments related to the Networx transition. As of fiscal
           year-end 2006, GSA accumulated approximately $142 million in a
           reserve dedicated to costs associated with the transition from
           FTS2001 to Networx. The uncertainty analysis we performed on GSA's
           cost estimate indicates that the funding GSA has already
           accumulated will most likely be adequate to pay for expected
           transition costs. By varying the transition traffic factor, our
           analysis indicates that the $142 million already retained should
           be adequate to cover expected expenses 96 percent of the time.
           Figure 2 illustrates the probability that a particular level of
           funding will be adequate to account for the total actual costs
           incurred. For example, the $151.5 million estimate represents a
           confidence level of 100 percent: there is almost no chance that
           costs will exceed the estimate. The full detail of our methodology
           is detailed in appendix II.

           Figure 2: Confidence That Certain Amounts of Funding Will Account
           for Actual Costs

           The merger of the IT and General Supply Funds provides GSA with
           additional flexibility, further reducing the need to retain the
           entire amount of the estimate. As discussed, legislation11
           combined the two funds into an Acquisition Services Fund, making
           their capital assets and balances available for the purposes of
           the new Federal Acquisition Service. The legislation also allows
           the Federal Acquisition Service to establish a reserve to retain
           surplus revenues from GSA's technology and service programs
           specifically for the purpose of offsetting losses and other
           unexpected contingencies. Further, at the end of each year, the
           statute requires GSA to return to the Treasury any funds not
           expended or held in a working capital reserve. During fiscal year
           2006, GSA's technology programs experienced losses, but its supply
           programs reported overall positive earnings of approximately $126
           million. If operations continue in this fashion, excess revenue
           will be available in the combined fund to offset losses or account
           for contingencies, such as the Networx transition, be retained
           within the Acquisition Services Fund, or be returned to the
           Treasury.

           With Networx contracts scheduled to be awarded starting in March
           2007, GSA will soon be in a position to reassess its main
           assumption, the transition traffic factor, and the resulting level
           of funding needed to meet anticipated commitments. Unless GSA
           revises its estimate, it risks unnecessarily retaining funds that
           could be reallocated to other agency priorities or returned to the
           Treasury.
			  
			  Conclusions

           While GSA achieved its goal of minimizing the risk that it would
           have inadequate funds to pay its transition commitments, the
           analysis used to develop the transition cost estimate was not
           sound because it was not sufficiently accurate, comprehensive,
           documented, or validated. The weaknesses can be attributed in part
           to a lack of a policy at GSA to ensure that such estimates are
           developed using best practices and make the best use of agency
           resources. While the quantifiable effect of the weaknesses in
           accuracy and comprehensiveness is small, the effect of potential
           inaccuracies resulting from the intentional use of an overly
           conservative assumption about a main cost driver could be more
           significant. Without the use of a cost estimation policy that
           reflects best practices, GSA could continue to produce similarly
           unsound estimates, increasing the risk that it will unnecessarily
           retain funds that could be reallocated for other purposes.

           Despite the weaknesses in its analysis, GSA has accumulated
           adequate funding to support its anticipated commitments related to
           the Networx transition. Our analysis indicates that it is highly
           unlikely that GSA will need more than the $142 million it has
           already accumulated. In addition, the merger of two revolving
           funds gives it increased flexibility in meeting costs. Once
           Networx contracts are awarded beginning in March 2007, GSA will be
           able to forecast the number of forced transitions more accurately,
           and, if necessary, reduce the amount of funding it plans to
           accumulate in the future or free already accumulated funds for
           other purposes. This reassessment will also be an opportunity for
           GSA to address the other weaknesses in its analysis that resulted
           from its deviation from best practices.
			  
			  Recommendations

           To improve GSA's program management, we are making two
           recommendations. First, to ensure that future cost estimates are
           sound and can be used as a reliable basis for decisions, we
           recommend that the GSA Administrator establish a policy for cost
           estimation efforts at GSA. Specifically, this policy should
           reflect best practices by requiring that estimates are:

           o accurate (not overly conservative, based on an assessment of the
           most likely costs, and adjusted properly for inflation);
           o comprehensive (their level of detail ensures that all pertinent
           costs are included and no costs are double-counted);
           o well-documented (can be easily repeated or updated and can be
           traced to original sources through auditing); and
           o validated (they have been cross-checked with an independent cost
           estimate and a level of uncertainty associated with the estimate
           has been identified.)

           Second, to ensure the most efficient use of federal funds, we
           recommend that the Administrator revise the transition cost
           estimate following the award of contracts under the Networx
           program. Specifically, this revision should reflect best
           practices, include a more precise transition traffic factor, and
           address the overestimation of service growth, the possible
           double-counting of a nonrecurring charge, and the effects of
           inflation during the extended delay in making awards. If the
           results of this new estimate indicate that the full $151.5 million
           is not needed to reasonably support the transition effort, GSA
           should reallocate any excess funds for other purposes allowable
           within the Acquisition Services Fund or return them to the
           Treasury.
			  
			  Agency Comments and Our Evaluation

           In written comments on a draft of this report, the GSA
           Administrator concurred with our recommendations and emphasized
           the importance of supporting a successful governmentwide
           telecommunications transition. To address our recommendations, she
           stated that GSA will issue improved policy guidance on cost
           estimating and review and adjust the cost estimate as additional
           information becomes available.

           The Administrator also commented that GSA's transition cost
           estimate was within 2 percent of our analysis using the same
           assumptions. However, while we identified only $3 million in
           quantifiable errors (roughly 2 percent of GSA's total estimate),
           our report also states that if the extent of transitioning
           services is significantly lower than GSA's intentionally
           conservative assumption, the actual costs of the transition could
           be considerably less than GSA's estimate. Specifically, if all
           incumbents are awarded Networx contracts and only GSA's assumption
           of a 16 percent voluntary transition occurs, the total cost of the
           transition could be reduced to approximately $40 million--over
           $110 million less than GSA's estimate.

           In addition, the Administrator stated that GSA does not concur
           with the entirety of the draft report. She presented two main
           objections. First, she questioned whether our reported findings
           were balanced given the facts and results presented. We clarified
           our report to ensure that our findings better reflect the
           information we discussed. Second, she raised a concern that we
           incorrectly suggested comparability between the pending Networx
           transition and the prior transition. We maintain that the two
           transitions are comparable, particularly because GSA's analysis is
           based, in part, on the results of the previous transition.
           Specifically, GSA's analysis relied on experiences and lessons
           learned from the previous transition to establish costs for the
           current estimate.
			  
1This included documentation available on a Transition Manager Web site
established by GSA, such as presentations, meeting minutes, and FTS2001
lessons learned.

2A Monte Carlo simulation allows the model's parameters to vary according
to their associated probability distribution. The result is a set of
estimated probabilities of achieving alternative outcomes, given the
uncertainty in the underlying parameters.

           Appendix III provides the full text of GSA's comments. GSA also
           provided technical comments that have been incorporated in this
           report, as appropriate.

6The IT Fund was authorized by the Paperwork Reduction Reauthorization Act
of 1986, Public Law 99-500 and 99-691; 40 U.S.C. 322.

7A revolving fund is established by Congress to finance a cycle of
businesslike operations through amounts received by the fund. A revolving
fund charges for the sale of products or services and uses the proceeds to
finance its spending, usually on a self-sustaining basis. Instead of
recording the collections in receipt accounts, the budget records the
collections and the outlays of revolving funds in the same account. A
revolving fund is a form of permanent appropriation.

8General Services Administration Modernization Act, Pub. L. No. 109-313,
120 Stat. 1734 (2006).

9GSA's supply programs provide agencies a source for commercial products
and services such as office supplies, vehicle purchasing, travel, and
furniture.

10For example, GSA provided data on contractor support costs incurred
during the previous transition.

11Public Law No. 109-313.

           As agreed with your office, unless you publicly announce the
           contents of this report earlier, we plan no further distribution
           until 30 days from the report date. At that time, we will send
           copies of this report to the GSA Administrator and interested
           congressional committees. We will also make copies available to
           others upon request. In addition, the report will be available at
           no charge on the GAO Web site at http://www.gao.gov .

           Should you or your offices have any questions about matters
           discussed in this report, please contact me at (202) 512-6240 or
           by e-mail at [email protected]. Contact points for our Offices of
           Congressional Relations and Public Affairs may be found on the
           last page of this report. GAO staff who made major contributions
           to this report are listed in appendix IV.

           Sincerely,

           Linda D. Koontz
			  Director, Information Management Issues
			  
			  Appendix I: Objectives, Scope, and Methodology

           Our objectives were to determine (1) the soundness of the analysis
           the General Services Administration (GSA) used to derive the
           estimate of funding that would be required for the transition and
           (2) whether GSA will have accumulated adequate funding to pay for
           its transition management costs.

           To determine the soundness of GSA's analysis, we conducted an
           intensive search of over 250 source documents of both government
           and industry literature for examples of best practices in the
           field of cost estimation. This included literature on cost
           estimation from the Society of Cost Estimating and Analysis, the
           Department of the U.S. Army, the Office of the Secretary of
           Defense, the National Aeronautics and Space Administration, and
           the Department of Energy. This indicated that high quality,
           reliable cost estimates are:

           o accurate (not overly conservative, based on an assessment of the
           most likely costs, and adjusted properly for inflation);
           o comprehensive (their level of detail ensures that all pertinent
           costs are included and no costs are double-counted);
           o well-documented (can be easily repeated or updated and can be
           traced to original sources through auditing); and
           o validated (they have been cross-checked with an independent cost
           estimate and a level of uncertainty associated with the estimate
           has been identified).

           To determine the extent to which GSA followed these practices, we
           analyzed documentation supporting the transition estimate,
           documentation provided by GSA on the previous transition estimate,
           and the Networx Request for Proposals. We also interviewed GSA
           Networx program managers and attended a GSA sponsored transition
           conference and meetings of the Interagency Management Council
           Transition Working Group.1

           To address whether GSA has accumulated adequate funding to pay for
           its transition management costs, we obtained and analyzed Cost and
           Capital Requirements Plans for the Information Technology Fund
           submitted to the Office of Management and Budget and legislation
           for GSA's Information Technology, General Supply, and Acquisition
           Services Funds. In addition, to determine the extent of funding
           held in GSA's related accounts, we analyzed financial statements
           for GSA's technology and service programs. To verify the
           reliability of these records, we obtained and analyzed the results
           of the most recent GSA financial audits and audit reports from
           GSA's Inspector General, and we interviewed GSA's independent
           financial auditor regarding the quality control procedures in
           place. The independent auditor did not specifically review the
           dollar amounts in the Information Technology or General Supply
           Funds for accuracy but did test the controls in place for
           compliance with laws and regulations. This auditor stated that
           there were no reportable findings associated with either fund, and
           it was reasonable to assume that these accounts were fairly
           stated. As a result, we determined the data were sufficiently
           reliable for the purposes of this report. We also interviewed
           officials from the Office of Management and Budget and officials
           from GSA's technology and service programs, Office of the Chief
           Acquisition Officer, Office of the Chief Financial Officer, and
           Office of the Chief Information Officer.

           To assess the adequacy of the level of funding already accumulated
           by GSA, we performed an uncertainty analysis for GSA's estimate
           using a Monte Carlo simulation.2 A Monte Carlo simulation provides
           a perspective on the potential variability of the cost estimate
           should the facts, circumstances, or assumptions change. We chose
           to vary only the transition traffic factor because it is the main
           driver of the costs in GSA's estimate. To carry out this
           simulation, we identified a minimum, maximum, and median value for
           the transition traffic factor based on information received from
           GSA.

           We conducted our work between May 2006 and January 2007 in
           accordance with generally accepted government auditing standards.
			  
			  Appendix II: Uncertainty Analysis for GSAï¿½s Transition Estimate

           An uncertainty analysis provides decision makers with a
           perspective on the potential variability of the estimate should
           the facts, circumstances, and assumptions change. By examining the
           effects of varying the estimates elements, a degree of uncertainty
           about the estimate can be expressed, possibly as an estimated
           range or qualified by some factor of confidence. For example, an
           estimate that produces a 100 percent confidence level indicates
           that, based on the methodology used to create that estimate, there
           is almost no chance that costs will exceed the estimate.

           We performed our uncertainty analysis on GSA's estimate using a
           Monte Carlo simulation.1 We chose to vary only the transition
           traffic factor because it is the main driver of the costs in GSA's
           estimate. To carry out this analysis, we identified a minimum,
           maximum, and median value for the transition traffic factor, based
           on information received from GSA.

           The transition traffic factor represents the possible percentage
           of services under FTS2001 that may transition to a different
           provider under the Networx contracts. For this factor, we chose a
           minimum of 3 percent, which represents the voluntary shift to a
           different vendor that occurred during GSA's previous transition to
           FTS2001. For the maximum, we used 76 percent, as this value was
           chosen by GSA officials as a worst-case scenario in an effort to
           mitigate the risks of underestimating costs. The median of these
           two numbers is 39.5 percent. Table 3 shows the variations of each
           factor used in our uncertainty analysis.

           Table 3: Risk Adjusted Factors

           Source: GAO analysis of GSA data.
			  
1See appendix I, footnote 2, for a description of Monte Carlo simulation.
			  
			  Appendix III: Comments from the GSA Administrator
			  
			  Appendix IV: GAO Contact and Staff Acknowledgments
			  
			  GAO Contact

           Linda D. Koontz (202) 512-6240 or [email protected]
			  
			  Staff Acknowledgments

           In addition to the contact named above, James R. Sweetman, Jr.,
           Assistant Director; Jamey A. Collins; Neil J. Doherty; Jennifer K.
           Echard; Wilfred B. Holloway; Ethan J. Iczkovitz; Frank Maguire;
           Karen A. Richey; Glenn D. Slocum; and Amos A. Tevelow made key
           contributions to this report.
			  
			  GAOï¿½s Mission

           The Government Accountability Office, the audit, evaluation and
           investigative arm of Congress, exists to support Congress in
           meeting its constitutional responsibilities and to help improve
           the performance and accountability of the federal government for
           the American people. GAO examines the use of public funds;
           evaluates federal programs and policies; and provides analyses,
           recommendations, and other assistance to help Congress make
           informed oversight, policy, and funding decisions. GAO's
           commitment to good government is reflected in its core values of
           accountability, integrity, and reliability.
			  
			  Obtaining Copies of GAO Reports and Testimony

           The fastest and easiest way to obtain copies of GAO documents at
           no cost is through GAO's Web site ( www.gao.gov ). Each
           weekday, GAO posts newly released reports, testimony, and
           correspondence on its Web site. To have GAO e-mail you a list of
           newly posted products every afternoon, go to www.gao.gov and
           select "Subscribe to Updates."
			  
			  Order by Mail or Phone

           The first copy of each printed report is free. Additional copies
           are $2 each. A check or money order should be made out to the
           Superintendent of Documents. GAO also accepts VISA and Mastercard.
           Orders for 100 or more copies mailed to a single address are
           discounted 25 percent. Orders should be sent to:

           U.S. Government Accountability Office 441 G Street NW, Room LM
           Washington, D.C. 20548

           To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax:
           (202) 512-6061
			  
			  To Report Fraud, Waste, and Abuse in Federal Programs

           Contact:

           Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail:
           [email protected] Automated answering system: (800) 424-5454 or
           (202) 512-7470
			  
			  Congressional Relations

           Gloria Jarmon, Managing Director, [email protected] (202)
           512-4400 U.S. Government Accountability Office, 441 G Street NW,
           Room 7125 Washington, D.C. 20548
			  
			  Public Affairs

           Paul Anderson, Managing Director, [email protected] (202)
           512-4800 U.S. Government Accountability Office, 441 G Street NW,
           Room 7149 Washington, D.C. 20548

(310761)

www.gao.gov/cgi-bin/getrpt?GAO-07-268 .

To view the full product, including the scope
and methodology, click on the link above.

For more information, contact Linda Koontz at (202) 512-6240 or
[email protected].

Highlights of [30]GAO-07-268 , a report to the Ranking Member, Committee
on Oversight and Government Reform, House of Representatives

February 2007

TELECOMMUNICATIONS

GSA Has Accumulated Adequate Funding for Transition to New Contracts but
Needs Cost Estimation Policy

The General Services Administration (GSA) and its customer agencies are
preparing to transition new governmentwide telecommunications contracts
known as the Networx program. GSA estimated the costs for which it is
responsible to be $151.5 million. This report addresses (1) the soundness
of the analysis GSA used to derive the estimate of funding that would be
required for the transition and (2) whether GSA will have accumulated
adequate funding to pay for transition costs. In performing this work, GAO
reviewed cost estimation best practices, analyzed relevant GSA documents,
and performed an uncertainty analysis on GSA's estimate.

[31]What GAO Recommends

GAO recommends that the GSA Administrator establish a cost estimation
policy that reflects best practices. In addition, GAO recommends that the
Administrator revise the transition cost estimate using best practices
after the award of the Networx contracts and reassess the funding needed
to meet its commitments. GSA concurred with these recommendations but
questioned the way GAO characterized the soundness of GSA's analysis and
whether this transition was comparable with the previous one. GAO
clarified its characterization of GSA's analysis.

GSA did not use sound analysis when estimating the amount of funding
needed to meet its transition-related commitments. Specifically, its
analysis was not sufficiently accurate, comprehensive, documented, or
validated. A primary weakness is that the estimate is largely based an
assumption--known as the transition traffic factor--that 76 percent of the
services provided under the current contracts would be moved to a
different provider under the Networx contracts. However, according to
program officials, this assumption is intentionally conservative and
represents a worst-case scenario that is unlikely to occur. Additionally,
GSA may have double-counted a cost and did not update its analysis to
reflect a nearly 2-year delay. Finally, GSA did not document significant
assumptions and data sources used in its analysis, or validate it. These
weaknesses can be attributed in part to the lack of a cost estimation
policy that reflects best practices. While GSA's intentionally
conservative approach minimizes the risk that it would have inadequate
funds to pay for committed transition costs, it increases the risk that
GSA will retain excess funds that could be used for other purposes.

GSA has accumulated adequate funding to support its anticipated transition
costs. As of fiscal year-end 2006, GSA had approximately $142 million in a
transition reserve. GAO analysis of the estimate indicates it is unlikely
that GSA will need more than it has already accumulated to fund the
transition. Specifically, the $142 million already retained will be
adequate to cover anticipated costs 96 percent of the time. The recent
merger of two GSA funds gives the agency additional flexibility that
reduces its need to accumulate the entire $151.5 million it estimated
would be needed (see table). With Networx contracts scheduled to be
awarded starting in March 2007, GSA will soon have the information
necessary to reassess the main assumption underlying its estimate--the
transition traffic factor--and address the weaknesses GAO identified. Once
this has been accomplished, GSA can reevaluate the funding needed to meet
anticipated commitments.

GSA's Transition Cost Estimate

Source: GSA.

References

Visible links

  30. http://www.gao.gov/cgi-bin/getrpt?GAO-07-268
*** End of document. ***