Superfund: EPA's Contract Cost-Estimating Initiatives Show Promise and
Should Be Monitored (Letter Report, 03/02/2001, GAO/GAO-01-302).
The Environmental Protection Agency (EPA), which manages the cleanup of
the nation's most hazardous abandoned sites through the Superfund
program, relies heavily on contractors to conduct its cleanup
activities. Currently, EPA spends about 50 percent of its approximately
$1.5 billion annual Superfund budget on contractors. With so much at
stake, it is critical that the government get the best contract price
for this cleanup work. EPA's and GAO's reviews have shown that the
agency has made significant progress over the past decade in addressing
the weakness of its cost-estimating processes. EPA regional work
assignment managers are currently developing independent estimates,
which contracting officers are using to negotiate the prices for cleanup
work. The agency's current initiatives should help the agency
successfully address the Army Corps of Engineers' and GAO's remaining
concerns by providing the managers with the training and tools they need
to develop better estimates. By incorporating some relatively simple
additional steps to more fully implement and better scrutinize the
effectiveness of the initiatives, the agency can better ensure that its
efforts improve cost estimates agencywide.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: GAO-01-302
TITLE: Superfund: EPA's Contract Cost-Estimating Initiatives Show
Promise and Should Be Monitored
DATE: 03/02/2001
SUBJECT: Environmental monitoring
Cost control
Contract costs
Projections
Prices and pricing
IDENTIFIER: Superfund Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Testimony. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-01-302
Report to the Administrator, Environmental Protection Agency
United States General Accounting Office
GAO
March 2001 SUPERFUND EPA's Contract CostEstimating Initiatives Show Promise
and Should Be Monitored
Page i GAO- 01- 302 EPA Contract Cost Estimating Letter 1
Results in Brief 2 Background 3 EPA Has Launched Initiatives to Address
Cost- Estimating
Weaknesses GAO and the Corps Identified 4 EPA Could Better Assess the
Ultimate Effectiveness of Its
Initiatives 7 Conclusions 8 Recommendations for Executive Action 9 Agency
Comments 9 Contents
Page 1 GAO- 01- 302 EPA Contract Cost Estimating
March 2, 2001 The Honorable Christine Todd Whitman Administrator,
Environmental Protection Agency
Dear Ms. Whitman: The Environmental Protection Agency (EPA), which manages
the cleanup of the nation's most hazardous abandoned sites through the
Superfund program, 1 relies heavily on contractors to conduct its cleanup
activities. Currently, EPA spends about 50 percent of its approximately $1.5
billion annual Superfund budget on contractors. With so much at stake, it is
critical that the government gets the best contract price for this cleanup
work.
EPA can best achieve this goal by developing an independent estimate of what
the work should cost and using this estimate to help negotiate the price it
will award the contractor for the work. However, in the early 1990s, we
found that EPA was not generating independent estimates but was generally
adopting the contractors' cost estimate as the price awarded for the work. 2
Since then, we have reviewed EPA's actions to address this problem about
every 2 years. In 1999, we reported that EPA had taken steps to generate
estimates for all of the contracted work we reviewed and to use these
estimates when negotiating the award price of each contract, although the
quality of some estimates remained a concern. 3 The regional work assignment
managers we contacted, who are generally responsible for preparing the
estimates, consistently said that additional training and access to cost
data on similar contracted work would help them better develop independent
estimates. We recommended that EPA take further action to address these
needs.
1 The Superfund program was created under the Comprehensive Environmental
Response, Compensation, and Liability Act of 1980. 2 Superfund: EPA Has Not
Corrected Long- Standing Contract Management Problems (GAO/ RCED- 92- 45,
Oct. 24, 1991).
3 Superfund: Progress Made by EPA and Other Federal Agencies to Resolve
Program Management Issues (GAO/ RCED- 99- 111, Apr. 29, 1999). We reviewed
the 35 highest- dollarvalue work assignments in three EPA regions.
United States General Accounting Office Washington, DC 20548
Page 2 GAO- 01- 302 EPA Contract Cost Estimating
In response to our recommendations, in 1999, EPA hired the U. S. Army Corps
of Engineers- an agency with considerable cost- estimating and contract
management expertise- to conduct a comprehensive review of EPA's cost-
estimating processes. 4 The Corps made a number of recommendations to EPA in
its review, including providing better training and estimating tools, such
as access to data on the actual costs of cleanup work already contracted.
The Corps also recommended that EPA reinforce its commitment to controlling
costs, consolidate its various cost guidance documents, and better document
the basis for any significant variances in the original estimates, awarded
prices, and actual costs incurred.
In light of the Corps' and our findings and recommendations, we initiated
work to determine (1) what initiatives EPA has taken in response to the two
reviews and (2) what means EPA will use to determine whether its initiatives
have improved the quality and usefulness of its estimates. To respond to
these objectives, we reviewed the Corps' and our previous documented
findings and recommendations to EPA; discussed current EPA initiatives to
improve cost estimates with Superfund program managers and EPA's Office of
Acquisition Management, which is responsible for the agency's contracting
practices; reviewed documentation on EPA's initiatives; and discussed the
initiatives with the Corps' officials who had reviewed EPA's estimating
processes and had interviewed contracting officials from three EPA regions
with large cleanup workloads. We did not review any individual estimates
because these had been incorporated into the Corps' comprehensive review. We
conducted our review from August 2000 through February 2001 in accordance
with generally accepted government auditing standards.
EPA has designed, but not fully implemented, two initiatives- a new training
course and new guidance- to address the Corps' and our concerns about its
cost- estimating processes. EPA's new training course on cost estimating has
been well received by the approximately 50 regional work assignment managers
(or 10 percent) who have taken the course in the 2 years since its
inception. However, the agency has not resolved how to make the training
available to the remaining staff. EPA also plans to release new guidance by
the end of 2001 on how managers
4 EPA/ USACE Assessment of Cost Procedures and Methodologies: Comprehensive
IGCE Review (U. S. Army Corps of Engineers, Walla Walla District, Dec.
1999). Results in Brief
Page 3 GAO- 01- 302 EPA Contract Cost Estimating
can prepare more detailed and complete estimates. The guidance will include
a Web- based tool with data on the actual, current costs of cleanup
activities. However, to make the data useful, the agency needs to ensure
that all data remain current. We are recommending that EPA resolve the
issues surrounding its training implementation and keeping cost data updated
and current.
Similarly, EPA could enhance the way it assesses its initiatives' impact on
estimate quality. The agency plans to continue its current approach- relying
primarily on the on- site reviews of estimates it conducts in each region
once every 3 years- to evaluate an estimate's quality and usefulness.
According to the Superfund managers in charge of contracting, limited
resources preclude more frequent on- site reviews. The managers also believe
that the current review process is successful because it indicates
improvements in estimates. While we recognize the resource limitations of
the Superfund program and the progress EPA has made to date on improving
estimates, we believe EPA could take relatively simple actions to augment
the regional reviews. For example, some regions currently collect data on
the estimated costs and awarded prices for their contracted work. Superfund
managers could analyze these data to look for systemic problems across the
agency that need to be addressed or successful best practices that should be
emulated. EPA's and Corps' contracting officials concurred that more
frequent monitoring of estimates would help EPA determine the initiatives'
effectiveness. Accordingly, we are recommending that the agency routinely
analyze data on its initiatives across regions to facilitate any necessary
adjustments and improvements.
Because of their experience with the technical aspects of cleanups, EPA
regional work assignment managers prepare detailed cost estimates for all
new work on Superfund projects before the agency contracts for that work.
These estimates should specify the scope of the work assignment (such as
conducting a cleanup feasibility study), the required hours, and the
expected cost of each major activity for that assignment. Regional contract
officers then use these estimates to negotiate with contractors the best
price that EPA can award for each work assignment, documenting any
significant differences between the estimate and the awarded price. The
agency also uses this process to negotiate any changes in the scope of a
work assignment as the cleanup proceeds that may lead to discrepancies
between the original awarded price and the final price of the contracted
work. The work assignment manager uses the estimate and any revisions to
develop the work plan and to manage the contractor's work. Background
Page 4 GAO- 01- 302 EPA Contract Cost Estimating
When we last examined EPA's cost estimates in April 1999, we reported that
11 (or about 31 percent) of the 35 estimates we reviewed fell substantially
below the awarded price- an indication that the estimates may be of poor
quality, according to the agency's Financial Manager's Financial Integrity
Act Report. In addition, in 29 percent of the cases, the awarded price
matched the contractor's estimate- an indication that EPA may not be using
the estimates to negotiate the best price for the government. In response to
our report, the agency said it was awaiting the results of the Corps' review
and would then take further action to correct any cost- estimating problems.
EPA is in the process of implementing two initiatives designed to address
the weaknesses with its cost- estimating processes identified by the Corps
and us. The initiatives are a new training course and new guidance,
including a Web- based tool that provides work assignment managers with data
on the actual, current costs of contracted work to help them develop more
complete and accurate estimates. EPA has yet to determine how it will make
the training available to all managers when needed or how it will ensure
that the actual cost data in the tool remain up- to- date.
In our April 1999 report, we noted that work assignment managers, who
prepare the estimates, cited two barriers that kept them from developing
better estimates. First, 15 of the 34 work assignment managers we
interviewed (about 44 percent) cited their inexperience with estimate
preparation. Although work assignment managers are chiefly responsible for
managing the technical aspects of cleanup work, they had relatively little
cost- estimating experience, because they tended to prepare only one or two
estimates each year. They told us that more and better training were needed
to compensate for this lack of experience. Second, all of the work
assignment managers cited the need to access data on the actual costs of
previously contracted work. They believed that better data on actual costs
could serve as a baseline when determining what new contracted work should
cost.
Similarly, the Corps recommended in its December 1999 report that EPA (1)
improve the training it provides work assignment managers, and (2) develop
tools that contain better information on actual costs of assignments to help
the work assignment managers generate better estimates. It also recommended
that EPA (1) consolidate its various costestimating guidance documents, (2)
encourage work assignment managers to make more of a commitment to using
their estimates to control contract costs, and (3) better document the
assumptions work assignment EPA Has Launched
Initiatives to Address Cost- Estimating Weaknesses GAO and the Corps
Identified
Page 5 GAO- 01- 302 EPA Contract Cost Estimating
managers used to develop each estimate and any significant differences in
the estimate, the awarded price, and actual costs incurred. Actual costs may
differ if changes in the scope of the work occur as the work proceeds.
To address the Corps' and our concerns, EPA designed a new training course
for regional work assignment managers and is currently in the process of
consolidating and revising its guidance on preparing estimates. The agency
offered its new half- day course targeted at preparing estimates as an
optional seminar during the last two annual conferences for work assignment
managers. According to EPA officials, about 50 (or 10 percent) of the
agency's 500 work assignment managers attended one of these two sessions.
The course offered instruction on which cost components to include in an
estimate and addressed pricing the components and the resources available to
help staff with this pricing activity. These resources included an
introduction to a database on historical costs for contracted work compiled
by the New York regional office and tables detailing the typical amount of
labor hours needed to complete various contracted cleanup activities. In
addition, the course showed managers how to adequately document the
assumptions and data used to generate EPA's estimate. The work assignment
managers who attended the course found it very helpful and suggested
improvements- such as additional instruction on documenting estimates- that
could make it even more useful, according to the course evaluations they
completed.
Superfund managers in charge of contracting practices have not yet decided
how to provide the training to all work assignment managers who need it when
they need it. In response to our questions, the managers said that they
planned to continue offering the course at the annual conference. However,
it is unlikely that a sufficient number of work assignment managers will
obtain the training they need in a timely and useful manner, since the
agency has only reached about 10 percent of the work assignment managers in
the past 2 years by relying on this method. To provide more opportunities
for work assignment managers to attend, Superfund managers also said they
are considering offering more than one session during each conference and
training key staff in each region, who could, in turn, train the work
assignment managers in that region. However, Superfund managers do not have
such a plan in place at this point.
According to the Atlanta region's cost estimator, who was instrumental in
developing the training course, it would be more effective for EPA to offer
future training in the regions rather than through the annual conferences
New Training Course
Page 6 GAO- 01- 302 EPA Contract Cost Estimating
because more useful and detailed cost information can be presented in that
forum. State project managers also attend the annual conferences, and unlike
EPA project managers, some of them are required to disclose to contractors
all of the information they used to develop their estimates. EPA does not
disclose how it prices estimates because that could diminish the
independence of the estimates and result in higher costs to the government.
By presenting the training in the regions exclusively to EPA project
managers, EPA can include the cost information without fearing that it will
be disclosed to contractors.
As its second initiative, the Superfund managers formed a workgroup-
composed of headquarters and regional contracting staff- to revise the
agency's guidance to regional staff on how to prepare independent cost
estimates. According to the managers, the new guidance will consolidate
eight separate guidance documents currently in use and will emphasize how
good estimates are critical to controlling contract costs. The guidance will
also take users through the steps of developing a thorough estimate,
providing them with checklists detailing the scope of a work assignment,
giving users instructions for adequately documenting the steps, and
identifying reliable sources of cost data. EPA has contracted with the Corps
to use its expertise and databases to develop the data on the actual costs
of similarly contracted work and other information, such as data on the
labor hours needed for cleanup activities, to be included in the guidance.
EPA also plans to ask cleanup estimators from other agencies, such as the U.
S. Navy, to review these data before they are made available to work
assignment managers. The Superfund managers intend to publish the guidance
in paper form and make it available as an interactive tool via EPA's Web
site by the end of 2001.
Data on the actual costs and the other information, such as the labor hours
needed, will only be useful if they are updated regularly and kept current.
To date, the Superfund managers have told us that their primary focus is to
develop and implement the revised guidance to ensure that the initial
version contains accurate data. They have not yet determined how they will
routinely keep the data current, including the data originally obtained from
the Corps, although they have discussed obtaining the Corps' assistance on
these issues. EPA plans to include a mechanism in the Webbased tool that
allows work assignment managers to provide feedback on the tool, which they
hope will help them keep the data current. However, because this feedback is
voluntary, it does not provide a comprehensive method for updating the
actual cost data. New Guidance
Page 7 GAO- 01- 302 EPA Contract Cost Estimating
EPA plans to continue its current method of evaluating the quality and
usefulness of estimates by relying primarily on its on- site reviews of
estimates in each region once every 3 years. Superfund managers believe the
reviews have been effective; furthermore, they do not believe they have the
resources needed to visit the regions more frequently. EPA could supplement
its regional reviews fairly easily by routinely analyzing information on
estimate quality that is now becoming available- such as regional data
comparing estimates to awarded prices- to look for systemic problems that
need addressing. According to contracting officials from EPA and the Corps,
additional monitoring would help EPA better ascertain the initiatives'
effectiveness in the regions.
Currently, Superfund managers and officials from EPA's Office of Acquisition
Management- the office responsible for the agency's overall contracting
practices- review all Superfund contracting activities in the regions by
visiting each of the 10 regions approximately once every 3 years to conduct
reviews. Superfund managers assured us that, in response to the concerns we
raised in our 1999 report regarding estimate quality, the reviews conducted
in the past 2 years have included an examination of the quality of a
region's estimates. The three most recent reviews, beginning with the review
of the Seattle region in June 1999, contain assessments of estimates and
recommendations for improvement.
In addition, Superfund managers told us that they use the periodic meetings
and monthly teleconferences of the Senior Regional Management Acquisition
Council as a forum to exchange information on issues or problems affecting
estimates. The Council includes representatives from EPA's Office of
Acquisition Management and the Superfund program in headquarters, and
regional work assignment managers. Furthermore, the managers told us that
the discussion of estimates is a regular topic on the agenda for the
Council's semiannual conferences.
Superfund managers plan to continue to rely on the regional reviews,
supplemented by the Council's and other discussions, to determine whether
the quality and usefulness of contract estimates have improved as a result
of the new guidance and training. They pointed out that recent regional
reviews have found that estimate quality has generally improved in the
regions. The managers believe that examining estimates in three to four
regions each year, supplemented by the periodic Council discussions,
provides them with a good indication of any problems that may be occurring
with estimates nationwide. Furthermore, they cited a lack of resources as a
barrier limiting more frequent regional reviews. EPA Could Better
Assess the Ultimate Effectiveness of Its Initiatives
Page 8 GAO- 01- 302 EPA Contract Cost Estimating
According to the director of the division in the Office of Acquisition
Management that oversees Superfund contracting activities, if the resources
for more frequent monitoring were available, EPA could better determine how
effective the initiatives are in the regions. Similarly, according to the
Corps' manager who oversaw the review of EPA's estimating practices, EPA
should review regional implementation of the new guidance within 6 months to
1 year of its taking effect to ensure that regional estimates are improving.
Monitoring would be particularly important in the early implementation
phases. However, these officials recognize that resource constraints are
likely to rule out more frequent regional reviews.
Superfund managers have at least two possible ways to augment the regional
reviews and better monitor the agency's initiatives. First, in their recent
regional reviews, managers have recommended that the regions begin tracking
data that compare the estimates, the awarded prices, and the final prices
paid for the work as a way to gauge estimate quality in that region. The
Atlanta region has already begun tracking its comparison data, according to
the cost estimator in that region, who found the analysis very helpful for
making improvements. Superfund managers have not yet required the remaining
regions to collect the data or to forward it to headquarters, where the
managers could analyze the information across regions. According to a Corps
contracting official, the Corps performs such an analysis to determine
whether its estimates are adequate nationwide. He believed that a similar
analysis could help Superfund managers quickly identify and correct any
systemic problems or incorporate any successful best practices.
Second, Superfund managers plan to include a comment section in the new
guidance, once it has been posted to the Superfund Web site, where regional
work assignment managers can voluntarily provide feedback on the guidance as
they begin to use it. The managers have not planned how they will analyze
and use this information but said they would develop a plan before launching
the new guidance by the end of 2001.
EPA's and GAO's reviews have shown that the agency has made significant
progress over the past decade in addressing the weakness of its
costestimating processes. EPA regional work assignment managers are
currently developing independent estimates, which contracting officers are
using to negotiate the prices for cleanup work. The agency's current
initiatives should help the agency successfully address the Corps' and our
remaining concerns by providing the managers with the training and tools
Conclusions
Page 9 GAO- 01- 302 EPA Contract Cost Estimating
they need to develop better estimates. By incorporating some relatively
simple additional steps to more fully implement and better scrutinize the
effectiveness of the initiatives, the agency can better ensure that its
efforts improve cost estimates agencywide.
In order to more fully implement and better evaluate the improvements
currently being made to EPA's cost- estimating processes, we recommend that
the Administrator, EPA, direct the Assistant Administrator, Office of Solid
Waste and Emergency Response, to complete plans on how to
? most effectively deliver improved, timely, and effective training to work
assignment managers nationwide,
? work with the Corps to keep the data on actual costs of contracted work in
its Web- based tool updated and current, and
? consolidate and routinely analyze regional data (which compare estimates,
awarded prices, and final prices paid to contractors) and the feedback from
work assignment managers on the Web- based tool to determine whether
systemic estimating problems exist that EPA needs to address or whether
best- estimating practices are available that it could adopt.
We provided a draft of this report to EPA for review and comment. We
subsequently met with designated Superfund managers, who agreed with our
recommendations and promised to take actions to begin their implementation.
Specifically, with respect to our first recommendation, they agreed that
regional training seminars could be tailored and timed to meet the specific
needs of the work assignment managers in the region. Therefore, they are
likely to develop region- specific training as the primary training vehicle.
In that regard, the Atlanta region's cost estimator is conducting a training
session in February 2001 for the work assignment managers in that region.
Project officers from other regions are also attending the training in
Atlanta, and the Superfund managers said they may consider whether it would
be best for those project officers to customize the training, and in turn,
train the work assignment managers in their own regions.
Responding to our second recommendation, Superfund managers agreed that the
Corps can most efficiently and effectively update cost data in the Web-
based tool. However, EPA has yet to work out the details of the Corps'
future involvement in that effort. Recommendations for
Executive Action Agency Comments
Page 10 GAO- 01- 302 EPA Contract Cost Estimating
Finally, Superfund managers agreed that uniform regional data would help
them judge the quality of estimates agencywide, identify any potential
problems, and address them in a timely manner. Therefore, they agreed to
formalize the requirements for reporting regional data to track cost
estimates, awarded prices, and the actual prices of contracted cleanup work.
We are sending copies of this report to appropriate congressional committees
and interested Members of Congress. We are also sending copies to the
Honorable Mitchell E. Daniels, Jr., Director, Office of Management and
Budget. In addition, we will make copies available to others on request.
If you or your staff have any questions about this report, please contact me
at (202) 512- 3841. Key contributors to this report were Eileen Larence,
Karla Springer, Elizabeth Erdmann, Jonathan S. McMurray, and Roger Bothun.
Sincerely yours, David G. Wood Director, Natural Resources
and Environment
(160548)
The first copy of each GAO report is free. Additional copies of reports are
$2 each. A check or money order should be made out to the Superintendent of
Documents. VISA and MasterCard credit cards are also accepted.
Orders for 100 or more copies to be mailed to a single address are
discounted 25 percent.
Orders by mail:
U. S. General Accounting Office P. O. Box 37050 Washington, DC 20013
Orders by visiting:
Room 1100 700 4 th St., NW (corner of 4 th and G Sts. NW) Washington, DC
20013
Orders by phone:
(202) 512- 6000 fax: (202) 512- 6061 TDD (202) 512- 2537
Each day, GAO issues a list of newly available reports and testimony. To
receive facsimile copies of the daily list or any list from the past 30
days, please call (202) 512- 6000 using a touchtone phone. A recorded menu
will provide information on how to obtain these lists.
Orders by Internet
For information on how to access GAO reports on the Internet, send an email
message with “info” in the body to:
Info@ www. gao. gov or visit GAO's World Wide Web home page at: http:// www.
gao. gov
Contact one:
? Web site: http:// www. gao. gov/ fraudnet/ fraudnet. htm
? E- mail: fraudnet@ gao. gov
? 1- 800- 424- 5454 (automated answering system) Ordering Information
To Report Fraud, Waste, and Abuse in Federal Programs
*** End of document. ***