GSA On-line Procurement Programs Lack Documentation and
Reliability Testing (21-DEC-01, GAO-02-229R).
The General Services Administration's (GSA) two online
procurement systems--GSA Advantage! and information Technology
Solutions Shop--lack basic management controls to ensure the
systems produce reliable data, can not be inappropriately
modified, and provide continuous service. These controls are
increasingly important since government agencies are expected to
increase their use of these systems and GSA relies heavily on
contractors for systems operations and maintenance. Moreover,
Congress and other oversight entities need adequate assurance
that these systems are reliable and accurate to assess the
effectiveness of government contracting programs.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-02-229R
ACCNO: A02675
TITLE: GSA On-line Procurement Programs Lack Documentation and
Reliability Testing
DATE: 12/21/2001
SUBJECT: Data integrity
Electronic commerce
Federal procurement
Internal controls
Procurement practices
Small business
Statistical data
GSA Advantage Program
GSA Information Technology Solutions
Shop Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Testimony. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-02-229R
United States General Accounting Office Washington, DC 20548
December 21, 2001
The Honorable Stephen A. Perry
Administrator
General Services Administration
1800 F Street NW
Washington, DC 20405
Subject: GSA On-line Procurement Programs Lack Documentation and Reliability
Testing
Dear Mr. Perry:
1
Earlier this year, we conducted a review of small business participation in
selected on-line procurement programs in fiscal years 1999 and 2000. During
the course of this review, we compared the small business share of
procurement dollars awarded by two GSA on-line procurement programs, GSA
Advantage! and Information Technology Solutions Shop (ITSS), to the small
business share of all federal contract dollars awarded and we assessed the
2
reliability of the data needed for our report. We evaluated the data by (1)
performing electronic tests of relevant fields and (2) requesting and
reviewing, if available, related program and system design documentation,
audit and system reviews, and reports. We determined that the data were
reliable enough for the purposes of that report. However, we did identify
several errors that GSA needed to correct to more accurately state total
sales and small business participation. We also identified serious
management control weaknesses related to data reliability as well as GSA's
ability to ensure the continuity of the operations for the on-line programs.
This letter discusses our concerns and recommends improvement actions.
BACKGROUND
The federal government is taking steps to increase its use of electronic
commerce, particularly in the area of conducting on-line procurement. For
example, as part of an effort to improve the functioning of the government
and achieve efficiencies of operations, the President has requested agencies
to expand their use of on-line procurement in fiscal year 2002.3
1 Electronic Commerce: Small Business Participation in Selected On-line
Procurement Programs 2 Generally accepted government auditing standards require that we assess
the reliability of any data used to support our
findings, conclusions, and recommendations.
3 OMB Memorandum No. 01-11, Feb. 14, 2001.
GAO-02-229R GSA Data Documentation Issues
GSA Advantage! and ITSS are two systems agencies can turn to for electronic
procurements. GSA Advantage! serves as an "electronic mall" in which
government buyers can search listings, compare prices, and purchase items
on-line. ITSS on-line procurement program is used for larger, complex
purchases. Customers enter requirements on-line, GSA staff solicit quotes,
issue purchase orders, and authorize payment electronically. GSA administers
both Advantage! and ITSS on-line procurement programs. Sales through these
two on-line programs have been small but growing. The two on-line programs
grew as a percentage of total federal procurement dollars from about
one-half percent in fiscal year 1999 to about 1 percent in fiscal year 2000.
The table below provides a description of each of the programs.
Table 1: GSA's Advantage! and ITSS Programs: Services, Products, and Sales
Dollars in thousands
Source:
GAO,
based
on
GSA
documentation
It
is
important
that
the
data
in
both
systems
be
as
accurate and reliable as possible. Government agencies, for example, need to
know that pricing data is accurate to make their purchases. Agencies also
rely on these systems for summary data on government contracting
opportunities, solicitations, awards, etc. The Congress and other oversight
entities need to know that data on such things as total sales, agency use,
and small business participation is reliable in order to assess the
effectiveness of government contracting programs.
LACK OF SYSTEM DOCUMENTATION
GSA program officials told us that they did not have system documentation
for either GSA Advantage! or ITSS. Documentation would normally include the
data processing or automated aspects of the system (such as application
program coding), operating instructions, and system policies, processes, and
procedures (such as procedures on "configuration control" or how changes to
the system are to be made). Documenting systems is a common management
practice and is required of GSA under the Joint Financial
Management Improvement Program's4 Federal Financial System Requirements to
completely document both management systems.
According to GSA Advantage! officials, there is no formal documentation of
the system because the system evolved over many years in a piecemeal fashion
among many developers -both government and contractors. The Chief Technology
Officer for ITSS also said that ITSS was developed without extensive
documentation and evolved over time, based on rapid prototyping and
deployment. GSA officials told us that they have just started efforts to
contract for services to document the systems.
It is important for GSA to take time now to document the overall system.
Documenting systems helps organizations ensure the data is reliably
collected and entered into a system and that changes made are appropriate
and completed in a timely and effective manner. Moreover, by documenting
such things as coding and system operations, organizations are better
positioned to ensure that they can continue operations in the event of a
disaster or emergency. This is especially important for GSA because it
relies heavily on contractors to operate and maintain the two systems.
LACK OF DATA ENTRY CONTROLS AND ROUTINE DATA VALIDATION
As part of our reliability assessment, we electronically analyzed data in
both GSA Advantage! and ITSS and found significant errors. For example:
* In the case of Advantage!, the database included $32.2 million that
represented 16 "test orders", which were used to train and provide
demonstrations to users, according to GSA officials. These data were not
marked as test cases and their inclusion would have skewed future reports,
including small business participation and data extracts. Currently, there
is no separate training environment to ensure that test orders are not
included with real orders.
* In the case of ITSS, we found that GSA was using an incorrect formula to
calculate total sales. As a result, initially fiscal year sales totaled $3.1
billion; as corrected, they summed to $2.3 billion. The ITSS Chief
Technology Officer told us that ITSS was not originally designed to provide
strong reporting capability. Consequently, those trying to generate reports
from the system need a thorough understanding of the data and data queries
to obtain correct information.
4 The Joint Financial Management Improvement Program (JFMIP) is a joint
undertaking of the US Department of Treasury, the General Accounting Office,
the Office of Management and Budget and the Office of Personnel Management,
working in cooperation with each other and other agencies to improve
financial management practices in Government. JFMIP's guidance covers
acquisition systems such as Advantage! and ITSS.
In each case, contract and program officials agreed with our analysis and
subsequently corrected the databases and/or reported numbers to reflect more
accurate numbers.
The data problems we identified may be attributable to the fact that GSA
lacks data entry or edit controls to catch this type of errors and does not
periodically validate data. Such controls are considered to be common
industry practice since they help ensure that data are accurate and
reliable. Although contractors told us that the system was designed to
"police itself"-that is, its customers, contracting officials and other
agencies would point out errors-the errors we discovered had not previously
been brought to GSA's attention. Moreover, GSA officials did not affirm that
they would be implementing routine data validation and other safeguards to
ensure data accuracy and integrity.
CONCLUSION
GSA's two online procurement systems-GSA Advantage! and ITSS-lack basic
management controls to ensure the systems produce reliable data, can not be
inappropriately modified, and provide continuous service. These controls are
increasingly important since government agencies are expected to increase
their use of these systems and GSA relies heavily on contractors for systems
operations and maintenance. Moreover, the Congress and other oversight
entities need adequate assurance that these systems are reliable and
accurate to assess the effectiveness of government contracting programs.
RECOMMENDATIONS FOR AGENCY ACTION
We recommend that the GSA Administrator direct managers responsible for
oversight of both GSA Advantage! and ITSS to:
* Fully document the procurement systems in accordance with guidelines for
government financial management systems, such as those in JFMIP;
* Establish procedures and deadlines for documenting any future changes to
either system;
* Establish procedures to prevent entry of test data into production
systems; and
* Develop procedures for periodic examination, validation, and timely
correction of GSA Advantage! and ITSS data.
AGENCY COMMENTS
We provided GSA with a draft of this report for review and comments. In oral
comments, GSA officials agreed with our recommendations and provided
technical comments, which we have incorporated as appropriate. GSA officials
also said that even though both systems are being reengineered, the issues
we raised continue to be relevant, and that they are initiating actions to
address our recommendations. The officials could not estimate when
implementation of the recommendations would be complete.
To evaluate the reliability of data provided by contractors and officials
responsible for Advantage! and ITSS, we:
* reviewed governmentwide guidance for developing and maintaining management
systems;
* reviewed existing agency information such as audit reports by the GSA
Inspector General, independent reviews and system documentation, if
available;
* performed electronic data testing of relevant fields such as checking for
missing data, the relationship of one field to another, values outside of a
designated range, and dates outside valid time frames; and
* discussed control weaknesses with officials and contractors.
We conducted our review from April 2001 to September 2001 in accordance with
generally accepted government auditing standards. In conducting our review
we met with officials at the General Services Administration, and
contractors for both Advantage! and ITSS on-line procurement programs.
This report contains recommendations to you. The head of a federal agency is
required under 31 U.S.C. 720 to submit a written statement of actions taken
on our recommendations to the Senate Committee on Governmental Affairs and
the House Committee on Government Reform not later than 60 days after the
date of this letter and to the Senate and House Committees on Appropriations
with the agency's first request for appropriations made more than 60 days
after the date of this letter.
We are sending copies of this letter to interested congressional committees.
This letter will also be available on GAO's home page at http://www.gao.gov.
Please contact me at (202) 512-4841 or Hilary Sullivan at (214) 777-5652 if
you or your staff have questions. Major contributors to this report were
Barbara Johnson, Rosa M. Johnson, and Beverly Ross.
Sincerely yours,
David E. Cooper Director, Acquisition and Sourcing Management
(120106)
*** End of document. ***