Managing For Results: Assessing the Quality of Program Performance Data
(Correspondence, 05/25/2000, GAO/GGD-00-140R).

Pursuant to a congressional request, GAO provided information on how
federal agencies should assess the quality of program performance data.

GAO noted that: (1) accuracy is the only one of several important areas
to consider when examining the quality of agency performance data; (2)
although certainly not exhaustive, the following key dimensions of data
quality illustrate some of the different types of quality concerns that
agencies consider when producing program performance data: (a)
accuracy--the extent to which data are free from significant error; (b)
validity--the extent to which the data adequately represent actual
performance; (c) completeness--the extent to which enough of the
required data elements are collected from a sufficient portion of the
target population or sample; (d) consistency--the extent to which data
are collected using the same procedures and definitions across
collectors and times; (e) timeliness--whether data about recent
performance are available when needed to improve program management and
report to Congress; and (f) ease of use--how readily intended users can
access data, aided by clear data definitions, user friendly software,
and easily-used access procedures; (3) when assessing how good data need
to be, it is important to recognize that no data are perfect; (4) data
need to be good enough to document performance, support decisionmaking,
and respond to the needs of internal and external stakeholders; (5)
decisions on "how good is good enough" may depend on the uses of the
data and the consequences of program or policy decisions based on those
data; (6) this may involve trade-offs among the key dimensions of
quality; (7) the key is for agencies to be aware of the data quality
limitations of the performance data, understand the trade-offs involved,
and reveal and discuss in performance reports the limitations and
trade-offs; (8) determinations on how good data need to be also depend
on other factors, such as the type of measure being used and the amount
of change expected in the data; (9) different types of performance data
may require different levels of accuracy; (10) developing standards for
key dimensions of data quality is a matter for individual agency
analysis and determination, taking into account: (a) the views of the
users of the data (internal and external stakeholders); and (b) relevant
professional standards and technical advice; (11) current users of
performance data may have valuable experience with the strengths and
weaknesses of existing data, therefore, they can provide insights into
the data credibility with external audiences; and (12) professional
standards provide another tool for agencies working to determine the
appropriate level of data quality for their performance measures.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  GGD-00-140R
     TITLE:  Managing For Results: Assessing the Quality of Program
	     Performance Data
      DATE:  05/25/2000
   SUBJECT:  Data integrity
	     Performance measures
	     Program evaluation
	     Strategic planning
	     Data collection

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved.                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                                            **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************
GAO/GGD-00-140R

United States General Accounting Office
GAO

Ordering Copies of GAO Reports
The first copy of each GAO report and testimony
is free. Additional copies are $2 each. Orders
should be sent to the following address,
accompanied by a check or money order made out
to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are
accepted, also. Orders for 100 or more copies to
be mailed to a single address are discounted 25
percent.
Order by mail:
U.S. General Accounting Office
P.O. Box 37050
Washington, DC 20013
or visit:
Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC
Orders may also be placed by calling (202) 512-
6000 or by using fax number (202) 512-6061, or
TDD (202) 512-2537.
Each day, GAO issues a list of newly available
reports and testimony. To receive facsimile
copies of the daily list or any list from the
past 30 days, please call (202) 512-6000 using a
touch-tone phone. A recorded menu will provide
information on how to obtain these lists.
Viewing GAO Reports on the Internet
For information on how to access GAO reports on
the INTERNET, send e-mail message with "info" in
the body to:
[email protected]
or visit GAO's World Wide Web Home Page at:
http://www.gao.gov

Reporting Fraud, Waste, and Abuse in Federal
Programs
To contact GAO FraudNET use:
Web site:
http://www.gao.gov/fraudnet/fraudnet.htm
E-Mail: [email protected]
Telephone: 1-800-424-5454 (automated answering
system)

United States
General Accounting Office
Washington, D.C. 20548-0001
Official Business
Penalty for Private Use $300
Address Correction Requested

                    Bulk Rate
               Postage & Fees Paid
                       GAO
                 Permit No. G100

(410581)

B-285312
Page 2GAO/GGD-00-140R Assessing the Quality of Program Perform
ance Data
B-285312

May 25, 2000

The Honorable Tillie K. Fowler
Chairman, Subcommittee on Oversight,
  Investigations, and Emergency Management
Committee on Transportation and Infrastructure
House of Representatives

Subject: Managing For Results: Assessing the Quality of
Program Performance Data

Dear Ms. Chairman:

On March 22, 2000, we testified at an oversight hearing that
the Subcommittee held on program data quality.1 This letter
responds to your request for additional information in order
to complete the record.

Question: In the testimony from the agencies we heard that
there will always be a degree of error in data. How would you
suggest agencies determine how accurate data need to be?

There is no easy answer to the question of how accurate
performance data need to be. Accuracy-the degree to which data
are free from significant error-is only one of several
important elements to consider when examining the quality of
agency performance data. Although certainly not exhaustive,
the following key dimensions of data quality illustrate some
of the different types of quality concerns that agencies
consider when producing program performance data:2

Accuracy-the extent to which the data are free from
significant error;

Validity-the extent to which the data adequately represent
actual performance;

Completeness-the extent to which enough of the required data
elements are collected from a sufficient portion of the target
population or sample;

Consistency-the extent to which data are collected using the
same procedures and definitions across collectors and times;

Timeliness-whether data about recent performance are available
when needed to improve program management and report to
Congress; and

Ease of Use-how readily intended users can access data, aided
by clear data definitions, user friendly software, and easily-
used access procedures.

When assessing how good data need to be, it is important to
recognize that no data are perfect. In general, data need to
be good enough to document performance, support
decisionmaking, and respond to the needs of internal and
external stakeholders. Decisions on "how good is good enough"
may depend on the uses of the data and the consequences of
program or policy decisions based on those data. This may
involve trade-offs among the dimensions of quality described
above.

For example, enhancing the completeness of a data collection
effort may have a negative impact on its timeliness when data
are to be obtained from a large number of independent
entities. Attempts to increase the timeliness of these data by
using a scientific sampling procedure to reduce the number of
entities providing data would reduce the completeness of the
coverage but may still provide adequate data for performance
measurement. Similarly, there can be trade-offs between
accuracy and timeliness, such as when an agency decides to use
or report not-yet-final data rather than waiting for final
figures. In this case, some accuracy may be knowingly
sacrificed for the more timely preliminary data. Budgetary
constraints may also play a role in determining the level of
data quality along a particular quality dimension.

The key is to for agencies to be aware of the data quality
limitations of the performance data, understand the trade-offs
involved, and reveal and discuss in performance reports the
limitations and trade-offs. Agencies may foster appropriate
use of data by clearly communicating how and to what extent
data limitations affect an assessment of performance. Making
Congress and stakeholders aware of significant data
limitations and their implications allows them to judge the
data credibility for their intended use and to use the data in
appropriate ways.

Determinations on how good data need to be also depend on
other factors, such as the type of measure being used and the
amount of change expected in the data. Different types of
performance data may require different levels of accuracy. For
example, audits of financial data, opinion surveys, and
assessments of air pollution in environmental measures may all
require different levels of accuracy. Within these areas,
professional judgement plays a role in determining acceptable
error levels. Data standards can also be influenced by the
expectations of people using the performance information. If
the amount of error present in performance data is not
significantly less than the amount of desired change that is
targeted for that measure, it is impossible to determine
whether the change is due to error or an actual change in
performance.

Developing standards for key dimensions of data quality, such
as accuracy, validity, consistency, and timeliness, is a
matter for individual agency analysis and determination,
taking into account, among other things, (1) the views of the
users of the data (internal and external stakeholders) and (2)
relevant professional standards and technical advice.

Current users of performance data may have valuable experience
with the strengths and weaknesses of existing data; therefore,
they can provide insights into the data credibility with
external audiences. For example, the Environmental Protection
Agency (EPA) seeks stakeholder feedback on the quality of its
performance data in several different ways, including
consulting with customers, posting feedback forms on its
Internet site, and requesting both users and providers to
verify data. To obtain "customer" feedback, EPA's Center for
Environmental Information and Statistics has conducted
meetings with national, regional, state, and local
environmental users to ask what information they need and how
they would like to access it. As part of this process, the
participants expressed concerns with the accuracy of data
entry, transmittal, and agency reporting.

Professional standards provide another tool for agencies
working to determine the appropriate level of data quality for
their performance measures. Several agencies have formed
groups to develop or apply data quality standards. Among these
is EPA's Science Advisory Board, established to provide
independent scientific and technical advice to the EPA
Administrator, conducts reviews to determine whether
performance data are of sufficient quality to support
environmental measures. The Department of Education has also
worked to use professional standards in setting and evaluating
the quality of performance data. When Education developed a
set of draft standards for use in evaluating the quality of
performance measures, it had the draft standards intensively
reviewed by the Department's Evaluation Review Panel, a group
of external evaluation experts from academia and state
agencies.

Because this response is primarily based on our previously
issued reports and testimonies, we did not seek agency comment
on a draft of this letter. We are sending a copy of this
letter to the Honorable James A. Trafficant, Jr., Ranking
Democratic Member, Subcommittee on Oversight, Investigations,
and Emergency Management, House Committee on Transportation
and Infrastructure. We will make copies available to others
upon request.

If you or your staff have any questions concerning this
letter, or if you need additional information, please contact
me on (202) 512-8676.

Sincerely yours,

J. Christopher Mihm
Associate Director, Federal Management and
  Workforce Issues
_______________________________
1 Managing for Results: Challenges in Producing Credible
Performance Information (GAO/T-GGD/RCED-00-134, March 22,
2000).
2 Performance Plans: Selected Approaches for Verification and
Validation of Agency Performance Information (GAO/GGD-99-139,
July 30, 1999).
*** End of document ***