Performance Plans: Selected Approaches for Verification and Validation of
Agency Performance Information (Letter Report, 07/30/1999,
GAO/GGD-99-139).
Under the Government Performance and Results Act, agencies' program
goals are to be spelled out in annual performance plans, and performance
against these goals is to be reported in annual performance reports. The
first performance reports are to be sent to the President and Congress
no later than March 31, 2000. This report identifies reasonable
approaches that agencies have proposed or adopted to verify and validate
performance information. GAO describes these approaches to help agency
managers pick appropriate techniques for assessing, documenting, and
improving the quality of their performance data.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: GGD-99-139
TITLE: Performance Plans: Selected Approaches for Verification
and Validation of Agency Performance Information
DATE: 07/30/1999
SUBJECT: Performance measures
Strategic planning
Accountability
Productivity in government
Data integrity
Agency missions
Reporting requirements
Congressional/executive relations
IDENTIFIER: GPRA
Government Performance and Results Act
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. This text was extracted from a PDF file. **
** Delineations within the text indicating chapter titles, **
** headings, and bullets have not been preserved, and in some **
** cases heading text has been incorrectly merged into **
** body text in the adjacent column. Graphic images have **
** not been reproduced, but figure captions are included. **
** Tables are included, but column deliniations have not been **
** preserved. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
United States General Accounting Office GAO Report
to the Chairman, Committee on Governmental Affairs, U.S. Senate
July 1999 PERFORMANCE PLANS Selected Approaches for
Verification and Validation of Agency Performance Information
GAO/GGD-99-139 United States General Accounting Office
General Government Division Washington, D.C. 20548 B-281215 July
30, 1999 The Honorable Fred Thompson Chairman, Committee on
Governmental Affairs United States Senate Dear Mr. Chairman: The
Government Performance and Results Act of 1993 (Results Act) seeks
to improve the effectiveness, efficiency, and accountability of
federal programs by requiring federal agencies to set goals for
program performance and to report on annual performance compared
with the goals. Annual program goals are to be set out in annual
performance plans, and performance against these goals is to be
reported in annual performance reports. The first performance
reports are to be submitted to the President and Congress no later
than March 31, 2000. In order to credibly report progress toward
intended results and to use the information for program
management, agencies will need to have sufficiently trustworthy
performance information. The Results Act requires agency
performance plans to "describe the means to be used to verify and
validate measured values" of performance. Verification includes
the assessment of data completeness, accuracy, and consistency and
related quality control practices. Its purpose is to ensure that
the data will be of sufficient quality to document performance and
support decision-making. Validation is the assessment of whether
the data are appropriate for the performance measure. In a
December 1997 letter to the Director of the Office of Management
and Budget, congressional leadership stated that performance plans
based on incomplete or inaccurate data would be of little use to
Congress or the executive branch. Agencies submitted annual
performance plans in spring 1998 and 1999, setting goals for
fiscal years 1999 and 2000, respectively. In response to
congressional requests, we have reviewed the fiscal year 1999 and
2000 performance plans of the 24 agencies covered by the Chief
Financial Officers (CFO) Act. Our analyses of the fiscal year 1999
performance plans concluded that most of the plans reviewed
provided limited confidence that agencies' performance data would
be credible.1 In 1Managing for Results: An Agenda to Improve the
Usefulness of Agencies' Annual Performance Plans, (GAO/GGD/AIMD-
98-228, Sept. 8, 1998). Page 1
GAO/GGD-99-139 Verification and Validation of Performance Data B-
281215 the report on their assessment of the 1999 performance
plans, the House leadership noted that "most agencies lack the
reliable data sources and systems needed to develop, validate and
verify performance information."2 The report also noted that the
problems in performance data were deep- seated and resolving them
would take much time and effort. Our assessment of the fiscal year
2000 performance plans identified a continuing lack of confidence
in performance information as a major concern. Ultimately,
performance plans will not be fully useful to congressional
decisionmakers unless and until this key weakness is resolved.3 In
this report, as you requested, our objective is to identify
reasonable approaches that agencies have proposed or adopted to
verify and validate performance information. This report describes
these approaches in order to help agency managers select
appropriate techniques for assessing, documenting, and improving
the quality of their performance data. Overall, we found examples
illustrating a wide range of possible Results in Brief
approaches for increasing the quality, validity, and credibility
of performance information. (See app. I for a discussion of how
agencies may decide on specific approaches.) These approaches
included a variety of senior management actions, agencywide
efforts, and specific program manager and technical staff
activities. These approaches can be organized into four general
strategies, as follows. Management can seek to improve the quality
of performance data by fostering an organizational commitment and
capacity for data quality (see app. II). Managers are ultimately
responsible for the quality of performance information. We found
examples of management communications and actions to encourage the
needed coordination, resource allocation, and attention to data
quality issues. Reporting efforts to build organizational
commitment to obtaining, maintaining, and using good information
and to developing the organization's capacity to do so can help
improve the credibility of performance information. Verification
and validation can include assessing the quality of existing
performance data (see app. III). Assessments might target specific
measures in the performance plan or more broadly assess major
2U.S. Congress, Seeking Honest Information for Better Decisions
(http://freedom.house.gov/results/ implement/implement4.asp, June
1998). 3 Managing for Results: Opportunities for Continued
Improvements in Agencies' Performance Plans (GAO/GGD/AIMD-99-215,
July 20, 1999). Page 2
GAO/GGD-99-139 Verification and Validation of Performance Data B-
281215 data systems to identify problems that may affect the use
of performance data. In our examples, assessments were conducted
internally, built into ongoing work processes and data systems, or
involved independent verification and external feedback.
Assessments of data quality are of little value unless agencies
are responding to identified data limitations (see app. IV).
Communicating significant data limitations and their implications
allows stakeholders to judge the data's credibility for their
intended use and to use the data in appropriate ways. In addition
to examples of reporting data limitations and their implications
in performance plans or other formats, we saw examples of efforts
to improve, supplement, or replace existing data. Building quality
into the development of performance data may help prevent future
errors and minimize the need to continually fix existing data (see
app. V). Reporting efforts to improve existing data systems or
processes can improve the credibility of performance information.
We found examples of efforts to build in data quality, including
involving stakeholders; providing feedback on data quality
problems; and using accepted practices in planning, implementing,
and reporting performance data. Within these general strategies
are more specific approaches that agencies may choose to adopt.
These specific approaches are listed in figure 1 and discussed in
more detail in appendixes II-V of this report. We identified a
wide range of reasonable approaches that agencies can use, where
appropriate, to improve the quality, usefulness, and credibility
of performance information. How an agency approaches data
verification and validation depends on the unique characteristics
of its programs, stakeholder concerns, performance measures, and
data resources. For example, different approaches may apply to the
information collected directly by a federal agency than to that
obtained from state sources. Verifying and validating information
on client satisfaction may require different approaches than
information obtained by direct measurement of environmental
conditions, for example. We expect that agencies will choose from
among the approaches described here or will develop different ones
to arrive at a systematic strategy suitable to their own situation
and performance information sources. Appendix I discusses a number
of key questions that can arise when agencies are deciding on the
effort to be devoted to verification and Page 3
GAO/GGD-99-139 Verification and Validation of Performance Data B-
281215 validation, the specific approaches to be adopted, and how
to credibly report on their verification and validation efforts.
Figure 1: Menu of Agency Approaches for Verifying and Validating
Performance Information Page 4 GAO/GGD-99-139
Verification and Validation of Performance Data B-281215 Because
of the need to develop a strategy that meets the unique
circumstances of each agency, the framework we present is not a
list of requirements or a checklist of steps to follow. Individual
agencies should not necessarily be expected to use all of the
approaches that we describe, and there may be other approaches
that we have not identified. We conducted our work in six
agencies: the Departments of Education, Scope and
Transportation, and Veterans Affairs and the Environmental
Protection Methodology Agency, National Science Foundation,
and Office of Personnel Management. These agencies were selected
after further review of the verification and validation
information contained in the 24 annual performance plans we had
assessed in 1999. We selected the agencies that we judged would
provide a wide range of examples of reasonable verification and
validation approaches and represented a variety of performance
measurement contexts. These six agencies differed in the extent to
which they * provided internal services to government or to the
public, * conducted and supported scientific research, or *
administered regulatory programs. They also varied in the extent
to which their programs were carried out through the states or
delivered directly. We identified examples of specific
verification and validation approaches based on our review of the
1999 and 2000 performance plans and discussions with agency
officials. Some of the additional examples identified by agency
officials pertain to verifying and validating the performance
information used for managing agency programs-not for assessing
progress toward performance plan goals. We reviewed agency
documents, where available, to confirm our examples and obtain
additional detail. We selected examples that appeared reasonable
and useful for other agencies to emulate because they were
consistent with accepted professional practice for managing data
quality. We found many more examples than are reported here, but
we restricted our choices to two or three examples per approach.
We included examples to represent different aspects of the
approaches being discussed and to provide for a balance among the
agencies reviewed. Where possible, our selection of examples drew
on our previous work on the adequacy of the agencies' performance
information in specific program Page 5 GAO/GGD-
99-139 Verification and Validation of Performance Data B-281215
areas. We did not do additional work to assess the adequacy or
extent of agencies' implementation of these approaches, the
quality of the performance information, or whether the approaches
had contributed to improving data quality. Although we identified
several specific reasonable approaches that these agencies had in
place or planned to implement, our separate assessment of the 24
CFO Act agencies' fiscal year 2000 plans (including the six
reviewed for this report) found that none provided full confidence
that their performance information would be credible. The
performance plans for the six agencies included in this report all
provided at least limited confidence in their performance
information--that is, they all addressed to varying extents, but
not completely, * their efforts to verify and validate
performance data, * actions to compensate for unavailable or low-
quality data, and * the implications of data limitations for
assessing performance. We developed the framework depicted in
figure 1 by analyzing our examples and by reviewing related
professional literature. We sought comments on the framework from
external professionals and agency officials and incorporated their
suggestions where appropriate. We focused our review on the
verification and validation of nonfinancial performance
information. Generally, financial performance information that is
derived from the same systems that produce financial statement
information is subject to the internal control standards, federal
financial systems requirements, and accounting standards
applicable to federal agencies' financial statement information.
We conducted our work between October 1998 and April 1999 in
accordance with generally accepted government auditing standards.
We did not seek comments on this report because it does not
provide an Agency Comments overall assessment of agency data
verification and validation efforts. However, we asked officials
in each of the six agencies to verify the accuracy of the
information presented. We incorporated their clarifications where
applicable. As agreed with your office, unless you publicly
announce its contents earlier, we plan no further distribution of
this report until 30 days from the date of this letter. At that
time, we will send copies to Senator Joseph I. Lieberman, Ranking
Minority Member of your Committee; Representative Richard K.
Armey, Majority Leader; Representative Dan Burton, Chairman, and
Representative Henry A. Waxman, Ranking Minority Member, House
Page 6 GAO/GGD-99-139 Verification and Validation
of Performance Data B-281215 Committee on Government Reform; and
Jacob Lew, Director, Office of Management and Budget. We will also
make copies available to others on request. This report was
prepared under the direction of Stan Divorski. Mary Ann Scheirer
was a key contributor. If you have any questions regarding this
report, please contact me or Stan Divorski at (202) 512-7997.
Sincerely yours, Susan S. Westin Associate Director, Advanced
Studies and Evaluation Methodology Page 7 GAO/GGD-
99-139 Verification and Validation of Performance Data Contents 1
Letter 12 Appendix I What Are Verification and
Validation? 12 Key
Questions About Why Are Verification and Validation
Important? 12 What Is Data
Quality?
13 Verification and How Good Do Data Need to Be?
13 Validation Does a Given Verification and
Validation Approach Apply 14 to All
Programs? Why Attend to Data Quality?
15 How Can Verification and Validation Efforts Be Clearly
16 Reported? 20 Appendix II Communicate Support for
Quality Data 20
Fostering Review Organizational Capacities and
Procedures for 21 Data Collection and
Use Organizational Facilitate Agencywide Coordination
and Cooperation 22 Commitment and
Assign Clear Responsibilities for Various Aspects of the
23 Data Capacity for Data Adopt Mechanisms That Encourage
Objectivity and 24 Quality
Independence in Collecting and Managing Data Provide Responsible
Staff With Training and Guidance 25
for Needed Skills and Knowledge An Example of Agencywide Capacity-
building at ED 26 28 Appendix III
Build Data Quality Assessment Into Normal Work
28 Assessing the Quality Processes, Including Ongoing Reviews
or Inspections Use Software Checks and Edits of Data on Computer
30 of Existing Data Systems and Review Their
Implementation Use Feedback From Data Users and Other Stakeholders
30 Compare With Other Sources of Similar Data or Program
32 Evaluations Obtain Verification by Independent Parties,
Including the 32 Office of the Inspector
General Consequences From Assessing the Quality of Existing
34 Data Page 8 GAO/GGD-99-139 Verification and
Validation of Performance Data Contents 35 Appendix IV
Report Data Limitations and Their Implications for
35 Responding to Data Assessing Performance Adjust or
Supplement Problematic Data
37 Limitations Use Multiple Data Sources With
Offsetting Strengths and 37 Limitations
Improve the Measure by Using Another Source or New
38 Methods of Measurement 39 Appendix V Use Prior
Research or Analysis to Identify Data Elements
40 Building Quality Into That Adequately Represent the
Performance to Be Measured the Development of Gain Agreement
Among Internal and External
40 Performance Data Stakeholders About a Set of Measures
That Are Valid for Their Intended Use Plan, Document, and
Implement the Details of the Data 41
Collection and Reporting Systems Provide Training and Quality
Control Supervision for All 44 Staff Who
Collect and Enter Data, Especially at Local Levels Provide
Feedback to Data Collectors on Types of Errors
45 Found by Data Checks Use Analytic Methods and Transformations
Appropriate 46 for the Data Type and
Measure Being Reported An Alternative Approach to Performance
Assessment at 46 NSF Figure 1: Menu of
Agency Approaches for Verifying and 4
Figures Validating Performance Information
Figure I.1: Sample Presentation Format From ED's Fiscal
18 Year 2000 Performance Plan Figure II.1: Approaches to Fostering
Organizational 20 Commitment and
Capacity Figure III.1: Approaches to Assessing the Quality of
28 Existing Data Figure IV.1: Approaches for Responding to Data
35 Limitations Figure V.1: Approaches to Building Data Quality
Into the 39 Development of Performance Data
Page 9 GAO/GGD-99-139 Verification and Validation of
Performance Data Contents Abbreviations ANSI American
National Standards Institute BTS Bureau of Transportation
Statistics DOT Department of Transportation ED
Department of Education EPA Environmental Protection
Agency GPRA Government Performance and Results Act HEDIS
Health Plan Employer Data and Information Set ISTEA
Intermodal Surface Transportation Efficiency Act IPBS
Integrated Performance and Benchmarking System NCES
National Center for Education Statistics NCQA National
Committee for Quality Assurance NHTSA National Highway
Traffic Safety Administration NSF National Science
Foundation OMB Office of Management and Budget OPM
Office of Personnel Management RIS Retirement and
Insurance Service SEA State educational agencies SEDCAR
Standards for Education Data Collection and Reporting VA
Department of Veterans Affairs VBA Veterans Benefits
Administration VHA Veterans Health Administration Page 10
GAO/GGD-99-139 Verification and Validation of Performance Data
Page 11 GAO/GGD-99-139 Verification and Validation of
Performance Data Appendix I Key Questions About Verification and
Validation Many questions can arise when agencies are deciding on
the effort that should be devoted to verification and validation
and the specific approaches appropriate to their agency and
program contexts. These questions can include * What are
verification and validation? * Why are verification and
validation important? * What is data quality? * How accurate do
data need to be? * Does a given verification and validation
approach apply to all programs? * Are Results Act requirements
the only reasons for attending to data quality? * How can
accountability for verification and validation be clearly
reported? Verification and validation refer to aspects of quality
control needed to What Are Verification ensure that users can
have confidence in the reported performance and Validation?
information. We define these terms as follows: * Verification is
the assessment of data completeness, accuracy, consistency,
timeliness, and related quality control practices. * Validation
is the assessment of whether data are appropriate for the
performance measure. We are not addressing here other aspects of
"validity," such as the appropriateness of the agency's choice of
performance measures in relation to its goals and objectives.1
Other GAO products discuss issues related to other aspects of
validation. For example, our Results Act: Evaluator's Guide
provides guidance on defining expected performance, and on validly
connecting missions, goals, and activities.2 Both verification and
validation help to ensure that data of sufficient Why Are
Verification quality will be available when needed to
document performance and and Validation support
decision-making. To be useful in reporting to Congress on the
fulfillment of Results Act requirements and in improving program
results, Important? the data must also be
"credible," that is, they must be seen by potential users to be of
sufficient quality to be trustworthy. 1The term "validation" can
be used in many different ways, including validation of the
appropriateness of the agency's overall goals and objectives,
given the agency's legislative mandates and mission; performance
measures as "validating" the program, e.g., to provide evidence of
program results; assessing whether the performance measures chosen
by the agency are clearly related to the target objectives; and
its more limited use in this report. 2The Results Act: An
Evaluator's Guide to Assessing Agency Annual Performance Plans
(GAO/GGD- 10.1.2, Apr. 1998). Page 12
GAO/GGD-99-139 Verification and Validation of Performance Data
Appendix I Key Questions About Verification and Validation
Reporting validation and verification procedures helps to ensure
that data will be credible to potential users. The Office of
Management and Budget (OMB) guidance states that "The means used
should be sufficiently credible and specific to support the
general accuracy and reliability of the performance information .
. ."(OMB Circular A-11, sec. 220.13). Attention to "credibility,"
in addition to more technical aspects of data quality, requires
careful consideration of the needs of the audiences for the
information. Choices among potential verification and validation
approaches involve What Is Data Quality? senior management's
making decisions about data quality. The approaches that agencies
use to verify and validate performance data should address key
dimensions of data quality. Specific agencies and professional
sources have developed data quality criteria specifically relevant
to their context and content area. The key dimensions of data
quality suggested below were developed for this report to
illustrate the types of quality concerns that agencies consider.
These are not intended to be exhaustive of all potential quality
considerations, nor substituted for agency-developed criteria.
Examples of data quality elements are the following: * Validity-
the extent to which the data adequately represent actual
performance. * Completeness-the extent to which enough of the
required data elements are collected from a sufficient portion of
the target population or sample. * Accuracy-the extent to which
the data are free from significant error. * Consistency-the
extent to which data are collected using the same procedures and
definitions across collectors and times. * Timeliness-whether
data about recent performance are available when needed to improve
program management and report to Congress. * Ease of use-how
readily intended users can access data, aided by clear data
definitions, user-friendly software, and easily used access
procedures. There is no easy answer to the question of how good
data need to be. No How Good Do Data data are perfect.
In general, data need to be good enough to document Need to Be?
performance and support decision-making. Decisions as to "how good
is good enough" may depend on the uses of the data and the
consequences of program or policy decisions based on those data.
These factors may involve trade-offs among the dimensions of data
quality presented above. On the one hand, emphasizing the
completeness of a planned data collection effort may reduce its
timeliness when data are to be obtained from a large number of
independent entities (such as school districts or industrial
establishments). On the other hand, seeking to increase timeliness
by using a scientific sampling procedure to reduce the number Page
13 GAO/GGD-99-139 Verification and Validation of
Performance Data Appendix I Key Questions About Verification and
Validation of entities providing data would reduce the
completeness of coverage of entities from which data are
collected, but may still provide adequate data for performance
measurement. Different levels of accuracy may be needed in
different circumstances. For Different Measures May
example, audits of financial data, assessments of the extent of
air pollution Require Different Levels of to use in
environmental performance measures, and opinion surveys may
Accuracy all require different levels of
accuracy. Within these areas, professional judgment plays a role
in determining acceptable error levels. In sample surveys, it is
recognized that the margin of error "should be dictated by how
much error the investigators feel they can live with in reporting
survey results."3 The amount of desired change in performance can
also influence the Amount of Change Desired determination
of a reasonable data standard. If the amount of error is not
Affects Data Standards sufficiently less than the amount
of change targeted, it will not be possible to determine whether
change in the measured value is due to error or actual changes in
performance. Agencies use different approaches to validate and
verify performance Does a Given measures for
different types of programs. For example, service delivery
Verification and assistance programs (e.g., of the
Department of Education) or administration of earned benefits
(e.g., benefits to federal employees Validation Approach
administered by the Office of Personnel Management) might use
Apply to All Programs? performance data from a survey of
beneficiaries, with data quality criteria derived from appropriate
procedures for sample surveys. In contrast, regulatory programs,
such as those of the Environmental Protection Agency, may measure
targeted pollutants, based on scientifically derived procedures
for assessing each pollutant. Determining the most appropriate
validation and verification approaches for each type of program is
a matter for individual agency diagnosis, analysis, and choice,
taking into account stakeholder views, the relevant professional
standards, and technical advice. Useful verification and
validation approaches will also vary with the data source being
used for the performance measure. For example, if agencies have
substantial direct control over data that they generate during
their normal operations (e.g., while processing claims for
benefits due), agency managers can directly supervise quality
control. In contrast, if agency partners, such as state or local
grantees, collect the performance data, 3Lu Ann Aday, Designing
and Conducting Health Surveys: A Comprehensive Guide (San
Francisco: Jossey-Bass, 1989), p. 120. Page 14
GAO/GGD-99-139 Verification and Validation of Performance Data
Appendix I Key Questions About Verification and Validation
substantial negotiation may be needed to agree on what data
elements are feasible to collect and how quality is to be ensured.
Agencies identified several influences, in addition to the Results
Act, that Why Attend to Data encourage attention to
data quality. These influences can be external or Quality?
internal in origin, and attending to them can help improve
programs. External influences include external critics,
legislative mandates, and the External Factors Can need
to comply with professional standards in each program delivery
area, Encourage Data Quality such as health care delivery.
For example, Department of Transportation (DOT) officials
identified several sources that stimulated its efforts to improve
data quality. These sources included the Results Act, the National
Performance Review, and provisions of the Intermodal Surface
Transportation Efficiency Act. This legislation set out explicit
data requirements for DOT and created the Bureau of Transportation
Statistics with the mission to compile statistics and improve the
quality of agency data. External studies by the National
Performance Review and the National Academy of Public
Administration identified a need for the Environmental Protection
Agency (EPA) to work to improve the quality of environmental data.
EPA has also been striving to increase the availability of
environmental data to the public and has recently established the
expansion of the public's right to know about their environment as
a strategic goal. The increased availability of data has brought
data quality issues to the surface, and external stakeholders have
identified inaccuracies in EPA data. Agency staff cited increased
use of data for program management as an Data Use Is an Internal
internal influence leading to better data quality. For example,
EPA Factor That Encourages identified the limited
operational use of some data as a root cause of errors Data
Quality in the data, as no operational unit was
using those data in managing its work. The agency described a
variety of potential uses for environmental data, including
regulatory compliance, public right-to-know, environmental status
and trends reporting, program management, and performance
accountability tracking, as called for under the Results Act. To
help achieve a better fit between its data and this variety of
uses, the agency has undertaken suitability assessments of key
data systems for uses other than those originally intended. Agency
attention to data quality can help managers increase efficiency or
Attention to Data Quality improve operations-for example, by
improving the flow of program tasks Can Help Improve Programs and
the coordination among related programs. Page 15
GAO/GGD-99-139 Verification and Validation of Performance Data
Appendix I Key Questions About Verification and Validation Tracing
data flow as a part of assessing the quality of existing data can
also lead to improvements in the "business processes" that rely on
that data.4 In examining the reasons for problems with data
quality, EPA identified a lack of correspondence between data
systems and overall business process management at the program
level as one of the factors. Veterans Benefits Administration
(VBA) officials noted that efforts were undertaken to reengineer
its business processes to better meet the needs of individual
veterans and to improve information about VBA's business and
veterans. These efforts included new procedures to credit benefit
processors for their work, changing their focus from an emphasis
on "timeliness" to a broader range of work quality criteria in
order to improve services to veterans and reduce the potential for
distorting measures. These criteria are reflected in a "balanced
scorecard" for its performance measures (see app. V). Validation
and verification are not isolated, technical concerns relevant
solely to the requirement of the Results Act. Use of data is part
of good management. Further, the production of data to inform both
business concerns and the public is a fundamental mission of such
government agencies as EPA and DOT. Therefore, fostering data
quality is fundamental to total agency management. Obtaining
agency commitment to and capacity for data that can be verified
and validated is a major management issue addressed in appendix
II. In our reports on agency fiscal year 1999 and 2000 performance
plans, we How Can Verification concluded that they
provided limited confidence that performance data and Validation
Efforts would be credible, observing that the plans lacked
specific information on verification and validation procedures and
on data limitations.5 Our Be Clearly Reported?
assessment of the 2000 performance plans noted that most did not
identify actions that agencies were undertaking to compensate for
the lack of quality data. Our report on practices that can improve
performance plan usefulness to congressional and other
decsionmakers identified a number of ways 4For a more detailed
discussion of the relationship between data quality control and
business processes, see Thomas C. Redman, Data Quality for the
Information Age (Norwood, MA: Artech House, 1996). 5GAO/GGD/AIMD-
98-228, Sept. 8, 1998 and GAO/GGD/AIMD-99-215, July 20, 1999. Page
16 GAO/GGD-99-139
Verification and Validation of Performance Data Appendix I Key
Questions About Verification and Validation agencies could
describe their capacity to gather and use performance
information.6 These practices include * identifying internal and
external sources for data, * describing efforts to verify and
validate performance data, * identifying actions to compensate
for unavailable or low-quality data, and * discussing
implications of data limitations for assessing performance. In
addition to these, our current review found examples where
agencies Highlight Verification and enhanced the
communication of verification and validation approaches by
Validation Procedures highlighting them and the data
source being verified and validated. We also observed
opportunities for agencies to enhance the credibility of their
performance plans through more emphasis on verification and
validation procedures already in place. For example, the
Department of Education (ED) in its fiscal year 2000 plan used a
format that reflects a number of practices to help communicate
verification and validation approaches. Figure 2 shows the format
ED used to explicitly define each indicator and comment on its
background, succinctly describe the implications of a data
limitation, briefly present verification and validation
information, and identify the data source. A similar format was
used by DOT in an appendix to provide information on the data
source, verification and validation procedures, and limitations
for each measure. In addition to presenting verification and
validation specific to individual measures and data sources, ED
and DOT used a separate section to highlight general verification
and validation procedures that applied across a number of measures
or data sources. 6Agency Performance Plans: Examples of Practices
That Can Improve Usefulness to Decisionmakers (GAO/GGD/AIMD-99-69,
Feb. 26, 1999). Page 17
GAO/GGD-99-139 Verification and Validation of Performance Data
Appendix I Key Questions About Verification and Validation Figure
I.1: Sample Presentation Format From ED's Fiscal Year 2000
Performance Plan Page 18 GAO/GGD-99-139
Verification and Validation of Performance Data Appendix I Key
Questions About Verification and Validation Some agencies'
performance plans provide descriptions of planned quality Report
Key Approaches control procedures, without including some
ongoing procedures whose Already in Place description
could have increased the credibility of their measures. For
example, ED's fiscal year 2000 plan does not describe the
extensive quality control procedures already in place for its
ongoing national student testing program, the National Assessment
of Educational Progress. The Department plans to use measures from
this program for several key indicators of major objectives, such
as Indicator 2: " Students in high poverty schools will show
continuous improvement in achieving proficiency levels comparable
to those for the nation." This program is managed by the National
Center for Educational Statistics, using credible procedures and
expert involvement that could have been summarized in ED's plan.
As another example, the Office of Personnel Management's (OPM)
plan does not mention several existing quality control procedures
in place for a management information system that will provide a
number of indicators for OPM's Retirement and Insurance Service.
The plan does briefly mention that verification is undertaken by
its Quality Assurance Division, but does not describe approaches
used by program management, such as the use of a "physical
inventory" to check work processing statistics and accuracy checks
on death claims processing. Page 19 GAO/GGD-99-
139 Verification and Validation of Performance Data Appendix II
Fostering Organizational Commitment and Capacity for Data Quality
Obtaining quality performance information is an agencywide
management issue, as well as one requiring special attention from
technical and program staff. Management needs to create a climate
that encourages the needed coordination, resource allocation, and
attention to data quality issues that enable improvements in data
quality. Several agencies are making efforts to stimulate such a
commitment to obtaining, maintaining, and using good information
and to developing the organization's capacity to do so. The
approaches that agencies are adopting to foster organizational
commitment and capacity are shown in figure II.1 and discussed
below. Figure II.1: Approaches to Fostering Organizational
Commitment and Capacity Senior agency executives play an important
role in fostering program Communicate Support management and staff
commitment to data quality. For agency staff and for Quality Data
mid-level managers to put priority on data quality, they need to
see that senior management values and will use quality performance
information for decision-making. We learned from agency officials
that data quality is a higher priority when program staff and
management see that data will be used for management. Senior
executives can provide confidence that they value and will use
good quality data by communicating its importance, making data
quality an organizational goal, creating a climate of managing for
results, and providing technical and financial support. For
example, in response to an audit by the Department of Veterans
Affairs VBA Senior Management (VA), Office of
the Inspector General, senior officials of the Veterans Emphasized
the Importance Benefits Administration (VBA) have used
presentations and written of Accurate Data
communications to emphasize to staff the importance of accurate
data. The Inspector General concluded that data on the timeliness
of processing veterans' claims for benefits were not accurate
enough to provide Page 20 GAO/GGD-99-139
Verification and Validation of Performance Data Appendix II
Fostering Organizational Commitment and Capacity for Data Quality
meaningful measures of VBA's performance. Senior management
acknowledged to their staff that data were inaccurate and
emphasized the implications of inaccurate data. They also asked
staff to undertake reviews to ensure the accuracy of management
reports. Management provided staff with a list of unacceptable
practices that influence data accuracy and were instructed to stop
those practices immediately. Agency officials reported that the
communications from senior management resulted in increased
attention to data quality and in improvements to data accuracy,
such as more accurate recording of the time taken to process
compensation claims. VBA has also moved to foster an
organizational commitment to data quality through establishing a
related organizational goal in its strategic plan, which is that
"VBA's data systems will be reliable, timely, accurate,
integrated, honest, and flexible."1 Progress in managing for
results also appears to have resulted in greater VHA Holds
Managers attention to data quality at the Veterans
Health Administration (VHA). One Accountable for Program
strategy employed by VHA to encourage managers to focus on results
was Performance the initiation of a
performance contract system. In this system, the Under Secretary
for Health negotiates performance agreements with all senior
executives in VHA that hold them accountable for quantifiable
performance targets. Although these targets do not include ones
for data quality, VHA officials told us that assessing managers'
performance against them has resulted in greater attention to data
quality. In another example, Environmental Protection Agency (EPA)
management EPA Provides Technical and created an initiative that
included the provision of technical and financial Financial
Support to support for improving data quality. EPA's
One-Stop Program is a long-term Improve Data Quality
effort to develop a coherent overall environmental reporting
system to address reporting burden and lack of integrated data.
EPA will provide technical support and financial assistance to
states for developing information management infrastructure and
processes, including the adoption of standard data elements.
Fostering organizational capacity to produce and maintain good
quality Review Organizational performance information may require
assessing existing organizational Capacities and
capacities and procedures using external or internal reviews.
Organizational capacities that might be assessed include the
appropriate Procedures for Data location of
responsibilities for integrating and coordinating data; sufficient
Collection and Use staff and expertise to fulfill
these responsibilities; appropriate hardware 11Veterans Benefits
Administration, Roadmap to Excellence: Planning the Journey (May
29, 1998). Page 21 GAO/GGD-
99-139 Verification and Validation of Performance Data Appendix II
Fostering Organizational Commitment and Capacity for Data Quality
and software; and resources for building, upgrading, and
maintaining the data systems. For example, EPA charged a task
force of senior managers with Internal EPA Review
redesigning the agency's internal management structure to better
meet its Proposed Restructuring of new information
demands. In carrying out its charge, the task force Information
Responsibilities consulted with EPA employees, external
stakeholders, and the states. The report of the task force
recommended establishing a single program manager for information
management and policy combined with strengthening information
resources management and technology functions. In response to the
recommendations, the agency has planned the establishment of an
Information Office that would bring together information
management functions previously housed separately. The new office
is to contain a new Quality and Information Council with a role
that includes the provision of agencywide strategic direction and
advice on quality and data collection. An Information Quality
Staff is to support the Council. Legislation establishing the
Bureau of Transportation Statistics (BTS) DOT Legislation Required
called for the National Academy of Sciences to conduct an external
review External Panel Review of of the adequacy of data
collection procedures and capabilities of the Data Collection
Capabilities Department of Transportation (DOT). A panel of
experts was subsequently appointed to examine the functions that
BTS could or did perform and the resources and capabilities it had
to carry out those functions. The study report noted a number of
areas for improvement, including the need to build and maintain a
strong statistical and technical staff, which is being implemented
by DOT. Coordination of data quality efforts across systems or
offices can be a key Facilitate Agencywide issue in agencies.
Reporting on annual performance goals may require Coordination and
integrating data from different systems, which requires
coordination across organizational units. Cooperation The agencies
we contacted have numerous data systems that collect the data used
for agency performance measures. We found that, often, these
systems were initially constructed to meet the management needs of
specific programs, sometimes in another organizational unit.
Consequently, they may collect different data elements or use
different data definitions and standards, even for the same data
element. The involvement of higher level administrators may be
needed to facilitate the necessary coordination among semi-
independent organizational units and to obtain agreement on the
division of responsibilities. The agencies we reviewed had
developed a variety of mechanisms to facilitate the Page 22
GAO/GGD-99-139 Verification and Validation of Performance Data
Appendix II Fostering Organizational Commitment and Capacity for
Data Quality coordination and cooperation needed for good quality
data in these circumstances. For example, the Veterans Health
Administration has established a Data VHA's Data Quality Council
Quality Council, chaired by the Deputy Under Secretary for Health,
with a to Ensure Coordination mandate to ensure open
discussion and greater collaboration in the area of data quality
policy development and implementation. The Council is to include
representatives from headquarters as well as regional and field
offices. Among its specific responsibilities is ensuring the
coordination and communication of major national data quality
issues and initiatives. The Department of Education (ED) is
initiating several activities to foster Several ED Groups to
agencywide implementation of data standards. Several groups
contribute Coordinate Agencywide to the coordination
of this effort, including a strategic planning team; a
Implementation of Data panel to review the individual
performance plans submitted by each ED Standards
office, including a review of data sources and quality; and a work
group to develop data quality standards. Coordinating the
perspectives of multiple organizational stakeholders may OPM Uses
Work Group to enhance the validity of data elements
chosen to provide a performance Coordinate Performance
measure. For example, the Office of Personnel Management (OPM)
used a Measure Development work group involving
several organizational units in developing and approving survey
items for assessing OPM's performance on federal personnel policy
issues. Many people "touch" performance data, including suppliers
and creators of Assign Clear the data, those
who store and process data, and those who use it. Data are
Responsibilities for more likely to be of high
quality when it is clear who is responsible for each step in data
creation and maintenance, from the initial specification Various
Aspects of the and definition of data elements to correctly
entering data about clients; Data
providing training for and supervision of those who enter data;
transferring data from initial to final formats; and appropriately
analyzing and reporting the performance measures. A primary
responsibility for the quality of a program's data rests with the
manager of that program. Often, performance information is
directly collected by the operating components and may be used for
managing those programs. Because these data are also used for
decision-making by other levels of management and for reporting to
Congress, managers' direct responsibility for data quality has
broader implications. Several agencies are explicitly holding
immediate program managers and their divisional administrators
accountable for the quality of data from their programs. Page 23
GAO/GGD-99-139 Verification and Validation of Performance Data
Appendix II Fostering Organizational Commitment and Capacity for
Data Quality For example, the Department of Education is planning
to hold managers ED Is Requiring Managers accountable by
requiring them to attest to the quality of the program data to
Certify That Their Data used for performance measures. ED
has developed detailed data quality Meet the Standards
standards and procedures for implementing this requirement. ED's
evaluation office will also provide support services, such as
training in the application of data standards for performance
measurement. If they cannot certify that the data for a
performance measure meet the standards, the managers are to
provide plans for bringing the data up to standard. EPA provides
an example of agency efforts to clearly assign EPA Assigns
Responsibility responsibilities for various aspects of data
quality. Headquarters' sponsors to Sponsors and Producers
of data in EPA's Comprehensive Environmental Response,
Compensation, of Data and Liability
Information System database are responsible for identifying and
defining needed data elements, and the regional manager who
produces the data is responsible for reviewing, verifying, and
validating the data for this system. An Information
Management/Program Measurement Center under EPA's Office of
Emergency and Remedial Response is assigned a variety of
responsibilities for the completeness, accuracy, integrity, and
accessibility of data. An organizational capacity for objectivity
and independence in key data Adopt Mechanisms
collection, management, and assessment processes can help create a
That Encourage climate that fosters data
quality. Fostering objectivity and independence as a protection
against bias is a major principle of several disciplines,
Objectivity and including auditing, scientific
research, and program evaluation. Independence in Collecting and
Managing Data We found instances of a deliberate management
strategy to introduce OPM's RIS Operates
mechanisms for fostering independence in data collection and
Independently From management. At the Office of
Personnel Management, the Retirement and Program Offices
Insurance Service's (RIS) Management Information Branch operates
independently from the relevant program offices and reports
directly to the RIS Associate Director through the Assistant
Director for Systems, Finance, and Administration. OPM's fiscal
year 2000 performance plan notes that this arrangement is part of
its strategy "to ensure the integrity of the performance
indicators" derived from its comprehensive management information
system, which is used to monitor and report output (business
process) measures. Page 24 GAO/GGD-99-139
Verification and Validation of Performance Data Appendix II
Fostering Organizational Commitment and Capacity for Data Quality
At the Department of Education, in addition to holding program
managers ED Is Planning Several responsible for their
program's data, several mechanisms are being Mechanisms for the
planned to ensure independent review of the data submitted,
including Independent Review of Data involvement and review by
staff of the National Center for Educational Statistics, Inspector
General's Office, and Planning and Evaluation Service. Both
external sources and our prior publications concerning results
Provide Responsible management have emphasized the
importance of training managers about Staff With Training and
measurement issues so they can implement performance management.2
Understandably managers whose prior job roles emphasized other
tasks, Guidance for Needed such as awarding and
managing portfolios of grants or contracts, may lack Skills and
Knowledge skills in using performance data-that is,
managing for results. Without assistance, management and staff may
not understand how measurement is best implemented or how to
correctly interpret performance information, especially how to
acknowledge its limitations. We have previously noted that " . . .
staff at all levels of an agency must be skilled in . . .
performance measurement and the use of performance information in
decision-making. Training has proven to be an important tool for
agencies that want to change their cultures."3 The Department of
Education's Inspector General reported on the status ED Plans to
Train Managers of ED's implementation of the Results Act before
March 1998. The Inspector General found that "not having a
sufficient number of staff qualified in information processing,
evaluation and reporting," and "the difficulty of analyzing and
interpreting performance measurement data" were two of the three
most frequently identified barriers to successful implementation
of the Results Act identified by 27 key staff they interviewed. ED
plans to provide several types of training opportunities for its
managers, including conducting workshops provided by a training
contractor, one-on-one "coaching" with managers of the largest
programs, and having evaluators review managers' self-ratings of
data quality to ask questions about weaknesses they may have
overlooked. 2Kathryn E. Newcome, "Comments on the Future for
Performance-Based Management in the U.S. Federal Government" in
Federal Committee on Statistical Methodology, Office of Management
and Budget, Statistical Policy Working Papers (#28: Seminar on
Interagency Coordination and Cooperation), pp. 148-49, and Report
of the Auditor General of Canada, "Moving Toward Managing for
Results" (Oct. 1997), pp. 11-26. 3Executive Guide: Effectively
Implementing the Government Performance and Results Act (GAO/GGD-
96-118, June 1996), p. 42. Page 25
GAO/GGD-99-139 Verification and Validation of Performance Data
Appendix II Fostering Organizational Commitment and Capacity for
Data Quality Some agencies are already developing training and
guidance for managers The VA Inspector General to build the
needed knowledge and skills. For example, the VA Office of Has
Issued a Program the Inspector General has issued a
program manager's guide on the Manager's Guide on
auditor's approach to auditable performance measures. By
highlighting the Auditable Performance requirements for
data collection and the common problems found in its audits, the
Inspector General intends to contribute to program managers'
Measures awareness and better enable them to
plan for their data collection and performance reporting
activities. The approaches we described in this section can be
part of an integrated An Example of strategy for
fostering the agency's commitment to enhanced data quality.
Agencywide Capacity- As discussed, the Department of Education is
embarking on a major effort to make high-quality performance data
an agencywide priority. A summary building at ED
of its plans illustrates an integrated strategy that incorporates
multiple approaches. Senior ED managers have made obtaining valid
and verifiable data a departmental priority. This is documented by
the inclusion of the following indicator in ED's fiscal year 2000
performance plan: "By 2000, all ED program managers will assert
that the data used for their program's performance measurement are
reliable, valid, and timely, or will have plans for improvement.
Several groups provide related coordination. These include a
strategic planning team; a panel to review all performance plans,
including a review of data sources and quality; and a work group
to develop data quality standards. In carrying out this priority,
the Department of Education * has developed an explicit set of
"Standards for Evaluating the Quality of Performance
Indicators/Measures," which includes definitions, examples of
meeting or failing to meet each standard, and possible methods for
data checking, for each of six standards; * is requiring program
managers to certify the data quality for each performance
indicator, using a standard rating system, or to provide plans for
bringing data quality up to the standards; * has developed an
explicit plan for implementing the data standards, which documents
the detailed steps to be followed; * is providing several types
of training for program managers on the data standards and their
implementation; * is using independent oversight by the
evaluation office and the Inspector General to provide concurrence
with program managers' assessment of the Page 26
GAO/GGD-99-139 Verification and Validation of Performance Data
Appendix II Fostering Organizational Commitment and Capacity for
Data Quality quality of their data and to issue a Program Data
Quality Report Card summarizing data status; * is developing new
integrated data systems for elementary and secondary data to
coordinate data definitions, collection, and reporting among ED's
programs and state and local education agencies; and * is
communicating with Congress on statutory changes needed to support
the Department's reporting for the Results Act; for example,
making recommendations for the reauthorization of the Elementary
and Secondary Education Act to limit data required from states to
the data elements essential for Results Act reporting. Page 27
GAO/GGD-99-139 Verification and Validation of Performance Data
Appendix III Assessing the Quality of Existing Data One strategy
for verifying and validating proposed performance data is to
assess the quality of current data to identify problems that may
affect their use. Assessments might target specific measures in
the performance plan or more broadly assess major data systems and
their problems. Examples of both broad and narrow data assessments
are presented below. If an assessment shows adequate data quality,
the agency will need to reassess performance data periodically to
verify that data quality is maintained. Such quality assessments
can examine each measure along the relevant dimensions of data
quality, such as those described in appendix I, for example. In
assessing data quality, the agency may first need to establish the
appropriate quality dimensions for its data because these may
differ for various types of programs and data sources. Several
approaches for assesing existing performance information are shown
in figure III.1 and discussed below. These are intended as an
initial "menu of approaches," not a checklist of requirements.
Each agency needs to choose approaches that fit the intended uses
of its performance information, nature of its data and data
systems, resources available for assessing the data, and initial
diagnosis of the extent of problems in current data systems.
Figure III.1: Approaches to Assessing the Quality of Existing
Data For some types of performance measures, quality assurance
procedures Build Data Quality can be built
into the agency's normal workflow and managerial oversight.
Assessment Into These may be
appropriate, for example, when the performance data are derived
from information systems used to manage the workflow, such as
Normal Work benefits or claims
determination. This approach is consistent with advice Processes,
Including from the business world to design
quality into data systems for managing Ongoing Reviews or
business processes by using methods that make data error detection
and Inspections correction a normal
part of agency operations. Page 28 GAO/GGD-99-139
Verification and Validation of Performance Data Appendix III
Assessing the Quality of Existing Data For example, the Office of
Personnel Management's (OPM) Retirement OPM's RIS Routinely
and Insurance Service (RIS) routinely verifies the accuracy of
federal Verifies the Accuracy of Its retiree benefit
calculations. Several times each year, its Quality Assurance
Calculations and Provides Division reviews a statistically
representative sample of completed Feedback to Managers
calculations and draws conclusions as to whether each claim was
merited or not. Division staff provide feedback from this
verification to help managers maintain quality control over the
accuracy of benefits awarded. OPM staff stated that they now have
a 95-percent accuracy rate from this process. The aggregate data
from these accuracy checks are also now used as a performance
measure to meet the requirements of the Results Act. A major role
of the Department of Veterans Affairs' (VA) Compensation VBA
Annually Reviews and Pension programs is to process
disability claims from veterans. This Cases Selected Randomly
program has built accuracy checks into its normal work processes
since From Regional Offices 1992. Under a new quality
review system, the Systematic Technical Accuracy Review (STAR),
VBA headquarters will annually review about 7,400 cases selected
from regional offices. In addition, VBA will require each regional
office to review samples of its own work products using STAR
procedures. The purpose of STAR is "to improve the accuracy of
compensation and pension claims by providing current and
diagnostic information about the accuracy of work being produced
at the field stations."1Data from the STAR system will also be
used for the program's performance reporting. The Environmental
Protection Agency's (EPA) Science Advisory Board EPA's Science
Advisory was established to provide independent
scientific and engineering advice Board Conducts Peer
to the EPA Administrator on the technical basis for EPA
regulations. Its Reviews members include
scientists, engineers, and economists from academia, industry, and
the environmental community. The board conducts scientific peer
reviews to assess the technical merit of agency positions. These
reviews include whether data are of sufficient quality to support
environmental measures and whether proposed measurement models and
methods are appropriate. 1Department of Veterans Affairs, Fiscal
Year 2000 Budget Submission, Departmental Performance Plan, p. 49.
Page 29 GAO/GGD-99-139
Verification and Validation of Performance Data Appendix III
Assessing the Quality of Existing Data Use of electronic data
management and processing systems normally Use Software Checks
includes a variety of automated checks on data values and
integrity. These and Edits of Data on can include built-
in "range checks," which notify the data manager if an entered
value falls outside of the expected range for that data element;
Computer Systems and consistency checks among several data
elements that should be Review Their consistent,
such as age and current enrollment in school; procedures for
Implementation ensuring the integrity of data files
and their links to other files; and overall system controls for
data security and integrity. Assessing the quality of existing
data in such computerized systems involves reviewing the
implementation of these procedures. Such a review would start with
the detailed documentation for each data system to assess the
completeness of the intended software checks and edits. The actual
procedures would then be reviewed to ensure that they are being
carried out consistently. Finally, the results from the data
checks would be examined. The assessment would include such things
as the percent of the data initially entered that fall outside the
range and consistency checks and the implementation of procedures
for correcting the data. Our publication, Assessing the
Reliability of Computer-Processed Data (GAO/OP-8.1.3, Sept. 1990),
discusses the issues concerned with this approach, and provides
several checklists of useful types of data tests and computer
system controls in its appendixes I and II. The Department of
Education's (ED) publication on Standards for Contractors Must
Apply Education Data Collection and Reporting contains a
major section on data ED's Standards on Data preparation
and reporting, with nine detailed sets of standards for these
Preparation and Reporting processes, including designing data
processing systems, testing data processing systems, and
documenting data processing activities. ED staff indicated that
the contractors who conduct the projects used to provide data for
performance measures must apply these standards. Current users of
data systems and their results may have valuable Use Feedback From
experience with the strengths and weaknesses of existing data and
can Data Users and Other provide insights into the data's
credibility with external audiences. These users and stakeholders
can include agency staff members in program or Stakeholders
statistical offices; providers of data, such as state agencies or
local grantees; academics or "think tank" staff who use the data
for policy analysis; and industry representatives who base plans
or decisions on comparative or trend statistics from the data.
Page 30 GAO/GGD-99-139 Verification and
Validation of Performance Data Appendix III Assessing the Quality
of Existing Data For example, the Environmental Protection Agency
seeks stakeholder EPA Has Several Methods feedback on the
quality and usefulness of its performance data in several for
Obtaining Data User ways: customer consultations, posting
feedback forms on its Internet site, Feedback
and sending data to users and providers for verification. To
obtain "customer" feedback, the Center for Environmental
Information and Statistics conducted meetings with national,
regional, state, and local environmental data users to ask what
information they need and how they would like to access it. In
addition, the participants expressed concerns with the accuracy of
data entry, transmittal, and agency reporting. EPA posts a wide
variety of environmental information on its Internet Web site. In
particular, the site's Envirofacts Warehouse provides a single
point of access to environmental data maintained by EPA. The site
and each data source link to a feedback form that invites
questions or comments about Envirofacts databases. EPA also
verifies some data by sending it back to its originators for
comment. The agency's pilot Sector Facility Indexing Project
includes information on inspections of regulated facilities and
noncompliance with regulations. As part of its process for
verifying the data, EPA sent each facility a copy of its
compliance and enforcement data for review and comment to make
sure mistakes were caught before the information was released.
Using stakeholders to provide feedback on data collection and VHA
Convened a Data management problems, the Veterans Health
Administration (VHA) Quality Summit to Elicit convened a 3-day
Data Quality Summit in December 1998 to bring together Feedback
From staff from its network of veterans hospitals,
information systems, and Stakeholders central
office staff. Prior to the summit, participants were asked to
prepare papers on data quality issues that they believed affected
the organization. Examples of issues identified by participants
included coding problems, data definitions, and data correction
and consistency. The Data Quality Summit obtained input on
potential solutions that would meet the needs of the multiple
users of the data systems. Follow-up work groups are to develop
action plans for needed improvements. For one of the National
Science Foundation's (NSF) science education NSF Uses a Contractor
to programs, a contractor was used to obtain feedback on the
validity of data Check Validity of Data reported annually by
each project. To confirm that these data elements Reported by
Projects incorporated the intended meaning, a contractor
conducted an informal telephone survey of 15 projects, asking
project evaluators about their understanding of the questions used
in reporting the data items. The Page 31
GAO/GGD-99-139 Verification and Validation of Performance Data
Appendix III Assessing the Quality of Existing Data contractor
also collected more detailed information about the procedures used
in collecting the data locally and identified problems that
projects were having with individual items. The contractor
reported that respondents generally understood the data
definitions, concepts, and time frames that had been established
to govern responses to individual items. A fourth approach to
assessing the quality of existing data is to compare Compare With
Other the performance information with data from
other sources. Comparisons Sources of Similar can
serve several different purposes. One purpose is to assess the
validity of the performance measure by comparing the data elements
or source to Data or Program be used with related
data elements from another source. Another purpose Evaluations
is to test the accuracy of data from an ongoing system with data
from a more rigorously collected source that may be available only
periodically, such as a one-time evaluation. One example is
provided by the Office of Personnel Management's OPM's RIS
Compares Retirement and Insurance Service, which
processes about 4 million paper Management Information
items each year to manage the federal retirement system. To cross-
check System Data With Periodic the accuracy of the
performance statistics in their management "Physical Inventories"
of information system, the central office staff reports
that they periodically request a "physical inventory" of pending
work in each local office at the Paper Documents
end of a week. They compare actual counts of hard-copy documents
on hand at that point with that office's statistics generated by
the management information system for that week. If there are
discrepancies, the central staff works with local managers to
avoid duplicate counting and other errors. Useful comparison among
data sources can include analysts' judgments. VHA Compared Data
From Staff at the Veterans Health Administration
reported that they compare Program Offices With data
from program offices with more aggregated data from their central
Centrally Aggregated Data systems for assessing health care
"capacity" indicators. If the comparison reveals inconsistencies
in these sources, they reconcile differences by contacting the
relevant program managers to learn reasons for the differences and
to reach consensus on the most accurate numbers for the intended
presentation. For data elements drawn from data systems in regular
use, a key Obtain Verification by assessment step could
be the verification of the accuracy of results by an Independent
Parties, external, independent examiner, such as a
professional body or the agency's Inspector General. A reported
result can be checked for Including the Office of completeness,
consistency, and accuracy by tracing the data, or a the Inspector
General representative sample of data, back to their
original source. Verification can involve analyzing whether the
end data match the initial data source. It Page 32
GAO/GGD-99-139 Verification and Validation of Performance Data
Appendix III Assessing the Quality of Existing Data can also
involve assessing whether data collection and transformation
procedures are fully documented and followed consistently. For
example, several measures for the Department of Education's ED's
Inspector General elementary and secondary education
assistance programs will be provided Traced Data Flow and
by state educational agencies (SEAs). To assess the accuracy,
Control in Several State completeness, methods of
calculation, and presentation of targeted Educational Agencies
educational data elements for example states, ED's Inspector
General conducted field assessments in four SEAs in early 1999.
The Inspector General's staff conducted interviews with state
officials and attempted to trace the data flow and data control
processes in place in each state. This exploratory work toward
data verification is intended to identify processes used by SEAs
to accumulate and report performance data to ED, to identify
limitations in the data submitted, and to describe any barriers to
improving data quality. This assessment also provides background
for a major redevelopment of joint data collection efforts between
ED and its state and local partners. The Inspector General for the
Department of Veterans Affairs has focused VA's Inspector General
Is on audits of key performance measures. With input from
management, the Assessing Critical Data Inspector
General identified a subset of 11 performance measures Elements
considered most critical for measuring the agency's performance.
The initial audit focused on data for three measures relating to
the timeliness achieved by the Veterans Benefits Administration in
processing claims from veterans for disability compensation and
pension benefits. The audit assessed the data for validity,
reliability, and integrity (the extent to which the data could not
be "gamed" or manipulated), in accordance with guidance contained
in our report, Assessing the Reliability of Computer Processed
Data (GAO/OP-8.1.3, Sept. 1990). The Inspector General compared
source documents with information on automated systems for three
random samples of claims completed in fiscal year 1997. The audit
found that "more than 30 percent of the records in each of the
three samples contained inaccurate or misleading data." VBA
administrators have cited the findings as an impetus for rigorous
data improvement efforts. Use of data that are "certified" by an
external, professional body is another OPM Uses External
means for independent verification. For example, the Office of
Personnel Certification of Health Care Management, which
administers the federal employees' health insurance Data
program, works closely with the professional organization for
improving quality in managed health care, the National Committee
for Quality Page 33 GAO/GGD-99-139 Verification
and Validation of Performance Data Appendix III Assessing the
Quality of Existing Data Assurance (NCQA). OPM requires its health
insurance carriers to submit scores for the Health Plan Employer
Data and Information Set (HEDIS), which is managed by NCQA. HEDIS
is a set of standardized health care quality measures used to
compare managed care health plans. To ensure that HEDIS quality
specifications are met, NCQA has developed a data auditing
procedure using licensed organizations and certified auditors for
assessing carriers' nonfinancial data elements. The audit includes
verifying a sample of HEDIS measures to confirm that HEDIS results
are based on accurate source information. The process results in a
certification rating of "Report," "Not report," or "Not
applicable" for each measure reviewed. As a result of using the
approaches outlined above, or from other data Consequences From
assessment procedures, the agency will be able to identify data of
Assessing the Quality adequate quality for some measures, as
well as gaps and limitations in some data elements planned for use
as performance measures. For some of Existing Data of the
limitations, the approaches identified in appendix IV can be
undertaken to provide more credible data. In other circumstances,
the agency may decide that it needs to substantially change its
data acquisition process or create a new data system to "build
quality" into performance data, which is addressed in appendix V.
Page 34 GAO/GGD-99-139 Verification and
Validation of Performance Data Appendix IV Responding to Data
Limitations Agencies have undertaken a variety of approaches to
assessing the quality of existing data, as discussed in appendix
III. Assessments of data quality do not lead to improved data for
accountability and program management unless steps are taken to
respond to the data limitations that are identified. Guidance for
assessing agencies' performance plans calls for them to identify
significant data limitations and to discuss the steps being taken
or proposed to address those limitations.1 In the report
summarizing observations on 1999 agency performance plans, we
found that "in general, agencies' annual performance plans did not
include discussions of known data limitations and strategies to
address them."2 Our assessment of the fiscal year 2000 plans found
that agencies generally do not identify actions they are taking to
compensate for the lack of quality data, nor do they discuss
implications for decision-making.3 Improving future performance
information, as outlined in appendix V, is one important response
to findings concerning data limitations. Appropriate agency
responses to directly address the data limitations, shown in
figure IV.1, are discussed below. Figure IV.1: Approaches for
Responding to Data Limitations Making stakeholders aware of
significant data limitations allows them to Report Data
judge the data's credibility for their intended use and to use the
data in Limitations and Their appropriate ways. All
data have limitations that may hinder their use for certain
purposes but still allow other uses. Stakeholders may not have
Implications for enough familiarity with the data
to recognize the significance of their Assessing Performance
shortcomings. Therefore, appropriate use of performance data may
be fostered by clearly communicating how and to what extent data
limitations 1The Results Act: An Evaluator's Guide to Assessing
Agency Plans (GAO/GGD-10.1.20, Apr. 1998), p. 45. 2GAO/GGD/AIMD-
98-228,Sept. 8, 1998. 3GAO/GGD/AIMD-99-215, July 20, 1999. Page 35
GAO/GGD-99-139 Verification and Validation of Performance Data
Appendix IV Responding to Data Limitations impact on assessments
of performance. Communicating the implications of data limitations
can involve specifically identifying appropriate and inappropriate
interpretations of the data. In response to a legislative
requirement that the Department of DOT's BTS Reports on Data
Transportation's (DOT) Bureau of Transportation Statistics (BTS)
identify Sources, Gaps, and information needs on
an ongoing basis, the Bureau published a report that Weakness
identified initial gaps in transportation statistics and proposed
strategic responses. The report identifies data gaps and
weaknesses in a variety of areas, such as transportation safety;
energy use; and the flow of people, goods, and vehicles. For
example, the report notes that transportation injuries are
underreported and that there are inconsistencies in how injuries
are reported, complicating the assessment of transportation
safety. The Department of Education (ED) includes a section on
"Limitations of ED Provides a Section on the Data" when
presenting each indicator in its performance plan. (See fig. Data
Limitations for Each I.1 in app. I for ED's presentation
format.) For some objectives, ED also Indicator in Its Annual Plan
discusses the reasons for and implications of these limitations as
they affect the verification and validation of the measure. In
addition, as part of their review and certification that data for
performance measures meet ED's data quality standards, program
managers are to identify any standards that are not met and steps
for correcting these limitations. An appendix in DOT's performance
plan for fiscal year 2000 contains a DOT's Performance Plan
section on limitations in the data sources for each performance
measure in Explains How Limited Data the plan. The discussions of
limitations for some performance measures May Still Be Useful
also include the implications of the limitations for performance
measurement. For example, the plan notes that because of the
judgment involved in assessing whether mariners are in distress,
the reported rate may overestimate the number of lives saved.
However, the plan argues that the reporting from year to year is
likely to be consistent, providing a reasonable estimate of
changes over time. In addition to describing some of its data
limitations, ED's performance ED Describes Challenges
plan provided a context for its efforts to address limitations by
describing Involved in Obtaining High- the challenges that
they faced. A detailed section on "measurement Quality Performance
Data challenges" describes data limitations derived from
the decentralized system of elementary and secondary education, in
which many national goals and objectives are under limited federal
control. Further, it discusses the need to measure programs with
overlapping goals but disjointed information systems as well as
identify knowledge gaps where the Department is attempting to
"measure the hard-to-measure." Page 36 GAO/GGD-
99-139 Verification and Validation of Performance Data Appendix IV
Responding to Data Limitations Sometimes, data limitations can be
overcome by conducting accepted Adjust or Supplement
statistical adjustments, such as statistical modeling or
estimating values Problematic Data for missing
data elements. However, statistical adjustments are sometimes
controversial and can be hard for nonspecialists to understand.
Appropriate use depends on a number of assumptions underlying each
adjustment procedure, whose application requires considerable
specialized expertise. One common data limitation is the inability
to get information on all cases DOT Uses Statistical
of interest. DOT reports the use of statistical adjustments to
compensate Adjustments to Compensate for this problem. Blood
alcohol consumption test results are not available for Missing
Data for all drivers and nonoccupants involved
in fatal crashes. Using important crash characteristics, such as
crash vehicle and person factors, the DOT's National Highway
Traffic Safety Administration (NHTSA) seeks to avoid an
undercounting of these fatalities by employing a statistical model
to adjust for missing data. Without this correction, the
percentage of alcohol- related highway fatalities would appear to
be lower than they actually are. The Schools and Staffing Survey,
used by ED for several performance ED Analyzes Nonresponses
measures, collects data from a variety of educational staff, such
as and Statistically Adjusts the teachers, administrators, and
librarians and for public and private schools. Reported Data
Even after rigorous survey administration procedures and telephone
follow-up, response rates differ among these components. To reduce
bias in reported results, the National Center for Educational
Statistics conducts analysis of the sources of nonresponse, then
uses statistical procedures to adjust the data reported. Comparing
information derived from data sources with different strengths Use
Multiple Data and weaknesses adds confidence to
judgments about performance. Sources With
Agencies may have access to two or more data sources that can
provide information on a given area of performance. Although each
data source Offsetting Strengths may have serious
limitations, confidence in results may be increased when and
Limitations each source provides the same
overall picture of performance. Combining data sources may also
provide a more complete picture of performance than can be
obtained from a single source. The Office of Personnel Management
is comparing results from three OPM Is Identifying
different surveys to identify consistencies and to stimulate
discussion of Consistencies and reasons for any
differences. The surveys examine federal employee and Discussing
Differences in human resource personnel satisfaction or
perceptions with regard to the Results of Three
human resource operations and OPM initiatives. They expect to
report on these analyses in next year's performance plan. Surveys
Page 37 GAO/GGD-99-139 Verification and
Validation of Performance Data Appendix IV Responding to Data
Limitations OPM's Employment Service operates an automated
Employment OPM Customer Satisfaction Information Service,
containing postings of federal job openings. The Data Validated by
Web system automatically collects several kinds of
data, including use statistics Master
and customer satisfaction feedback. The results concerning
customer satisfaction are validated qualitatively by the system
Web master from complaints received and any technical problems
identified with regard to recent system enhancements. Some data
limitations can be addressed by replacing the data source. In
Improve the Measure some cases, improving data
collection and management procedures, as by Using Another
described in appendix V, may correct the problem. Comparing data
with its original source and correcting the errors in existing
data may also be Source or New possible, for
example, if the limitations occur because of inaccurate data
Methods of coding and entry. However,
fixing existing data can be expensive, and Measurement
unless stakeholders require the historical data as a baseline, the
resources may be better used to find new information or new
methods of measurement. The National Highway Traffic Safety
Administration uses private industry NHTSA Uses Data From
data on vehicle registration rather than federal estimates,
believing that the Private Industry former more
closely reflect the actual mix of vehicles on highways. Federal
statistics are obtained from state information systems, which may
overcount certain vehicles if they have been transferred from one
state to another and show up in both states' files. The Veterans
Benefits Administration (VBA) has recently changed its VBA Has
Changed Its method for estimating the accuracy rate
for the processing of veterans' Method for Estimating the
compensation and pension claims. This rate is one of its
performance Accuracy Rate for Claims measures. VBA's
system for measuring accuracy had indicated an Processing
estimated 95-percent accuracy rate for the claims processing
activity. However, questions arose because the processing of
veterans' appeals of these initial decisions reversed about 19
percent of the appealed decisions and remanded about 47 percent
back for further development and reconsideration. The Systematic
Technical Accuracy Review (STAR) was implemented to improve the
accuracy of the work of compensation and benefits officers and to
provide information for measuring annual performance goals
concerning accuracy. Pilot tests of the new STAR system found only
a 64- percent accuracy rate in claims processing decisions.
Compared to the earlier system, the STAR system focuses more on
decisions that are likely to contain processing errors and uses a
stricter standard for computing accuracy rates. Page 38
GAO/GGD-99-139 Verification and Validation of Performance Data
Appendix V Building Quality Into the Development of Performance
Data Improving data quality by detecting and correcting errors
with existing data will not necessarily prevent future errors.
Assessments of existing data elements or systems to be used for
performance measures may reveal that improvements are needed in
current data systems or that new systems are needed. Agency
performance plans are expected to indicate any changes or
improvements being made to modify, improve, or expand the
capability of existing data systems or processes, according to the
Office of Management and Budget's (OMB) Circular A-11 guidance for
performance plans. Reporting data design and collection procedures
may be particularly useful when data are collected episodically,
rather than on an ongoing basis. In these circumstances, it may
not be feasible to verify the data by comparing them to an
original source or alternative data sources, such as in a
nonrecurring survey. Figure V.1 lists approaches that agencies can
take to build quality into their performance data. Figure V.1:
Approaches to Building Data Quality Into the Development of
Performance Data Page 39 GAO/GGD-99-139
Verification and Validation of Performance Data Appendix V
Building Quality Into the Development of Performance Data Several
agencies use findings from basic research to assess the validity
of Use Prior Research or potential data elements for measuring
intended performance. Research Analysis to Identify may
illuminate the relationships between the agency's strategies and
outcomes in the content area of the performance measure. Or,
appropriate Data Elements That measuring tools and
data collection procedures may be drawn from this Adequately
Represent literature or adapted to become more
compatible with the agency's needs. the Performance to Be Measured
For example, the Environmental Protection Agency's (EPA) Safe
Drinking EPA Uses Research on Water Program uses
research on the health risks associated with specific Health Risk
in Drinking levels of exposure to set standards for
maximum contaminant levels. The Water
agency measures its annual progress in ensuring that Americans
will have clean and safe drinking water by estimating and
reporting the percentage of the population served by water systems
that meet all health-based standards. The Office of Personnel
Management (OPM) uses several sample surveys OPM Used Research on
to assess federal agency human resources staff satisfaction, for
example, Customer Satisfaction with OPM technical
assistance and guidance materials. To develop valid items for its
survey instruments, OPM's Personnel Resources and Development
Center reviewed extensive research on "customer satisfaction" in
the fields of organizational psychology, management, and
marketing. From this literature, OPM identified nine underlying
service dimensions of customer satisfaction, including the
courtesy, knowledge, and timeliness of the service staff as well
as the extent of choice and quality for the specific service. OPM
developed a set of survey scales with 30 core items for these nine
dimensions, along with four general items about overall quality
and satisfaction. The core items were pretested with staff in
three agencies before the measures were included in OPM's customer
satisfaction surveys, used to provide measures in OPM's
performance plan. Selecting or developing valid data elements can
also be enhanced by Gain Agreement involving
others who collect or use the resulting data (stakeholders). This
Among Internal and step is particularly useful when
staff outside the agency will be the primary data collectors, such
as staff in state or local agencies or grantees. Such External
Stakeholders consultation helps to establish consensus on the data
elements that are About a Set of valid measures
of the underlying concept and that take into account the Measures
That Are varied local circumstances and resource
availability affecting the Valid for Their
consistency of data collection. Intended Use Page 40
GAO/GGD-99-139 Verification and Validation of Performance Data
Appendix V Building Quality Into the Development of Performance
Data For example, the Department of Education (ED) is facilitating
the An ED Panel Is Developing development of an Integrated
Performance and Benchmarking System an Integrated System for
(IPBS) for its elementary and secondary education programs.
Currently, State Reporting most ED programs have
separate reporting systems, with considerable overlap in the types
of information collected, but not always common definitions of key
terms, such as "student." Most states also collect similar types
of data, but often not in ways that allow them to compare with
other states. The IPBS initiative will seek agreement among states
and ED program managers for a common core of data elements. It is
currently in an exploratory phase, with representatives from two
states cochairing a panel in conjunction with staff from ED to
develop a system plan. Full national implementation is intended by
2004. ED also expects to award financial grants to states for
implementing the needed improvements. Further, such consultation
may be desirable even within an agency, to avoid overemphasis on
any single measure. Collecting data on only a limited aspect of
total performance may encourage management and staff to look for
ways to make performance appear better than it actually is.
Obtaining within-agency agreement on a more balanced set of
performance measures may help to minimize distortions that can
result from overemphasis on a single measure. For example,
Veterans Benefits Administration (VBA) officials told us that VBA
Is Developing an improved data quality was an
anticipated benefit of their adopting a Expanded, More Balanced
"balanced scorecard" approach to performance measurement. For this
Set of Performance approach, VBA staff are
developing an array of measures that capture the Measures
various elements of VBA's strategic vision, including measures of
timeliness, accuracy, unit cost, employee development, and
customer satisfaction. The new set of measures expands VBA's
previous emphasis on timeliness and productivity. Although
improved data quality is not the primary purpose for adopting a
more balanced set of measures, the officials we talked to believed
that this would be one benefit of the approach. Agencies find that
developing new or revised data systems involves a Plan, Document,
and number of aspects that need to be carefully
planned and carried out for the Implement the Details resulting
data to be valid and verifiable. These aspects can include the
exact specifications of the data elements to be collected, the
population or of the Data Collection sample of entities
from which to collect data in each location, the detailed and
Reporting Systems steps for data collection and manipulation in
each location, training for local data collectors, oversight
procedures for local supervision of data collection, and quality
standards to be employed in that oversight. After these efforts,
the subsequent reporting of validation and verification Page 41
GAO/GGD-99-139 Verification and Validation of Performance Data
Appendix V Building Quality Into the Development of Performance
Data methods in the performance plan could focus on the methods
employed to build in data quality, for example, by documenting the
steps undertaken in developing and implementing the data
collection system. To obtain consistent data from different
locales, detailed plans for the data ED's Even Start Family
definitions and data collection procedures are needed. This
planning may Literacy Program Illustrates involve local partners,
if they will be responsible for data collection. For Planning for
Data example, ED's Even Start Family Literacy
Program provides a Definitions and Data
multicomponent program for low-income families in more than 600
locations across the nation. ED staff report they have involved
the Collection grantees in developing
data definitions and changes in the data collection procedures as
they have evolved since 1989. ED uses a contractor to work with
the local projects in developing its reporting system, which has
five data reporting forms for various population groups in the
program. The data collection system is documented in a detailed
user's manual, which contains an explanation of every question in
every form, as well as instructions for using the automated data
entry system. The contractor maintains a toll-free telephone line
for answering questions about the data forms and communicates
immediately with a grantee if its data submission appears to
contain errors. Plans for data processing at a central level also
need to be developed and documented to ensure consistency among
multiple staff and over time, as turnover occurs among staff.
These plans include how the data will be transferred from the
individual collection sites, how it will be stored and processed,
and how it will be aggregated into the needed performance
measures. These developmental steps involve the technical staff
and data processing specialists from the several organizational
levels that will collect and manage the data. Implementing the
plans will also involve using the software checks and edits for
data on computer systems that are discussed in appendix III. For
example, EPA's Toxics Release Inventory is a database containing
EPA's Toxic Release industry-reported data or
estimates about releases of listed chemicals that Inventory
Illustrates exceed certain amounts. EPA's Center for
Environmental Information and Planning and Control of
Statistics indicates that every facility uses the same forms for
reports and Data Processing at the that input forms
are checked centrally for completeness, valid formats, chemical
identification numbers, and internal consistency. The agency
Central Level runs computer checks against
the reported data. When potential errors are identified,
facilities are notified to allow for correction. Page 42
GAO/GGD-99-139 Verification and Validation of Performance Data
Appendix V Building Quality Into the Development of Performance
Data The development of new or revised data systems can be aided
by utilizing ED Developed Quality relevant expertise and
professional standards to advise on both the Standards for
Performance content of the measures and the technical aspects
of information systems. Measures Using Relevant For example,
ED developed a brief set of draft "Standards for Evaluating
Expertise and Professional the Quality of Performance Indicator
Measures" that all the Department's programs will be required to
follow when reporting their performance Standards
data. To develop these standards, ED drew on internal expertise
from several disciplines, including both educational statistics
and auditing; used a contractor to collect examples of quality
standards; and had draft standards reviewed intensively by the
Department's Evaluation Review Panel, a group of external
evaluation experts from academia and state agencies. EPA requires
all its environmental programs to be supported by quality EPA
Environmental Data systems that comply fully with standards
for environmental data collection Collection Must
developed by the American Society for Quality and authorized by
the Demonstrate Conformity American National Standards
Institute (ANSI).1 EPA's policy requires the With ANSI Standards
development of Quality Management Plans for programs and Project
Assurance Project Plans for individual projects, as recommended in
the standards. The ED's National Center for Education Statistics
(NCES) has detailed ED's NCES Has Detailed standards and
specifications for designing, conducting, and analyzing Survey
Standards and educational surveys, including those
collecting data used for performance Specifications
measures.2 These standards are built into new contracts for data
collection, and quality control procedures are monitored by each
contract's technical officer, then documented in technical reports
for each survey. Each project also has a technical review panel,
which reviews the details of survey design and quality control
during data collection. Some agencies are trying to minimize data
entry and transmittal errors by NSF Is Using an Electronic
using or planning for electronic data systems, rather than using
paper- Web-Based Process to based data collection forms,
for initial data entry and transmittal to a Obtain Final Reports
From central location. For example, the National Science
Foundation (NSF) is Grantees implementing an
electronic, Web-based process for the submission of final reports
from its research grants. The on-line report format includes a
number of features to ensure appropriate data entry, including
hypertext 1American Society for Quality Control, American National
Standard: Specifications and Guidelines for Quality Systems for
Environmental Data Collection and Environmental Technology
Programs (ANSI/ ASQC E4-1994, Jan. 3, 1995). 2National Center for
Education Statistics, NCES Statistical Standards (Washington,
D.C.: Department of Education (NCES 92-021r)), 1992; Westat, Inc.,
and NCES, SEDCAR: Standards for Education Data Collection and
Reporting (Washington, DC: Department of Education (NCES 92-
022r)), 1991. Page 43
GAO/GGD-99-139 Verification and Validation of Performance Data
Appendix V Building Quality Into the Development of Performance
Data explanations and definitions as to what is needed in each
entry, automatic range checks to flag improper entries, and
immediate feedback to the grantee regarding invalid entries. The
electronically submitted reports are then reviewed by NSF's
content area project officers, who check whether entries are
reasonable and consistent with other information about that
project. NSF staff believe that this electronic report submission
leads to quicker turnaround time for data submission; simplified
and more accurate data entry; data more relevant for program
management; and therefore, more use of the data by program
managers. The quality of any information system, and of the
performance measures Provide Training and derived from
it, depends on the quality of the data being entered. To obtain
Quality Control the necessary consistency and
accuracy in data collection and entry, several agencies are
providing training and supervision of data collectors Supervision
for All as a part of their data quality procedures.
Such training helps to ensure Staff Who Collect and that
those collecting and entering the data have a common understanding
Enter Data, Especially of the meaning of each data element, the
conventions for using each at Local Levels
categorization or coding rule, the frequency with which data are
to be entered, and so on. If data collection will use electronic
forms, hands-on experience with sample cases to code and enter
during the training is desirable. For example, the ED's Even Start
program provides annual training on ED's Even Start Program
data collection issues for new grantees and new data management
staff Provides Annual Training who supervise local data
collection and entry. As an example of the on Data Collection
training activities covered, Even Start's training agenda for
spring 1998 included * orientation to the roles of various staff
members and contractors involved with the data collection used for
performance measures, * discussion of the use and findings from
similar data in prior evaluation reports to illustrate the
importance of accurate data collection, * directions and answers
to frequently asked questions about six types of data collection
forms, * tips about local data entry methods and schedules, and *
demonstration sessions and opportunities for hands-on practice in
using electronic data entry forms. In addition to this annual
training, evaluation and data collection concerns are discussed in
meetings of grantees. Page 44 GAO/GGD-99-139
Verification and Validation of Performance Data Appendix V
Building Quality Into the Development of Performance Data To
engage staff across the country in data quality issues, the
Department VHA Has Produced a of Veterans
Affairs' (VA) Veterans Health Administration (VHA) produced
Professional Training Video a training video called "I am Joe's
Data" for initial use at its Data Quality Summit. The video shows
the "travel" of data from a typical patient in a VA hospital
through various processing steps and illustrates the diverse ways
in which the data are used. This video is being distributed to
staff in VA hospitals to help them understand the importance of
their roles in the accuracy of data that is ultimately presented
to Congress. When data quality checks are performed on information
systems with Provide Feedback to ongoing data
collection, agencies may provide feedback about the types Data
Collectors on and frequencies of errors found to
those collecting and entering data. The feedback might list errors
found in the data submitted by the specific data Types of Errors
Found collection unit and provide comparison with error rates from
other units or by Data Checks from all units.
Sometimes the specific data entries with errors can be corrected;
in other cases, obtaining accurate data in the future may be the
objective. Feedback about data problems is sometimes combined with
feedback showing the actual aggregated data results from that
unit, so operating organizations see their concrete results along
with any data problems. For example, the ED's Even Start
contractor provides immediate ED's Even Start Contractor
telephone feedback about any data submission errors. Each project
also Provides Immediate receives a "project
profile" that summarizes the data for its own project, Telephone
Feedback on compared with other similar projects,
state averages, and national Errors
averages. The ED evaluation officer reported that such feedback
contributes to data quality by encouraging projects to get their
data in on time, with less data cleaning needed, and to be more
involved in properly implementing any changes needed in the data
collection procedures. Another example of the use of feedback
comes from the Office of OPM Uses Agency Data to
Personnel Management, which oversees the life insurance program
for Verify Contractor Claims federal employees.
According to OPM staff, initial death claims processing Processing
Data is done by a contracted life insurance
company, but OPM does a computerized "paid claims match," using
agency records to verify the contractor's claims processing data.
The results of these reviews are sent back to the contractor for
investigation of any discrepancies, and results are fed back to
the relevant managers within OPM. These data are also used in
training new staff on the types of cases that may lead to errors
in the adjudication of claims. Page 45 GAO/GGD-
99-139 Verification and Validation of Performance Data Appendix V
Building Quality Into the Development of Performance Data Before
reporting data as performance measures, it is often necessary to
Use Analytic Methods aggregate data from multiple
locations, to transform the raw data into a and Transformations
ratio or percentage, or otherwise process the data. Appropriate
for the Data Type and Measure Being Reported For example, when
reporting its performance measure for highway DOT Measures the
Rate of fatalities, the Department of Transportation (DOT)
uses the rate of Fatalities per Vehicle Miles highway-related
fatalities per 100 million vehicle miles traveled rather than the
"raw" number of fatalities. This ratio adjusts for a greater risk
of fatalities each year due to an expected approximately 2.2-
percent annual increase in miles driven. An example of aggregating
multiple data elements comes from the VHA Plans to Use Indexes
Veterans Health Administration, which plans to use several
indexes, to Report on the Quality of including the Chronic
Disease Care Index and the Prevention Index, to VA Health Care
Delivery report on the quality of its health care
delivery. An index that includes information on a number of health
areas allows the agency to provide an overall assessment of
performance. VA's fiscal year 2000 performance plan indicates that
both indexes measure how well VA follows nationally recognized
guidelines and recommendations for delivering clinical care to
veterans with chronic diseases and for primary prevention and
early detection. For each index, data about multiple relevant
conditions are extracted from a sample of individual patient
charts. The data are aggregated to form the indexes and
statistically evaluated for validity and reliability. Not all
agencies are depending entirely on using or building new An
Alternative quantitative data systems. The
National Science Foundation is developing Approach to
an alternative format for performance reporting that relies on
qualitative assessments by external reviewers, as permitted under
OMB's Circular A Performance 11 guidance.
NSF's procedures for these assessments illustrate some Assessment
at NSF issues in verifying and validating
qualitative methods to build quality into this alternative
practice. NSF is a federal agency that supports basic scientific
research and science education. It operates primarily by awarding
grants and cooperative agreements to individuals and groups in
research institutions. For its alternative assessment approach,
which is used for four major outcome goals on the advancement of
science, NSF developed descriptive standards to characterize
"successful" and "minimally effective" performance. For Page 46
GAO/GGD-99-139 Verification and Validation of Performance Data
Appendix V Building Quality Into the Development of Performance
Data example, for Outcome Goal 1: "Discoveries at and across the
frontier of science and engineering," the standards state the
program is * "successful when NSF awards lead to important
discoveries; new knowledge and techniques, both expected and
unexpected, within and across traditional disciplinary boundaries;
and high-potential links across these boundaries" and *
"minimally effective when there is a steady stream of outputs of
good scientific quality." A committee of external reviewers for
each scientific program will assess Committees of External
the program's research grant results by applying these standards,
using as Reviewers to Assess NSF evidence summary reports
and examples of results prepared by program Research Grant Results
staff. Each committee's report will be reviewed by several higher
level entities: the Directorate's chartered Advisory Committee of
external scientists, the Directorate's senior management, and the
Office of the Director. These procedures build on similar prior
peer review that focused primarily on improving the processes of
grantmaking, which has been very useful as a management tool,
according to an NSF official. NSF has built into its alternative
assessment procedures several methods Validation and Verification
to increase the credibility of reports of program performance.
First, NSF Built in for NSF's issued explicit
guidelines on how the review committees will be convened
Alternative Assessment and managed to help make the
process systematic. Second, the guidelines require that the
reviewers be "credible, independent experts who are able to
provide balanced and impartial assessments," with diversity among
scientific, institutional, geographic, and demographic
characteristics. Finally, the sequential layers of review for
scientific programs help to validate the judgments made in the
initial steps. The external review assessments ultimately depend,
however, on the selection of final project reports and other
materials provided by the program staff to reviewers. NSF guidance
does not require that the review include a balanced sample of
projects closed out during the years being reviewed; instead,
"examples may be selected to reflect the most significant
accomplishments in a program's portfolio of support." Agency
officials report that the reviewers will have access to all
information systems and will be encouraged to make their own
choice of examples. NSF intends to review this process and make
changes. Page 47 GAO/GGD-99-139 Verification and
Validation of Performance Data Page 48 GAO/GGD-99-139
Verification and Validation of Performance Data Ordering
Information The first copy of each GAO report and testimony is
free. Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order made out
to the Superintendent of Documents, when necessary. VISA and
MasterCard credit cards are accepted, also. Orders for 100 or more
copies to be mailed to a single address are discounted 25 percent.
Order by mail: U.S. General Accounting Office P.O. Box 37050
Washington, DC 20013 or visit: Room 1100 700 4th St. NW (corner of
4th and G Sts. NW) U.S. General Accounting Office Washington, DC
Orders may also be placed by calling (202) 512-6000 or by using
fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO
issues a list of newly available reports and testimony. To receive
facsimile copies of the daily list or any list from the past 30
days, please call (202) 512-6000 using a touch-tone phone. A
recorded menu will provide information on how to obtain these
lists. For information on how to access GAO reports on the
INTERNET, send e-mail message with "info" in the body to:
[email protected] or visit GAO's World Wide Web Home Page at:
http://www.gao.gov United States General Accounting Office
Bulk Rate Washington, D.C. 20548-0001 Postage & Fees Paid GAO
Permit No. G100 Official Business Penalty for Private Use $300
Address Correction Requested (966712)
*** End of document. ***