Managing for Results: Challenges in Producing Credible Performance
Information (Testimony, 03/22/2000, GAO/T-GGD/RCED-00-134).

This testimony presents information on the challenges that federal
agencies face in producing credible performance information and how
those challenges may affect performance reporting. GAO recently reported
that it is unlikely that agencies will have the reliable information
needed to assess whether goals are being met or specifically how
performance can be improved. (See GAO/GGD-00-52, Feb. 2000.) This
testimony (1) provides a governmentwide perspective on the credibility
of agencies' performance information; (2) discusses some of the
challenges faced by agencies, including the Department of
Transportation, the Environmental Protection Agency, and the General
Services Administration, in producing credible performance data; and (3)
highlights how agencies can use their performance reports to address
data credibility issues.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  T-GGD/RCED-00-134
     TITLE:  Managing for Results: Challenges in Producing Credible
	     Performance Information
      DATE:  03/22/2000
   SUBJECT:  Agency missions
	     Accountability
	     Interagency relations
	     Performance measures
	     Strategic planning
	     Data integrity
	     Reporting requirements
	     Congressional/executive relations
	     Program evaluation
	     Productivity in government
IDENTIFIER:  GPRA
	     Government Performance and Results Act

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  This text was extracted from a PDF file.        **
** Delineations within the text indicating chapter titles,      **
** headings, and bullets have not been preserved, and in some   **
** cases heading text has been incorrectly merged into          **
** body text in the adjacent column.  Graphic images have       **
** not been reproduced, but figure captions are included.       **
** Tables are included, but column deliniations have not been   **
** preserved.                                                   **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                                            **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************

GAO/T-GGD/RCED-00-134

MANAGING FOR RESULTS: Challenges in Producing Credible Performance Information (GAO/T-GGD/RCED-00-134)
MANAGING FOR RESULTS
Challenges in Producing Credible Performance Information
Statement of J. Christopher Mihm, Associate Director Federal Management and Workforce Issues General Government Division
United States General Accounting Office
GAO Testimony Before the Subcommittee on Oversight, Investigations,
and Emergency Management Committee on Transportation and Infrastructure House of Representatives
For Release on Delivery Expected at 10: 00 a. m. EST Wednesday March 22, 2000
GAO/ T- GGD/ RCED- 00- 134
Statement Managing for Results: Challenges in Producing Credible Performance Information
Page 1 GAO/ T- GGD/ RCED- 00- 134
Madam Chairman, Mr. Traficant, and Members of the Subcommittee: I am pleased to be here today to discuss the challenges federal agencies face in producing credible performance information including the Department of Transportation (DOT), the Environmental Protection Agency (EPA), and the General Services Administration (GSA) and the opportunities the Government Performance and Results Act (GPRA) provides for generating information to help Congress and other decisionmakers.
As you know, GPRA was passed in part out of Congress' frustration over the fact that congressional policymaking, spending decisions, and oversight had been seriously handicapped by agencies' lack of clear goals and adequate program performance and cost information. To remedy that situation, GPRA requires agencies to set multiyear strategic goals and corresponding annual performance goals, measure performance toward the achievement of those goals, and publicly report on their progress.
About 100 agencies, including DOT, EPA, and GSA, published a first set of strategic plans in 1997 and, as required, will issue updated plans by this September. These agencies also issued annual performance plans for fiscal years 1999 and 2000. Some agencies, such as EPA, have already published their plans for fiscal year 2001, and other plans will soon become available.
By the end of this month, agencies are to publish their first annual performance reports that, for the first time, will provide important information on the overall performance of federal programs. The issuance of these reports, therefore, represents a new and potentially more substantive stage in the implementation of GPRA. Performance reports offer Congress the opportunity to systematically assess agencies' actual performance on a governmentwide basis and to consider the specific steps that can be taken to improve performance and reduce costs.
Last month, we released our report on the challenges agencies face in producing credible performance information and how those challenges may affect performance reporting. 1 In that report, we state that it appears unlikely that agencies will consistently have for their first performance reports the reliable information needed to assess whether goals are being met or specifically how performance can be improved. Our work over the past several years has identified limitations in agencies' abilities to
1 Managing for Results: Challenges Agencies Face in Producing Credible Performance Information (

GAO/GGD-00-52
, Feb. 4, 2000).
Statement Managing for Results: Challenges in Producing Credible Performance Information
Page 2 GAO/ T- GGD/ RCED- 00- 134
produce credible data and identify performance improvements. These limitations are substantial and long- standing, and they will not be quickly or easily resolved. These limitations are, therefore, likely to be reflected in agencies' initial performance reports as they have been in the performance plans prepared to date.
As agreed with the Subcommittee, my statement today will address three topics. First, I will provide a governmentwide perspective on the credibility of agencies' performance information based on our assessment of agencies' performance plans. Second, I will discuss some of the challenges agencies face including DOT, EPA, and GSA in producing credible performance data. Third, I will highlight how agencies can use their performance reports to address data credibility issues. My comments today are based on our large body of work on agencies' performance data problems and related issues.
To efficiently and effectively operate, manage, and oversee programs and activities, agencies need reliable information during their planning efforts to set realistic goals and later, as programs are being implemented, to gauge their progress toward achieving those goals. Credible performance information is also essential for Congress and other decisionmakers to accurately assess agencies' progress towards the achievement of their goals.
As shown in figure 1, our analysis of agencies' fiscal year 2000 performance plans noted that most agencies including EPA and GSA provided only limited confidence that their performance information would be credible. 2 DOT was one of four agencies that provided general confidence that their performance information would be credible. 3
2 Managing for Results: Opportunities for Continued Improvements in Agencies' Performance Plans (GAO/ GGD/ AIMD- 99- 215, July 20, 1999). 3 In addition to DOT, the Department of Education, the Department of Justice, and the Social Security Administration provided general confidence in their fiscal year 2000 plans that their performance information would be credible. Confidence in the
Credibility of Agencies' Performance Information Is Limited
Statement Managing for Results: Challenges in Producing Credible Performance Information
Page 3 GAO/ T- GGD/ RCED- 00- 134
Source: GAO/ GGD/ AIMD- 99- 215, July 20, 1999.
Congress and other decisionmakers must have assurance that the program and financial data being used will be sufficiently timely, complete, accurate, useful, and consistent if these data are to inform decisionmaking. However, most agencies lacked information in their fiscal year 2000 performance plans on the procedures they would use to verify and validate performance information.
In addition, most agencies failed to include discussions of strategies to address known data limitations. We reported that when performance data are unavailable or of low quality, a performance plan would be more useful to Congress and other decisionmakers if it briefly discussed how the agency plans to deal with such limitations. Without such a discussion, Congress and other decisionmakers will have difficulty determining the implications for assessing the subsequent achievement of performance goals.
Agencies need to build the capacity to gather and use performance information to successfully measure and report progress toward intended results. However, our work over the past several years has identified limitations in agencies' abilities to produce credible performance data.
Figure 1: Confidence in Performance Data
Challenges Agencies Face in Producing Credible Performance Information
Statement Managing for Results: Challenges in Producing Credible Performance Information
Page 4 GAO/ T- GGD/ RCED- 00- 134
For example, in our June 1998 review of GSA's building security upgrade program, we found limitations with data quality in GSA's building security update tracking system, which is used to track the status of security upgrades at federal buildings. 4 We found that the system contained errors for 24 of the 53 buildings reviewed. A similar study conducted by GSA's Office of Inspector General (OIG) found that the status of security upgrades was not accurately reflected in 65 of the 120 buildings it reviewed. In 1999, we reexamined 31 of the buildings reviewed by either the OIG or us and found that GSA corrected many of the inaccuracies previously identified. 5 However, we found that the tracking system continued to contain errors. We found errors concerning 10 buildings, about one- third of those we visited. Data inaccuracies included overstating the number or extent of security upgrades completed and misstating the operating status of security improvements. Without valid data, it is unlikely that GSA will be able to assess, first, the costs versus the benefits of upgrades; and, second, the extent to which completed upgrades have contributed to increased security or reduced vulnerability to the greatest threats to federal office buildings.
In general, we have found that limitations in performance data relate to 3 areas: (1) program design issues that may make it difficult to collect timely and consistent national data; (2) the relatively limited level of agencies' program evaluation capabilities; and (3) long- standing weaknesses in agencies' financial management capabilities, particularly in viewing program performance in relation to program cost.
Program design features have implications for the availability of consistent and reliable performance information. In several federal mission areas, devolution of program responsibility from the federal level has shifted both program management and accountability responsibilities toward the states.
Collecting consistent data to provide an overall, national picture of performance can be challenging when programs are implemented and results achieved through networks of intergovernmental partnerships. To illustrate, EPA depends on the state and local agencies it is working with to provide the performance information that indicates whether important environmental results are being achieved. For example, the state water
4 General Services Administration: Many Building Security Upgrades Made But Problems Have Hindered Program Implementation (GAO/ T- GGD- 98- 141, June 4, 1998). 5 General Services Administration: Status of Efforts to Improve Management of Building Security Upgrade Program (GAO/ T- GGD/ OSI- 00- 19, Oct. 7, 1999.) Program Design Affects the
Availability of Consistent and Reliable Information
Statement Managing for Results: Challenges in Producing Credible Performance Information
Page 5 GAO/ T- GGD/ RCED- 00- 134
quality reports required by the Clean Water Act are a key source of information for measuring progress in cleaning up the nation's lakes, rivers, and streams. However, EPA has found that the wealth of environmental data EPA and states collect is often difficult to compile in a meaningful way. 6
As provided in the Clean Water Act, Congress left the primary monitoring responsibility to the states for measuring progress in cleaning up the nation's lakes, rivers, and streams. However, inconsistencies in water quality assessments and in assessment methodologies from state to state make it difficult, first, to aggregate the data and; second, to use the information to conclusively determine whether the quality of rivers, lakes, and streams is getting better or worse over time. Absent this information, it has been difficult for EPA to set priorities, evaluate the success of its programs and activities, and report on its accomplishments in a credible and informed way.
Similar challenges confront DOT in determining the results of its efforts. DOT's Federal Highway Administration's (FHWA) efforts to increase the percentage of roads that meet pavement performance standards for good ride quality provide an example. To measure ride quality, FHWA uses the International Roughness Index (IRI), a quantitative measure of the amount a vehicle moves up and down while traveling over pavement. FHWA relies on the states to take and report IRI measurements. In September 1999, we reported that the IRI data are not consistent or accurate. 7 We found that IRI data were not comparable between states, because states differed in the devices, procedures, and mathematical simulations they used to calculate the index. These differences reduce the accuracy of the IRI data disseminated by FHWA and limit the ability to make state- to- state comparisons. As we recommended, in December 1999, FHWA adopted new standards that should improve the consistency of state data.
The problems at EPA and DOT in obtaining consistent and reliable information are not isolated to these two agencies. Our governmentwide survey of mid- and upper level federal managers, conducted in late 1996 and 1997, found that 34.8 percent of mangers identified the use of differing definitions as a factor that hindered their measurement or use of
6 Major Management Challenges and Program Risks: Environmental Protection Agency (GAO/ OCG- 9917, January 1999). 7 Transportation Infrastructure: Better Data Needed to Rate the Nation's Highway Conditions (GAO/ RCED- 99- 264, Sept. 27, 1999).
Statement Managing for Results: Challenges in Producing Credible Performance Information
Page 6 GAO/ T- GGD/ RCED- 00- 134
performance measures to either a great or a very great extent. 8 At the request of the Senate Subcommittee on Oversight of Government Management, Restructuring, and the District of Columbia, we are currently surveying federal managers again to follow up on whether there have been any changes in their opinions on a wide range of management issues, including the quality and consistency of performance data.
A fundamental challenge confronting Congress and agencies in improving federal performance is ensuring data- derived understandings exist of the contributions that specific programs agencies implement make to achieving results. Such understandings are important for Congress and agencies to ensure that agencies have the best mix of programs and strategies in place to achieve results as well as pinpoint and act on improvement opportunities. In this regard, program evaluation studies are important for assessing the contributions programs are making to results, determining factors affecting performance, and identifying improvement opportunities. However, we continue to be concerned that many federal agencies lack the capacity to undertake program evaluations. The absence of program evaluation capacity is a major concern, because a federal environment that focuses on results where federal efforts are often but one factor among many that determine whether goals are achieved depends on program evaluation to provide vital information about the effect of the federal effort.
Further, in our assessment of agencies' fiscal year 2000 plans, we noted that few agencies indicated how their strategies would contribute to accomplishing results. Similar to most other agencies, EPA and GSA provided general discussions of how resources and strategies will be used to achieve results, but DOT was the only agency offering specific discussions. DOT listed in its fiscal year 2000 performance plan an overall strategy for achieving each of its performance goals, as well as specific activities and initiatives. 9 For example, DOT expects to increase transit ridership through (1) investments in transit infrastructure, (2) financial assistance to metropolitan planning organizations and state departments of transportation for planning activities, (3) research on improving train control systems, and (4) fleet management to provide more customer service.
8 The Government Performance and Results Act: 1997 Governmentwide Implementation Will Be Uneven (GAO/ GGD- 97- 109, June 2, 1997). 9 Results Act: Observations on the Department of Transportation's Fiscal Year 2000 Performance Plan (GAO/ RCED- 99- 153, May 7, 1999). Program Evaluation Is
Essential, but Federal Capacity Is Limited
Statement Managing for Results: Challenges in Producing Credible Performance Information
Page 7 GAO/ T- GGD/ RCED- 00- 134
However, even in DOT's case, there is ample room for continued progress. For example, DOT identified the rehabilitation of approximately 200 airport runways in the year 2000 as one of the activities contributing to the performance goal concerning the condition of runway pavement. We reported that there is a lack of information identifying the point at which rehabilitation or maintenance of pavement can be done before relatively rapid deterioration sets in. As a result, the Federal Aviation Administration (FAA) is not in a position to determine which projects are being proposed at the most economical time. 10 As we recommended, FAA plans to require airport sponsors to submit specific pavement condition information when applying for runway rehabilitation projects to aid FAA in setting priorities for airport improvement program funds.
Strong program evaluation capacity is needed to provide feedback on how well an agency's activities and programs contributed to achieving its results. Good evaluation information about program effects is difficult to obtain. Each of the tasks involved measuring results, ensuring the consistency and quality of data collected, establishing the casual connection between results and program activities, and separating out the influence of extraneous factors raises formidable technical or logistical problems that are not easily resolved. Thus, evaluating program impact generally requires a planned study and, often, considerable time and expense.
Conclusions about what the government is accomplishing with the taxpayers' money cannot be drawn without linking performance with program and cost information. Viewing program performance in relation to program cost as envisioned by GPRA for instance, by establishing the unit cost per output or outcome achieved can help Congress in its oversight, authorization, and appropriations capacities. Unfortunately, program and cost information has not always been present or reliable enough either to use in decisionmaking or to provide the requisite public accountability for the use of taxpayers' money.
This Subcommittee's September 30, 1999, hearing focused specifically on the issue of the quality of financial data. In our statement for that hearing, we noted that major reforms, such as the Chief Financial Officers (CFO) Act, set expectations for agencies to develop and deploy more modern financial management systems and to routinely produce sound cost and
10 Airfield Pavement: Keeping Nation's Runways in Good Condition Could Require Substantially Higher Spending (GAO/ RCED- 98- 226, July 31, 1998). Viewing Program
Performance in Relation to Program Cost
Statement Managing for Results: Challenges in Producing Credible Performance Information
Page 8 GAO/ T- GGD/ RCED- 00- 134
operating performance information, among other things. 11 More fundamentally, the Federal Financial Management Improvement Act (FFMIA) focused, among other things, on ensuring that agency financial management systems routinely provide reliable, useful, and timely financial information. The overhauling of financial and related management information systems is the overarching challenge for agencies in generating timely, reliable data throughout the year. With such information, Congress and other decisionmakers will be better positioned to invest scarce resources, reduce costs, oversee programs, and hold agency managers accountable for the way they run government programs. For fiscal year 1999, auditors found that financial management systems for 19 of the 22 agencies reporting to date, including DOT, EPA, and GSA, did not substantially comply with FFMIA's requirements. 12
The March 2000 performance reports for DOT, GSA, EPA, and other agencies provide them with an opportunity to show the progress they have made in addressing data credibility issues. As far back as our earliest assessment of agencies' efforts to implement GPRA, and more recently in our reviews of agencies' strategic and performance plans, we identified data credibility issues as a persistent and continuing challenge for agencies. 13 In passing GPRA, Congress emphasized that the usefulness of agencies' performance information depends, to a large degree, on the reliability and validity of their data.
During this past year, we issued several reports on practices and approaches that agencies have proposed or adopted that address data credibility issues. 14 For example, we reported that applied practices, such as identifying actions to compensate for unavailable or low- quality data and discussing implications of data limitations for assessing performance,
11 Financial Management: Financial Audit Results at GSA, EPA, and DOT (GAO/ T- AIMD- 99- 301, Sept. 30, 1999). 12 The three agencies in compliance were the Department of Energy, the National Aeronautics and Space Administration, and the National Science Foundation. 13 GAO/ GGD/ AIMD- 99- 215, July 20, 1999; Managing for Results: An Agenda To Improve the Usefulness of Agencies' Annual Performance Plans (GAO/ GGD/ AIMD- 98- 228, Sept. 8, 1998); Managing for Results: Agencies' Annual Performance Plans Can Help Address Strategic Planning Challenges (GAO/ GGD- 9844, Jan. 30, 1998); GAO/ GGD- 97- 109, June 2, 1997; and GPRA Performance Reports (GAO/ GGD- 96- 66R, Feb. 14, 1996).
14 Managing for Results: Strengthening Regulatory Agencies' Performance Management Practices (GAO/ GGD- 00- 10, Oct. 28, 1999); Performance Plans: Selected Approaches for Verification and Validation of Agency Performance Information (GAO/ GGD- 99- 139, July 30, 1999); and Agency Performance Plans: Examples of Practices That Can Improve Usefulness to Decisionmakers (GAO/ GGD/ AIMD- 99- 69, Feb. 26, 1999). Performance Reports
Provide Opportunities to Show Progress in Addressing Data Credibility Issues
Statement Managing for Results: Challenges in Producing Credible Performance Information
Page 9 GAO/ T- GGD/ RCED- 00- 134
can help agencies describe to Congress and others the agencies' capacity to gather and use performance information.
To illustrate, the Department of Transportation stated in its fiscal years 1999 and 2000 performance plans that one of the most significant limitations of both internal and external data is timeliness. One way that DOT plans to deal with this limitation is to compile preliminary estimates from the portion of data that is available in time to report on the performance measures. According to DOT, fatality data from the first 6 months of the year could be compared with data from the first 6 months of the previous year for an initial performance measurement.
In our report on reasonable approaches to verify and validate performance information, we identified a wide range of possible approaches that can be organized into four general strategies, as follows:
 Management can seek to improve the quality of performance data by fostering an organizational commitment and capacity for data quality.
 Verification and validation can include assessing the quality of existing performance data.
 Assessments of data quality are of little value unless agencies are responding to identified data limitations.
 Building quality into the development of performance data may help prevent future errors and minimize the need to continually fix existing data.
These approaches can help agencies improve the quality, usefulness, and credibility of performance information.
In summary, Madam Chairman, sound performance data are key to strengthening decisionmaking in agencies and in Congress and pinpointing specific opportunities for improved performance. As stated earlier, the issuance of the first performance reports will provide important information on the overall performance of federal programs. Discussing data credibility and related issues in performance reports can provide important contextual information to Congress and agencies to help them address the weaknesses in this area. For example, this sort of discussion in an agency's performance report can alert Congress to the problems the agency has had in collecting results- oriented performance information. Agencies can also alert Congress to the cost and data quality trade- offs associated with various collection strategies, such as relying on sources outside the agency to provide performance data and the degree to which those data are expected to be reliable. Summary
Statement Managing for Results: Challenges in Producing Credible Performance Information
Page 10 GAO/ T- GGD/ RCED- 00- 134
Madam Chairman, this concludes my prepared statement. I would be pleased to respond to any questions that you or other Members of the Subcommittee may have.
Contacts and Acknowledgments
For further information regarding this testimony, please contact J. Christopher Mihm at (202) 512- 8676. Individuals making key contributions to this testimony include Dottie Self, Peter Del Toro, Janet Barbee, Kate Sigggerud, Ralph Running, and Susan Swearingen.
Page 11 GAO/ T- GGD/ RCED- 00- 134
Viewing GAO Reports on the Internet For information on how to access GAO reports on the INTERNET, send e- mail message with info in the body to:
info@ www. gao. gov or visit GAO's World Wide Web Home Page at: http:// www. gao. gov Reporting Fraud, Waste, and Abuse in Federal Programs To contact GAO's Fraud Hotline use: Web site: http:// www. gao. gov/ fraudnet/ fraudnet. htm E- Mail: fraudnet@ gao. gov Telephone: 1- 800- 424- 5454 (automated answering system)
(410563)
MANAGING FOR RESULTS: Challenges in Producing Credible Performance
Information (GAO/T-GGD/RCED-00-134) MANAGING FOR RESULTS
Challenges in Producing Credible Performance Information Statement
of J. Christopher Mihm, Associate Director Federal Management and
Workforce Issues General Government Division United States General
Accounting Office GAO Testimony Before the Subcommittee on
Oversight, Investigations, and Emergency Management Committee on
Transportation and Infrastructure House of Representatives For
Release on Delivery Expected at 10: 00 a. m. EST Wednesday March
22, 2000 GAO/ T- GGD/ RCED- 00- 134 Statement Managing for
Results: Challenges in Producing Credible Performance Information
Page 1 GAO/ T- GGD/ RCED- 00- 134 Madam Chairman, Mr. Traficant,
and Members of the Subcommittee: I am pleased to be here today to
discuss the challenges federal agencies face in producing credible
performance information including the Department of Transportation
(DOT), the Environmental Protection Agency (EPA), and the General
Services Administration (GSA) and the opportunities the Government
Performance and Results Act (GPRA) provides for generating
information to help Congress and other decisionmakers. As you
know, GPRA was passed in part out of Congress' frustration over
the fact that congressional policymaking, spending decisions, and
oversight had been seriously handicapped by agencies' lack of
clear goals and adequate program performance and cost information.
To remedy that situation, GPRA requires agencies to set multiyear
strategic goals and corresponding annual performance goals,
measure performance toward the achievement of those goals, and
publicly report on their progress. About 100 agencies, including
DOT, EPA, and GSA, published a first set of strategic plans in
1997 and, as required, will issue updated plans by this September.
These agencies also issued annual performance plans for fiscal
years 1999 and 2000. Some agencies, such as EPA, have already
published their plans for fiscal year 2001, and other plans will
soon become available. By the end of this month, agencies are to
publish their first annual performance reports that, for the first
time, will provide important information on the overall
performance of federal programs. The issuance of these reports,
therefore, represents a new and potentially more substantive stage
in the implementation of GPRA. Performance reports offer Congress
the opportunity to systematically assess agencies' actual
performance on a governmentwide basis and to consider the specific
steps that can be taken to improve performance and reduce costs.
Last month, we released our report on the challenges agencies face
in producing credible performance information and how those
challenges may affect performance reporting. 1 In that report, we
state that it appears unlikely that agencies will consistently
have for their first performance reports the reliable information
needed to assess whether goals are being met or specifically how
performance can be improved. Our work over the past several years
has identified limitations in agencies' abilities to 1 Managing
for Results: Challenges Agencies Face in Producing Credible
Performance Information (  GAO/GGD-00-52 , Feb. 4, 2000).
Statement Managing for Results: Challenges in Producing Credible
Performance Information Page 2 GAO/ T- GGD/ RCED- 00- 134 produce
credible data and identify performance improvements. These
limitations are substantial and long- standing, and they will not
be quickly or easily resolved. These limitations are, therefore,
likely to be reflected in agencies' initial performance reports as
they have been in the performance plans prepared to date. As
agreed with the Subcommittee, my statement today will address
three topics. First, I will provide a governmentwide perspective
on the credibility of agencies' performance information based on
our assessment of agencies' performance plans. Second, I will
discuss some of the challenges agencies face including DOT, EPA,
and GSA in producing credible performance data. Third, I will
highlight how agencies can use their performance reports to
address data credibility issues. My comments today are based on
our large body of work on agencies' performance data problems and
related issues. To efficiently and effectively operate, manage,
and oversee programs and activities, agencies need reliable
information during their planning efforts to set realistic goals
and later, as programs are being implemented, to gauge their
progress toward achieving those goals. Credible performance
information is also essential for Congress and other
decisionmakers to accurately assess agencies' progress towards the
achievement of their goals. As shown in figure 1, our analysis of
agencies' fiscal year 2000 performance plans noted that most
agencies including EPA and GSA provided only limited confidence
that their performance information would be credible. 2 DOT was
one of four agencies that provided general confidence that their
performance information would be credible. 3 2 Managing for
Results: Opportunities for Continued Improvements in Agencies'
Performance Plans (GAO/ GGD/ AIMD- 99- 215, July 20, 1999). 3 In
addition to DOT, the Department of Education, the Department of
Justice, and the Social Security Administration provided general
confidence in their fiscal year 2000 plans that their performance
information would be credible. Confidence in the Credibility of
Agencies' Performance Information Is Limited Statement Managing
for Results: Challenges in Producing Credible Performance
Information Page 3 GAO/ T- GGD/ RCED- 00- 134 Source: GAO/ GGD/
AIMD- 99- 215, July 20, 1999. Congress and other decisionmakers
must have assurance that the program and financial data being used
will be sufficiently timely, complete, accurate, useful, and
consistent if these data are to inform decisionmaking. However,
most agencies lacked information in their fiscal year 2000
performance plans on the procedures they would use to verify and
validate performance information. In addition, most agencies
failed to include discussions of strategies to address known data
limitations. We reported that when performance data are
unavailable or of low quality, a performance plan would be more
useful to Congress and other decisionmakers if it briefly
discussed how the agency plans to deal with such limitations.
Without such a discussion, Congress and other decisionmakers will
have difficulty determining the implications for assessing the
subsequent achievement of performance goals. Agencies need to
build the capacity to gather and use performance information to
successfully measure and report progress toward intended results.
However, our work over the past several years has identified
limitations in agencies' abilities to produce credible performance
data. Figure 1: Confidence in Performance Data Challenges Agencies
Face in Producing Credible Performance Information Statement
Managing for Results: Challenges in Producing Credible Performance
Information Page 4 GAO/ T- GGD/ RCED- 00- 134 For example, in our
June 1998 review of GSA's building security upgrade program, we
found limitations with data quality in GSA's building security
update tracking system, which is used to track the status of
security upgrades at federal buildings. 4 We found that the system
contained errors for 24 of the 53 buildings reviewed. A similar
study conducted by GSA's Office of Inspector General (OIG) found
that the status of security upgrades was not accurately reflected
in 65 of the 120 buildings it reviewed. In 1999, we reexamined 31
of the buildings reviewed by either the OIG or us and found that
GSA corrected many of the inaccuracies previously identified. 5
However, we found that the tracking system continued to contain
errors. We found errors concerning 10 buildings, about one- third
of those we visited. Data inaccuracies included overstating the
number or extent of security upgrades completed and misstating the
operating status of security improvements. Without valid data, it
is unlikely that GSA will be able to assess, first, the costs
versus the benefits of upgrades; and, second, the extent to which
completed upgrades have contributed to increased security or
reduced vulnerability to the greatest threats to federal office
buildings. In general, we have found that limitations in
performance data relate to 3 areas: (1) program design issues that
may make it difficult to collect timely and consistent national
data; (2) the relatively limited level of agencies' program
evaluation capabilities; and (3) long- standing weaknesses in
agencies' financial management capabilities, particularly in
viewing program performance in relation to program cost. Program
design features have implications for the availability of
consistent and reliable performance information. In several
federal mission areas, devolution of program responsibility from
the federal level has shifted both program management and
accountability responsibilities toward the states. Collecting
consistent data to provide an overall, national picture of
performance can be challenging when programs are implemented and
results achieved through networks of intergovernmental
partnerships. To illustrate, EPA depends on the state and local
agencies it is working with to provide the performance information
that indicates whether important environmental results are being
achieved. For example, the state water 4 General Services
Administration: Many Building Security Upgrades Made But Problems
Have Hindered Program Implementation (GAO/T-GGD-98-141, June 4,
1998). 5 General Services Administration: Status of Efforts to
Improve Management of Building Security Upgrade Program (GAO/ T-
GGD/ OSI- 00- 19, Oct. 7, 1999.) Program Design Affects the
Availability of Consistent and Reliable Information Statement
Managing for Results: Challenges in Producing Credible Performance
Information Page 5 GAO/ T- GGD/ RCED- 00- 134 quality reports
required by the Clean Water Act are a key source of information
for measuring progress in cleaning up the nation's lakes, rivers,
and streams. However, EPA has found that the wealth of
environmental data EPA and states collect is often difficult to
compile in a meaningful way. 6 As provided in the Clean Water Act,
Congress left the primary monitoring responsibility to the states
for measuring progress in cleaning up the nation's lakes, rivers,
and streams. However, inconsistencies in water quality assessments
and in assessment methodologies from state to state make it
difficult, first, to aggregate the data and; second, to use the
information to conclusively determine whether the quality of
rivers, lakes, and streams is getting better or worse over time.
Absent this information, it has been difficult for EPA to set
priorities, evaluate the success of its programs and activities,
and report on its accomplishments in a credible and informed way.
Similar challenges confront DOT in determining the results of its
efforts. DOT's Federal Highway Administration's (FHWA) efforts to
increase the percentage of roads that meet pavement performance
standards for good ride quality provide an example. To measure
ride quality, FHWA uses the International Roughness Index (IRI), a
quantitative measure of the amount a vehicle moves up and down
while traveling over pavement. FHWA relies on the states to take
and report IRI measurements. In September 1999, we reported that
the IRI data are not consistent or accurate. 7 We found that IRI
data were not comparable between states, because states differed
in the devices, procedures, and mathematical simulations they used
to calculate the index. These differences reduce the accuracy of
the IRI data disseminated by FHWA and limit the ability to make
state- to- state comparisons. As we recommended, in December 1999,
FHWA adopted new standards that should improve the consistency of
state data. The problems at EPA and DOT in obtaining consistent
and reliable information are not isolated to these two agencies.
Our governmentwide survey of mid- and upper level federal
managers, conducted in late 1996 and 1997, found that 34.8 percent
of mangers identified the use of differing definitions as a factor
that hindered their measurement or use of 6 Major Management
Challenges and Program Risks: Environmental Protection Agency
(GAO/ OCG- 9917, January 1999). 7 Transportation Infrastructure:
Better Data Needed to Rate the Nation's Highway Conditions
(GAO/RCED-99-264, Sept. 27, 1999). Statement Managing for Results:
Challenges in Producing Credible Performance Information Page 6
GAO/ T- GGD/ RCED- 00- 134 performance measures to either a great
or a very great extent. 8 At the request of the Senate
Subcommittee on Oversight of Government Management, Restructuring,
and the District of Columbia, we are currently surveying federal
managers again to follow up on whether there have been any changes
in their opinions on a wide range of management issues, including
the quality and consistency of performance data. A fundamental
challenge confronting Congress and agencies in improving federal
performance is ensuring data- derived understandings exist of the
contributions that specific programs agencies implement make to
achieving results. Such understandings are important for Congress
and agencies to ensure that agencies have the best mix of programs
and strategies in place to achieve results as well as pinpoint and
act on improvement opportunities. In this regard, program
evaluation studies are important for assessing the contributions
programs are making to results, determining factors affecting
performance, and identifying improvement opportunities. However,
we continue to be concerned that many federal agencies lack the
capacity to undertake program evaluations. The absence of program
evaluation capacity is a major concern, because a federal
environment that focuses on results where federal efforts are
often but one factor among many that determine whether goals are
achieved depends on program evaluation to provide vital
information about the effect of the federal effort. Further, in
our assessment of agencies' fiscal year 2000 plans, we noted that
few agencies indicated how their strategies would contribute to
accomplishing results. Similar to most other agencies, EPA and GSA
provided general discussions of how resources and strategies will
be used to achieve results, but DOT was the only agency offering
specific discussions. DOT listed in its fiscal year 2000
performance plan an overall strategy for achieving each of its
performance goals, as well as specific activities and initiatives.
9 For example, DOT expects to increase transit ridership through
(1) investments in transit infrastructure, (2) financial
assistance to metropolitan planning organizations and state
departments of transportation for planning activities, (3)
research on improving train control systems, and (4) fleet
management to provide more customer service. 8 The Government
Performance and Results Act: 1997 Governmentwide Implementation
Will Be Uneven (GAO/GGD-97-109, June 2, 1997). 9 Results Act:
Observations on the Department of Transportation's Fiscal Year
2000 Performance Plan (GAO/RCED-99-153, May 7, 1999). Program
Evaluation Is Essential, but Federal Capacity Is Limited Statement
Managing for Results: Challenges in Producing Credible Performance
Information Page 7 GAO/ T- GGD/ RCED- 00- 134 However, even in
DOT's case, there is ample room for continued progress. For
example, DOT identified the rehabilitation of approximately 200
airport runways in the year 2000 as one of the activities
contributing to the performance goal concerning the condition of
runway pavement. We reported that there is a lack of information
identifying the point at which rehabilitation or maintenance of
pavement can be done before relatively rapid deterioration sets
in. As a result, the Federal Aviation Administration (FAA) is not
in a position to determine which projects are being proposed at
the most economical time. 10 As we recommended, FAA plans to
require airport sponsors to submit specific pavement condition
information when applying for runway rehabilitation projects to
aid FAA in setting priorities for airport improvement program
funds. Strong program evaluation capacity is needed to provide
feedback on how well an agency's activities and programs
contributed to achieving its results. Good evaluation information
about program effects is difficult to obtain. Each of the tasks
involved measuring results, ensuring the consistency and quality
of data collected, establishing the casual connection between
results and program activities, and separating out the influence
of extraneous factors raises formidable technical or logistical
problems that are not easily resolved. Thus, evaluating program
impact generally requires a planned study and, often, considerable
time and expense. Conclusions about what the government is
accomplishing with the taxpayers' money cannot be drawn without
linking performance with program and cost information. Viewing
program performance in relation to program cost as envisioned by
GPRA for instance, by establishing the unit cost per output or
outcome achieved can help Congress in its oversight,
authorization, and appropriations capacities. Unfortunately,
program and cost information has not always been present or
reliable enough either to use in decisionmaking or to provide the
requisite public accountability for the use of taxpayers' money.
This Subcommittee's September 30, 1999, hearing focused
specifically on the issue of the quality of financial data. In our
statement for that hearing, we noted that major reforms, such as
the Chief Financial Officers (CFO) Act, set expectations for
agencies to develop and deploy more modern financial management
systems and to routinely produce sound cost and 10 Airfield
Pavement: Keeping Nation's Runways in Good Condition Could Require
Substantially Higher Spending (GAO/RCED-98-226, July 31, 1998).
Viewing Program Performance in Relation to Program Cost Statement
Managing for Results: Challenges in Producing Credible Performance
Information Page 8 GAO/ T- GGD/ RCED- 00- 134 operating
performance information, among other things. 11 More
fundamentally, the Federal Financial Management Improvement Act
(FFMIA) focused, among other things, on ensuring that agency
financial management systems routinely provide reliable, useful,
and timely financial information. The overhauling of financial and
related management information systems is the overarching
challenge for agencies in generating timely, reliable data
throughout the year. With such information, Congress and other
decisionmakers will be better positioned to invest scarce
resources, reduce costs, oversee programs, and hold agency
managers accountable for the way they run government programs. For
fiscal year 1999, auditors found that financial management systems
for 19 of the 22 agencies reporting to date, including DOT, EPA,
and GSA, did not substantially comply with FFMIA's requirements.
12 The March 2000 performance reports for DOT, GSA, EPA, and other
agencies provide them with an opportunity to show the progress
they have made in addressing data credibility issues. As far back
as our earliest assessment of agencies' efforts to implement GPRA,
and more recently in our reviews of agencies' strategic and
performance plans, we identified data credibility issues as a
persistent and continuing challenge for agencies. 13 In passing
GPRA, Congress emphasized that the usefulness of agencies'
performance information depends, to a large degree, on the
reliability and validity of their data. During this past year, we
issued several reports on practices and approaches that agencies
have proposed or adopted that address data credibility issues. 14
For example, we reported that applied practices, such as
identifying actions to compensate for unavailable or low- quality
data and discussing implications of data limitations for assessing
performance, 11 Financial Management: Financial Audit Results at
GSA, EPA, and DOT (GAO/T-AIMD-99-301, Sept. 30, 1999). 12 The
three agencies in compliance were the Department of Energy, the
National Aeronautics and Space Administration, and the National
Science Foundation. 13 GAO/ GGD/ AIMD- 99- 215, July 20, 1999;
Managing for Results: An Agenda To Improve the Usefulness of
Agencies' Annual Performance Plans (GAO/ GGD/ AIMD- 98- 228, Sept.
8, 1998); Managing for Results: Agencies' Annual Performance Plans
Can Help Address Strategic Planning Challenges (GAO/ GGD- 9844,
Jan. 30, 1998); GAO/GGD-97-109, June 2, 1997; and GPRA Performance
Reports (GAO/GGD-96-66R, Feb. 14, 1996). 14 Managing for Results:
Strengthening Regulatory Agencies' Performance Management
Practices (GAO/GGD-00-10, Oct. 28, 1999); Performance Plans:
Selected Approaches for Verification and Validation of Agency
Performance Information (GAO/GGD-99-139, July 30, 1999); and
Agency Performance Plans: Examples of Practices That Can Improve
Usefulness to Decisionmakers (GAO/ GGD/ AIMD- 99- 69, Feb. 26,
1999). Performance Reports Provide Opportunities to Show Progress
in Addressing Data Credibility Issues Statement Managing for
Results: Challenges in Producing Credible Performance Information
Page 9 GAO/ T- GGD/ RCED- 00- 134 can help agencies describe to
Congress and others the agencies' capacity to gather and use
performance information. To illustrate, the Department of
Transportation stated in its fiscal years 1999 and 2000
performance plans that one of the most significant limitations of
both internal and external data is timeliness. One way that DOT
plans to deal with this limitation is to compile preliminary
estimates from the portion of data that is available in time to
report on the performance measures. According to DOT, fatality
data from the first 6 months of the year could be compared with
data from the first 6 months of the previous year for an initial
performance measurement. In our report on reasonable approaches to
verify and validate performance information, we identified a wide
range of possible approaches that can be organized into four
general strategies, as follows:  Management can seek to improve
the quality of performance data by fostering an organizational
commitment and capacity for data quality.  Verification and
validation can include assessing the quality of existing
performance data.  Assessments of data quality are of little value
unless agencies are responding to identified data limitations.
Building quality into the development of performance data may help
prevent future errors and minimize the need to continually fix
existing data. These approaches can help agencies improve the
quality, usefulness, and credibility of performance information.
In summary, Madam Chairman, sound performance data are key to
strengthening decisionmaking in agencies and in Congress and
pinpointing specific opportunities for improved performance. As
stated earlier, the issuance of the first performance reports will
provide important information on the overall performance of
federal programs. Discussing data credibility and related issues
in performance reports can provide important contextual
information to Congress and agencies to help them address the
weaknesses in this area. For example, this sort of discussion in
an agency's performance report can alert Congress to the problems
the agency has had in collecting results- oriented performance
information. Agencies can also alert Congress to the cost and data
quality trade- offs associated with various collection strategies,
such as relying on sources outside the agency to provide
performance data and the degree to which those data are expected
to be reliable. Summary Statement Managing for Results: Challenges
in Producing Credible Performance Information Page 10 GAO/ T- GGD/
RCED- 00- 134 Madam Chairman, this concludes my prepared
statement. I would be pleased to respond to any questions that you
or other Members of the Subcommittee may have. Contacts and
Acknowledgments For further information regarding this testimony,
please contact J. Christopher Mihm at (202) 512- 8676. Individuals
making key contributions to this testimony include Dottie Self,
Peter Del Toro, Janet Barbee, Kate Sigggerud, Ralph Running, and
Susan Swearingen. Page 11 GAO/ T- GGD/ RCED- 00- 134 Viewing GAO
Reports on the Internet For information on how to access GAO
reports on the INTERNET, send e- mail message with info in the
body to: info@ www. gao. gov or visit GAO's World Wide Web Home
Page at: http:// www. gao. gov Reporting Fraud, Waste, and Abuse
in Federal Programs To contact GAO's Fraud Hotline use: Web site:
http:// www. gao. gov/ fraudnet/ fraudnet. htm E- Mail: fraudnet@
gao. gov Telephone: 1- 800- 424- 5454 (automated answering system)
(410563)

*** End of document. ***