Managing for Results: Challenges Agencies Face in Producing Credible
Performance Information (Letter Report, 02/04/2000, GAO/GGD-00-52).

In the past, congressional policymaking, spending decisions, and
oversight as well as agencies' decisionmaking have all been seriously
handicapped by the lack of clear goals and sound performance information
at federal agencies. To remedy that situation, the Government
Performance and Results Act of 1993 requires agencies to set multiyear
strategic goals, measure progress toward achieving those goals, and
report on their progress. Agencies' annual performance reports are
intended to provide important information to agency managers,
policymakers, and the public on what each agency accomplished with the
resources it was given. This report identifies some of the challenges
that agencies face in producing credible performance information and how
those challenges may affect performance reporting-the next phase in
implementing the Results Act. GAO (1) discusses whether the weaknesses
it has identified in agencies' performance plans imply challenges for
the performance reports, (2) illustrates some of the challenges agencies
face in producing credible performance data, and (3) describes how
performance reports can be used to address data credibility issues.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  GGD-00-52
     TITLE:  Managing for Results: Challenges Agencies Face in
	     Producing Credible Performance Information
      DATE:  02/04/2000
   SUBJECT:  Agency missions
	     Accountability
	     Interagency relations
	     Performance measures
	     Program evaluation
	     Strategic planning
	     Data integrity
	     Productivity in government
	     Reporting requirements
IDENTIFIER:  Dept. of Education Safe and Drug-Free Schools and
	     Communities Program
	     HHS Temporary Assistance for Needy Families Program
	     Head Start Program
	     DOD Financial Management Improvement Plan
	     GPRA
	     Government Performance and Results Act

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  This text was extracted from a PDF file.        **
** Delineations within the text indicating chapter titles,      **
** headings, and bullets have not been preserved, and in some   **
** cases heading text has been incorrectly merged into          **
** body text in the adjacent column.  Graphic images have       **
** not been reproduced, but figure captions are included.       **
** Tables are included, but column deliniations have not been   **
** preserved.                                                   **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                                            **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************

GAO/GGD-00-52

MANAGING FOR RESULTS: Challenges Agencies Face in Producing Credible Performance Information (GAO/GGD-00-52)
MANAGING FOR RESULTS
Challenges Agencies Face in Producing Credible Performance Information
United States General Accounting Office
GAO Report to Congressional Requesters
February 2000 

GAO/GGD-00-52

United States General Accounting Office General Government Division Washington, D. C. 20548
Page 1 GAO/ GGD- 00- 52 Performance Information Challenges
B- 284548 February 4, 2000 The Honorable Fred Thompson Chairman Committee on Governmental Affairs United States Senate
The Honorable Dick Armey Majority Leader House of Representatives
The Honorable Dan Burton Chairman Committee on Government Reform House of Representatives
The Honorable John R. Kasich Chairman Committee on the Budget House of Representatives
The Government Performance and Results Act (Results Act or GPRA) seeks to shift the focus of federal management and decisionmaking from a preoccupation with the number of tasks completed or services provided to a more direct consideration of the results of programs that is, the real differences the tasks or services make to the nation or individual taxpayer. The Results Act originated in part from Congress's frustration over the fact that congressional policymaking, spending decisions, and oversight and agencies' decisionmaking all had been seriously handicapped by the lack of clear goals and sound performance information. To remedy that situation, the Results Act requires agencies to set multiyear strategic goals in their strategic plans and corresponding annual goals in their performance plans, measure performance toward the achievement of those goals, and report on their progress in their annual performance reports. These reports are intended to provide important information to agency managers, policymakers, and the public on what each agency accomplished with the resources it was given.
This report responds to your request that we identify some of the challenges agencies face in producing credible performance information and how those challenges may affect performance reporting the next
B- 284548 Page 2 GAO/ GGD- 00- 52 Performance Information Challenges
phase of the Results Act implementation. As agreed, our objectives were to (1) discuss whether the weaknesses we have identified in agencies' performance plans imply challenges for the performance reports, (2) illustrate some of the challenges agencies face in producing credible performance data, and (3) describe how performance reports can be used to address data credibility issues.
The Results Act is aimed at improving performance of government programs by requiring agencies to clarify their missions, establish goals and strategies for reaching them, measure performance, and report on their accomplishments. Beginning with fiscal year 1999, the head of each agency is to prepare and submit to Congress and the president a report on program performance. The first of these annual reports is to be submitted no later than March 31, 2000. These reports are to contain two main parts: a report on the actual performance achieved as compared with the performance goals expressed in the performance plan and the plans and schedules to achieve those goals that were not met. If a performance goal becomes impractical or infeasible to achieve, the agency is to explain why that is the case and what legislative, regulatory, or other actions are needed to accomplish the goal, or whether the goal ought to be modified or discontinued. Finally, the reports should also relate performance measurement information to program evaluation findings, in order to give a clear picture of an agency's performance and its efforts at improvement. 1
Based on our previously issued work, it appears unlikely that agencies consistently will have for their first performance reports the reliable performance information needed to assess whether performance goals are being met or specifically how performance can be improved. Our work over the past several years has identified limitations in agencies' abilities to produce credible data and identify performance improvement opportunities. These limitations are substantial, long- standing, and will not be quickly or easily resolved. They are likely to be reflected in agencies' initial performance reports as they have been in the performance plans prepared to date.
For example, policy decisions made when designing federal programs, particularly intergovernmental programs, may make it difficult to collect timely and consistent national data. In administering programs that are a joint responsibility with state and local governments, Congress and the
1 For specific information on the requirements of the Government Performance and Results Act, see Public Law 103- 62, August 3, 1993; and Senate Report 103- 58, June 16, 1993. Also, see Office of Management and Budget Circular No. A- 11, Part 2, July 1999, for information on the preparation and submission of strategic plans, annual performance plans, and annual program performance reports. Background
Results in Brief
B- 284548 Page 3 GAO/ GGD- 00- 52 Performance Information Challenges
executive branch continually balance the competing objectives of collecting uniform program information to assess performance with giving states and localities the flexibility needed to effectively implement intergovernmental programs.
In addition, the relatively limited level of agencies' program evaluation capabilities a perennial problem suggests that many agencies are not well positioned to undertake necessary evaluations. Program evaluations are important to providing information on the extent to which an agency's efforts contributed to results and to highlight opportunities to improve those results.
Finally, long- standing weaknesses in agencies' financial management capabilities make it difficult for decisionmakers to effectively assess and improve many programs' financial performance.
In order to help agency managers select appropriate techniques for assessing, documenting, and improving the quality of their performance data, we have reported on some reasonable approaches that agencies had proposed or adopted to verify and validate performance information. 2 These approaches include senior management actions, agencywide efforts, and specific program manager and technical staff activities. We noted that agencies could use these approaches, where appropriate, to improve the quality, usefulness, and credibility of performance information.
In that regard, performance reports provide agencies with an opportunity to show the progress they have made in addressing data credibility issues. The Results Act requires agencies to describe in their annual performance plans how they will verify and validate the performance information that will be collected. Including information in performance reports that describes the quality of the reported performance data and the implications of missing data can be equally important and can provide key contextual information to Congress and other users of the performance reports.
To address our objectives, we relied on our large body of work on agencies' performance data problems and related issues. We drew examples from our reviews of agencies' efforts to implement the Results Act, such as our reviews of agencies' fiscal years 1999 and 2000 performance plans, and products we have issued on major management
2 Performance Plans: Selected Approaches for Verification and Validation of Agency Performance Information (GAO/ GGD- 99- 139, July 30, 1999). Scope and
Methodology
B- 284548 Page 4 GAO/ GGD- 00- 52 Performance Information Challenges
challenges and risks, such as our performance and accountability series and high- risk series. 3
Because this report is based primarily on our previously issued reports, we did not obtain agency comments. Our work on this report was done from October 1999 to January 2000, in Washington, D. C., in accordance with generally accepted government auditing standards.
Agencies need reliable information during their planning efforts to set realistic goals and later, as programs are being implemented, to gauge their progress toward achievement of those goals. 4 In our assessments of annual performance plans, we identified challenges that will affect agencies' abilities to reliably report on the achievement of program goals and, in cases where goals are not met, either identify opportunities for improvement or whether goals need to be adjusted. 5
For example, we concluded in our review of fiscal year 1999 performance plans that future plans would be more useful if they would, among other things, (1) more fully articulate how strategies and resources will lead to improved performance and (2) provide much greater confidence that performance information will be credible and useful for decisionmaking. Although, on the whole, the fiscal year 2000 plans showed moderate improvements over the fiscal year 1999 plans, we identified these two issues as continuing key weaknesses common among agencies' plans.
Most of the fiscal year 2000 plans related strategies and programs to performance goals; however, few plans indicated how the strategies would contribute to accomplishing the expected level of performance. Agencies need to understand and articulate how what they do on a day- to- day basis contributes to mission- related results. Such an understanding is important for agencies to pinpoint opportunities to improve performance and design and implement appropriate initiatives. This information is also helpful to congressional and other decisionmakers in assessing the degree to which strategies are appropriate and reasonable.
3 See, for example, Managing for Results: Opportunities for Continued Improvements in Agencies' Performance Plans (GAO/ GGD/ AIMD- 99- 215, July 20, 1999); Managing for Results: An Agenda To Improve the Usefulness of Agencies' Annual Performance Plans (GAO/ GGD/ AIMD- 98- 228, Sept. 8, 1998); Performance and Accountability Series: Major Management Challenges and Program Risks (GAO/ OCG- 99- 1 through GAO- OCG- 99- 21, January 1999); and High- Risk Series: An Update (GAO/ HR99- 1, January 1999).
4 The Government Performance and Results Act: 1997 Governmentwide Implementation Will Be Uneven (GAO/ GGD- 97- 109, June 2, 1997). 5 GAO/ GGD/ AIMD- 99- 215, July 20, 1999; and GAO/ GGD/ AIMD- 98- 228, September 8, 1998. Challenges in
Performance Planning Will Affect Performance Reporting
Few Plans Indicated How Strategies Would Contribute to Accomplishing Expected Level of Performance
B- 284548 Page 5 GAO/ GGD- 00- 52 Performance Information Challenges
The outcomes of many federal programs are the result of the interplay of several factors, and only some of these are within a program's control. 6 Further, on a daily basis, agencies do not produce outcomes, but rather outputs, such as activities, products, or services that are intended to contribute to outcomes. Thus, a key analytic challenge for agencies is knowing how their programmatic efforts contribute to their desired outcomes. The inconsistent attention to this critical element undermined the value of agencies' performance plans and, unless addressed, it also will severely limit the value of their performance reports.
In 1997, we reported that the Department of Justice's Immigration and Naturalization Service (INS) lacked data on the overall effectiveness of its southwest border strategy. 7 For example, data were insufficient to indicate whether illegal aliens were deterred from entering the United States, whether there had been a decrease in attempted reentries by those who had previously been apprehended, and whether the strategy had reduced border violence. We noted that, despite the investment of billions of dollars in the strategy, INS had amassed only a partial picture of the effects of increased border control and did not know whether the investment was producing the intended results. We reported that a comprehensive, systematic evaluation of the agency's strategy to deter illegal entry along the southwest border would provide INS with information on whether its border enforcement strategy had produced the intended results. After our report, INS contracted with independent research firms for an evaluation.
Another weakness that we identified in our review of agencies' fiscal year 1999 and fiscal year 2000 performance plans was the limited confidence they provided in the credibility of performance information. Credible performance information is essential for accurately assessing agencies' progress towards the achievement of their goals the cornerstone of performance reporting. As shown in figure 1, our analysis of agencies' fiscal year 2000 performance plans noted that most of the plans provided only limited confidence that performance information would be credible. Only the plans for the Department of Education, the Department of Justice, the Department of Transportation, and the Social Security Administration provided general confidence that their performance information would be credible.
6 Managing for Results: Analytic Challenges in Measuring Performance (GAO/ HEHS/ GGD- 97- 138, May 30, 1997). 7 Illegal Immigration: Southwest Border Strategy Results Inconclusive; More Evaluation Needed (GAO/ GGD- 98- 21, Dec. 11, 1997). Few Plans Provided
Confidence That Performance Information Would Be Credible
B- 284548 Page 6 GAO/ GGD- 00- 52 Performance Information Challenges
Source: GAO/ GGD/ AIMD- 99- 215, July 20, 1999.
Decisionmakers must have assurance that the program and financial data being used will be sufficiently timely, complete, accurate, useful, and consistent if these data are to inform decisionmaking. However, like the fiscal year 1999 performance plans, most of the fiscal year 2000 plans lacked information on the procedures the agencies would use to verify and validate performance information.
Similar to our findings with the fiscal year 1999 plans, we also found that, in general, the fiscal year 2000 plans failed to include discussions of strategies to address known data limitations. We reported that when performance data are unavailable or of low quality, a performance plan would be more useful to decisionmakers if it briefly discussed how the agency plans to deal with such limitations. Without such a discussion, decisionmakers will have difficulty determining the implications for assessing the subsequent achievement of performance goals that agencies include in their performance reports.
Figure 1: Confidence in Performance Data
B- 284548 Page 7 GAO/ GGD- 00- 52 Performance Information Challenges
In order to successfully measure and report progress toward intended results, agencies need to build the capacity to gather and use performance information. However, our work over the past several years has identified limitations in agencies' abilities to produce credible performance data. Specifically, those limitations relate to program design issues that may make it difficult to collect timely and consistent national data, the relatively limited level of agencies' program evaluation capabilities, and long- standing weaknesses in agencies' financial management capabilities.
In several program areas, devolution of program responsibility from the federal level and consolidation of individual federal programs into more comprehensive, multipurpose grant programs have shifted both program management and accountability responsibilities toward the states. These programs vary greatly in the kind and degree of flexibility afforded to state or local entities, distribution of accountability across levels of government, and availability of direct measures of program performance. In our report on grant program design features, we noted that relatively few flexible programs collected uniform data on the outcomes of state or local service activities. 8 Collecting such data requires conditions such as uniformity of activities, objectives, and measures that do not exist under many flexible program designs. 9
For instance, we reported that the block grants enacted as part of the Omnibus Budget Reconciliation Act of 1981 carried no uniform federal information and reporting requirements. 10 States collected a wide range of program information, but the collection efforts were designed to meet the needs of the individual states. Congress had limited information on program activities, services delivered, and clients served. As a result, it was difficult, in many cases, to aggregate state experiences and speak from a national perspective on the block grant activities or their effects. Similarly, without uniform information definitions and collection methodologies, it was difficult to compare state efforts or draw meaningful conclusions about the relative effectiveness of different strategies.
8 Grant Programs: Design Features Shape Flexibility, Accountability, and Performance Information (GAO/ GGD- 98- 137, June 22, 1998). 9 The recently passed Federal Financial Assistance Management Improvement Act of 1999 is designed, among other things, to establish greater commonality in reporting requirements and procedures across federal financial assistance programs and to foster improvements in the quality of the data collected. The act recognizes that changes in current law may be needed to accomplish these purposes and directs that recommendations concerning such changes be reported to Congress no later than May 2001.
10 Block Grants: Issues in Designing Accountability Provisions (GAO/ AIMD- 95- 226, Sept. 1, 1995). Agencies Face
Limitations in Producing Credible Performance Data
Program Design Features Have Implications for the Availability of Performance Information
B- 284548 Page 8 GAO/ GGD- 00- 52 Performance Information Challenges
Education is one of many agencies where the interest in having enough information for accountability and federal program management continually competes with the aim of providing local agencies with the flexibility needed to implement their programs on the basis of their local needs. 11 The Safe and Drug- Free Schools program, for example, allows a wide range of activities, such as drug prevention instruction for students; staff training; general violence- prevention instruction; and special one- time events, such as guest speakers and drug- and alcohol- free social activities. 12 States are also permitted to define the information they collect on program activities and effectiveness. Under the Safe and Drug- Free Schools and Communities Act, Education oversees state programs and state agencies monitor local programs. Under the act, each state may establish its own reporting requirements for local education agencies. Although these requirements have some common elements, state requirements vary widely. With no requirements that states use consistent measures, Education faces, as our work has shown, a difficult challenge in assembling the required state reports to develop a nationwide picture of the program's effectiveness.
Because states, localities, or nongovernmental organizations operate many of its programs, the Department of Health and Human Services (HHS) experiences similar challenges. 13 The Personal Responsibility and Work Opportunity Reconciliation Act dramatically altered the nation's system for providing assistance to the poor. Among the many changes, the act replaced the existing entitlement program for poor families (Aid to Families With Dependent Children) with fixed block grants to the states to provide Temporary Assistance for Needy Families (TANF). Under the TANF block grant, states have flexibility in designing and implementing their own assistance programs within federal guidelines. Meanwhile, HHS has a broad range of responsibilities for ensuring accountability from the states.
The welfare reform law gives HHS administrative and oversight responsibilities, the performance of which will rely on state- provided data. HHS needs to ensure that it receives comparable and reliable data from the
11 Department of Education: Information Needs Are at the Core of Management Challenges Facing the Department (GAO/ T- HEHS- 98- 124, Mar. 24, 1998); and Balancing Flexibility and Accountability: Grant Program Design in Education and Other Areas (GAO/ T- GGD/ HEHS- 98- 94, Feb. 11, 1998).
12 Safe and Drug- Free Schools: Balancing Accountability With State and Local Flexibility (GAO/ HEHS98- 3, Oct. 10, 1997). 13 Department of Health and Human Services: Strategic Planning and Accountability Challenges (GAO/ T- HEHS- 98- 96, Feb. 26, 1998).
B- 284548 Page 9 GAO/ GGD- 00- 52 Performance Information Challenges
states to help it fulfill its oversight responsibilities, which include ensuring that states enforce the federal 5- year time limit on receiving welfare benefits, meet minimum work participation rates, and maintain a certain level of state welfare spending. Enforcing the time limit, for example, will be difficult because information on the total amount of time that someone has received TANF is not always available in individual states, let alone across states. In addition, the law gives HHS authority to assess penalties if states fail to comply with certain requirements and provides for states to receive bonuses if they meet certain performance standards. HHS needs to collect state data to determine performance penalties and bonuses. In view of the increased flexibility of states in designing their programs, obtaining comparable and reliable data to assess the effect of welfare reform on children and families could be difficult for HHS.
The Environmental Protection Agency (EPA) provides another example of a federal agency that depends on the state and local agencies it is working with to provide the performance information that indicates whether results are being achieved. For example, the state water quality reports required by the Clean Water Act are a key source of information for measuring progress in cleaning up the nation's lakes, rivers, and streams. However, EPA has found that the wealth of environmental data EPA and states collect are often difficult to compile in a meaningful way. 14
As contained in the Clean Water Act, Congress left the primary monitoring responsibility to the states for measuring progress in cleaning up the nation's lakes, rivers, and streams. However, inconsistencies in water quality assessments and in assessment methodologies from state to state make it difficult to aggregate the data and to use the information to conclusively determine whether the quality of rivers, lakes, and streams is getting better or worse over time. Absent this information, it has been difficult for EPA to set priorities, evaluate the success of its programs and activities, and report on its accomplishments in a credible and informed way.
The unavailability of reliable performance information can also be traced to a lack of standards and of common definitions for terms used to evaluate programs. For example, we reported that the agencies involved in wetlands- related activities inconsistently used terms such as protection, restoration, rehabilitation, improvement, and enhancement in describing
14 Major Management Challenges and Program Risks: Environmental Protection Agency (GAO/ OCG- 9917, January 1999).
B- 284548 Page 10 GAO/ GGD- 00- 52 Performance Information Challenges
and reporting on their accomplishments. 15 Even when the same terms are used, the agencies do not define them in the same way. As a result, the consistency and reliability of data on the status of wetlands acreage are questionable. Thus, neither the progress made toward achieving the governmentwide goal of no net loss of the nation's remaining wetlands nor the contributions made by the agencies in achieving this goal can be accurately measured.
Weaknesses in the availability of direct measures of performance can be overcome by drawing on information from other sources, such as program evaluation studies, research on the effectiveness of service delivery, or aggregate data such as vital statistics that describe the general status of a population. In this regard, our report on grant program design features noted that 13 of the 21 flexible grant programs reviewed used such sources along with, or as a substitute for, performance measures collected from program operations. 16 We found that agencies that made use of multiple sources had information that covered more aspects of program performance than those that relied upon a single source.
Program evaluation studies are important for assessing how well programs are working, determining factors affecting performance, and identifying improvement opportunities. In our report on the analytic challenges facing agencies in measuring performance, we stated that supplementing performance data with impact evaluations might help provide agencies with a more complete picture of program effectiveness. 17 Evaluations can play a critical role in helping to address those measurement and analysis difficulties agencies face that stem from two features common to many federal programs: the interplay of federal, state, and local government activities, and objectives and the aim to influence complex systems or phenomena whose outcomes are largely outside government control. Furthermore, systematic evaluation of how a program was implemented can provide important information about why a program did or did not succeed as well as suggest ways to improve it.
However, as we reported in our assessment of agencies' fiscal year 1999 performance plans, we continue to be concerned about the lack in many federal agencies of the capacity to undertake the program evaluations that will be vital to the success of the Results Act. In our earlier review of
15 Wetlands Overview: Problems With Acreage Data Persist (GAO/ RCED- 98- 150, July 1, 1998). 16 GAO/ GGD- 98- 137, June 22, 1998. 17 GAO/ HEHS/ GGD- 97- 138, May 30, 1997. Program Evaluation Is
Essential, But Federal Capacity Is Limited
B- 284548 Page 11 GAO/ GGD- 00- 52 Performance Information Challenges
agencies' strategic plans, we found that many agencies had not given sufficient attention to how program evaluations will be used in implementing the Results Act and improving performance. 18 In another report, we noted that agencies' program evaluation capabilities would be challenged to meet the new demands for information on program results. 19 We found that the resources allocated for conducting program evaluations were small and unevenly distributed across the 13 departments and 10 independent agencies we surveyed for that report.
Good evaluation information about program effects is difficult to obtain. Each of the tasks involved measuring outcomes, ensuring the consistency and quality of data collected, establishing the causal connection between outcomes and program activities, and separating out the influence of extraneous factors raises formidable technical or logistical problems that are not easily resolved. Thus, evaluating program impact generally requires a planned study and, often, considerable time and expense.
The experiences of the Head Start program illustrate the importance and difficulty of systematic program evaluation. Head Start, administered by HHS' Administration for Children and Families, is one of the most popular federal early childhood programs and has long enjoyed both congressional and public support. 20 Between fiscal years 1990 and 1998, annual Head Start funding nearly tripled, from $1. 5 billion to almost $4.4 billion. Head Start's purpose is to improve the social competence of children in lowincome families, and in the past 33 years, the program has provided a comprehensive set of services to about 16 million low- income children. Educational, medical, nutritional, mental health, dental, social, and other services have been provided to low- income children and their families in all 50 states, the District of Columbia, Puerto Rico, and the U. S. territories, as well as to migrant and Native American populations. Given the size of the Head Start program and the efforts to expand the program's annual enrollment to one million children by 2002, investing in studies that will assess its impact is important. Specifically, the challenge for HHS is to determine whether the same outcomes would have occurred if children
18 Managing for Results: Agencies' Annual Performance Plans Can Help Address Strategic Planning Challenges (GAO/ GGD- 98- 44, Jan. 30, 1998). 19 Program Evaluation: Agencies Challenged by New Demand for Information on Program Results (GAO/ GGD- 98- 53, Apr. 24, 1998). 20 Head Start: Challenges in Monitoring Program Quality and Demonstrating Results (GAO/ HEHS- 98186, June 30, 1998).
B- 284548 Page 12 GAO/ GGD- 00- 52 Performance Information Challenges
and families were in other kinds of early childhood programs, or none at all.
HHS has substantially strengthened its emphasis on determining whether Head Start has achieved its purpose. In part in response to the direction of Congress, HHS has new initiatives that will, in the next few years, provide information not previously available on outcomes, such as gains made by children and their families while in the program. In addition, the program is currently designing an impact study to assess whether children and their families would have achieved these gains without participating in Head Start. Congress has required that HHS submit a final report on the impact of the Head Start program by September 30, 2003.
The long- standing inability of many federal agencies to accurately record and report financial management data on both a year- end and an ongoing basis for decisionmaking and oversight purposes continues to be a serious weakness. Without reliable data on costs, decisionmakers cannot effectively evaluate programs' financial performance or control and reduce costs. Under the Chief Financial Officers (CFO) Act, agencies are expected to develop and deploy modern financial management systems and to routinely produce sound cost and operating performance information, among other things. Further, the Federal Financial Management Improvement Act (FFMIA) focuses on ensuring greater attention to making much needed improvements in financial management systems. The primary purpose of FFMIA is to ensure that agency financial management systems routinely provide reliable, useful, and timely financial information. With such information, government leaders will be better positioned to invest scarce resources, reduce costs, oversee programs, and hold agency managers accountable for the way they run government programs.
Table 1 shows the financial statement audit results for fiscal year 1998 for the 24 CFO Act agencies. Financial Management
Capabilities Are Needed to Support Decisionmaking and Accountability
B- 284548 Page 13 GAO/ GGD- 00- 52 Performance Information Challenges
Opinons Agencies
Department of Housing and Urban Development
Department of the Interior
Department of Labor
Department of State
Environmental Protection Agency
Federal Emergency Management Agency
General Services Administration
National Aeronautics and Space Administration
National Science Foundation
Nuclear Regulatory Commission
Small Business Administration
Social Security Administration Unqualified audit opinions:
The financial statements are reliable in all material respects.
Department of Health and Human Services
Department of Energy
Department of the Treasury
Department of Veterans Affairs Qualified audit opinions:
Except for some item( s), which are mentioned in the auditor's report, the financial statements are reliable in all material respects.
Department of Agriculture
Department of Defense
Department of Education
Department of Justice
Department of Transportation
U. S. Agency for International Development Disclaimers:
The auditor does not know if the financial statements are reliable in all material respects.
Department of Commerce received an unqualified opinion on its balance sheet and a disclaimer on its other financial statements.
Office of Personnel Management's Retirement Program, Life Insurance Program, and Health Benefits Insurance Program received unqualified opinions; the Revolving Funds and the Salaries and Expenses Accounts received disclaimers. Other:
Source: Individual agency reports on results of audits of fiscal year 1998 financial statements, as of October 1999.
In addition, financial management systems for 21 of the 24 agencies were found by auditors not to comply substantially with FFMIA's requirements for fiscal year 1998. The three agencies in compliance were the Department of Energy, National Aeronautics and Space Administration, and the National Science Foundation.
Table 1: Audit Opinions for the 24 CFO Agencies' Fiscal Year 1998 Financial Statements
B- 284548 Page 14 GAO/ GGD- 00- 52 Performance Information Challenges
For some agencies, the preparation of financial statements requires considerable reliance on ad hoc programming and analysis of data produced by inadequate financial management systems that are not integrated or reconciled, and that often require significant audit adjustments. The key for agencies is to take steps to continuously improve internal controls and underlying financial and management information systems. These systems must generate timely, accurate, and useful information on an ongoing basis. The overhauling of financial and related management information systems is the overarching challenge for agencies in generating timely, reliable data throughout the year. The following examples illustrate serious financial management weaknesses and systems problems.
While the Department of Defense (DOD) is responsible for vast operations with an estimated $1 trillion in assets, nearly $1 trillion in liabilities, and a net cost of operations of $280 billion in fiscal year 1998 no major part of the department has been able to pass the test of an independent audit because of pervasive financial management weaknesses. Such weaknesses led us in 1995 to put DOD financial management on our list of high- risk areas vulnerable to waste, fraud, and mismanagement a designation that continued unchanged in our more recent high- risk update. 21 These financial management weaknesses limit the reliability and timeliness of DOD's currently available financial information. DOD management and/ or auditors have repeatedly found DOD systems to be inadequate for measuring the cost of operations and programs. For example:
 DOD has acknowledged that the lack of a cost accounting system is the single largest impediment to controlling and managing weapon systems costs, including costs of acquiring, managing, and disposing of weapon systems.
 DOD is unable to provide actual data on the cost associated with functions to be considered for A- 76 outsourcing competitions, including the capital costs associated with operations.
 DOD has long- standing problems accumulating and reporting the full costs associated with working capital fund operations, which provide goods and services in support of the military services.
As a result of our assessment of DOD's fiscal year 2000 performance plan, we noted that the lack of adequate cost information impairs the development of cost- based performance measures and indicators across
21 GAO/ HR- 99- 1, January 1999; and High- Risk Series: An Overview (GAO/ HR- 95- 1, February 1995).
B- 284548 Page 15 GAO/ GGD- 00- 52 Performance Information Challenges
virtually the entire spectrum of DOD's program operations. 22 While DOD developed 43 unclassified performance measures and indicators to measure a wide variety of activities from force levels to asset visibility these measures and indicators contained few efficiency measures based on cost.
In our most recent testimony on DOD financial management, we reported that DOD has started to devote additional resources to correcting its financial management weaknesses. 23 DOD's Financial Management Improvement Plans represent an ambitious undertaking and are an important step toward long- term improvements in the department's accountability. However, eliminating DOD's financial management weaknesses represents a major challenge because they are pervasive and entrenched in an extremely large decentralized organization.
As another example, in January 1999, we designated the Federal Aviation Administration's (FAA) financial management a high- risk area because of serious and long- standing accounting and financial reporting weaknesses. 24 These weaknesses render FAA vulnerable to waste, fraud, and abuse; undermine its ability to manage its operations; and limit the reliability of financial information it provides to Congress.
Beginning with fiscal year 1994, the Department of Transportation's Office of the Inspector General has audited FAA's financial statements and has consistently been unable to determine whether the financial information is reliable. This pattern of negative financial audit results has continued with its most recent report a disclaimer of opinion on FAA's fiscal year 1998 financial statements, citing as a primary reason the inability to verify property, plant, and equipment (PP& E) reported at a cost of $11.9 billion. 25
We previously reported that many problems in the PP& E accounts affect FAA's ability to efficiently and effectively manage programs that use these assets. 26 We also reported that many problems in these accounts result
22 Observations on the Department of Defense's Fiscal Year 2000 Performance Plan (GAO/ NSIAD- 99178R, July 20, 1999). 23 Department of Defense: Status of Financial Management Weaknesses and Actions Needed to Correct Continuing Challenges (GAO/ T- AIMD/ NSIAD- 99- 171, May 4, 1999). 24 GAO/ HR- 99- 1, January 1999.
25 Federal Aviation Administration: Financial Management Issues (GAO/ T- AIMD- 99- 122, Mar. 18, 1999). 26 Financial Management: Federal Aviation Administration Lacked Accountability for Major Assets (GAO/ AIMD- 98- 62, Feb. 18, 1998).
B- 284548 Page 16 GAO/ GGD- 00- 52 Performance Information Challenges
from the lack of a reliable system for accumulating project cost accounting information. The inadequacy of FAA's cost accounting system is a weakness that prevents the agency from having reliable and timely information about the full cost of program activities. The lack of cost accounting information also limits FAA's ability to, among other things, meaningfully evaluate performance in terms of efficiency and costeffectiveness.
FAA senior management have indicated that they recognize the urgency of correcting their financial management deficiencies and have taken steps to address them, including efforts to continue to develop a cost accounting system, which FAA expects will be fully operational in 2001. 27 However, as we reported in July 1999, while FAA has taken steps that are likely to lead to or already have resulted in improved accountability for FFA assets, much still remains to be done. 28
Agencies' March 2000 performance reports will provide them with an opportunity to show the progress they have made in addressing data credibility issues. As far back as our earliest assessment of agencies' efforts to implement the Results Act, and more recently in our reviews of agencies' strategic and performance plans, we identified data credibility issues as a persistent and continuing challenge for agencies. 29 In passing the Results Act, however, Congress emphasized that the usefulness of agencies' performance information depends, to a large degree, on the reliability and validity of their data.
During this past year, we issued several reports on practices and approaches that agencies have proposed or adopted that address data credibility issues. 30 For example, we reported that applied practices, such as identifying actions to compensate for unavailable or low- quality data and discussing implications of data limitations for assessing performance, can help agencies describe their capacity to gather and use performance
27 GAO/ T- AIMD- 99- 122, March 18, 1999. 28 FAA Financial Management: Further Actions Needed to Achieve Asset Accountability (GAO/ AIMD99- 212, July 30, 1999). 29 GAO/ GGD/ AIMD- 99- 215, July 20, 1999; GAO/ GGD/ AIMD- 98- 228, September 8, 1998; GAO/ GGD- 98- 44, January 30, 1998; GAO/ GGD- 97- 109, June 2, 1997; and GPRA Performance Reports (GAO/ GGD- 96- 66R, Feb. 14, 1996).
30 Managing for Results: Strengthening Regulatory Agencies' Performance Management Practices (GAO/ GGD- 00- 10, Oct. 28, 1999); GAO/ GGD- 99- 139, July 30, 1999; and Agency Performance Plans: Examples of Practices That Can Improve Usefulness to Decisionmakers (GAO/ GGD/ AIMD- 99- 69, Feb. 26, 1999). Performance Reports
Provide Opportunities to Show Progress in Addressing Data Credibility Issues
B- 284548 Page 17 GAO/ GGD- 00- 52 Performance Information Challenges
information. To illustrate, the Department of Transportation stated in its fiscal year 1999 performance plan that one of the most significant limitations of both internal and external data is timeliness. One way the department plans to deal with this limitation is to compile preliminary estimates from the portion of data that is available in time to report on the performance measures. According to the plan, fatality data from the first 6 months of the year could be compared with data from the first 6 months of the previous year for an initial performance measurement.
In our report on reasonable approaches to verify and validate performance information, we identified a wide range of possible approaches that can be organized into four general strategies, as follows:
 Management can seek to improve the quality of performance data by fostering an organizational commitment and capacity for data quality.
 Verification and validation can include assessing the quality of existing performance data.
 Assessments of data quality are of little value unless agencies are responding to identified data limitations.
 Building quality into the development of performance data may help prevent future errors and minimize the need to continually fix existing data.
These approaches can help agencies improve the quality, usefulness, and credibility of performance information. However, as noted earlier, making stakeholders aware of significant data limitations allows them to judge the data's credibility for their intended use and to use the data in appropriate ways. All data have limitations that may hinder their use for certain purposes but still allow them to be used for others. Stakeholders may not have enough familiarity with the data to recognize the significance of their shortcomings. Therefore, appropriate use of performance data may be fostered by clearly communicating how and to what extent data limitations affect assessments of performance. For example, we noted that when the Department of the Treasury's Customs Service field staff realized management was using performance data to make decisions, the staff began providing explanations for any incorrect data. Customs also said it required each office to establish a data quality function, responsible for verification and validation, which would be inspected annually.
A federal environment that focuses on results relies on new types of information that are different from those that have traditionally been collected by federal agencies. Obtaining more credible results- oriented performance information is essential for agencies to plan their efforts and Conclusion
B- 284548 Page 18 GAO/ GGD- 00- 52 Performance Information Challenges
gauge progress toward the achievement of their goals. However, as we previously reported, agencies have encountered some difficult analytic and technical challenges in obtaining timely and reliable results- oriented performance information and in ensuring that program evaluations that allow for the informed use of that information are undertaken. 31
The Results Act requires agencies to describe in their annual performance plans how they will verify and validate the performance information that will be collected. Including such information in performance reports can be equally important in helping to assure report users of the quality of the performance data. Discussing data credibility and related issues in performance reports can provide important contextual information to Congress and agencies to help them address the weaknesses in this area. For example, this sort of discussion in an agency's performance report can alert Congress to the problems the agency has had in collecting needed results- oriented performance information. Agencies can also alert Congress to the cost and data quality trade- offs associated with various collection strategies, such as relying on sources outside the agency to provide performance data and the degree to which those data are expected to be reliable. Finally, in order to give a clear picture of the agency's performance and its efforts at improvement, annual reports on performance can also relate performance measurement information to program evaluation findings.
We are sending copies of this report to Senator Joseph I. Lieberman, Ranking Minority Member, Senate Governmental Affairs Committee; Representative Richard A. Gephardt, Minority Leader, House of Representatives; Representative Henry A. Waxman, Ranking Minority Member, House Government Reform Committee; Representative John M. Spratt, Jr., Ranking Minority Member, House Budget Committee; and the Honorable Jacob J. Lew, Director, Office of Management and Budget. Copies will be made available to others on request.
31 See, for example, GAO/ GGD- 97- 109, June 2, 1997, and GAO/ HEHS/ GGD- 97- 138, May 30, 1997.
B- 284548 Page 19 GAO/ GGD- 00- 52 Performance Information Challenges
Please contact me at (202) 512- 8676 if you have any questions. Dottie Self was the key contributor to this report.
J. Christopher Mihm Associate Director, Federal Management and Workforce Issues
Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.
Order by mail: U. S. General Accounting Office P. O. Box 37050 Washington, DC 20013
or visit: Room 1100 700 4 th St. NW (corner of 4 th and G Sts. NW) U. S. General Accounting Office Washington, DC
Orders may also be placed by calling (202) 512- 6000 or by using fax number (202) 512- 6061, or TDD (202) 512- 2537.
Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512- 6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.
For information on how to access GAO reports on the INTERNET, send e- mail message with info in the body to:
info@ www. gao. gov or visit GAO's World Wide Web Home Page at: http:// www. gao. gov
United States General Accounting Office Washington, D. C. 20548- 0001
Official Business Penalty for Private Use $300
Address Correction Requested Bulk Rate
Postage & Fees Paid GAO Permit No. G100
(410505)
MANAGING FOR RESULTS: Challenges Agencies Face in Producing
Credible Performance Information (GAO/GGD-00-52) MANAGING FOR
RESULTS Challenges Agencies Face in Producing Credible Performance
Information United States General Accounting Office GAO Report to
Congressional Requesters February 2000   GAO/GGD-00-52  United
States General Accounting Office General Government Division
Washington, D. C. 20548 Page 1 GAO/GGD-00-52 Performance
Information Challenges B-284548 February 4, 2000 The Honorable
Fred Thompson Chairman Committee on Governmental Affairs United
States Senate The Honorable Dick Armey Majority Leader House of
Representatives The Honorable Dan Burton Chairman Committee on
Government Reform House of Representatives The Honorable John R.
Kasich Chairman Committee on the Budget House of Representatives
The Government Performance and Results Act (Results Act or GPRA)
seeks to shift the focus of federal management and decisionmaking
from a preoccupation with the number of tasks completed or
services provided to a more direct consideration of the results of
programs that is, the real differences the tasks or services make
to the nation or individual taxpayer. The Results Act originated
in part from Congress's frustration over the fact that
congressional policymaking, spending decisions, and oversight and
agencies' decisionmaking all had been seriously handicapped by the
lack of clear goals and sound performance information. To remedy
that situation, the Results Act requires agencies to set multiyear
strategic goals in their strategic plans and corresponding annual
goals in their performance plans, measure performance toward the
achievement of those goals, and report on their progress in their
annual performance reports. These reports are intended to provide
important information to agency managers, policymakers, and the
public on what each agency accomplished with the resources it was
given. This report responds to your request that we identify some
of the challenges agencies face in producing credible performance
information and how those challenges may affect performance
reporting the next B-284548 Page 2 GAO/GGD-00-52 Performance
Information Challenges phase of the Results Act implementation. As
agreed, our objectives were to (1) discuss whether the weaknesses
we have identified in agencies' performance plans imply challenges
for the performance reports, (2) illustrate some of the challenges
agencies face in producing credible performance data, and (3)
describe how performance reports can be used to address data
credibility issues. The Results Act is aimed at improving
performance of government programs by requiring agencies to
clarify their missions, establish goals and strategies for
reaching them, measure performance, and report on their
accomplishments. Beginning with fiscal year 1999, the head of each
agency is to prepare and submit to Congress and the president a
report on program performance. The first of these annual reports
is to be submitted no later than March 31, 2000. These reports are
to contain two main parts: a report on the actual performance
achieved as compared with the performance goals expressed in the
performance plan and the plans and schedules to achieve those
goals that were not met. If a performance goal becomes impractical
or infeasible to achieve, the agency is to explain why that is the
case and what legislative, regulatory, or other actions are needed
to accomplish the goal, or whether the goal ought to be modified
or discontinued. Finally, the reports should also relate
performance measurement information to program evaluation
findings, in order to give a clear picture of an agency's
performance and its efforts at improvement. 1 Based on our
previously issued work, it appears unlikely that agencies
consistently will have for their first performance reports the
reliable performance information needed to assess whether
performance goals are being met or specifically how performance
can be improved. Our work over the past several years has
identified limitations in agencies' abilities to produce credible
data and identify performance improvement opportunities. These
limitations are substantial, long- standing, and will not be
quickly or easily resolved. They are likely to be reflected in
agencies' initial performance reports as they have been in the
performance plans prepared to date. For example, policy decisions
made when designing federal programs, particularly
intergovernmental programs, may make it difficult to collect
timely and consistent national data. In administering programs
that are a joint responsibility with state and local governments,
Congress and the 1 For specific information on the requirements of
the Government Performance and Results Act, see Public Law 103-
62, August 3, 1993; and Senate Report 103- 58, June 16, 1993.
Also, see Office of Management and Budget Circular No. A- 11, Part
2, July 1999, for information on the preparation and submission of
strategic plans, annual performance plans, and annual program
performance reports. Background Results in Brief B-284548 Page 3
GAO/GGD-00-52 Performance Information Challenges executive branch
continually balance the competing objectives of collecting uniform
program information to assess performance with giving states and
localities the flexibility needed to effectively implement
intergovernmental programs. In addition, the relatively limited
level of agencies' program evaluation capabilities a perennial
problem suggests that many agencies are not well positioned to
undertake necessary evaluations. Program evaluations are important
to providing information on the extent to which an agency's
efforts contributed to results and to highlight opportunities to
improve those results. Finally, long- standing weaknesses in
agencies' financial management capabilities make it difficult for
decisionmakers to effectively assess and improve many programs'
financial performance. In order to help agency managers select
appropriate techniques for assessing, documenting, and improving
the quality of their performance data, we have reported on some
reasonable approaches that agencies had proposed or adopted to
verify and validate performance information. 2 These approaches
include senior management actions, agencywide efforts, and
specific program manager and technical staff activities. We noted
that agencies could use these approaches, where appropriate, to
improve the quality, usefulness, and credibility of performance
information. In that regard, performance reports provide agencies
with an opportunity to show the progress they have made in
addressing data credibility issues. The Results Act requires
agencies to describe in their annual performance plans how they
will verify and validate the performance information that will be
collected. Including information in performance reports that
describes the quality of the reported performance data and the
implications of missing data can be equally important and can
provide key contextual information to Congress and other users of
the performance reports. To address our objectives, we relied on
our large body of work on agencies' performance data problems and
related issues. We drew examples from our reviews of agencies'
efforts to implement the Results Act, such as our reviews of
agencies' fiscal years 1999 and 2000 performance plans, and
products we have issued on major management 2 Performance Plans:
Selected Approaches for Verification and Validation of Agency
Performance Information (GAO/GGD-99-139, July 30, 1999). Scope and
Methodology B-284548 Page 4 GAO/GGD-00-52 Performance Information
Challenges challenges and risks, such as our performance and
accountability series and high- risk series. 3 Because this report
is based primarily on our previously issued reports, we did not
obtain agency comments. Our work on this report was done from
October 1999 to January 2000, in Washington, D. C., in accordance
with generally accepted government auditing standards. Agencies
need reliable information during their planning efforts to set
realistic goals and later, as programs are being implemented, to
gauge their progress toward achievement of those goals. 4 In our
assessments of annual performance plans, we identified challenges
that will affect agencies' abilities to reliably report on the
achievement of program goals and, in cases where goals are not
met, either identify opportunities for improvement or whether
goals need to be adjusted. 5 For example, we concluded in our
review of fiscal year 1999 performance plans that future plans
would be more useful if they would, among other things, (1) more
fully articulate how strategies and resources will lead to
improved performance and (2) provide much greater confidence that
performance information will be credible and useful for
decisionmaking. Although, on the whole, the fiscal year 2000 plans
showed moderate improvements over the fiscal year 1999 plans, we
identified these two issues as continuing key weaknesses common
among agencies' plans. Most of the fiscal year 2000 plans related
strategies and programs to performance goals; however, few plans
indicated how the strategies would contribute to accomplishing the
expected level of performance. Agencies need to understand and
articulate how what they do on a day- to- day basis contributes to
mission- related results. Such an understanding is important for
agencies to pinpoint opportunities to improve performance and
design and implement appropriate initiatives. This information is
also helpful to congressional and other decisionmakers in
assessing the degree to which strategies are appropriate and
reasonable. 3 See, for example, Managing for Results:
Opportunities for Continued Improvements in Agencies' Performance
Plans (GAO/ GGD/ AIMD- 99- 215, July 20, 1999); Managing for
Results: An Agenda To Improve the Usefulness of Agencies' Annual
Performance Plans (GAO/ GGD/ AIMD- 98- 228, Sept. 8, 1998);
Performance and Accountability Series: Major Management Challenges
and Program Risks (GAO/OCG-99-1 through GAO- OCG- 99- 21, January
1999); and High- Risk Series: An Update (GAO/ HR99- 1, January
1999). 4 The Government Performance and Results Act: 1997
Governmentwide Implementation Will Be Uneven (GAO/GGD-97-109, June
2, 1997). 5 GAO/ GGD/ AIMD- 99- 215, July 20, 1999; and GAO/ GGD/
AIMD- 98- 228, September 8, 1998. Challenges in Performance
Planning Will Affect Performance Reporting Few Plans Indicated How
Strategies Would Contribute to Accomplishing Expected Level of
Performance B-284548 Page 5 GAO/GGD-00-52 Performance Information
Challenges The outcomes of many federal programs are the result of
the interplay of several factors, and only some of these are
within a program's control. 6 Further, on a daily basis, agencies
do not produce outcomes, but rather outputs, such as activities,
products, or services that are intended to contribute to outcomes.
Thus, a key analytic challenge for agencies is knowing how their
programmatic efforts contribute to their desired outcomes. The
inconsistent attention to this critical element undermined the
value of agencies' performance plans and, unless addressed, it
also will severely limit the value of their performance reports.
In 1997, we reported that the Department of Justice's Immigration
and Naturalization Service (INS) lacked data on the overall
effectiveness of its southwest border strategy. 7 For example,
data were insufficient to indicate whether illegal aliens were
deterred from entering the United States, whether there had been a
decrease in attempted reentries by those who had previously been
apprehended, and whether the strategy had reduced border violence.
We noted that, despite the investment of billions of dollars in
the strategy, INS had amassed only a partial picture of the
effects of increased border control and did not know whether the
investment was producing the intended results. We reported that a
comprehensive, systematic evaluation of the agency's strategy to
deter illegal entry along the southwest border would provide INS
with information on whether its border enforcement strategy had
produced the intended results. After our report, INS contracted
with independent research firms for an evaluation. Another
weakness that we identified in our review of agencies' fiscal year
1999 and fiscal year 2000 performance plans was the limited
confidence they provided in the credibility of performance
information. Credible performance information is essential for
accurately assessing agencies' progress towards the achievement of
their goals the cornerstone of performance reporting. As shown in
figure 1, our analysis of agencies' fiscal year 2000 performance
plans noted that most of the plans provided only limited
confidence that performance information would be credible. Only
the plans for the Department of Education, the Department of
Justice, the Department of Transportation, and the Social Security
Administration provided general confidence that their performance
information would be credible. 6 Managing for Results: Analytic
Challenges in Measuring Performance (GAO/ HEHS/ GGD- 97- 138, May
30, 1997). 7 Illegal Immigration: Southwest Border Strategy
Results Inconclusive; More Evaluation Needed (GAO/GGD-98-21, Dec.
11, 1997). Few Plans Provided Confidence That Performance
Information Would Be Credible B-284548 Page 6 GAO/GGD-00-52
Performance Information Challenges Source: GAO/ GGD/ AIMD- 99-
215, July 20, 1999. Decisionmakers must have assurance that the
program and financial data being used will be sufficiently timely,
complete, accurate, useful, and consistent if these data are to
inform decisionmaking. However, like the fiscal year 1999
performance plans, most of the fiscal year 2000 plans lacked
information on the procedures the agencies would use to verify and
validate performance information. Similar to our findings with the
fiscal year 1999 plans, we also found that, in general, the fiscal
year 2000 plans failed to include discussions of strategies to
address known data limitations. We reported that when performance
data are unavailable or of low quality, a performance plan would
be more useful to decisionmakers if it briefly discussed how the
agency plans to deal with such limitations. Without such a
discussion, decisionmakers will have difficulty determining the
implications for assessing the subsequent achievement of
performance goals that agencies include in their performance
reports. Figure 1: Confidence in Performance Data B-284548 Page 7
GAO/GGD-00-52 Performance Information Challenges In order to
successfully measure and report progress toward intended results,
agencies need to build the capacity to gather and use performance
information. However, our work over the past several years has
identified limitations in agencies' abilities to produce credible
performance data. Specifically, those limitations relate to
program design issues that may make it difficult to collect timely
and consistent national data, the relatively limited level of
agencies' program evaluation capabilities, and long- standing
weaknesses in agencies' financial management capabilities. In
several program areas, devolution of program responsibility from
the federal level and consolidation of individual federal programs
into more comprehensive, multipurpose grant programs have shifted
both program management and accountability responsibilities toward
the states. These programs vary greatly in the kind and degree of
flexibility afforded to state or local entities, distribution of
accountability across levels of government, and availability of
direct measures of program performance. In our report on grant
program design features, we noted that relatively few flexible
programs collected uniform data on the outcomes of state or local
service activities. 8 Collecting such data requires conditions
such as uniformity of activities, objectives, and measures that do
not exist under many flexible program designs. 9 For instance, we
reported that the block grants enacted as part of the Omnibus
Budget Reconciliation Act of 1981 carried no uniform federal
information and reporting requirements. 10 States collected a wide
range of program information, but the collection efforts were
designed to meet the needs of the individual states. Congress had
limited information on program activities, services delivered, and
clients served. As a result, it was difficult, in many cases, to
aggregate state experiences and speak from a national perspective
on the block grant activities or their effects. Similarly, without
uniform information definitions and collection methodologies, it
was difficult to compare state efforts or draw meaningful
conclusions about the relative effectiveness of different
strategies. 8 Grant Programs: Design Features Shape Flexibility,
Accountability, and Performance Information (GAO/GGD-98-137, June
22, 1998). 9 The recently passed Federal Financial Assistance
Management Improvement Act of 1999 is designed, among other
things, to establish greater commonality in reporting requirements
and procedures across federal financial assistance programs and to
foster improvements in the quality of the data collected. The act
recognizes that changes in current law may be needed to accomplish
these purposes and directs that recommendations concerning such
changes be reported to Congress no later than May 2001. 10 Block
Grants: Issues in Designing Accountability Provisions (GAO/AIMD-
95-226, Sept. 1, 1995). Agencies Face Limitations in Producing
Credible Performance Data Program Design Features Have
Implications for the Availability of Performance Information B-
284548 Page 8 GAO/GGD-00-52 Performance Information Challenges
Education is one of many agencies where the interest in having
enough information for accountability and federal program
management continually competes with the aim of providing local
agencies with the flexibility needed to implement their programs
on the basis of their local needs. 11 The Safe and Drug- Free
Schools program, for example, allows a wide range of activities,
such as drug prevention instruction for students; staff training;
general violence- prevention instruction; and special one- time
events, such as guest speakers and drug- and alcohol- free social
activities. 12 States are also permitted to define the information
they collect on program activities and effectiveness. Under the
Safe and Drug- Free Schools and Communities Act, Education
oversees state programs and state agencies monitor local programs.
Under the act, each state may establish its own reporting
requirements for local education agencies. Although these
requirements have some common elements, state requirements vary
widely. With no requirements that states use consistent measures,
Education faces, as our work has shown, a difficult challenge in
assembling the required state reports to develop a nationwide
picture of the program's effectiveness. Because states,
localities, or nongovernmental organizations operate many of its
programs, the Department of Health and Human Services (HHS)
experiences similar challenges. 13 The Personal Responsibility and
Work Opportunity Reconciliation Act dramatically altered the
nation's system for providing assistance to the poor. Among the
many changes, the act replaced the existing entitlement program
for poor families (Aid to Families With Dependent Children) with
fixed block grants to the states to provide Temporary Assistance
for Needy Families (TANF). Under the TANF block grant, states have
flexibility in designing and implementing their own assistance
programs within federal guidelines. Meanwhile, HHS has a broad
range of responsibilities for ensuring accountability from the
states. The welfare reform law gives HHS administrative and
oversight responsibilities, the performance of which will rely on
state- provided data. HHS needs to ensure that it receives
comparable and reliable data from the 11 Department of Education:
Information Needs Are at the Core of Management Challenges Facing
the Department (GAO/T-HEHS-98-124, Mar. 24, 1998); and Balancing
Flexibility and Accountability: Grant Program Design in Education
and Other Areas (GAO/ T- GGD/ HEHS- 98- 94, Feb. 11, 1998). 12
Safe and Drug- Free Schools: Balancing Accountability With State
and Local Flexibility (GAO/ HEHS98- 3, Oct. 10, 1997). 13
Department of Health and Human Services: Strategic Planning and
Accountability Challenges (GAO/T-HEHS-98-96, Feb. 26, 1998). B-
284548 Page 9 GAO/GGD-00-52 Performance Information Challenges
states to help it fulfill its oversight responsibilities, which
include ensuring that states enforce the federal 5- year time
limit on receiving welfare benefits, meet minimum work
participation rates, and maintain a certain level of state welfare
spending. Enforcing the time limit, for example, will be difficult
because information on the total amount of time that someone has
received TANF is not always available in individual states, let
alone across states. In addition, the law gives HHS authority to
assess penalties if states fail to comply with certain
requirements and provides for states to receive bonuses if they
meet certain performance standards. HHS needs to collect state
data to determine performance penalties and bonuses. In view of
the increased flexibility of states in designing their programs,
obtaining comparable and reliable data to assess the effect of
welfare reform on children and families could be difficult for
HHS. The Environmental Protection Agency (EPA) provides another
example of a federal agency that depends on the state and local
agencies it is working with to provide the performance information
that indicates whether results are being achieved. For example,
the state water quality reports required by the Clean Water Act
are a key source of information for measuring progress in cleaning
up the nation's lakes, rivers, and streams. However, EPA has found
that the wealth of environmental data EPA and states collect are
often difficult to compile in a meaningful way. 14 As contained in
the Clean Water Act, Congress left the primary monitoring
responsibility to the states for measuring progress in cleaning up
the nation's lakes, rivers, and streams. However, inconsistencies
in water quality assessments and in assessment methodologies from
state to state make it difficult to aggregate the data and to use
the information to conclusively determine whether the quality of
rivers, lakes, and streams is getting better or worse over time.
Absent this information, it has been difficult for EPA to set
priorities, evaluate the success of its programs and activities,
and report on its accomplishments in a credible and informed way.
The unavailability of reliable performance information can also be
traced to a lack of standards and of common definitions for terms
used to evaluate programs. For example, we reported that the
agencies involved in wetlands- related activities inconsistently
used terms such as protection, restoration, rehabilitation,
improvement, and enhancement in describing 14 Major Management
Challenges and Program Risks: Environmental Protection Agency
(GAO/ OCG- 9917, January 1999). B-284548 Page 10 GAO/GGD-00-52
Performance Information Challenges and reporting on their
accomplishments. 15 Even when the same terms are used, the
agencies do not define them in the same way. As a result, the
consistency and reliability of data on the status of wetlands
acreage are questionable. Thus, neither the progress made toward
achieving the governmentwide goal of no net loss of the nation's
remaining wetlands nor the contributions made by the agencies in
achieving this goal can be accurately measured. Weaknesses in the
availability of direct measures of performance can be overcome by
drawing on information from other sources, such as program
evaluation studies, research on the effectiveness of service
delivery, or aggregate data such as vital statistics that describe
the general status of a population. In this regard, our report on
grant program design features noted that 13 of the 21 flexible
grant programs reviewed used such sources along with, or as a
substitute for, performance measures collected from program
operations. 16 We found that agencies that made use of multiple
sources had information that covered more aspects of program
performance than those that relied upon a single source. Program
evaluation studies are important for assessing how well programs
are working, determining factors affecting performance, and
identifying improvement opportunities. In our report on the
analytic challenges facing agencies in measuring performance, we
stated that supplementing performance data with impact evaluations
might help provide agencies with a more complete picture of
program effectiveness. 17 Evaluations can play a critical role in
helping to address those measurement and analysis difficulties
agencies face that stem from two features common to many federal
programs: the interplay of federal, state, and local government
activities, and objectives and the aim to influence complex
systems or phenomena whose outcomes are largely outside government
control. Furthermore, systematic evaluation of how a program was
implemented can provide important information about why a program
did or did not succeed as well as suggest ways to improve it.
However, as we reported in our assessment of agencies' fiscal year
1999 performance plans, we continue to be concerned about the lack
in many federal agencies of the capacity to undertake the program
evaluations that will be vital to the success of the Results Act.
In our earlier review of 15 Wetlands Overview: Problems With
Acreage Data Persist (GAO/RCED-98-150, July 1, 1998). 16 GAO/GGD-
98-137, June 22, 1998. 17 GAO/ HEHS/ GGD- 97- 138, May 30, 1997.
Program Evaluation Is Essential, But Federal Capacity Is Limited
B-284548 Page 11 GAO/GGD-00-52 Performance Information Challenges
agencies' strategic plans, we found that many agencies had not
given sufficient attention to how program evaluations will be used
in implementing the Results Act and improving performance. 18 In
another report, we noted that agencies' program evaluation
capabilities would be challenged to meet the new demands for
information on program results. 19 We found that the resources
allocated for conducting program evaluations were small and
unevenly distributed across the 13 departments and 10 independent
agencies we surveyed for that report. Good evaluation information
about program effects is difficult to obtain. Each of the tasks
involved measuring outcomes, ensuring the consistency and quality
of data collected, establishing the causal connection between
outcomes and program activities, and separating out the influence
of extraneous factors raises formidable technical or logistical
problems that are not easily resolved. Thus, evaluating program
impact generally requires a planned study and, often, considerable
time and expense. The experiences of the Head Start program
illustrate the importance and difficulty of systematic program
evaluation. Head Start, administered by HHS' Administration for
Children and Families, is one of the most popular federal early
childhood programs and has long enjoyed both congressional and
public support. 20 Between fiscal years 1990 and 1998, annual Head
Start funding nearly tripled, from $1. 5 billion to almost $4.4
billion. Head Start's purpose is to improve the social competence
of children in lowincome families, and in the past 33 years, the
program has provided a comprehensive set of services to about 16
million low- income children. Educational, medical, nutritional,
mental health, dental, social, and other services have been
provided to low- income children and their families in all 50
states, the District of Columbia, Puerto Rico, and the U. S.
territories, as well as to migrant and Native American
populations. Given the size of the Head Start program and the
efforts to expand the program's annual enrollment to one million
children by 2002, investing in studies that will assess its impact
is important. Specifically, the challenge for HHS is to determine
whether the same outcomes would have occurred if children 18
Managing for Results: Agencies' Annual Performance Plans Can Help
Address Strategic Planning Challenges (GAO/GGD-98-44, Jan. 30,
1998). 19 Program Evaluation: Agencies Challenged by New Demand
for Information on Program Results (GAO/GGD-98-53, Apr. 24, 1998).
20 Head Start: Challenges in Monitoring Program Quality and
Demonstrating Results (GAO/ HEHS- 98186, June 30, 1998). B-284548
Page 12 GAO/GGD-00-52 Performance Information Challenges and
families were in other kinds of early childhood programs, or none
at all. HHS has substantially strengthened its emphasis on
determining whether Head Start has achieved its purpose. In part
in response to the direction of Congress, HHS has new initiatives
that will, in the next few years, provide information not
previously available on outcomes, such as gains made by children
and their families while in the program. In addition, the program
is currently designing an impact study to assess whether children
and their families would have achieved these gains without
participating in Head Start. Congress has required that HHS submit
a final report on the impact of the Head Start program by
September 30, 2003. The long- standing inability of many federal
agencies to accurately record and report financial management data
on both a year- end and an ongoing basis for decisionmaking and
oversight purposes continues to be a serious weakness. Without
reliable data on costs, decisionmakers cannot effectively evaluate
programs' financial performance or control and reduce costs. Under
the Chief Financial Officers (CFO) Act, agencies are expected to
develop and deploy modern financial management systems and to
routinely produce sound cost and operating performance
information, among other things. Further, the Federal Financial
Management Improvement Act (FFMIA) focuses on ensuring greater
attention to making much needed improvements in financial
management systems. The primary purpose of FFMIA is to ensure that
agency financial management systems routinely provide reliable,
useful, and timely financial information. With such information,
government leaders will be better positioned to invest scarce
resources, reduce costs, oversee programs, and hold agency
managers accountable for the way they run government programs.
Table 1 shows the financial statement audit results for fiscal
year 1998 for the 24 CFO Act agencies. Financial Management
Capabilities Are Needed to Support Decisionmaking and
Accountability B-284548 Page 13 GAO/GGD-00-52 Performance
Information Challenges Opinons Agencies Department of Housing and
Urban Development Department of the Interior Department of Labor
Department of State Environmental Protection Agency Federal
Emergency Management Agency General Services Administration
National Aeronautics and Space Administration National Science
Foundation Nuclear Regulatory Commission Small Business
Administration Social Security Administration Unqualified audit
opinions: The financial statements are reliable in all material
respects. Department of Health and Human Services Department of
Energy Department of the Treasury Department of Veterans Affairs
Qualified audit opinions: Except for some item( s), which are
mentioned in the auditor's report, the financial statements are
reliable in all material respects. Department of Agriculture
Department of Defense Department of Education Department of
Justice Department of Transportation U. S. Agency for
International Development Disclaimers: The auditor does not know
if the financial statements are reliable in all material respects.
Department of Commerce received an unqualified opinion on its
balance sheet and a disclaimer on its other financial statements.
Office of Personnel Management's Retirement Program, Life
Insurance Program, and Health Benefits Insurance Program received
unqualified opinions; the Revolving Funds and the Salaries and
Expenses Accounts received disclaimers. Other: Source: Individual
agency reports on results of audits of fiscal year 1998 financial
statements, as of October 1999. In addition, financial management
systems for 21 of the 24 agencies were found by auditors not to
comply substantially with FFMIA's requirements for fiscal year
1998. The three agencies in compliance were the Department of
Energy, National Aeronautics and Space Administration, and the
National Science Foundation. Table 1: Audit Opinions for the 24
CFO Agencies' Fiscal Year 1998 Financial Statements B-284548 Page
14 GAO/GGD-00-52 Performance Information Challenges For some
agencies, the preparation of financial statements requires
considerable reliance on ad hoc programming and analysis of data
produced by inadequate financial management systems that are not
integrated or reconciled, and that often require significant audit
adjustments. The key for agencies is to take steps to continuously
improve internal controls and underlying financial and management
information systems. These systems must generate timely, accurate,
and useful information on an ongoing basis. The overhauling of
financial and related management information systems is the
overarching challenge for agencies in generating timely, reliable
data throughout the year. The following examples illustrate
serious financial management weaknesses and systems problems.
While the Department of Defense (DOD) is responsible for vast
operations with an estimated $1 trillion in assets, nearly $1
trillion in liabilities, and a net cost of operations of $280
billion in fiscal year 1998 no major part of the department has
been able to pass the test of an independent audit because of
pervasive financial management weaknesses. Such weaknesses led us
in 1995 to put DOD financial management on our list of high- risk
areas vulnerable to waste, fraud, and mismanagement a designation
that continued unchanged in our more recent high- risk update. 21
These financial management weaknesses limit the reliability and
timeliness of DOD's currently available financial information. DOD
management and/ or auditors have repeatedly found DOD systems to
be inadequate for measuring the cost of operations and programs.
For example:  DOD has acknowledged that the lack of a cost
accounting system is the single largest impediment to controlling
and managing weapon systems costs, including costs of acquiring,
managing, and disposing of weapon systems.  DOD is unable to
provide actual data on the cost associated with functions to be
considered for A- 76 outsourcing competitions, including the
capital costs associated with operations.  DOD has long- standing
problems accumulating and reporting the full costs associated with
working capital fund operations, which provide goods and services
in support of the military services. As a result of our assessment
of DOD's fiscal year 2000 performance plan, we noted that the lack
of adequate cost information impairs the development of cost-
based performance measures and indicators across 21 GAO/HR-99-1,
January 1999; and High- Risk Series: An Overview (GAO/HR-95-1,
February 1995). B-284548 Page 15 GAO/GGD-00-52 Performance
Information Challenges virtually the entire spectrum of DOD's
program operations. 22 While DOD developed 43 unclassified
performance measures and indicators to measure a wide variety of
activities from force levels to asset visibility these measures
and indicators contained few efficiency measures based on cost. In
our most recent testimony on DOD financial management, we reported
that DOD has started to devote additional resources to correcting
its financial management weaknesses. 23 DOD's Financial Management
Improvement Plans represent an ambitious undertaking and are an
important step toward long- term improvements in the department's
accountability. However, eliminating DOD's financial management
weaknesses represents a major challenge because they are pervasive
and entrenched in an extremely large decentralized organization.
As another example, in January 1999, we designated the Federal
Aviation Administration's (FAA) financial management a high- risk
area because of serious and long- standing accounting and
financial reporting weaknesses. 24 These weaknesses render FAA
vulnerable to waste, fraud, and abuse; undermine its ability to
manage its operations; and limit the reliability of financial
information it provides to Congress. Beginning with fiscal year
1994, the Department of Transportation's Office of the Inspector
General has audited FAA's financial statements and has
consistently been unable to determine whether the financial
information is reliable. This pattern of negative financial audit
results has continued with its most recent report a disclaimer of
opinion on FAA's fiscal year 1998 financial statements, citing as
a primary reason the inability to verify property, plant, and
equipment (PP& E) reported at a cost of $11.9 billion. 25 We
previously reported that many problems in the PP& E accounts
affect FAA's ability to efficiently and effectively manage
programs that use these assets. 26 We also reported that many
problems in these accounts result 22 Observations on the
Department of Defense's Fiscal Year 2000 Performance Plan (GAO/
NSIAD- 99178R, July 20, 1999). 23 Department of Defense: Status of
Financial Management Weaknesses and Actions Needed to Correct
Continuing Challenges (GAO/ T- AIMD/ NSIAD- 99- 171, May 4, 1999).
24 GAO/HR-99-1, January 1999. 25 Federal Aviation Administration:
Financial Management Issues (GAO/T-AIMD-99-122, Mar. 18, 1999). 26
Financial Management: Federal Aviation Administration Lacked
Accountability for Major Assets (GAO/AIMD-98-62, Feb. 18, 1998).
B-284548 Page 16 GAO/GGD-00-52 Performance Information Challenges
from the lack of a reliable system for accumulating project cost
accounting information. The inadequacy of FAA's cost accounting
system is a weakness that prevents the agency from having reliable
and timely information about the full cost of program activities.
The lack of cost accounting information also limits FAA's ability
to, among other things, meaningfully evaluate performance in terms
of efficiency and costeffectiveness. FAA senior management have
indicated that they recognize the urgency of correcting their
financial management deficiencies and have taken steps to address
them, including efforts to continue to develop a cost accounting
system, which FAA expects will be fully operational in 2001. 27
However, as we reported in July 1999, while FAA has taken steps
that are likely to lead to or already have resulted in improved
accountability for FFA assets, much still remains to be done. 28
Agencies' March 2000 performance reports will provide them with an
opportunity to show the progress they have made in addressing data
credibility issues. As far back as our earliest assessment of
agencies' efforts to implement the Results Act, and more recently
in our reviews of agencies' strategic and performance plans, we
identified data credibility issues as a persistent and continuing
challenge for agencies. 29 In passing the Results Act, however,
Congress emphasized that the usefulness of agencies' performance
information depends, to a large degree, on the reliability and
validity of their data. During this past year, we issued several
reports on practices and approaches that agencies have proposed or
adopted that address data credibility issues. 30 For example, we
reported that applied practices, such as identifying actions to
compensate for unavailable or low- quality data and discussing
implications of data limitations for assessing performance, can
help agencies describe their capacity to gather and use
performance 27 GAO/T-AIMD-99-122, March 18, 1999. 28 FAA Financial
Management: Further Actions Needed to Achieve Asset Accountability
(GAO/ AIMD99- 212, July 30, 1999). 29 GAO/ GGD/ AIMD- 99- 215,
July 20, 1999; GAO/ GGD/ AIMD- 98- 228, September 8, 1998;
GAO/GGD-98-44, January 30, 1998; GAO/GGD-97-109, June 2, 1997; and
GPRA Performance Reports (GAO/GGD-96-66R, Feb. 14, 1996). 30
Managing for Results: Strengthening Regulatory Agencies'
Performance Management Practices (GAO/GGD-00-10, Oct. 28, 1999);
GAO/GGD-99-139, July 30, 1999; and Agency Performance Plans:
Examples of Practices That Can Improve Usefulness to
Decisionmakers (GAO/ GGD/ AIMD- 99- 69, Feb. 26, 1999).
Performance Reports Provide Opportunities to Show Progress in
Addressing Data Credibility Issues B-284548 Page 17 GAO/GGD-00-52
Performance Information Challenges information. To illustrate, the
Department of Transportation stated in its fiscal year 1999
performance plan that one of the most significant limitations of
both internal and external data is timeliness. One way the
department plans to deal with this limitation is to compile
preliminary estimates from the portion of data that is available
in time to report on the performance measures. According to the
plan, fatality data from the first 6 months of the year could be
compared with data from the first 6 months of the previous year
for an initial performance measurement. In our report on
reasonable approaches to verify and validate performance
information, we identified a wide range of possible approaches
that can be organized into four general strategies, as follows:
Management can seek to improve the quality of performance data by
fostering an organizational commitment and capacity for data
quality.  Verification and validation can include assessing the
quality of existing performance data.  Assessments of data quality
are of little value unless agencies are responding to identified
data limitations.  Building quality into the development of
performance data may help prevent future errors and minimize the
need to continually fix existing data. These approaches can help
agencies improve the quality, usefulness, and credibility of
performance information. However, as noted earlier, making
stakeholders aware of significant data limitations allows them to
judge the data's credibility for their intended use and to use the
data in appropriate ways. All data have limitations that may
hinder their use for certain purposes but still allow them to be
used for others. Stakeholders may not have enough familiarity with
the data to recognize the significance of their shortcomings.
Therefore, appropriate use of performance data may be fostered by
clearly communicating how and to what extent data limitations
affect assessments of performance. For example, we noted that when
the Department of the Treasury's Customs Service field staff
realized management was using performance data to make decisions,
the staff began providing explanations for any incorrect data.
Customs also said it required each office to establish a data
quality function, responsible for verification and validation,
which would be inspected annually. A federal environment that
focuses on results relies on new types of information that are
different from those that have traditionally been collected by
federal agencies. Obtaining more credible results- oriented
performance information is essential for agencies to plan their
efforts and Conclusion B-284548 Page 18 GAO/GGD-00-52 Performance
Information Challenges gauge progress toward the achievement of
their goals. However, as we previously reported, agencies have
encountered some difficult analytic and technical challenges in
obtaining timely and reliable results- oriented performance
information and in ensuring that program evaluations that allow
for the informed use of that information are undertaken. 31 The
Results Act requires agencies to describe in their annual
performance plans how they will verify and validate the
performance information that will be collected. Including such
information in performance reports can be equally important in
helping to assure report users of the quality of the performance
data. Discussing data credibility and related issues in
performance reports can provide important contextual information
to Congress and agencies to help them address the weaknesses in
this area. For example, this sort of discussion in an agency's
performance report can alert Congress to the problems the agency
has had in collecting needed results- oriented performance
information. Agencies can also alert Congress to the cost and data
quality trade- offs associated with various collection strategies,
such as relying on sources outside the agency to provide
performance data and the degree to which those data are expected
to be reliable. Finally, in order to give a clear picture of the
agency's performance and its efforts at improvement, annual
reports on performance can also relate performance measurement
information to program evaluation findings. We are sending copies
of this report to Senator Joseph I. Lieberman, Ranking Minority
Member, Senate Governmental Affairs Committee; Representative
Richard A. Gephardt, Minority Leader, House of Representatives;
Representative Henry A. Waxman, Ranking Minority Member, House
Government Reform Committee; Representative John M. Spratt, Jr.,
Ranking Minority Member, House Budget Committee; and the Honorable
Jacob J. Lew, Director, Office of Management and Budget. Copies
will be made available to others on request. 31 See, for example,
GAO/GGD-97-109, June 2, 1997, and GAO/ HEHS/ GGD- 97- 138, May 30,
1997. B-284548 Page 19 GAO/GGD-00-52 Performance Information
Challenges Please contact me at (202) 512- 8676 if you have any
questions. Dottie Self was the key contributor to this report. J.
Christopher Mihm Associate Director, Federal Management and
Workforce Issues Ordering Information The first copy of each GAO
report and testimony is free. Additional copies are $2 each.
Orders should be sent to the following address, accompanied by a
check or money order made out to the Superintendent of Documents,
when necessary. VISA and MasterCard credit cards are accepted,
also. Orders for 100 or more copies to be mailed to a single
address are discounted 25 percent. Order by mail: U. S. General
Accounting Office P. O. Box 37050 Washington, DC 20013 or visit:
Room 1100 700 4 th St. NW (corner of 4 th and G Sts. NW) U. S.
General Accounting Office Washington, DC Orders may also be placed
by calling (202) 512- 6000 or by using fax number (202) 512- 6061,
or TDD (202) 512- 2537. Each day, GAO issues a list of newly
available reports and testimony. To receive facsimile copies of
the daily list or any list from the past 30 days, please call
(202) 512- 6000 using a touchtone phone. A recorded menu will
provide information on how to obtain these lists. For information
on how to access GAO reports on the INTERNET, send e- mail message
with info in the body to: info@ www. gao. gov or visit GAO's World
Wide Web Home Page at: http:// www. gao. gov United States General
Accounting Office Washington, D. C. 20548- 0001 Official Business
Penalty for Private Use $300 Address Correction Requested Bulk
Rate Postage & Fees Paid GAO Permit No. G100 (410505)

*** End of document. ***