Managing For Results: Enhancing Agency Use of Performance
Information for Management Decision Making (09-SEP-05,
GAO-05-927).
The Government Performance and Results Act (GPRA) of 1993 has
laid a foundation of results-oriented agency planning,
measurement, and reporting in the federal government. Performance
planning and measurement have slowly, yet increasingly, become a
part of agencies' cultures. For planning and performance
measurement to be effective, federal managers need to use
performance information to identify performance problems and look
for solutions, develop approaches that improve results, and make
other important management decisions. According to GAO's periodic
surveys, federal managers reported having more performance
measures in 2003 than in 1997. However, the data also showed that
managers' reported use of performance information for program
management activities has remained essentially unchanged from
1997 levels. GAO was asked to identify (1) how federal agencies
can use performance information to make management decisions and
(2) practices that can enhance or facilitate the use of
performance information to make management decisions.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-05-927
ACCNO: A36217
TITLE: Managing For Results: Enhancing Agency Use of Performance
Information for Management Decision Making
DATE: 09/09/2005
SUBJECT: Decision making
Information resources management
Internal controls
Management information systems
Performance management
Performance measures
Strategic planning
Performance plans
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-05-927
* Report to the Chairman, Subcommittee on Government Management, Finance
and Accountability, Committee on Government Reform, House of
Representatives
* September 2005
* MANAGING FOR RESULTS
* Enhancing Agency Use of Performance Information for Management
Decision Making
* Contents
* Results in Brief
* Background
* Agencies Can Use Performance Information to Manage for Results
* Performance Information Can Be Used to Identify Problems in
Programs and Take Corrective Action
* Performance Information Can Be Used to Develop Agency
Strategy, Plan and Budget, Identify Priorities and Allocate
Resources
* Performance Information Can Be Used to Recognize and Reward
Performance
* Performance Information Can Be Used to Identify and Share
More Effective Processes and Approaches to Program
Implementation
* Management Practices Can Enhance the Use of Performance
Information
* Demonstrating Management Commitment
* Aligning Agencywide Goals, Objectives and Measures
* Improving the Usefulness of Performance Information to
Better Meet Management's Decision-Making Needs
* Developing Agency Capacity
* Communicating Performance Information Frequently and
Effectively
* Conclusions
* Agency Comments
* Scope and Methodology
* GAO Contact and Staff Acknowledgments
* Bibliography
* Related GAO Products
United States Government Accountability Office
Report to the Chairman, Subcommittee
on Government Management, Finance and Accountability, Committee on Government
Reform, House of Representatives
September 2005
MANAGING FOR RESULTS
Enhancing Agency Use of Performance Information for Management Decision Making
a
MANAGING FOR RESULTS
Enhancing Agency Use of Performance Information for Management Decision
Making
What GAO Found
Agencies can use performance information to make various types of
management decisions to improve programs and results. Agencies can also
implement a number of practices that can enhance or facilitate the use of
performance information. GAO identified four broad types of management
decisions for which federal managers can use performance information and
five different types of practices that can contribute to greater use of
performance information. (See figure.)
Practices That Can Enhance or Facilitate the Use of Performance
Information for Management Decision Making
The five federal agencies that GAO examined for this report provide
examples of how agencies can use performance information for key
management decisions and practices that can enhance or facilitate such
use. While agencies face different management conditions and challenges
and operate under different authorities, the general uses and practices
highlighted in this report could be adapted by other agencies. Helpful
next steps would be for agency experiences in using performance
information to be more widely shared; and, for agencies to be encouraged
to adapt the practices to their unique situations.
United States Government Accountability Office
Contents
Letter 1
Results in Brief 3
Background 5
Agencies Can Use Performance Information to
Manage for
Results 7
Management Practices Can Enhance the Use of
Performance
Information 18
Conclusions 33
Agency Comments 33
Appendixes
Appendix I: Scope and Methodology 35
Appendix II: GAO Contact and Staff Acknowledgments 40
Bibliography
Related GAO Products
Table Table 1: Background Information on Agencies Selected to
Illustrate
Performance Information Uses and Practices 36
Figures Figure 1: Practices That Can Enhance or Facilitate the Use of
Performance Information for Management Decision
Making 3
Figure 2: Example of Using Performance Information to Manage
for Results 4
Figure 3: The Percentage of Federal Managers Who Reported
Using Information Obtained from Performance Measures
to a Great or Very Great Extent for Various Management
Activities Has Not Increased Significantly Since 6
1997
Figure 4: Uses of Performance Information 8
Figure 5: Practices That Can Enhance or Facilitate the Use of
Performance Information for Management Decision
Making 18
Figure 6: SBA Data Validation Table 25
Figure 7: NOAA Quad Chart Template 29
Figure 8: NOAA Display Board Template 30
Contents
Figure 9: FAA Scorecard for Runway Incursions
Abbreviations
BIC Business Information Center
COO Chief Operating Officer
DOC Department of Commerce
DOL Department of Labor
DOT Department of Transportation
EBSA Employee Benefits Security Administration
EMS Enforcement Management System
ETA Employment and Training Administration
FAA Federal Aviation Administration
FARS Fatality Analysis Reporting System
FFMP Flash Flood Monitoring and Prediction Program
GPRA Government Performance and Results Act
GS General Schedule
NBLP National Business Learning Partnership
NHTSA National Highway Traffic Safety Administration
NOAA National Oceanic and Atmospheric Administration
NWS National Weather Service
OMB Office of Management and Budget
OPA Office of Participant Assistance
OSI Organizational Success Increase
PEP Performance Enhancement Project
SBA Small Business Administration
SES Senior Executive Service
VA Department of Veterans Affairs
VHA Veterans Health Administration
WFO Weather Forecast Office
WIA Workforce Investment Act
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
A
United States Government Accountability Office Washington, D.C. 20548
September 9, 2005
The Honorable Todd Platts Chairman, Subcommittee on Government
Management, Finance and Accountability Committee on Government Reform
House of Representatives
Dear Mr. Chairman:
The federal government is in a period of profound transition and faces an
array of challenges and opportunities to enhance performance, ensure
accountability, and position the nation for the future. These 21st century
challenges include the nation's large, long-term fiscal imbalance,
evolving national and homeland security threats, increasing global
interdependence, the global shift to market-oriented knowledge-based
economies, an aging and more diverse population, and rapid advances in
science and technology. In the face of these pressures, it is vital to
maximize the performance of federal agencies in achieving their long-term
goals. The Government Performance and Results Act of 1993 (GPRA)1 planning
and reporting requirements can provide the essential information needed to
assess federal agencies' performance, hold agencies accountable for
achieving results, and help the baseline review of federal programs,
operating functions, and activities needed to respond to the nation's
long-term structural fiscal imbalance. GPRA requires executive agencies to
develop strategic plans, prepare annual performance plans, measure
progress toward the achievement of the goals in the annual performance
plans, and report annually on their progress in program performance
reports.
For planning and performance measurement to be effective, federal managers
need to use performance information to identify performance problems and
look for solutions, develop approaches that improve results, and make
other important management decisions. A 2004 GAO report that reviewed the
effectiveness of GPRA found that while the percentage of federal managers
who report having performance measures for their programs has increased
over time, their use of performance information in
1GPRA, Pub. L. No. 103-62 (Aug. 3, 1993), was enacted to help resolve the
long-standing management problems that undermined the federal government's
efficiency and effectiveness and provide greater accountability for
results.
making key management decisions, such as adopting new program approaches
or changing work processes, has not.2 Federal agencies also appear to
differ considerably in the extent to which they use performance
information to manage.
Identifying practices associated with the increased use of performance
information could be helpful to federal agencies seeking to better use
performance information to inform their decisions and ultimately achieve
results. Although performance information could also be helpful for
external purposes, such as congressional decision making, transparency,
and public accountability, this report focuses on use of performance
information by federal agency managers.
This report responds to your request that we identify (1) how federal
agencies can use performance information to make management decisions and
(2) practices that can facilitate the use of performance information.
To address the objectives, we reviewed relevant literature, including
previous GAO reports, spoke to experts in using performance information,
and held group discussions with federal program managers. We also
interviewed individuals within five federal agencies and reviewed
documentation to illustrate in greater detail how program managers have
used performance information to make decisions and specific agency
practices that facilitated those uses. We selected the five agencies-the
Departments of Commerce, Labor, Transportation, and Veterans Affairs, and
the Small Business Administration-on the basis of a number of factors,
including a relatively high agency score on our 2000 GPRA survey regarding
the extent to which agency managers reported using performance
information, and relatively good agency marks in the September 2004
Executive Branch Management Scorecard (President's Management Agenda). We
also considered information obtained from interviews with experts, and
diversity in terms of program type and size. It should be noted that we
did not attempt to identify all agencies and programs that could have
illustrated these uses and practices. In addition, we reviewed existing
relevant information regarding the quality of performance information
where available but did not systematically assess the quality of the
performance information used in the examples we cite. Finally, for most of
the examples, we describe how performance
2GAO, Results-Oriented Government: GPRA Has Established a Solid Foundation
for Achieving Greater Results, GAO-04-38 (Washington, D.C.: Mar. 10,
2004).
Page 2 GAO-05-927 Managing for Results
Results in Brief
information was used to make decisions, but we did not attempt to verify
that the use ultimately resulted in improved outcomes. (See appendix I for
a more detailed discussion of our scope and methodology and for further
information on the five agencies.) We conducted our work in accordance
with generally accepted government auditing standards from September 2004
through August 2005.
Agencies can use performance information to make various types of
management decisions to improve programs and results. Agencies can also
implement a number of practices that can enhance or facilitate the use of
performance information. We identified four broad types of management
decisions for which federal managers can use performance information and
five different types of practices that can contribute to greater use of
performance information. (See fig. 1.)
Figure 1: Practices That Can Enhance or Facilitate the Use of Performance
Information for Management Decision Making
Improved results
Uses
Identify Develop Recognize Identify and
problems and strategy
and reward share effective
take corrective and allocate
performance approaches
action resources
Practices
Aligning
Improving the Developing Communicating Demonstrating agencywide usefulness
of capacity to use performance management goals, information
commitment objectives, performance performance
frequently
information
and measures information and effectively
Source: GAO.
The five federal agencies that GAO examined for this report provide
examples of how agencies can use performance information for key
management decisions and practices that can enhance or facilitate such
use. While agencies face different management conditions and challenges
and operate under different authorities, the general uses and practices
highlighted in this report could be adapted by other agencies. Helpful
next steps would be for agency experiences in using performance
information to be more widely shared; and, for agencies to be encouraged
to adapt the practices to their unique situations. In figure 2 we describe
how an agency employed several practices to use performance information to
make management decisions.
Figure 2: Example of Using Performance Information to Manage for Results
Over the past 20 years, the National Highway Traffic Safety Administration
(NHTSA) has used performance information to identify, develop, and share
effective programs and strategies that increase safety belt usage, and as
a result has contributed to an increase in seat belt usage nationally from
11 percent in 1985 to 80 percent in 2004. With a mission to save lives and
prevent injuries, NHTSA's evaluation office analyzes information provided
by the annual National Occupant Protection Usage Survey, in which safety
belt use is directly observed at locations across the country, and the
Fatality Analysis Reporting System, a database of all fatal crashes in the
United States. From its analysis, NHTSA estimated that safety belts are 50
percent effective at preventing fatalities of front seat occupants in
crashes in which these motorists would otherwise die. In an effort to
increase usage nationally, NHTSA conducted several small studies in the
1980s and 1990s on how to best increase safety belt usage. NHTSA's
research showed that educating the public on the safety benefits of seat
belts was not very effective and that laws requiring seat belt use, while
somewhat effective, only increased usage to about 40 percent. One of
NHTSA's studies, Buckle Up America, revealed that active enforcement
combined with media campaigns highlighting enforcement, as opposed to
safety, could increase seat belt usage incrementally by 4 to 8 percent a
year.
However, NHTSA officials said these studies alone did not influence states
to change their expenditures with regard to safety belt use. Building on
the findings of these studies, NHTSA then developed and piloted the Click
It or Ticket safety belt campaign, which included both a paid media
enforcement message along with periods of active police enforcement of
safety belt use. The campaign, co-sponsored by the Airbag and Safety Belt
Safety Campaign of the National Safety Councila, was piloted in three
states. Doubledigit increases in safety belt use were seen in all three
states, demonstrating the effectiveness of this strategy in changing
motorists' safety belt use behavior. Ultimately, performance information
documenting the program's effectiveness in multiple states began to
persuade officials from other states to adopt the program. By 2003, 43
states, the District of Columbia, and Puerto Rico had participated in the
twice yearly national Click It or Ticket campaign. As a result, the
national average for safety belt use is up to 80 percent. NHTSA is now
focusing on identifying and developing strategies to target specific
segments of the population found to have the lowest safety belt usage
rates, such as drivers of pickup trucks.
Source: GAO.
aThe National Safety Council is a nonprofit, nongovernmental,
international public service organization dedicated to protecting life and
promoting health.
We provided a draft of this report to the Secretaries of Commerce, Labor,
Transportation, and Veterans Affairs, and the Administrator of the Small
Background
Business Administration for their review and comment. Technical comments
provided were incorporated, where appropriate, in the report.
GPRA helped create a governmentwide focus on results by establishing a
statutory framework for performance management and accountability, with
the necessary infrastructure to generate meaningful performance
information. GPRA introduced planning and reporting requirements that seek
to shift the focus of federal management and decision making from a
preoccupation with the number of program tasks or activities completed or
services provided to a more direct consideration of the results of
programs. The Act was intended to improve federal program effectiveness,
accountability, and service delivery. GPRA requires federal agencies to
develop strategic plans with long-term, outcome-oriented goals and
objectives, annual goals linked to the long-term goals, and annual reports
on the results achieved. Agencies are required to measure progress toward
the achievement of the goals in annual performance plans and report
annually on their progress in program performance reports.
As we reported in our 10-year retrospective report on the effectiveness of
GPRA, the Act's requirements have laid a solid foundation of
results-oriented agency planning, measurement, and reporting that has
begun to address the purposes of GPRA.3 Performance planning and
measurement have slowly yet increasingly become a part of agencies'
cultures. According to three government-wide, random sample surveys of
federal managers conducted by us in 1997, 2000, and 2003, managers
reported having significantly more of the types of performance measures
called for by GPRA, particularly outcome-oriented performance measures, in
2003 than in 1997, when GPRA went into effect governmentwide.
The benefit of collecting performance information is only fully realized
when this information is actually used by managers to make decisions
oriented toward improving results. While our surveys found that managers
reported having more performance measures in 2003 than in 1997, the data
showed that the use of performance information for program management
activities did not increase significantly from 1997 levels. In our survey,
we asked managers about their use of performance information for specific
purposes, such as "setting program priorities", "allocating resources", or
3 GAO-04-38.
"adopting new program approaches." As shown in figure 3, the majority of
managers who expressed an opinion reported using performance information
for these uses to a great or very great extent in 2003. While the reported
use of performance information for these uses to a great or very great
extent fluctuated somewhat between 1997 and 2003, the extent of use in
2003 was not significantly different from 1997 levels for any category of
use except "adopting new program approaches or changing work processes",
where use actually decreased.
Figure 3: The Percentage of Federal Managers Who Reported Using
Information Obtained from Performance Measures to a Great or Very Great
Extent for Various Management Activities Has Not Increased Significantly
Since 1997 Percent 100 90
80
70 66 66
62
6160 60
60
59
60
56
56
53
53 53
51
51
50
40
30
20
10
0 Setting program Allocating Adopting new Setting individual Rewarding
staff I priorities resources program approaches job expectations manage or
supervise or changing for my staff work processesa Key management
activities
1997 2000 2003 Source: GAO.
aThere was a statistically significant difference between the 1997 and
2003 surveys.
Agencies Can Use Performance Information to Manage for Results
From our 2000 survey,4 we also know that use of performance information
has varied widely among agencies. For example, while 56 percent of
managers overall reported using performance information when setting
program priorities in 2000, the percentage of managers within specific
agencies reporting this use to a great or very great extent ranged from 26
percent to 64 percent. Similarly, wide variation among agencies was seen
in managers' reported use of performance information for other types of
management decisions. For example, the reported use of performance
information to a great or very great extent in agencies ranged from 24 to
66 percent when allocating resources, from 25 to 64 percent when adopting
new program approaches or changing work processes, and from 16 to 66
percent when setting individual job expectations.
Federal agencies can use performance information to make various types of
management decisions to improve programs and results. We have previously
reported that leading organizations that have progressed toward
results-oriented management use performance information as a basis for
decision making.5 The full benefit of collecting performance information
is realized only when managers actually use it to manage. From our review
of the literature, agency documents and interviews with experts and agency
officials, we identified four categories of management decisions for which
federal agency managers can use performance information. (See fig. 4).6
Managers can use performance information to identify problems in existing
programs, to try to identify the causes of problems, and/or to develop
corrective actions. They can also use program performance information to
develop strategies, plan and budget, identify priorities, and make
resource
4GAO, Managing for Results: Federal Managers' Views on Key Management
Issues Vary Widely Across Agencies, GAO-01-592 (Washington, D.C.: May 25,
2001). The 2000 survey provided the most current data on specific
agencies, while the 2003 survey provided governmentwide data.
5GAO, Executive Guide: Effectively Implementing the Government Performance
and Results Act, GAO/GGD-96-118 (Washington, D.C.: June 1996); and GAO,
Results-Oriented Cultures: Creating a Clear Linkage between Individual
Performance and Organizational Success, GAO-03-488 (Washington, D.C.: Mar.
14, 2003).
6The four use categories we identified through our independent analysis
are similar to the four types of uses identified in How Federal Programs
Use Outcome Information: Opportunities for Federal Managers, by Harry
Hatry, Elaine Morley, Shelli Rossman and Joseph Wholey, published by the
IBM Endowment for the Business of Government in May 2003. Furthermore,
these categories are consistent with, but not identical to, the types of
uses captured in questions for our surveys of federal agency managers
described earlier.
Performance Information Can Be Used to Identify Problems in Programs and
Take Corrective Action
allocation decisions to affect programs in the future. Agency managers can
also use performance information to reward individuals or organizations
who meet or exceed expectations. Finally, managers can use performance
information to identify more effective approaches to program
implementation and share those approaches more widely across the agency.
Figure 4: Uses of Performance Information
Improved results
Uses
Identify Develop Recognize Identify and
problems and strategy
and reward share effective
take corrective and allocate
performance approaches
action resources
Source: GAO.
Agencies can use performance information to identify problems or
weaknesses in programs, to try to identify factors causing the problems,
and to modify a service or process to try to address problems. We have
reported that leading organizations that have progressed toward
results-oriented management have used performance information to identify
gaps in performance, improve organizational processes, and improve their
performance.7 We found several examples where federal program managers
used performance data to identify problems and corrective actions at the
agencies we examined.
Officials in the Office of Participant Assistance (OPA) at the Employee
Benefits Security Administration (EBSA) in the Department of Labor (DOL)
told us that when data for a performance measure of customer satisfaction
indicated a problem, they changed a process to try to improve
7 GAO-04-38, GAO-01-592, and GAO/GGD-96-118.
performance. One of EBSA's functions is to provide information and
assistance for individuals covered by private retirement and health and
welfare plans, plan sponsors, and members of the employee benefits
community. EBSA has contracted with an outside organization since 2003 to
collect data through a specialized survey on customer satisfaction with
EBSA services. Data indicated that customers whose cases were referred to
the enforcement side of EBSA were reporting lower satisfaction rates. An
analysis of data for specific survey questions identified the need for
improved communication. Investigating an inquiry or complaint could
sometimes take up to 2 years, and a customer might not receive any
information during that time. As a result of the performance data, EBSA
changed a standard operating procedure in January 2005 so that customers
would be kept informed about the status of their open inquiries or
complaints. Under the new procedure, when an inquiry is referred to the
enforcement staff for investigation, the OPA benefits advisor informs the
inquirer about the referral and sets realistic expectations for future
contact by an investigator. Investigative staff are to make initial
contact with the customer no later than 30 days following receipt of the
referral and maintain contact with the inquirer on a quarterly basis until
the matter is resolved. Managers plan to continue to monitor customer
satisfaction to see whether the changes lead to improved performance.
Through the Performance Enhancement Project (PEP), the Office of Workforce
Investment in DOL has used performance information to identify the
technical assistance needs of state and local employment and training
programs and has then provided targeted assistance to try to improve
performance. The PEP project, which began in October 2002, has provided
technical assistance and training for targeted state and local workforce
organizations that receive funds under the Workforce Investment Act
(WIA).8 Poorly performing states, as shown by the performance measures
(supplemented by analysis from regional DOL staff), have received
technical assistance designed to improve their performance through PEP.
DOL's Office of Workforce Investment sent a questionnaire to all the
regional offices and held a conference with them at
8Pub. L. No. 105-220 (Aug. 7, 1998). The Workforce Investment Act of 1998
required 17 employment and training programs, funded by four different
agencies, to centralize service delivery through a one-stop-center system.
WIA also established performance measures that look at a broad array of
participant outcomes, such as job placement and retention, earnings, skill
gains, and customer satisfaction.
Page 9 GAO-05-927 Managing for Results
the close of the first PEP project9 to assess the effectiveness of the
technical assistance that had been given. This information was used to
improve the second year's project. Online tutorials have also been
developed under PEP in response to requests from states.10
Officials in the Veterans Health Administration (VHA) network that
includes facilities located in Maryland, Virginia, West Virginia, and
Pennsylvania,11 stated that in response to one of the network facilities
not meeting the heart failure performance measure target, all of the
facilities adopted a new process to improve the network's performance.
According to an official, in the first quarter of 2004, the network was at
risk of not meeting the entire set of 10 cardiovascular performance
measures if one of its facilities did not improve its low score on the
heart failure performance measure by the end of the year.12 The heart
failure performance measure is a cumulative performance measure that
assesses whether discharged heart failure patients receive six types of
instructions, including instructions for monitoring weight after discharge
and taking medications. The instructions are critical because patients'
noncompliance with physicians' instructions is often a cause for
rehospitalization. An agency official stated that the entire network
implemented a new set of mandatory data fields in patients' electronic
medical records. The data fields include the six types of instructions and
have to be filled out when a facility discharges a heart failure patient.
According to this official, the result of the implementation of the new
mandatory fields in the patients' electronic records was that the facility
that was not meeting the heart failure performance measure target in the
first quarter improved its score to the "fully satisfactory" category in
the fourth quarter.
9The first PEP project ended in June 2004. The second PEP project began in
July 2004 and is currently scheduled to end in August 2005. 10A tutorial
became available online at www.workforcetools.org in December 2004.
11VHA networks are regional VHA units composed of health care facilities.
12As will be discussed in more detail later in this report, if one of the
facilities is failing a performance measure, then the entire network fails
the performance measure.
Performance Information Can Be Used to Develop Agency Strategy, Plan and
Budget, Identify Priorities and Allocate Resources
Agencies can use performance information to make decisions that affect
future strategies, planning and budgeting, identifying priorities, and
allocating resources. We have previously reported that outcome-based
performance information should be used for the allocation of resources and
in deciding among competing priorities in a results-oriented management
system.13 In addition, linking cost with performance information infuses
performance concerns into planning and budgetary deliberations, prompting
agencies to reassess their performance goals and strategies and to more
clearly understand the cost of performance. Performance information also
allows program managers to compare their programs' results with goals and
thus determine where to target program resources to improve performance.
When managers are faced with reduced resources, the same analysis can help
them target reductions to minimize negative impacts on program results. We
found a number of examples among our case agencies of managers using
performance information to identify priorities and allocate resources, and
to shape future program strategy.
The National Highway Traffic Safety Administration (NHTSA) has used
performance information to set priorities in budgeting and to target
resources. The Senior Associate Administrator for Traffic Injury Control
annually sends out a memo that identifies agency priorities that support
the Department of Transportation's (DOT) strategic goals and are based on
performance information for traffic-injury-related measures, such as
safety belt use and impaired driver rates.14 The memo directs managers to
focus their programs and proposals to support these priorities. In
addition, NHTSA used performance information on alcohol-related injuries
and fatalities to target grant funding and specific program strategies to
states with the highest impaired driver rates. Using information reported
through the Fatality Analysis Reporting System (FARS),15 NHTSA identified
13GAO, Results-Oriented Government: Using GPRA to Address 21st Century
Challenges, GAO-03-1166T (Washington, D.C.: Sept. 18, 2003); GAO,
Results-Oriented Budget Practices in Federal Agencies, GAO-01-1084SP
(Washington, D.C.: August 2001); and GAO, Human Capital: Key Principles
for Effective Strategic Workforce Planning, GAO-04-39 (Washington, D.C.:
December 2003).
14Impaired driving is driving under the influence of alcohol or other
drugs.
15FARS is a database containing information on fatal crashes in the United
States, including specific details about crash circumstances and vehicle
and victim demographic information. It is maintained by NHTSA's National
Center for Statistics and Analysis.
13 states that represented 46 percent of the total number of
alcohol-related fatalities in the United States. Officials said NHTSA
worked closely with these 13 states to implement the impaired-driving
program strategies it had designed. The strategies involved
state-sponsored, high-visibility law enforcement activities in conjunction
with a NHTSA-designed media campaign. A NHTSA official said that the
agency brought senior policy and law enforcement officials from these
states together and presented the performance information and evaluations
showing impaired driving rates and the effectiveness of the high
visibility law enforcement and media campaign. In order to receive
funding, the targeted states had to agree to implement the program that
NHTSA had designed, including the media campaign. After the publicized
crackdown periods, NHTSA provided the targeted states with additional
evaluation resources.
The Small Business Administration (SBA) used performance and cost
information to reallocate resources to better meet priorities. In the
1990s, SBA created Business Information Centers (BIC), a community
partnership service to provide entrepreneurs with access to computers and
the Internet to help them access business information and develop business
plans. Officials said that, as a partnership using hardware and software
and library resources donated from private technology companies, the
program was designed to require minimal SBA employee support or capital
costs. However, after analyzing financial and human capital resources,
including the data collected through their annual survey of how much
support SBA employees provide for various programs, they found that
substantial employee resources were being devoted to the BICs, on the
order of $11 million. This cost information led SBA officials to
reconsider the service. Since computers had become more accessible to
entrepreneurs than when the BIC program was designed, and because
assistance to entrepreneurs could be provided more efficiently though
other programs, SBA decided to end its participation in the program and
has phased out its role in the community partnerships.
Performance Information Can Be Used to Recognize and Reward Performance
Agencies can use performance information to affect pay decisions and to
reward individuals, and in some cases, grantees.16 Using performance
information this way reinforces accountability and creates incentives for
achieving results. We have previously reported that high-performing
public-sector organizations create a clear "line of sight" between
individual performance and organizational success17 showing how team,
unit, and individual performance can contribute to overall organizational
results. There is growing recognition that the government needs to
transform how it compensates its employees to achieve greater results. To
this end, some agencies have implemented performance agreements as a
vehicle for bringing results-oriented performance information into
evaluations of performance. A revised performance-based pay system for
members of the Senior Executive Service (SES) across the federal
government has been in effect since January 2004, as authorized by
Congress and overseen by the Office of Personnel Management (OPM), where
pay adjustments for SES members are to be based on the alignment of
individual performance and contributions to the agency's mission and
strategic goals, among other things. The human capital reforms under way
at the Departments of Defense and Homeland Security are intended to help
them manage their human capital to achieve results. We found several
examples where case agencies used performance information to recognize and
reward performance, as described next.
The Federal Aviation Administration (FAA) uses performance information to
affect pay decisions. FAA has linked employee annual pay increases to the
achievement of agencywide goals through the performance-based pay
system.18 At the end of fiscal year 2004, 78 percent of employees were
included in the agency's performance-based pay system, which allocates
16Agencies can also use performance information to recognize and reward
contractors for achieving results. However, we did not analyze any
examples of performance-based contracting for this report.
17GAO-03-488.
18We previously reviewed FAA's human capital reforms and found that FAA
had little data with which to assess the effects of its reform effort. See
GAO, Human Capital Management: FAA's Reform Effort Requires a More
Strategic Approach, GAO-03-156 , (Washington, D.C.: Feb. 3, 2003).
increases on the basis of individual and organizational performance.19 At
the individual level, employees can receive two types of increases.
Employees earn the first increase for meeting individual goals and the
second increase for achieving outstanding individual performance. At the
organizational level, employees are eligible for the organizational
success increase (OSI). This is an agencywide pay increase based on the
performance of the agency as a whole on the 31 key performance measures in
the agency's strategic plan, called the Flight Plan.20 The OSI funding
pool consists of the amount of the general increase for General Schedule
(GS) employees in other federal agencies plus an additional 1 percent
(which reflects a portion of the funds previously spent on within-grade
increases). Receiving the full OSI depends on whether actual performance
meets the goals. In calendar year 2004, FAA employees received an OSI
increase of
2.13 percent. FAA also offers Short Term Incentives to FAA senior staff
for achieving key performance targets and specified accomplishments in
implementing Flight Plan initiatives.
VHA uses performance information to create incentives for network
directors. We previously reported that the use of performance information
in performance agreements helped VHA define directors' accountability for
specific goals and monitor progress during the year, and contributed to
their evaluations. VHA accomplishes this through the performance-based
review process, which links a director's performance rating and
compensation to the performance of the network that he or she oversees.
Specifically, a director's annual ratings and bonuses are based on how
well he or she meets the goals in the performance contract. As detailed in
the contract, the network director's compensation is affected by that
network's scores on performance measures, which include clinical waiting
times, the percentage of patients receiving cancer screenings, and patient
satisfaction. Half of the network director's performance evaluation is
based upon the cumulative score on the network's performance measures.
The Employment and Training Administration (ETA) uses performance
information to provide incentives for and apply sanctions to state
programs that receive grants under Title I of the Workforce Investment
Act. The authority to hold grantees accountable for performance in this
manner is
19Section 347 of the Fiscal Year 1996 Department of Transportation
Appropriations Act, Pub. L. No. 104-50 (Nov. 15, 1995), provided FAA with
special flexibilities with regard to its human capital policies. 49 U.S.C.
S: 40122.
20FAA's 2005-2009 Flight Plan can be found online at http://www.faa.gov.
Page 14 GAO-05-927 Managing for Results
Performance Information Can Be Used to Identify and Share More Effective
Processes and Approaches to Program Implementation
provided for in the WIA legislation.21 Each state negotiates performance
goals with DOL prior to the beginning of the program year for the 17 WIA
indicators measuring employment, retention, earnings, attainment of
credentials, attainment of skills, and customer satisfaction.22 States are
eligible for incentive awards if they have exceeded performance goals set
for programs funded under 3 separate programs, one of which is WIA.23 In
general, financial sanctions may be levied if performance is less than 80
percent of the negotiated performance levels for 2 consecutive years or if
the state fails to submit an accurate and complete annual performance
report.24 Specific mitigating factors may be considered, such as
performance relative to other states, improvement efforts, and economic
conditions, among others.
We have reported that high-performing organizations continuously assess
and benchmark performance and efforts to improve performance.25 They
evaluate their efforts using fact-based understandings of how their
activities contribute to accomplishing the mission and broader results,
and optimize their efforts through continuous improvement. Managers can
use performance information to identify and increase the use of program
approaches that are working well, and consider alternative processes in
areas where goals are not being met. Pilots and demonstration projects
2129 U.S.C. S: 2871(g). The authority to provide financial incentives and
levy sanctions based on performance is specific to WIA legislation and
associated WIA programs. Similar authority may or may not be provided for
in legislation for other grant programs.
22We have reported on quality issues regarding the performance data
generated by state WIA programs. In our report, Workforce Investment Act:
States and Local Areas Have Developed Strategies to Assess Performance,
but Labor Could Do More to Help, GAO-04-657 (Washington, D.C.: June 1,
2004) we discuss OMB's proposed common measures, and data quality issues
such as the lack of consistent definitions which leads to variations in
reporting.
23The two other programs are adult education and vocational education,
authorized by the Adult Education and Family Literacy Act and the Carl D.
Perkins Vocational and Technical Education Act, respectively.
24More specific criteria for sanctions and incentives implementation can
be found in DOL's Training and Employment Guidance Letter No. 19-02,
located online at
http://wdr.doleta.gov/directives/attach/TEGL19-02.cfm.
25GAO, Highlights of a GAO Forum: High Performing Organizations: Metrics,
Means, and Mechanisms for Achieving High Performance in the 21st Century
Public Management Environment, GAO-04-343SP (Washington, D.C.: Feb. 13,
2004).
can be used to identify innovative ways to improve performance-by allowing
for experiences to be rigorously evaluated, and shared systematically with
others, and by allowing for new procedures to be adjusted as
appropriate-before receiving wider application. Organizations can also use
the process improvement technique of benchmarking-comparing their
processes with those of private and public organizations that are thought
to be the best in their fields. By benchmarking its own processes against
those of leading business and government entities, an organization can
learn how much change it needs to make and what changes might be the right
ones. We found examples of these types of uses of performance information
in the case agencies, as described next.
On the basis of performance information from a pilot test, the National
Weather Service (NWS) adopted a more effective flash flood monitoring and
prediction tool on a national basis. NWS provides flash flood warning
services and verifies the timeliness and accuracy of its warnings. It aims
to improve its performance in this area through the flash flood
performance goals entitled, flash flood warning lead time and flash flood
warning accuracy, as outlined in its strategic plan.26 A prototype flash
flood monitoring application was developed and first tested at the
Pittsburgh Weather Forecast Office (WFO) beginning in 1995. On the basis
of improved performance data achieved with the prototype and case studies
of several flash flood events in the Pittsburgh area, NWS decided to
develop and test an enhanced version of the pilot system. The enhanced
version, called the Flash Flood Monitoring and Prediction (FFMP) Program,
continuously monitors rainfall rates and hydrologic conditions to provide
automated alerts when a dangerous flood situation may be developing on a
given stream or catchment. In 2002 NWS began implementing FFMP nationwide,
with the aim of improving flash flood services and helping NWS to meet its
flash flood performance goals. Initial training on how to use the FFMP
system was provided for all WFOs and additional training is provided at
routinely offered courses. According to a recent survey, nearly all of the
WFOs were using the system. It is expected that implementing the system
will facilitate improvements in the accuracy and timeliness of official
NWS flash flood warnings, and enable forecasters to provide more specific
location information within a flash flood warning.
26The NWS Strategic Plan for 2005-2010 can be found online at
http://www.nws.noaa.gov/sp/.
Page 16 GAO-05-927 Managing for Results
ETA has used performance information on state and local workforce
investment areas to identify and nominate participants for the National
Business Learning Partnership's (NBLP) peer-to-peer, mentoring initiative.
The NBLP, which started in October of 2003, links local workforce areas27
that want to improve their services (proteges) with workforce areas that
have exceeded performance standards (mentors). The program focuses on the
delivery of workforce services tailored to the needs of business and
industry in that specific workforce area. Overall, 19 mentor sites and 25
protege sites have participated. Examples of the types of performance and
related information considered by ETA in selecting workforce areas to
nominate as mentors include the following: successful engagement with
businesses to meet the workforce needs of local industry; high performance
on the 17 legislated WIA performance indicators measuring employment,
retention, earnings, attainment of credentials, attainment of skills, and
customer satisfaction; and the transferability of mentor practices to
other areas. Proteges were selected on the basis of their desire to
improve their performance and services to businesses, and also on the
basis of whether they had problems meeting the WIA performance measures.
Each protege site was matched with a mentor site to help it develop a work
plan. The proteges and mentors conducted at least two site visits to each
other's respective areas to study the local workforce investment
operations and share information. In addition to peer-to-peer
consultation, NBLP case studies have been developed to provide a wider
audience with access to participants' experiences and learning. Each case
study includes practices and principles to improve performance outcomes by
addressing the workforce needs of businesses and industries, as well as a
guide to facilitate shared learning and promote action.28 To assess the
success of these partnerships, each protege's performance on selected WIA
measures will be compared from performance year 2003 through performance
year 2005. Additional measures of success will include such considerations
as contacts with businesses, resources leveraged, and workers trained and
placed in high-growth careers.
27Workforce investment areas are designated within states by governors.
28Three case studies have been published so far, and another five are
being developed.
Page 17 GAO-05-927 Managing for Results
Management Practices Can Enhance the Use of Performance Information
Agencies can adopt or apply a number of practices that can enhance the use
of performance information for policy and program decisions aimed at
improving results. The concept of practices as used in this report
includes administrative or management tools, systems, or processes that
agencies can implement. From our interviews with experts and agency
officials and our review of related literature and agency documents, we
identified five types of practices that can facilitate greater use of
performance information. As illustrated in figure 5, the five types of
practices are demonstrating management commitment; aligning agency goals,
objectives and measures; improving the usefulness of performance
information to better meet management's needs; developing agency capacity
to effectively use performance information; and frequently and effectively
communicating performance information within the agency. We discuss the 5
practices in detail following the figure.
Figure 5: Practices That Can Enhance or Facilitate the Use of Performance
Information for Management Decision Making
Improved results
Uses
Identify Develop Recognize Identify and
problems and strategy
and reward share effective
take corrective and allocate
performance approaches
action resources
Practices
Aligning Improving the Developing Communicating Demonstrating agencywide
usefulness of capacity to use performance management goals, information
commitment objectives, performance performance
frequently
information
and measures information and effectively
Source: GAO.
Demonstrating Management Commitment
The commitment of agency managers to results-oriented management is
critical to increased use of performance information for policy and
program decisions. As we previously reported,29 demonstrating the
willingness and ability to make decisions and manage programs on the basis
of results, and inspiring others to embrace such a model, are important
indicators of management's commitment. Agency officials from our case
illustration programs also identified management commitment as central to
encouraging the use of performance information in decisions. Officials
from these agencies described how management showed this type of
commitment by leading frequent, regular performance review meetings to
discuss progress made toward the achievement of results, and by involving
staff from different organizational levels in performance review meetings.
These methods assisted agencies in identifying performance problems and in
developing performance improvement plans based on collected performance
information.
SBA conducts monthly meetings with the associate administrator of each
mission and functional office to review that office's performance. The
administrator, deputy administrator, and chief operating officer (COO)
participate. Each SBA office has performance measures organized along
three components of performance-office strategic goals, production goals,
and project goals-on a scorecard. Prior to meeting, the COO's staff review
the monthly and cumulative performance information for each office and ask
each office to respond in writing to questions or concerns based on the
data. This analysis is the basis for the regular performance review
meetings. Officials said management's commitment to regularly reviewing
performance increases performance ownership among staff and competition
among the offices to meet performance targets.
Similarly, FAA management demonstrates commitment through monthly,
day-long, agencywide performance review meetings that are led by the
Administrator and key associate administrators who serve as goal leads for
each FAA Flight Plan goal area: Increased Safety, Greater Capacity,
International Leadership, and Organizational Excellence. At these
meetings, officials said that they discuss performance for the agency's 31
key performance metrics and the strategic initiatives supporting each.
When a business line is not meeting the performance targets for specific
29GAO, Managing for Results: Federal Managers' Views Show Need for
Ensuring Top Leadership Skills, GAO-01-127 (Washington, D.C.: Oct. 20,
2000).
Page 19 GAO-05-927 Managing for Results
Aligning Agencywide Goals, Objectives and Measures
metrics, officials report on efforts planned or under way to improve
performance. For metrics where performance targets have been met,
officials discuss the actions that were taken to achieve the targets.
Officials said that during this performance review, the Administrator
identifies IOUs, outlining agreed-upon actions to be implemented.
Officials said that they provide updates on the status of these IOUs at
the following performance review meeting.
At the Department of Veterans Affairs (VA), managers are involved in
reviewing performance at different levels of the organization. Senior
management hold monthly departmentwide meetings to review performance.
According to officials, the meetings are chaired by the Deputy Secretary
and are attended by the heads of the three administrations: VHA, the
Veterans Benefit Administration, and the National Cemetery Administration,
as well as all of the staff offices within VA. Each administration reports
financial status, workload, and key performance measures. The Deputy
Secretary has recently implemented reporting by exception at the monthly
performance review. Reporting by exception requires that administration
and staff offices offer an explanation for less-than-desired performance,
describe plans or ongoing efforts to improve performance and discuss when
results were expected. At the following month's performance review, status
information is presented to demonstrate progress. In addition, success
stories and the activities used to achieve them are also reported.
According to officials, within VHA, mid- and lower-level managers are also
involved in regular performance review meetings, including meetings
involving the Deputy Under Secretary and the directors of the regional
networks, meetings of network directors and their facility directors, and
meetings between facility directors and their staff. In addition,
according to officials, each of the facilities in one of the networks we
visited, which includes VA facilities in California, Nevada, Hawaii and
the Philippines, assigns a "champion" who is responsible for monitoring
and presenting results for a specific performance measure at weekly
facility management meetings.
Agencies can encourage greater use of performance information by aligning
agencywide goals and objectives, and by aligning program performance
measures at each operating level with those goals and objectives. GPRA
requires that agencies use performance measurement to reinforce the
connection between their long-term strategic goals and the day-to-day
activities of their managers and staff. As we reported
Page 20 GAO-05-927 Managing for Results
previously,30 in order to meet the GPRA requirements, an agency should
cascade its goals and objectives throughout the organization and should
align performance measures to the objectives from the executive level down
to the operational levels. This forms hierarchies of goals and objectives
and performance information that are appropriate to the managerial
responsibilities and controls at each level of the organization. This
alignment increases the usefulness of the performance information
collected to decision makers at each level, and reinforces the connection
between strategic goals and the day-to-day activities of managers and
staff. We have also reported that a greater focus on results can be
created by cascading organizational goals and objectives down to the
individual performance level. Earlier in this report, we described how
agencies can use performance information to recognize and reward
performance and encourage the achievement of results. Alignment
facilitates the linking of individual performance to organizational
performance.
FAA promotes alignment by requiring that all organizations use the same
alignment framework and approach in preparing their annual business plans.
Goals cascade throughout each of FAA's 16 organizations to each individual
employee. The agency goals detailed in FAA's strategic plan (or Flight
Plan) drive new strategic initiatives and the ongoing, day-to-day
operations of FAA, referred to as their core business functions. Key
elements required in the business plans for each of FAA's four lines of
business and 12 offices are developed using a systematic alignment
approach that guides the organizations through the step-by-step process of
defining and developing performance elements that support the Flight Plan
goals. Organizations align themselves with the Flight Plan through this
process, and targets cascade down through the organization into employee
performance plans. These plans identify specific work expectations and
outcomes, and are required for every FAA employee. This alignment
facilitates FAA recognizing and rewarding performance by using performance
information to affect pay decisions, as described earlier in this report.
VHA's performance measures are aligned through different levels of the
organization. The same performance measures are used to assess VHA's
performance from the VHA-wide level down through the network levels to the
individual facilities. For example, the performance measure of the
percentage of patients receiving breast cancer screening is used at all
three
30GAO/GGD-96-118.
Improving the Usefulness of Performance Information to Better Meet
Management's Decision-Making Needs
levels. As mentioned earlier in this report, the performance score of a
facility can have an effect upon the entire network to which it belongs.
As an illustration of how performance is "rolled up," officials stated
that in order to get a fully satisfactory rating at the network level, 80
percent of the hospitals in that network must have met their goals.
Officials stated that if any facility is at the failing level for a
performance measure, the entire network will also be rated as failing the
measure. In addition, alignment contributes to VHA's ability to use
performance information to provide incentives for network directors,
through the performance-based review process described earlier.
To ensure that performance information will be both useful and used in
decision making throughout the organization, agencies need to consider
users' differing policy and management information needs. Practices that
improve the usefulness of performance information can help to meet those
needs. We reported previously31 that to be useful, performance information
must meet users' needs for completeness, accuracy, consistency,
timeliness, validity, and ease of use. Other attributes that affect the
usefulness of information include, but are not limited to, relevance,
credibility, and accessibility. We have reported previously on a number of
practices that improve the usefulness of performance information to
different users.32 Measures should be selected specifically on the basis
of their ability to inform the decisions made at each organizational
level, and should be appropriate to the responsibilities and control at
each level. In that regard, involving managers in the development of
performance goals and measures is critical to increasing the relevance and
therefore the usefulness of performance information to their day-to-day
activities. Agency officials in our case illustrations identified a number
of practices that increased the usefulness of performance information,
including its relevance, ease of use, timeliness, and accessibility.
As discussed earlier, EBSA contracted with an outside organization to
collect more useful customer satisfaction information. By increasing the
scope and depth of customer satisfaction data, the information has been
made more relevant, and therefore, more useful to management's day-to-
31GAO-04-38.
32GAO, Performance Plans: Selected Approaches for Verification and
Validation of Agency Performance Information, GAO/GGD-99-139 (Washington,
D.C.: July 30, 1999).
Page 22 GAO-05-927 Managing for Results
day activities. Prior to this contract, EBSA used another contractor that
provided a general high-level index of customer satisfaction that provided
comparability to other organizations. However, in order to provide this
comparability, the survey questions had to be standardized in a manner
that was not as relevant for EBSA as for other organizations. In addition,
the first customer satisfaction survey did not provide a means for EBSA to
disaggregate data to identify specific aspects of its performance.
Recognizing that the information did not fully meet their needs, EBSA
staff identified a new contractor to create and conduct a specialized
customer satisfaction survey of clients that will provide comparability
from year to year even if it does not provide comparability among other
organizations. Survey questions are now tailored to EBSA's needs. Specific
questions can be added and data disaggregated and analyzed as needed to
focus in on potential problem areas. For example, data can be broken down
by categories such as regional office, benefit advisor, and type of
inquiry.
The Enforcement Office of the Department of Labor's EBSA Office
implemented a new Enforcement Management System (EMS) that officials say
improved the usefulness of performance information, making it more timely,
accessible, and easy to use. The system, implemented in 1999, manages
information related to investigations into the potential violation of
benefits laws. These investigations are conducted by 385 investigators
located at EBSA field offices across the country. Prior to the development
of the EMS, officials said that the Enforcement Office had a simple
database system located at headquarters and that field offices used a
separate case management system.33 The EMS was custommade for the
Enforcement Office. It tracks information about every investigation
nationwide throughout its lifecycle, providing both field and headquarters
staff access to real-time information. According to officials, not only
does the system make information more timely and accessible, it has
increased the ease of use and relevance of performance information by
incorporating user-friendly windows technology and allowing users to
define their own information needs with customized queries and reports. In
addition, over 30 standardized statistical and case reports are available.
Standardized statistical reports allow managers to view, by selected
geographic area, summaries of such information as the different
investigation types (e.g., fiduciary and criminal), the cases referred for
litigation, and closed cases
33We reviewed EBSA's previous database system in GAO, Pension and Welfare
Benefits Administration: Opportunities Exist for Improving Management of
the Enforcement Program, GAO-02-232 (Washington, D.C., Mar. 15, 2002).
Page 23 GAO-05-927 Managing for Results
along with the associated results. Headquarters provides additional
analysis of the information regarding overall progress toward policy
goals, allowing adjustments to be made in the field offices.
SBA has involved its program managers in the development of the agency's
performance measurement system, allowing offices to determine their own
goals and measures. This practice increases the usefulness of performance
information to the activities of offices and programs, and SBA officials
credit it with creating buy-in and ownership of the performance
measurement system. Measures and targets must meet certain requirements as
overseen by the chief operating officer. For example, measures should
support the strategic goals; measures should be valid, measurable, and
verifiable; and the set of measures as a whole should contain both
qualitative and quantitative measures. At the beginning of each year when
the measures and targets are being selected, assistance is provided for
any program upon request to help with selecting appropriate measures.
SBA has also initiated a practice aimed at documenting and improving the
overall usefulness of performance information and increasing its use.
Programs are required to identify the intended use of each measure in
decisions related to policy and to provide an assessment of how fit the
information is for this use. The importance of this practice is that it
promotes the selection of performance measures that better meet
managements' needs, providing information that is relevant, and
sufficiently complete, accurate, consistent, and timely for management's
decision-making needs. To facilitate and document the practice, SBA's
Office of Analysis, Planning and Accountability developed an assessment
tool for its indicators in the form of the Data Validation Table,"34 shown
in figure 6. These tables are updated annually for every indicator
published in the SBA Performance and Accountability Report. In addition to
documenting the intended use, the office must provide an assessment of the
information's completeness, accuracy, consistency, and timeliness.
Validation Tables also document the associated office and program; what
outcomes are being assessed; how each outcome is aligned with agency
goals; the system in which the data are kept; and any limitations to the
use of the information in the intended manner and associated remedies,
among other items.
34Tables for specific SBA metrics can be found online at
http://www.sba.gov/PAR.pdf/strategicgoal1.html.
Page 24 GAO-05-927 Managing for Results
Figure 6: SBA Data Validation Table
Source: SBA.
Developing Agency Capacity
The practice of building analytical capacity to use performance
information-both in terms of staff trained to do analysis and availability
of research and evaluation resources-is critical to using performance
information in a meaningful fashion. Our 2003 GPRA survey found that there
was a positive relationship between agencies providing training and
development on setting program performance goals and the use of
performance information when setting or revising performance goals. We
have previously reported35 on the need for agencies to expend resources on
effective training and professional development to equip federal employees
to work effectively. Specifically, managers and staff need competencies
and skills to plan strategically, develop robust measures of performance,
and analyze what the performance data mean. Performance management
literature also states that training is a key factor in improving
employees' capabilities and enabling employee involvement in achieving
performance improvements. We found examples where our case illustration
agencies built capacity through providing staff with training in the
performance management system and by providing access to technical
assistance, such as having designated evaluation support staff or
performance measurement experts available.
The VHA network that includes health care facilities located in
California, Nevada, Hawaii, and the Philippines builds agency capacity by
providing annual training on performance measures for facility managers.
The training is coordinated by a performance measurement committee
composed of clinical leadership and quality management staff from each of
the facilities. The committee's mission includes providing education on
the performance measures beyond the annual training, as well as sharing
and implementing strategies to facilitate meeting the performance
measurement targets. At the annual training conference, all of the
performance measures are discussed, including existing measures as well as
any new measures. The training lasts for a day or two, and facility
managers can participate in person, on the telephone, or by
videoconference. In addition, the performance measurement committee is
also a resource throughout the year for questions on the performance
measures.
NHTSA has established capacity to use performance information through its
research and evaluation office. According to one official, NHTSA has
35GAO-04-38.
Communicating Performance Information Frequently and Effectively
had internal analytic capacity through a research office for more than 25
years. The evaluation office provides the capacity to conduct research and
evaluations on the effectiveness of NHTSA's Traffic Injury Control Office
programs and initiatives, which address issues such as safety related to
alcohol- and drug-impaired driving, pedestrians, bicycles, motorcycles,
school buses, emergency medical services, older drivers, and driver
fatigue and inattention. We previously reported that NHTSA's evaluation
office provides the capacity for a three-phase evaluation process.36
First, studies identify the nature of the problem and possible solutions.
Second, cost-benefit analyses identify the expected consequences or
alternative approaches. Third, follow-up studies assess the consequences
of policy or regulatory changes, since effects of some changes may not be
apparent until 5 or more years after the introduction of changes.
According to a NHTSA official, it is important to have this capacity
because the Traffic Injury Control Office does not have any regulatory
authority, and needs the analysis and evaluations to demonstrate that the
proposed programs are effective in order to persuade states to implement
them. In addition, we have previously reported that evaluations can
improve agency use of performance information by providing performance
information that may otherwise be unavailable; by validating the accuracy
of performance data; by explaining the reasons for observed performance;
or, finally, by identifying ways to improve performance.
Improving the communication of performance information among staff and
stakeholders can facilitate the use of performance information by agency
managers. Improvements can be achieved through frequent and routine
communication, and the use of effective communication tools, such as
visual aids. We previously reported that frequent, regular communication
is key for managers to inform staff and other stakeholders of their
commitment to achieve the agency's goals and to keep these goals in mind
as they pursue their day-to-day activities. 37 Frequently reporting
performance information also allows managers to review the information in
time to take action to make improvements. Program managers can also
communicate performance information upward through the management
hierarchy, and across operating units. We found a number of vehicles
36GAO, Program Evaluation: An Evaluation Culture and Collaborative
Partnerships Help Build Agency Capacity, GAO-03-454 (Washington, D.C.: May
2, 2003).
37GAO/GGD-96-118.
Page 27 GAO-05-927 Managing for Results
through which agencies communicated performance information effectively.
For example, agencies maintained frequent, routine communication among
managers and staff through visual tools such as poster displays,
performance scorecards, and intranet sites. Some agencies provided
performance updates through regular e-mail or the distribution of monthly
performance review meeting minutes. In addition, officials said
publicizing performance information can inspire ownership of a unit's
performance, as well as competition between units. For example, scorecards
or other visual presentations of performance information enable staff to
compare and analyze performance, resulting in a competitive interest in
improving performance. We found examples in selected agencies of the
practice of frequent and routine communication and using visual tools to
communicate performance information.
NWS uses a number of display tools to convey performance information to
staff and management at different levels of the organization. Quad charts
and display boards are two examples. Quad charts arose out of the National
Oceanic and Atmospheric Administration's (NOAA) efforts to improve
budgeting and planning systems, and have also been used at the NWS. They
are one-page summaries that provide pertinent information for
decision-making purposes. One particular type of quad chart is prepared to
inform agency executives about program performance. These quad charts are
prepared by each major program and presented during NOAA's quarterly
executive panels. These panels are composed of the Deputy Under Secretary
of NOAA and the deputies of the line offices. A one-page chart is divided
into four quadrants containing the following four areas: Performance
Parameters (metrics and threshold values for selected performance
indicators); Schedule of Activities (including schedule status); Key
Issues and Risks; and Funding (budgetary issues). For each of these
topics, status is also color-coded in green, yellow, or red. An example of
this type of quad chart is shown in figure 7. Similar types of quad charts
are prepared by managers requesting program adjustments (such as changes
in funding) and for selected NWS and NOAA programs. Some quad charts also
include GPRA performance measures when applicable to a NOAA or NWS
program.
Figure 7: NOAA Quad Chart Template
Source: NOAA.
NWS also uses display boards at all of its field and regional offices to
communicate information to all staff. As shown in figure 8, display boards
convey national, regional, and site performance information on GPRA
measures relevant to specific site activities. These charts allow site
personnel to compare the site's performance with performance measured at
the regional and national levels. Sites can update their own performance
measures as often as monthly, while national data are updated annually.
Comparable site-level information is available for all sites on the NOAA
Intranet.
Figure 8: NOAA Display Board Template
Source: NOAA.
FAA communicates performance information to all staff through a monthly
performance scorecard. The scorecard is organized into a table, and
includes the actual and targeted performance levels for the agency's 31
key metrics, as identified in FAA's strategic plan, or Flight Plan. Each
metric is color coded, using the stoplight system of red, yellow, or
green, according to how closely actual performance met the targeted
performance for each month. The scorecard is posted on FAA's Intranet and
is updated monthly. The same information is reported in more detail on a
quarterly basis on FAA's external Web site.38 An example of one of the
metrics in the quarterly report, Runway Incursions,39 is shown in figure
9.
38The FAA Quarterly Performance Report can be found online at
http://www.faa.gov/about/plans_reports/performance/performancetargets/book0/section0/
section.html.
39A runway incursion is any occurrence in the airport runway environment
involving an aircraft, vehicle, person, or object on the ground that
creates a collision hazard or results in a loss of required separation
with an aircraft taking off, intending to take off, landing, or intending
to land.
Figure 9: FAA Scorecard for Runway Incursions
Source: FAA.
Conclusions
Our 2003 survey of federal managers found that, despite the fact that
managers reported having more performance measures, the use of performance
information for program management activities did not increase
significantly from 1997 levels when GPRA was first implemented
governmentwide. Creating results-oriented cultures in which performance
information is routinely used to make key management decisions will
require the sustained attention and commitment of top agency leadership
and more widespread adoption of the practices identified in this report.
The five federal agencies that we examined for this report provide
examples of how agencies can use performance information for key
management decisions and practices that can facilitate such use. The
specific ways in which the case agencies used performance information or
implemented the practices may not be appropriate for wholesale adoption
throughout the federal government because agencies face different
management conditions and challenges, and operate under different
authorities. Nevertheless, the general uses and practices highlighted in
this report are universal and could be adapted by each agency. Helpful
next steps would be for the relevant experiences of agencies in using
performance information and adopting practices that facilitate use of
performance information to be more widely shared, and for agencies to be
encouraged to adapt practices to their unique situations.
The Departments of Commerce, Labor, Transportation, and Veterans
Agency Comments
Affairs, and the Small Business Administration provided technical comments
that were incorporated where appropriate throughout the report.
As agreed with your office, we will send copies of this report to the
Director of the Office of Management and Budget and other interested
parties. We will also make copies available to others upon request. In
addition, the report will be available at no charge on GAO's Web site at
http://www.gao.gov.
Please contact me on (202) 512-6543 if you or your staff have any
questions about this report. Contact points for our Office of
Congressional Relations and Public Affairs may be found on the last page
of this report. Other contacts and staff acknowledgments are listed in
appendix II.
Sincerely yours,
Bernice Steinhardt Director, Strategic Issues Appendix I
Scope and Methodology
To identify uses of performance information and practices that encourage
the use of performance information, we reviewed our prior reports and
other relevant literature and interviewed agency officials. In addition,
we interviewed experts in using performance information from the following
organizations: the School of Public Policy at the University of Maryland;
the School of Policy, Planning and Development at the University of
Southern California; the Institute for the Study of Public Policy
Implementation at American University; the School of Public Policy and
Public Administration at George Washington University; the Urban
Institute; the IBM Center for the Business of Government; and the Mercatus
Center.
To identify examples of ways in which managers use performance information
and practices that enhance their use of that information, we selected five
federal agencies that were identified as having a greater likelihood of
using performance information: the Departments of Commerce (DOC), Labor
(DOL), Transportation (DOT), and Veterans Affairs (VA); and the Small
Business Administration (SBA). Among the factors that we considered in
guiding our selection processs were that the agencies received relatively
high scores on the 2000 GAO Government Performance and Results Act survey
results regarding the extent to which managers reported using performance
information,1 relatively good marks on the September 2004 Executive Branch
Management Scorecard (President's Management Agenda), and/or
recommendations from interviews with experts. We also considered diversity
in terms of program type and size.
Table 1 contains brief background information on the agencies within the
selected departments that are highlighted in our case examples.
1 GAO-01-592.
Appendix I Scope and Methodology
Table 1: Background Information on Agencies Selected to Illustrate Performance
Information Uses and Practices
Department or Selected program
agency Selected agencies or offices type
DOL Employment and Training Administration Block/ formula
(ETA) ETA provides job training, employment grants
and labor market information, and income
maintenance services. Within ETA, the
Office of Workforce Investment (OWI) is
responsible for providing national
leadership, oversight, policy guidance, and
technical assistance for the one-stop
public employment service system and the
youth and adult employment and training
programs funded under the Workforce
Investment Act (WIA).
Employee Benefits Security Administration Regulatory-based
(EBSA) EBSA is tasked with protecting the
integrity of pensions, health plans, and
other employee benefits. EBSA's Office of
Participant Assistance (OPA) provides
information and assistance for individuals
covered by private retirement and health
and welfare plans, plan sponsors, and
members of the employee benefits community.
EBSA's goal is to enable participants to
better understand and exercise their rights
under the law, and when possible, recover
any benefits to which they may be entitled.
They also seek to give practitioners the
opportunity to better understand and comply
with the law. The Office of Enforcement is
tasked with protecting pension and welfare
benefits under ERISA. The Office of
Enforcement's investigative field staff
conduct investigations to detect and
correct violations of Title I of ERISA and
related criminal laws.
VA Veterans Health Administration (VHA) The mission of Direct federal
VHA is to assist veterans by providing comprehensive
healthcare services including primary care,
specialized care, and related medical and social
support. Comprehensive health care services are
provided for veterans through VHA's field facilities
and Veterans Integrated Service Networks. VHA's
medical system consists of 21 of these integrated
networks of care whose focus is pooling and aligning
resources.
SBA Small Business Administration (SBA) The principal Direct federal;
mission of the Small Business Administration is to credit;
maintain and strengthen the nation's economy by block/formula
aiding, counseling, assisting and protecting the grants;
interests of small businesses and by helping competitive grant
families and businesses recover from national
disasters. Within SBA, the Office of Capital Access
assists small businesses in obtaining loans
necessary for growth by being a gap lender,
assisting small businesses that otherwise would not
qualify for un-guaranteed financing in obtaining
equity, or taking advantage of procurement
opportunities. The Office of Government Contracting
and Business Development promotes increased small
business participation in the federal procurement
market for goods and services and fulfills SBA's
statutory mission to ensure that a fair share of
federal procurement goes to small businesses. The
Office of Entrepreneurial Development, through a
network of partners and its own on-line Small
Business Training Network, provides counseling,
training and business information to small
entrepreneurs. These resources help to start and
grow small businesses throughout the country. The
Office of Disaster Assistance provides affordable,
timely, and accessible financial assistance for
homeowners, renters, and businesses of all sizes
affected by disaster. Financial assistance is
available in the form of low-interest, long-term
loans.
Appendix I Scope and Methodology
(Continued From Previous Page)
Department Selected program or agency Selected agencies or offices type
DOT National Highway Traffic Safety Administration (NHTSA) Block/formula
grant; NHTSA's mission is to save lives, prevent injuries, and reduce
economic costs due to regulatory-based road traffic crashes. NHTSA
establishes and enforces automobile safety regulations, including crash
avoidance and crashworthiness standards, and consumer protection
standards, including fuel efficiency standards, bumper standards, and
regulations relating to odometer tampering and domestic content labeling.
NHTSA also investigates safety defects and carries out the duties and the
powers of DOT to provide for aspects of highway safety other than highway
safety design features, such as driver performance.
Federal Aviation Administration (FAA) Direct federal; The mission of the
FAA is to improve the safety and efficiency of aviation. FAA competitive
grant; promotes safe flight of civil aircraft in air commerce by
prescribing standards for the regulatory-based design; material;
construction; quality of work; and performance of aircraft, aircraft
engines, and propellers. In addition, FAA issues certificates such as
those for airworthiness and airport operations. FAA also helps develop and
maintain a safe and efficient nationwide system of public-use airports
that meets the present and future needs of civil aeronautics. Finally, FAA
licenses commercial space launches and the operation of nonfederal launch
sites within the United States and by U.S. citizens abroad.
DOC National Oceanic and Atmospheric Administration (NOAA) Direct federal
The mission of NOAA is to understand and predict changes in the earth's
environment and to conserve and manage coastal and marine resources to
meet our nation's economic, social, and environmental needs. A part of
NOAA, the National Weather Service (NWS) provides weather, hydrologic, and
climate forecasts and warnings for the United States, its territories,
adjacent waters, and ocean areas; for the protection of life and property;
and for the enhancement of the national economy. NWS produces and
maintains data in a national information database, the products and
infrastructure of which are used by other governmental agencies, the
private sector, the public, and the international community.
Source: GAO.
Within each of the five selected agencies, we conducted interviews with
program/policy management and staff to obtain their suggestions regarding
specific programs in which performance information has been used for
management. We then interviewed staff involved with those programs in more
detail and reviewed related documents to obtain further information about
how those programs have used performance information and any associated
practices that have contributed to those uses. We selected specific
examples to highlight each use and practice and to illustrate particular
aspects of how that use or practice was implemented within specific agency
program contexts and across various program types. We did not attempt to
identify all agencies and programs that could have illustrated these uses
and practices. In addition, while we did review existing relevant
information regarding performance data quality where available, we did not
systematically assess the quality of the performance
Appendix I Scope and Methodology
information used in the examples we cite. Finally, for most of the
examples, we describe how performance information was used to make
decisions, but we did not attempt to verify that the use ultimately
resulted in improved outcomes.
From our review of the literature and interviews with experts and staff
from the five agencies, we developed a conceptual framework identifying
four categories of uses of performance information and five categories of
practices that contribute to using performance information. The framework
and categories of uses and practices are consistent with the themes
identified in our March 2003 report on results-oriented cultures and
effective performance management, and are consistent with additional GAO
reports on GPRA and results-oriented government.2 The four use categories
in our framework that we identified through our independent analysis are
similar to the four types of uses identified in How Federal Programs Use
Outcome Information: Opportunities for Federal Managers, by Harry Hatry,
Elaine Morley, Shelli Rossman, and Joseph Wholey, published by the IBM
Endowment for The Business of Government in May 2003. We recognize that
alternative categories of uses and practices could have been developed and
that there may be additional uses and practices that we did not identify.
To provide additional support and further illustrate the uses and
practices, we conducted two discussion groups. The first discussion group
included about 15 agency officials from the 5 selected case agencies and
was intended to gain further perspective on those agencies' uses and
practices. To broaden the range of experiences we could draw upon, we also
conducted a second discussion group consisting of about 15 representatives
from 5 different agencies that were not in the pool of selected case
agencies: the Departments of Agriculture, Health and Human Services,
Housing and Urban Development, the Interior, and Justice. For each
discussion group, we asked agencies to nominate potential participants who
had hands-on experience with using performance information, at least 3
years of management experience, and were at a GS13 level or higher. We
selected participants on the basis of their self-described experience
using performance information and selected them to reflect a range of
programs within agencies.
2 GAO-01-592 , GAO-03-488 , and GAO-04-38 .
Appendix I Scope and Methodology
We conducted our review from September 2004 through August 2005 in
accordance with generally accepted government auditing standards.
Appendix II
GAO Contact and Staff Acknowledgments
Bernice Steinhardt (202) 512-6543 or [email protected].
GAO Contact
In addition to the contact name above, Elizabeth Curda, Assistant
Director;
Acknowledgments
Maya Chakko; Chelsa Gurkin; Anne Inserra; Anne Marie Morillon; and Susan
Wilschke made significant contributions to this report. In addition, Tom
Beall provided key assistance.
Bibliography
Ammons, David N. Municipal Benchmarks: Assessing Local Performance and
Establishing Community Standards. Thousand Oaks, California: Sage
Publications, 1996.
Fountain, James, Wilson Campbell, Terry Patton, Paul Epstein, Mandi Cohn,
Mark Abrahams and Jonathan Walters. Reporting Performance Information:
Suggested Criteria for Effective Communication.
Governmental Accounting Standards Board: Norwalk, Connecticut, 2003.
Hatry, Harry P., Performance Measurement: Getting Results. Washington,
D.C.: The Urban Institute, 1999.
------"Public and Private Agencies Need to Manage for Results, Not Just
Measure Them." Washington, D.C.: The Urban Institute, August 31, 2004.
http://www.urban.org/url.cfm?ID=900731
Hatry, Harry P., Elaine Morley, Shelli B. Rossman and Joseph S. Wholey.
How Federal Programs Use Outcome Information: Opportunities for Federal
Managers. Washington, D.C.: IBM Center for the Business of Government,
2003.
Morley, Elaine, Harry P. Hatry and Jake Cowan. Making Use of Outcome
Information for Improving Services: Recommendations for Nonprofit
Organizations. Washington, D.C.: The Urban Institute, 2002.
Newcomer, Kathryn, Edward T. Jennings, Jr., Cheryle Broom and Allen Lomax.
Meeting the Challenges of Performance-Oriented Government. Washington,
D.C.: American Society for Public Administration, 2002.
Osborne, David and Peter Hutchinson. The Price of Government: Getting the
Results We Need in an Age of Permanent Fiscal Crisis. New York: Basic
Books, 2004.
Wholey, Joseph S. Overcoming the Challenges in Managing for Results.
Washington, D.C.: IBM Center for the Business of Government, 2004.
Related GAO Products
Workforce Investment Act: States and Local Areas Have Developed Strategies
to Assess Performance, but Labor Could Do More to Help. GAO- 04-651 .
Washington, D.C.: June 1, 2004.
Results-Oriented Government: GPRA Has Established a Solid Foundation for
Achieving Greater Results. GAO-04-38. Washington, D.C.: March 10, 2004.
Highlights of a GAO Forum: High Performing Organizations: Metrics, Means,
and Mechanisms for Achieving High Performance in the 21st Century Public
Management Environment. GAO-04-343SP. Washington, D.C.: February 13, 2004.
Human Capital: Key Principles for Effective Strategic Workforce Planning.
GAO-04-39. Washington, D.C.: December 11, 2003.
Results-Oriented Government: Using GPRA to Address 21st Century
Challenges. GAO-03-1166T. Washington, D.C.: September 18, 2003.
Program Evaluation: An Evaluation Culture and Collaborative Partnerships
Help Build Agency Capacity. GAO-03-454 . Washington, D.C.: May 2, 2003.
Results-Oriented Cultures: Creating a Clear Linkage between Individual
Performance and Organizational Success. GAO-03-488. Washington, D.C.:
March 14, 2003.
Human Capital Management: FAA's Reform Effort Requires a More Strategic
Approach. GAO-03-156 . Washington, D.C.: February 3, 2003.
Pension and Welfare Benefits Administration: Opportunities Exist for
Improving Management of the Enforcement Program. GAO-02-232. Washington,
D.C.: March 15, 2002.
Results-Oriented Budget Practices in Federal Agencies. GAO-01-1084SP.
Washington, D.C.: August 2001.
Managing for Results: Federal Managers' Views on Key Management Issues
Vary Widely Across Agencies. GAO-01-592 . Washington, D.C.: May 25, 2001.
Related GAO Products
Managing for Results: Federal Managers' Views Show Need for Ensuring Top
Leadership Skills. GAO-01-127. Washington, D.C.: October 20, 2000.
Performance Plans: Selected Approaches for Verification and Validation of
Agency Performance Information. GAO/GGD-99-139. Washington, D.C.: July 30,
1999.
Executive Guide: Effectively Implementing the Government Performance and
Results Act. GAO/GGD-96-118 . Washington, D.C.: June 1996.
Results-Oriented Cultures: Insights for U.S. Agencies from Other
Countries' Performance Management Initiatives. GAO-02-862 . Washington,
D.C.: August 2, 2002.
Managing for Results: Efforts to Strengthen the Link Between Resources and
Results at the Veterans Health Administration. GAO-03-10 . Washington,
D.C.: December 10, 2002.
Results-Oriented Cultures: Implementation Steps to Assist Mergers and
Organizational Transformations. GAO-03-669 . Washington, D.C.: July 2,
2003.
Human Capital: Senior Executive Performance Management Can Be
Significantly Strengthened to Achieve Results. GAO-04-614. Washington,
D.C.: May 26, 2004.
Workforce Investment Act: Labor Should Consider Alternative Approaches to
Implement New Performance and Reporting Requirements. GAO-05- 539 .
Washington, D.C.: May 27, 2005.
The Government Performance and Results Act: 1997 Governmentwide
Implementation Will Be Uneven. GAO/GGD-97-109 . Washington, D.C.: June 2,
1997.
Tax Administration: IRS Needs to Further Refine Its Tax Filing Season
Performance Measures. GAO-03-143 . Washington, D.C.: November 22, 2002.
Architect of The Capitol: Midyear Status Report on Implementation of
Management Review Recommendations. GAO-04-966 . Washington, D.C.: August
31, 2004.
Related GAO Products
Managing For Results: Emerging Benefits From Selected Agencies' Use of
Performance Agreements. GAO-01-115. Washington, D.C.: October 30, 2000.
Human Capital: Observations on Final DHS Human Capital Regulations.
GAO-05-391T. Washington, D.C.: March 2, 2005.
Managing For Results: Strengthening Regulatory Agencies' Performance
Management Practices. GAO/GGD-00-10 . Washington, D.C.: October 28, 1999.
Human Capital: A Self-Assessment Checklist for Agency Leaders.
GAO/OCG-00-14G . Washington, D.C.: September 2000.
Managing For Results: Challenges Agencies Face in Producing Credible
Performance Information. GAO/GGD-00-52 . Washington, D.C.: February 4,
2000.
Human Capital: Preliminary Observations on Proposed DOD National Security
Personnel System Regulations. GAO-05-432T. Washington, D.C.: March 15,
2005.
GAO's Mission
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
The fastest and easiest way to obtain copies of GAO documents at no cost
is through GAO's Web site ( www.gao.gov ). Each weekday, GAO posts GAO
Reports and newly released reports, testimony, and correspondence on its
Web site. To
have GAO e-mail you a list of newly posted products every afternoon, go to
www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
Contact:
To Report Fraud, Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: [email protected]
Federal Programs Automated answering system: (800) 424-5454 or (202)
512-7470
Gloria Jarmon, Managing Director, [email protected] (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125 Relations
Washington, D.C. 20548
Paul Anderson, Managing Director, [email protected] (202) 512-4800
Public Affairs
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
*** End of document. ***