Managing for Results: Strengthening Regulatory Agencies' Performance
Management Practices (Letter Report, 10/28/1999, GAO/GGD-00-10).

Pursuant to a congressional request, GAO provided information on
performance management and measurement practices that could position
regulatory agencies to more effectively implement the Government
Performance and Results Act of 1993.

GAO noted that: (1) GAO gathered information from 23 federal and state
organizations that GAO or other credible sources identified as using or
planning to use a variety of useful practices to enhance specific
aspects of their performance management and measurement processes; (2)
GAO has grouped these practices into the following five categories: (a)
restructure the organization's management approach to become more
performance-oriented; (b) establish relationships outside of the
organization to enhance performance; (c) refine performance goals,
measures, and targets to better translate activities into results; (d)
strengthen analytical capabilities and techniques to better meet
performance management information needs; and (e) assess
performance-based management efforts on a continuous basis to identify
areas for improvement; (3) the organizations, although they had
different missions, sizes, and organizational structures, said they
consistently recognized that these practices are important in their
efforts to develop a stronger results orientation; (4) the organizations
used the practices to varying degrees; (5) thus, the practices did not
appear to be a function of any particular organizational characteristic;
(6) GAO believes the practices would be readily transferable to the
federal financial institution regulatory agencies or other governmental
agencies seeking to improve their implementation of the Results Act; (7)
in addition, the practices are consistent with those identified in GAO's
previous reports; and (8) these reports, for example, described
approaches agencies were taking to address analytical and technical
challenges in measuring program performance, align employee performance
with organizational missions and goals, and address the influence of
external factors in developing performance measures.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  GGD-00-10
     TITLE:  Managing for Results: Strengthening Regulatory Agencies'
	     Performance Management Practices
      DATE:  10/28/1999
   SUBJECT:  Regulatory agencies
	     Accountability
	     Performance measures
	     Strategic planning
	     Private sector practices
IDENTIFIER:  GPRA
	     Government Performance and Results Act

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved.                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                                            **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************

United States General Accounting Office
GAO

Report to the Chairman, Committee on Banking and

Financial Services, House of Representatives

October 1999

GAO/GGD-00-10

MANAGING FOR RESULTS
Strengthening Regulatory Agencies' Performance

Management Practices

Ordering Information

The first copy of each GAO report and testimony
is free. Additional copies are $2 each. Orders
should be sent to the following address,
accompanied by a check or money order made out
to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are
accepted, also. Orders for 100 or more copies to
be mailed to a single address are discounted 25
percent.

Order by mail:

U.S. General Accounting Office
P.O. Box 37050
Washington, DC 20013

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-
6000 or by using fax number (202) 512-6061, or
TDD (202) 512-2537.

Each day, GAO issues a list of newly available
reports and testimony. To receive facsimile
copies of the daily list or any list from the
past 30 days, please call (202) 512-6000 using a
touch-tone phone. A recorded menu will provide
information on how to obtain these lists.

For information on how to access GAO reports on
the INTERNET, send e-mail message with "info" in
the body to:

[email protected]

or visit GAO's World Wide Web Home Page at:

http://www.gao.gov

United States
General Accounting Office
Washington, D.C. 20548-0001
Official Business
Penalty for Private Use $300
Address Correction Requested

                    Bulk Rate
               Postage & Fees Paid
                       GAO
                 Permit No. G100

(233584)

Contents
Page 6          GAO/GGD-00-10 Managing for Results
Letter                                                                      1
                                                                             
Appendix I                                                                 10
Practice Category 1:
Restructure Management
Approach
                           Summary                                         10
                                                                             
Appendix II                                                                24
Practice Category 2:
Establish Relationships
Outside of the
Organization
                           Summary                                         24
                                                                             
Appendix III                                                               32
Practice Category 3:
Refine Performance
Goals, Measures, and
Targets
                           Summary                                         32
                                                                             
Appendix IV                                                                45
Practice Category 4:
Strengthen Analytical
Capabilities and
Techniques
                           Summary                                         45
                                                                             
Appendix V                                                                 55
Practice Category 5:
Continue Improving
Performance-Based
Management
                           Summary                                         55
                                                                             
Bibliography                                                               57
                                                                             
Related GAO Products                                                       62
                                                                             
Tables                     Table IV.1: Florida Measure                     47
                           Definitions
                                                                             
Figures                    Figure I.1:  Example of Mission                 16
                           Statement Used to Develop Strategic
                           Goals-U.S. Coast Guard
                           Figure I.2:  Example of Relating                17
                           Efforts to Outcomes-U.S. Coast Guard
                           Figure II.1: Example of a Survey for            27
                           Client Input on Agency
                           Performance-Florida Department of
                           Banking and Finance
                           Figure II.2: Matrix Showing                     28
                           Crosscutting Program
                           Relationships-Nuclear Regulatory
                           Commission
                           Figure III.1:  Constructing National            34
                           Highway Traffic Safety
                           Administration's Program Logic Model
                           Figure III.2:  Decisionmaking Tiers             39
                           Figure III.3:  Examples of Performance          41
                           Comparisons
                           Figure III.4:  Examples of Targets Set          44
                           Using Baseline Data-U.S. Coast Guard
                           Figure IV.1:  National Highway Traffic          51
                           Safety Administration Matrix to
                           Define Problems and Strategies
                                                                             

Abbreviations

BCI       Border Coordination Initiative
DOT       Department of Transportation
EPA       Environmental Protection Agency
FAA       Federal Aviation Administration
FDA       Food and Drug Administration
FDBF      Florida Department of Banking and
Finance
FDI       Florida Department of Insurance
FGFFC     Florida Game and Freshwater Fish
Commission
FHCA      Florida Agency for Health Care
Administration
FSIS      Food Safety and Inspection Service
INS       Immigration and Naturalization Service
MDES      Minnesota Department of Economic
Security
MDOC      Minnesota Department of Corrections
MDOR      Minnesota Department of Revenue
MDOT      Minnesota Department of Transportation
MNA       Mission Needs Analysis Process
MOEA      Minnesota Office of Environmental
Assistance
MPCA      Minnesota Pollution Control Agency
NHTSA     National Highway Traffic Safety
Administration
NOAA      National Oceanic and Atmospheric
Administration
NRC       Nuclear Regulatory Commission
NSF       National Science Foundation
OPPAGA    Office of Program, Policy Analysis and
Government Accountability
OSFI      Office of the Superintendent of
Financial Institutions
OSHA      Occupational Safety and Health
Administration
PBPM      planning, budgeting, and performance
management
TDOB      Texas Department of Banking
MOE       Measures of Effectiveness

B-281899

Page 5          GAO/GGD-00-10 Managing for Results
     B-281899

October 28, 1999

The Honorable James A. Leach
Chairman, Committee on Banking
 and Financial Services
House of Representatives
 
 Dear Mr. Chairman:

     As part of your oversight of federal agencies
that are responsible for regulating financial
institutions,1 you expressed concern over the
progress these agencies have made in implementing
the Government Performance and Results Act of 1993
(Results Act). At your request, in September and
November 1998, we provided you with our
observations on how the financial institution
regulatory agencies could improve their annual
performance plans.2 Subsequently, on November 10,
1998, you requested that we undertake a
performance management and measurement "best
practices"3 study to identify model approaches
that might help these agencies in their efforts to
better implement the Results Act.

     This report summarizes information provided
in a May 26, 1999, briefing to representatives
from the federal financial institution regulatory
agencies, your office, and other federal
representatives. In the briefing, we presented
performance management and measurement practices
that could position the agencies to more
effectively implement the Results Act. Details on
these practices are presented in appendixes I
through V. Additionally, a bibliography and a list
of related GAO products are included at the end of
this report.

Results in Brief
We gathered information from 23 federal and state
organizations that we or other credible sources
identified as using or planning to use a variety
of useful practices to enhance specific aspects of
their performance management and measurement
processes. We have grouped these practices into
the following five categories, which are also
detailed in appendixes I through V:

ï¿½    Restructure the organization's management
approach to become more performance-oriented.
ï¿½    Establish relationships outside of the
organization to enhance performance.
ï¿½    Refine performance goals, measures, and
targets to better translate activities into
results.
ï¿½    Strengthen analytical capabilities and
techniques to better meet performance management
information needs.
ï¿½    Assess performance-based management efforts
on a continuous basis to identify areas for
improvement.

Each appendix describes individual practices
within each category and provides examples of how
one or more of the state and federal organizations
we reviewed said they have implemented or plan to
implement these practices.

The organizations, although they had different
missions, sizes, and organizational structures,
said they consistently recognized that these
practices are important in their efforts to
develop a stronger results orientation.  The
organizations used the practices to varying
degrees. Thus, the practices did not appear to be
a function of any particular organizational
characteristic. We believe the practices would be
readily transferable to the federal financial
institution regulatory agencies or other
governmental agencies seeking to improve their
implementation of the Results Act.  In addition,
the practices are consistent with those identified
in our previous reports, which are listed at the
end of this report. These reports, for example,
described approaches agencies were taking to
address analytical and technical challenges in
measuring program performance, align employee
performance with organizational missions and
goals, and address the influence of external
factors in developing performance measures.

Background
Congress passed the Results Act as part of a
legislative framework to instill performance-based
management in the federal government. The Results
Act establishes a management system to set agency
goals for program performance and to measure
results against those goals. In enacting the Act,
Congress and the administration realized that the
transition to results-oriented management would
not be easy. For that reason, the Act provided for
a phased approach to implementation.

Implementing the Results Act in a regulatory
environment is particularly challenging. In the
past, regulatory agencies have cited numerous
barriers to their efforts to establish results-
oriented goals and measures. These barriers
included problems in obtaining data to demonstrate
results, accounting for factors outside of the
agency's control that affect results, and dealing
with the long time periods often needed to see
results.

Over the past several years, we have issued
reports that identified practices for improving
Results Act implementation in federal agencies.
These reports have focused on, among other things,
overcoming barriers specific to regulatory
agencies, improving the usefulness of annual
performance plans to decisionmakers, and measuring
program results that are under limited federal
control. These reports point out the depth and
scope of management practices needed to
successfully implement performance-based
management as envisioned under the Results Act.

     At your request, during the last year, we
reviewed the annual performance plans of the
federal financial institution regulatory agencies.
Our review identified several ways in which the
agencies could improve their plans, including
making performance goals more results-oriented;
more clearly linking performance goals and
measures with strategic goals; more fully
describing crosscutting efforts with other
agencies; more fully explaining how strategies and
resources would be used to achieve agency goals;
and providing more details on data verification
and validation efforts and data limitations.4

Scope and Methodology
To identify practices potentially useful to the
federal financial institution regulatory agencies,
we first gathered views from these agencies to
identify issues they believed we should consider
in our work. Overall, the agencies were interested
in the specific activities, organizational
support, and incentives that would comprise a
successful performance management approach, and
they wanted us to use specific examples whenever
possible. To develop this information, we selected
several government organizations-mostly regulatory
in nature-that our past work or other credible
sources identified as having, or planning to
implement, all or parts of performance management
and measurement practices that other agencies
might find useful.

At the federal level, the organizations we
reviewed included the Federal Aviation
Administration, the Food and Drug Administration,
the U.S. Customs Service, the U.S. Coast Guard,
the National Highway Traffic Safety
Administration, the Environmental Protection
Agency, the Occupational Safety and Health
Administration, the Food Safety and Inspection
Service, the National Science Foundation, and the
Nuclear Regulatory Commission. In addition, we
reviewed the performance management approach of
Canada's Office of the Superintendent of Financial
Institutions.

At the state level, the organizations we reviewed
included the following five Florida state
agencies: the Office of Program, Policy Analysis,
and Government Accountability; the Department of
Banking and Finance; the Department of Insurance;
the Game and Freshwater Fish Commission; and the
Agency for Health Care Administration. We also
reviewed the following six Minnesota state
agencies: the Department of Economic Security, the
Pollution Control Agency, the Department of
Transportation, the Office of Environmental
Assistance, the Department of Corrections, and the
Department of Revenue. Lastly, we reviewed the
Texas Department of Banking.

We collected data on the performance management
and measurement efforts of these organizations. We
did not directly verify the accuracy of these
data.  Because of the recent adoption of the
practices, we were unable to determine the extent
to which the organizations' emphasis on results
was directly traceable to specific practices.
However, we did ask the organizations to review
our results for accuracy and completeness. Each
organization agreed with our characterization of
the information discussed in this report. To
supplement the organizational information, we
reviewed available public and private sector
performance management and measurement literature.

From the organizations we studied, supplemented by
our literature review, we identified
organizational performance management and
measurement practices that appeared useful for the
federal financial institution regulatory agencies.
As part of our methodology, we also asked several
government performance management experts to
review these practices and our interpretation of
the information. These experts generally concurred
with our observations and characterizations.

We did our work between September 1998 and May
1999 in accordance with generally accepted
government auditing standards.

We are providing copies of this report to
Representative John J. LaFalce, Ranking Minority
Member of this Committee; the Honorable John D.
Hawke, Jr., Comptroller of the Currency; the
Honorable Alan Greenspan, Chairman, Board of
Governors of the Federal Reserve System; the
Honorable Donna Tanoue, Chairman, Federal Deposit
Insurance Corporation; the Honorable Norman E.
D'Amours, Chairman, National Credit Union
Administration; and the Honorable Ellen Seidman,
Director, Office of Thrift Supervision. We will
also make copies available to others on request.

This report was prepared under the direction of
Kane Wong, Assistant Director. Key contributors to
this assignment were Sharon Caudle and Patrick
Ward. Please contact me at (202) 512-8678 or Mr.
Wong at (415) 904-2000 if you or your staff have
any questions.

Sincerely yours,

Richard J. Hillman
Associate Director, Financial Institutions
  and Markets Issues
 
_______________________________
1For purposes of this report, the federal
financial institution regulatory agencies include
the Federal Deposit Insurance Corporation, the
Federal Reserve Board, the National Credit Union
Administration, the Office of the Comptroller of
the Currency, and the Office of Thrift
Supervision.
2The Results Act: Observations on FDIC's Annual
Performance Plan (GAO/GGD-98-190R, Sept. 15,
1998), The Results Act: Observations on the
Federal Reserve's 1998-99 Biennial Performance
Plan (GAO/GGD-99-9R, Nov. 9, 1998), The Results
Act: Observations on NCUA's Annual Performance
Plan (GAO/GGD-98-192R, Sept. 15, 1998), The
Results Act: Observations on OCC's Annual
Performance Plan for Calendar Year 1998 (GAO/GGD-
98-189R, Sept. 15, 1998), and The Results Act:
Observations on OTS' Annual Performance Plan (GAO-
GGD-98-191R, Sept. 29, 1998).
3"Best practices" are generally recognized as the
processes, practices, and systems identified in
organizations that could provide models for other
organizations.
4GAO/GGD-98-190R, GAO/GGD-99-9R, GAO/GGD-98-192R,
GAO/GGD-98-189R, and GAO-GGD-98-191R.

Appendix I
Practice Category 1: Restructure Management
Approach
Page 20         GAO/GGD-00-10 Managing for Results
Summary
In the first practice category,1 the organizations
said they restructured their management approach
to become more performance-oriented. Practices in
this area included the following:

1.   Strengthen the organizations' performance-
  based management approach.
  
2.   Enhance organization ownership and
coordination of performance management efforts.
3.   Redesign responsibility and accountability
structures.
Practice 1: Strengthen Management Approach
As a first practice, the organizations said they
strengthened their performance-based management
approach. The characteristics of this practice
included the following:

ï¿½    Make implementation of performance-based
management and achievement of outcomes top
organizational priorities.
ï¿½    Conduct a comprehensive assessment of the
organizations' internal and external environments.
ï¿½    Establish a high-level performance management
support capability.
ï¿½    Develop core performance-based management
competencies through skill development and
operational pilots.

Make Implementation and Achievement Top Priorities
The organizations said top management in the
organizations was committed to successful
implementation of performance-based management and
the achievement of agreed-upon outcomes, making
them top agency priorities. Top management buy-in
and commitment, a high level of involvement, and
consistency in leadership characterized the
organizations. According to the organizations, top
managers were involved in all aspects of
performance-based management, from developing a
performance monitoring and evaluation system to
identifying and assessing key measures. For
example:

ï¿½    The Federal Aviation Administration (FAA)
said that to demonstrate support, the agency head
needed to attend key meetings, support requests
for the time and resources to work on strategic
planning, and be willing to talk with managers and
staff about performance issues and processes.
ï¿½    To show high-level commitment, the U.S.
Customs Service (Customs Service) said the agency
put an Assistant Commissioner in charge of the
performance management redesign in its trade
compliance program.

The organizations we studied also said they used
several methods of communication to explain the
purpose, processes, implementation strategies, and
staff responsibilities for performance management
and measurement. For example:

ï¿½    The Minnesota Office of Environmental
Assistance (MOEA) said that the agency stressed
outcomes in major policy reports.  MOEA shared
performance information with staff and detailed
how the information supported major agency
decisions.
ï¿½    To set the tone for major changes in its
management approach, the Minnesota Pollution
Control Agency (MPCA) said it developed
descriptions of how the organization would be
changing its management approach by, for example,
developing shared goals, using environmental
outcomes, creating alliances with others, and
becoming a learning organization.

In addition, the organizations said top management
ensured that the organizations were accountable
for the implementation of performance-based
management by rigorously tracking and evaluating
action items designed to implement strategic plans
and meet performance expectations. Variances
between actual performance and expected
performance targets were promptly identified and
acted upon. For example:

ï¿½    The Food Safety and Inspection Service (FSIS)
said line managers were (1) involved in preparing
plans for their own areas and (2) regularly
discussed with top management the activities under
way to implement the plans.
ï¿½    The National Science Foundation (NSF) and the
Minnesota Department of Economic Security (MDES)
said they used management meetings to discuss
performance management. NSF said it had a
permanent slot on the Director's agenda to address
Results Act implementation. MDES said the agency
ensured that performance management issues were
discussed regularly at management meetings.

Conduct a Comprehensive Assessment of Internal and
External Environments
As a first step toward developing a tightly
integrated and comprehensive strategic management
approach, the organizations said they performed
comprehensive internal and external assessments.
These assessments generally included identifying
customers and stakeholders and assessing the
agency's mission, vision statement, and operating
principles. For example:

ï¿½    FAA said its overall assessment process was
to examine FAA's legislation, define mission
areas, and define measures for the intended
outcomes.
ï¿½    At the beginning of its performance
management efforts, MDES said it conducted a self-
assessment using the Minnesota Quality Award
criteria, which were based directly on Malcolm
Baldrige award criteria. According to MDES, this
effort allowed the agency to systematically assess
core processes and, at the same time, more
directly engage agency leadership in performance-
based management.
ï¿½    The Food and Drug Administration (FDA) said
that after passage of the FDA Modernization Act of
1997, it conducted an assessment of its external
and internal environments to identify challenges
the agency would face over the next several years.
According to the agency, the assessment included
identifying FDA's statutory requirements and
public expectations, evaluating environmental
factors affecting the agency's future actions, and
reviewing current program performance.

Establish a High-Level Organizational Capability
The organizations said they recognized that even
when performance-based management had the
involvement and support of top management, there
was still a need for a central point to facilitate
and support performance management. Therefore,
they established central units or individuals to
help coordinate the organizations' performance
management efforts, ensure consistency across the
organizations, and provide training on
methodologies and approaches. These individuals or
units were to act as internal consultants, and
they were not responsible for developing plans,
goals, targets, or measures. Those aspects were
clearly the responsibility of program managers.
For example:

ï¿½    The Customs Service said its Office of Budget
Formulation and Planning served as the facilitator
for Results Act implementation, with function
managers and process owners developing their own
performance plans.
ï¿½    The Minnesota Department of Transportation
(MDOT) said it used its Data Services Office to
facilitate the agency's performance measurement
efforts. In addition, the office acted as a
clearinghouse for best practices in measurement
and a source for benchmarking data.

Develop Core Performance-Based Management
Competencies
The organizations said they recognized that to
successfully implement results-oriented
organizational strategies, they needed managers
and staff to be competent in at least the basics
of performance management. According to the
organizations, the competencies were needed for
two purposes. The first purpose was to understand
the rationale of performance management and how
measurement could be used. The second purpose was
to go beyond understanding and actually put
performance management and measurement to use in
directly improving organizational and program
performance. The organizations said they sought to
build the necessary competencies through training,
on-the-job activities, and the use of pilot
efforts in lead programs or organizational units.

The organizations said they offered in-depth
training on performance management. For example:

ï¿½    The National Highway Traffic Safety
Administration (NHTSA) said it conducted a 3-day
"train the trainer" seminar on Results Act-related
areas early in its Results Act efforts. NHTSA also
joined the Consortium for Continuous Improvement,
which the agency has found to be a useful source
of training materials.
ï¿½    MDES said that it used a consultant to
provide a 3-day training session for all managers
and supervisors on various topics, such as quality
concepts and performance requirements. Later, the
agency provided 40 hours of similar training for
the remaining 1,500 employees.
ï¿½    The Florida Department of Banking and Finance
(FDBF) said it provided performance measurement
training as a part of new employee orientations
and provided supervisors with training in
performance management basics. FDBF also said it
conducted yearly refresher training for staff.

According to the organizations, pilot efforts
within the organizations were used to motivate and
provide "lessons learned" for other parts of the
organizations or to focus on specific performance
management tasks. For example:

ï¿½    FAA said one FAA line of business2 with more
experience in performance management served as a
model for the rest of the agency's lines of
business for Results Act implementation.
ï¿½    MDOT said it used divisions "ahead of the
curve" in its performance management efforts. The
agency said these divisions were more practiced in
using performance data to make decisions and
support funding requests, and they motivated other
divisions to improve their performance management
efforts.
ï¿½    The Environmental Protection Agency (EPA)
said its Region 10 was leading a pilot to
establish baselines for water quality and measure
the effects of EPA activities on dairy farmer
behavior and water quality.

Practice 2: Enhance Ownership and Commitment
As a second practice, the organizations said they
enhanced the ownership of and commitment to their
performance management efforts across their
organizations. The characteristics of this
practice included the following:

ï¿½    Have management and staff fully participate
in performance management decisionmaking and agree
on program mission areas, goals, measures, and
targets.
ï¿½    Develop a well-defined and tightly focused
mission statement to direct agency efforts.
ï¿½    Develop a "family" of plans to implement
strategies across the organization.
ï¿½    Align the organizational structure and
coordinating management interrelationships.

Have Management and Staff Fully Participate
The organizations said they ensured that managers
and staff at all levels extensively participated
in the development of goals, targets, and
measures, seeking to secure agreement between
management and staff on the program mission areas,
goals, targets, and measures that would be used
for detailed planning and program management. In
addition, the managers and staff actively
participated in the implementation and tracking of
performance results. For example:

ï¿½    FAA said that its unions and key managers
used a National Partnership Council to address
issues of common concern, and that the Council
participated in the development of the agency's
strategic plan. In addition, FAA said all of its
employee associations received the plan for
comments. FAA program managers also had to agree
on agency mission areas and measures. Further, the
agency said there was a conscious effort to focus
on the most important results or outcomes,
resulting from a long process of negotiation with
lines of business to agree on the goals and
measures.
ï¿½    The Texas Department of Banking (TDOB) said
it used an iterative process to share a draft
strategic plan with field staff, obtain comments,
and then redistribute the draft. TDOB's process
was repeated three times before final agreement
was reached. The agency also said it included key
divisional staff on a strategic planning task
force.
ï¿½    Canada's Office of the Superintendent of
Financial Institutions (OSFI) said the agency's
Performance Measure Advisory Committee-tasked with
developing a performance measurement framework-met
with directors and executives to understand what
measures would be useful to them.
ï¿½    The Florida Agency for Health Care
Administration (FHCA) said it made extensive use
of workgroups, using them to develop measures,
educate staff, enhance interdivisional
communication, and obtain feedback from the
legislature on FHCA's measures.

Use a Well-Defined and
Tightly Focused Mission Statement
The organizations said they built mission
statements that were focused, yet would encompass
all of the programs within the organizations. The
organizations said they found that a clear,
concise mission statement formed the foundation
for a coordinated, balanced set of strategic
goals, performance measures, and strategies to
implement the goals. Without such a mission
statement, the organizations said they found it
difficult to develop an appropriate hierarchy of
goals and strategies across the organizations and
to clearly relate the associated outputs and
outcomes to the organizations' missions. For
example:

ï¿½    The Nuclear Regulatory Commission (NRC) said
it started its Results Act efforts with the
agency's mission statement, using it to formulate
"what if" questions in linking strategic goals to
its mission. That is, the agency said it assessed
whether pursuing a particular strategic goal would
lead to fulfilling its mission.
ï¿½    TDOB said it reduced its mission statement
from two paragraphs to one sentence by focusing on
key outcomes and removing any discussion of goals
and strategies. The new mission statement was "to
promote a stable state banking and financial
services environment and provide the public with
convenient, safe, and competitive financial
services."
ï¿½    FAA said its mission statement-to provide a
safe, secure, and efficient global aerospace
system that contributes to national security and
the promotion of U.S. aerospace safety-translated
into its three mission-based strategic goals on
safety, security, and system efficiency.
ï¿½    As shown in figure I.1, the U.S. Coast Guard
said it translated its mission statement-to
protect the people, the environment, and the
maritime security of the United States-into five
strategic goals that described the outcomes the
agency sought to achieve or influence over the
long term. These goals included the major outcome
areas of safety, protection of natural resources,
mobility, maritime security, and national defense.
The Coast Guard said that the agency's programs,
policies, facilities, processes, procedures,
activities, and requirements all should ultimately
be linked to achieving the agency's mission,
vision, and strategic goals. Figure I.2 shows a
diagram used by the Coast Guard to illustrate the
relationship of effort-what the Coast Guard
does-to outcomes-why the Coast Guard undertakes
its efforts. The Coast Guard said it had five
traditional roles that are categorized into
specific mission areas for organizational and
administrative purposes. Activities performed in
each mission area were to contribute to one or
more of the agency's strategic goals, leading to
accomplishment of the Coast Guard's mission.

Figure I.1:  Example of Mission Statement Used to
Develop Strategic Goals-U.S. Coast Guard

Source: U.S. Coast Guard.

Figure I.2:  Example of Relating Efforts to
Outcomes-U.S. Coast Guard

Source: U.S. Coast Guard.

Design a Family of Plans
Across the Organization
The organizations said they saw a need to direct
and coordinate performance management and
measurement efforts within each of their
organizations, and found that using a family of
plans encouraged the direct linkage of strategic
goals and measures to operational and support
goals, measures, and related activities. By using
performance plans at all levels of an
organization, each level's goals could be
carefully integrated with those of the other
levels, allowing all of the organization's
strategies and activities to be oriented toward
achieving the principal strategic goals. The
organizations also said that a family of plans
helped prevent the occurrence of contradictory
goals across the organization. In addition, they
said a hierarchy of plans was developed to
recognize that different management levels need
different performance information, and that goals,
objectives, and measures become more meaningful if
they relate to the appropriate level of
responsibility and control. The use of measures
targeted at different decisionmaking tiers is
discussed further in practice 7.

Following are examples of organizations' use of a
family of plans:

ï¿½    The Coast Guard said it had a complete family
of plans illustrating how planning efforts at
various levels of the Coast Guard were intended to
be linked. The agency said the family of plans was
designed to facilitate the communication of vital
information for decisionmaking purposes between
the Coast Guard's operational and logistics
components in the field and at headquarters.
According to the agency, the family of plans
started with the Coast Guard 2020-a broad internal
and external environmental scan-which outlined
probable challenges and opportunities that the
Coast Guard may face in the coming decades. The
strategic plan served as the implementation
vehicle for Coast Guard 2020 and was developed to
provide focus and alignment for the development of
business plans and specialized plans. The
performance plan showed how the agency intended to
translate the resources it has, and those it was
requesting, into performance outcomes for a
specific budget year. Other parts of the family of
plans included (1) the performance report that
detailed the annual level of performance actually
achieved; (2) the annual budget that detailed the
resources needed to fund operations and logistics
activities; (3) specialized plans for major
capital assets, such as workforce and systems; (4)
business plans for headquarters strategies,
measures, objectives, and resources; and (5)
regional strategic assessments that were
assessments of risk, threat, opportunity, and
demand as well as resource requirements and major
issues from area and district commanders for
incorporation into headquarters business plans and
the agency strategic plan.
ï¿½    FAA said it used a linkage across several
primary documents, as follows: (1) FAA's strategic
plan coordinated with the DOT plan, (2) FAA's
annual performance plan, (3) FAA's line of
business and performance plans, and (4) FAA's
performance agreement with the DOT Secretary.
According to FAA, the agency's lines of business
prepared their own business plans, complete with
their own performance measures. The plans
described the work and work-related activities
that each major organizational unit would
undertake in the next several years.
ï¿½    NRC said it developed operating plans for
each of its programs, covering program
commitments, significant information technology
initiatives, program assumptions, self-assessments
and evaluations, and a summary of quarterly
resource changes. The program commitments included
various items, such as planned accomplishments,
resources, milestones, measures, status,
activities, and quarterly targets.
ï¿½    MDOT said it operated from a family of plans
that included a statewide strategic plan, an
agencywide strategic plan, district strategic
plans, and an upcoming agencywide business plan.

Change Organizational
Structure and/or Management Interrelations
The organizations we studied said they followed
the principal that "form follows function," or in
this case, form follows the intended results
described in the mission statement and the
strategic goals. The organizations said they
evaluated whether their organizational structures
aided or hindered meeting performance
expectations, and made the necessary changes. For
example:

ï¿½    Using a business process improvement
methodology, the Customs Service said it
identified its core and mission support processes
and managed through those processes. Its outcome-
oriented core processes included trade compliance
(the commercial importation of merchandise),
passenger processing (the processing of passengers
entering and leaving the United States by vehicle,
vessel, and air), and outbound processing (the
commercial exportation of merchandise). Mission
support processes included information and
technology, financial, and human resources
processes. The agency said it assigned a process
owner to each of its processes to ensure
accountability.
ï¿½    FDBF said it defined separate program areas
by first identifying the ultimate outcomes desired
by the agency, such as consumer protection or bank
safety and soundness.
ï¿½    The Minnesota Department of Corrections
(MDOC) said the agency organized its divisions by
customer, such as adult facilities and juvenile
facilities, with each division required to relate
its efforts to the agencywide outcome goals.
ï¿½    MPCA said it was restructuring its divisions
so that it could attack environmental problems
more holistically and avoid duplication. Instead
of being organized around four "media"
divisions-Air Quality, Ground Water and Solid
Waste, Hazardous Waste, and Water Quality-the
agency was becoming geographically organized,
allowing different district offices to focus their
resources on the most serious problems faced in
their area.

The organizations said they also recognized that
coordination among managers was important in
achieving performance goals and, therefore,
crafted operating procedures and activities to
foster this coordination, especially when several
program or functional areas shared similar goals.
For example:

ï¿½    FAA said the success of the FAA plan required
assigning both accountability and coordination
within the organization. FAA lead and support
organizations were designated for each goal and
project. According to FAA, each lead organization
called together managers of key supporting
organizations to discuss interrelationships, what
is required by whom and when, and how the goal or
project would be accomplished.
ï¿½    FSIS said that the performance management
efforts helped program managers work together by
laying out common goals and describing how
managers should interact with each other to meet
the goals. This provided a structure to implement
the goals and identified assignments and tasks to
implement them.

Practice 3: Redesign Responsibility and
Accountability
Structures
As a third practice, the organizations said they
redesigned organizational responsibility and
accountability structures. The characteristics of
this practice included the following:

ï¿½    Target daily activities and projects to
support strategies to implement goals.
ï¿½    Integrate performance goals, measures, and
costs into budget structures and decisionmaking.
ï¿½    Link performance appraisals to performance
accountability and responsibility.

Target Daily Activities to Implement Performance
Goals
The organizations said the family of plans
previously mentioned was to align strategic goals
and lower-level organizational goals. As a further
alignment step, the organizations said they
clearly linked organizational strategies-which
were to reflect daily activities and projects-to
strategic goals. According to the organizations,
this approach encouraged an outcome orientation
down to the task level in the organization,
thereby building a clearly linked hierarchy of
goals, objectives, measures, and implementation
activities. Performance management became a clear
part of how the organizations were to manage day
to day in such areas as securing and justifying
resources and defining accountability. For
example:

ï¿½    EPA said it used more detailed strategic plan
objectives with subobjectives for annual
performance goals. Shorter term activities
indicated the contribution of lower levels of the
organization to the strategic goals.
ï¿½    FSIS said its strategic plan was coordinated
with an agency database containing 26 activities
that were used to track, on a monthly basis,
specific implementation tasks assigned to staff
for the agency's new meat and poultry safety
program.
ï¿½    NHTSA said each program office had defined
milestones for activities and projects supporting
strategies for the annual performance plan, such
as completing rulemaking by a certain date.
ï¿½    The Minnesota Department of Revenue (MDOR)
said the agency used information from each of its
functional areas in its daily management to
compare various measures to current targets as
well as the results achieved in prior years.
According to the agency, the performance
information was contained in reports available to
all employees over the agency's network, all
updated at specific times. For example, the agency
said it weekly updated the report on the number of
days taken to process refunds for paper tax
returns. The report provided information on the
year-to-date count for specific processing times,
including the target of processing the returns
within 90 days if received by April 1, and 120
days if received after April 1. The report
contrasted this current information with the
processing time performance of the past 2 calendar
years.

Integrate Performance Management Into Budgetary
Processes
The organizations said they recognized that one of
the biggest incentives for performance-based
management was seeing information on results
integrated into budgetary structures and
decisionmaking. The organizations said they
aligned their budgets with program activities
which, in turn, were tied to program goals,
targets, and measures. The organizations said they
were very specific in defining the direct and
indirect costs of the activities that produced
outputs, especially those that could be linked to
intermediate or final outcomes. For example:

ï¿½    TDOB said it used performance measurement to
allocate and prioritize the use of its limited
resources. When TDOB did not meet targets in one
area, the agency said it could divert resources
from an area where the agency had exceeded its
performance targets.
ï¿½    The Coast Guard said it was using activity-
based costing to identify direct and indirect
costs in program areas and developing standard
rates for time and for fully loaded costs.
According to the Coast Guard, this will enable it
to identify all costs to the activity level and
link them to outcome areas. The agency said it was
also refining its resource allocation process to
align its assets with mission requirements.
ï¿½    NRC said it was implementing an integrated
and disciplined system to improve its processes
for planning, budgeting, and performance
management (PBPM). According to NRC, the PBPM
system was to establish a process for defining
agency goals, develop cost-effective strategies to
achieve those goals, determine the resources
needed to implement the agency's strategic
direction, and measure and assess the agency's
progress.
ï¿½    FAA said it was in the process of developing
a cost accounting system that linked outputs to
activities and costs. FAA said it already had a
formal agency needs assessment process in place,
called the Mission Needs Analysis Process (MNA).
This formal process, for example, called for an
assessment of the need for new systems and
equipment to meet the mission and strategic goals
of the agency that related to future National
Airspace System capital requirements. According to
FAA, each organization within the agency was to
conduct its own assessments to establish its
mission needs. In addition, FAA said it was
developing a cost and performance management
system to track the costs incurred in performing
services, to allow management to measure the
effectiveness of its workforce, and to rank return
on assets.
ï¿½    The Customs Service said it also was
developing a resource allocation model to identify
the effect of increases in operational staff on
requirements for support staff. According to the
agency, the model was to be used to develop a
budget and a basis for requesting funding. The
model allowed the user to perform an "what-if"
analysis, such as changes in the amount of
overtime, minimum staffing constraints, or
staffing allocations.

Use Performance Agreements and Appraisals
The organizations said they incorporated
performance management into performance agreements
and appraisals by clearly specifying performance
responsibilities and accountabilities. The
organizations said they did this to (1) increase
the visibility and importance of performance
management results and (2) encourage managers and
staff to pay attention to performance information
and outcomes. In addition to the performance
agreements and appraisals, the organizations said
they established a firm link, at the individual
and/or unit level, between the performance
expectations and incentives. The incentives
included both monetary and nonmonetary incentives.
Nonmonetary incentives included recognition
awards, regular performance reports to the
organization, and managerial flexibility in
exchange for more performance accountability. For
example:

ï¿½    NSF said agency staff were determined that
whatever performance management system was
implemented, it would be consistent with normal
program management and a useful management tool.
As a result, senior executive service performance
objectives contained at least one Results Act-
related element.
ï¿½    FAA said that each month its lines of
business managers reported on their progress in
reaching the goals set forth in the performance
agreement between the FAA head and the Department
of Transportation (DOT) Deputy Secretary. The
agency said the performance agreement contained
measures used in the performance plan and specific
projects and milestones that were not in the
performance plan. FAA also said it was piloting,
at the senior executive service level, a program
tying incentives, such as bonuses, to key
performance measures in the strategic plan.
Agreements with executive-level staff were to
specify how the executives would contribute to
FAA's three primary goals, weighted by the areas
in which the executives had responsibilities.
ï¿½    NHTSA said it used an employee performance
management program that focused on results, not
behaviors and characteristics, and provided a
linkage between individual performance results and
required outcomes of organizational performance.
According to NHTSA, outcomes in the employee's
performance plan were desired end-results that the
employee and the organization were trying to
achieve. The employee performance plan was to link
the work of the employee to the organization's
goals. NHTSA said the outcomes were generally
beyond the employee's ability to control, but
defined what the employee and the organization
were working toward. The employee performance plan
contained performance targets (equivalent to
performance standards) for each outcome. The
agency said the targets were those activities that
the employee should accomplish by the end of the
performance period that contributed to the
attainment of the outcome or desired end-result.
The employee was accountable for the performance
targets, not the outcomes per se.

_______________________________
1 The 5 practice categories in appendixes I
through V contain 12 practices that are numbered
consecutively throughout the appendixes.
2Within FAA, there are six lines of business: Air
Traffic Services, Research and Acquisition,
Regulation and Certification, Airports, Civil
Aviation Security, and Commercial Space
Transportation.

Appendix II
Practice Category 2: Establish Relationships
Outside of the Organization
Page 30         GAO/GGD-00-10 Managing for Results
Summary
In the second practice category, the organizations
said they established relationships outside of the
organization for the purpose of enhancing
performance. They then worked through these
relationships to implement specific programs.
Practices in this area included the following:

4.  Establish results-oriented collaborative
relationships with regulated
     entities and program delivery partners.

5.  Establish partnerships with other
organizations involved in
     implementing crosscutting programs.

Practice 4: Establish Results-Oriented
Collaborative
Relationships
In the fourth practice, the organizations said
they established, at a program level, results-
oriented collaborative relationships with
regulated institutions and program delivery
partners. Specific practices included the
following:

ï¿½    Work with regulated entities in identifying
the organization's mission, performance goals,
targets, and measures.
ï¿½    Involve regulated entities in the prevention
aspect of performance.
ï¿½    Build consensus among program delivery
partners on performance targets, measures, and
data use.
ï¿½    Obtain periodic input from regulated entities
and delivery partners on the organization's
performance management efforts.

Involve Regulated Entities in Identifying Mission,
Goals, Targets, and Measures
The organizations said they recognized the need
for the entities they regulated to be involved in
developing the organizations' mission, goals, and
targets, as well as selecting performance measures
and helping determine their use in program
operations. According to the organizations, the
consultative process was meaningful and involved
extensive dialogue with the regulated entities.
For example:

ï¿½    FAA held annual sessions with its
stakeholders, called "Challenger Sessions," in
which FAA discussed its mission and goals with
representatives from industry groups, such as
airlines and manufacturers, as well as user
groups, such as pilots. The agency said the
stakeholders also played a role in defining FAA's
strategic goals and strategies. For example, FAA
said that the agency established air traffic
control preferred routes to minimize conflicts in
congested airspace. However, the FAA routes often
differed significantly from the routes that
airline pilots and flight planners would prefer to
optimize their operations on the basis of their
own objectives and constraints. In response, FAA
said it began working closely with airlines to
share air traffic information so that
collaborative decisions could be made.
ï¿½    The Coast Guard said it conducted regional
strategic assessments that examined the demand for
agency services and attempted to anticipate
current and future demands for services and
resources. These assessments examined partnerships
with such entities as ports and waterways.
ï¿½    The Customs Service said it worked with
industries and communities to achieve outcomes. To
illustrate, the agency said it worked with local
community and Mexican government officials to
measure delays entering the United States from
Mexico and develop strategies to reduce the delays
but still ensure compliance.
ï¿½    MDOR said the agency validated its initial
strategic objectives with over 70 customers and
external stakeholders and used focus groups as
part of its target-setting processes, thereby
getting customer feedback on what constituted
acceptable targets.

Involve Regulated Entities
in the Prevention Aspect of Performance
The organizations said they took the approach that
regulated entities should join with the regulators
in the prevention of problems and the
"coproduction" of performance results. The
organizations said they recognized that the
regulated entities themselves should assume more
responsibility for identifying and addressing
performance issues. According to the
organizations, this approach was to allow them to
target resources more effectively. For example:

ï¿½    FAA said the agency wanted to reduce the
fatal commercial aviation accident rate by 80
percent, but FAA could not do so solely through
its role as regulator and enforcer. The agency
said it worked with the aviation community,
analyzing data to agree on the top accident
categories and then conducting a detailed root-
cause analysis. According to FAA, this approach
had been instrumental in helping stakeholders
throughout the aviation community-both the
regulators and the regulated entities-reach
agreement on the key accident categories on which
to focus. FAA said that most important was the
subsequent stakeholder commitment to effective
implementation of intervention strategies to
prevent future accidents in these categories.
ï¿½    FSIS said the agency's new meat and poultry
food safety program provided a more specific and
critical approach to the control of
microbiological hazards in foods than the
approaches provided by traditional inspection and
quality control. The program required meat and
poultry slaughter and processing plants to adopt a
system of process controls to prevent chemical,
physical, and biological food safety hazards.
According to FSIS, the program allowed FSIS
inspectors to use a combination of a scientific
approach for monitoring meat company management
practices and traditional carcass-by-carcass
inspections. FSIS said its employees analyzed
company meat and poultry safety practices,
monitored company records, and conducted
laboratory tests to ensure that company testing
was accurate.
ï¿½    The Occupational Safety and Health
Administration (OSHA) said the agency's pilot
Maine Top 200 Program encouraged employers to
identify hazards themselves and take corrective
action before the hazards led to injury and
illness. According to the agency, this pilot
program enabled OSHA to focus on workplaces where
the largest number of serious injuries and
illnesses occurred.

Build Consensus Among Program Delivery Partners
on Targets, Measures, and
Data Use
Similar to their approach with regulated entities,
the organizations said they recognized the need
for the involvement of the organizations on which
they relied in delivering program products and
services, such as state regulatory agencies. The
organizations said they saw the delivery partners
as key stakeholders and sought to establish
agreement on program mission and objectives, the
most useful performance measures, and the
potential use of performance measures. For
example:

ï¿½    EPA said its Office of Enforcement and
Compliance Assurance initiated the agency's
National Performance Measures Strategy in January
1997 to develop an enhanced set of performance
measures for EPA's enforcement and compliance
assurance program. EPA said it consulted with
state environmental agencies on the measures,
which resulted in several key ideas. For example,
the states believed that EPA's enforcement and
compliance assurance program should (1) place more
emphasis on the use of outcomes and environmental
indicators to measure performance and (2) reduce
its emphasis on outputs as a measure of
performance.
ï¿½    OSHA said it relied on 23 states and 2 U.S.
territories to operate their own OSHA-approved
occupational safety and health programs. According
to OSHA, these states were integral partners in
OSHA's mission of ensuring the safety and health
of workers. In developing its strategic plan, OSHA
said it directly involved these organizations in
reviewing the plan. The agency said this approach
clarified the role of states and how OSHA's
strategic plan applied to state programs.

Obtain Periodic Input on Performance Management
Efforts
On an ongoing basis, the organizations said they
secured input from their regulated entities and
delivery partners on the effectiveness of program
operations. In addition, they gathered information
from these partners on program mission statements
and goals. For example:

ï¿½    MDES said it conducted quarterly customer
satisfaction surveys through its workforce
centers, with some centers going even further and
conducting focus groups with staff and clients.
MDES said it used the survey data to update its
mission, values, and goals.
ï¿½    FDBF said it obtained feedback from its
customers-the banks it regulated-on how they were
regulated by conducting short surveys on various
areas, such as the bank examination process,
examination reporting, and examination team
competency. A portion of the survey is shown in
figure II.1.

Figure II.1: Example of a Survey for Client Input
on Agency Performance-Florida Department of
Banking and Finance

Source: Florida Department of Banking and Finance.

ï¿½    MDOR said the agency conducted postaudit
customer surveys to obtain information on the
organization's audit process. The agency said the
surveys solicited responses on how auditors
treated the agency's customers, not the outcome of
the audit, and revealed that most employers do
want to comply with tax laws.

Practice 5: Establish Rigorous Partnerships Across
Organizations
Under the fifth practice, the organizations said
they sought to identify other organizations
involved in implementing crosscutting programs,
then establish rigorous partnerships with those
organizations. Specific practices included the
following:

ï¿½    Identify the contributions of each
organization and develop common outcomes.
ï¿½    Establish a leadership role to coordinate
cooperative efforts across common goal areas.
ï¿½    Use tools to facilitate common data sharing
across partner organizations.

Identify Crosscutting Relationships, Common
Outcomes, and Agency Contributions
The organizations said they recognized the
importance of interactions among agencies involved
in similar or related programs. They identified
organizations with related programs, their
relationship with those organizations, and common
outcome expectations. For example, FSIS said it
included a specific goal to establish effective
working partnerships with other public health
agencies and stakeholders to support the
President's National Food Safety Initiative, which
called for a reduction in foodborne illnesses. The
agency said specific activities were to include
(1) collaborative monitoring with the Center for
Disease Control, FDA, and state public health
departments; (2) establishing cooperative
agreements with states for risk assessment; (3)
developing standard operating procedures for
coordination in cases of foodborne illness
outbreaks and other food safety emergencies; and
(4) coordinating strategies with the Department of
Health and Human Services, the United States
Department of Agriculture, and private sector
groups to expand communication of food safety
information to the general public.

According to the organizations, their crosscutting
efforts also involved coordinating their program
activities and linking their goals and strategies
with those of the partner organizations. For
example, NRC's annual performance plan identified
areas of mutual interest with other agencies, the
related NRC programs, and the crosscutting
strategies NRC would use to address the shared
responsibilities with these other agencies. NRC
presented most of this information in the form of
a matrix in its annual plan. A portion of that
matrix, describing part of NRC's interaction with
the Department of Energy, is shown in figure II.2.

Figure II.2: Matrix Showing Crosscutting Program
Relationships-Nuclear Regulatory Commission

Source: U.S. Nuclear Regulatory Commission.

The organizations said they involved crosscutting
partners, jointly and equally, in developing
common or consistent outcome goals, associated
measures, targets, and data collection procedures
as well as in determining the types of decisions
they would be used to make. The organizations said
they also used interagency agreements and
performance partnerships to clarify each partner's
roles and responsibilities. For example:

ï¿½    NRC said that in most instances, it had or
was developing memorandums of understanding or
other agreements with other agencies to ensure
that areas of mutual interest and cooperation were
treated in a consistent, coordinated, and
complementary way that avoided unnecessary
duplication or conflict.
ï¿½    MDOC said that it completed literature
reviews and other secondary research to ensure
that the agency was measuring many of the same
indicators as other criminal justice agencies. As
a result, the agency said it had the capability to
benchmark with other agencies.

In addition, the organizations said they worked to
identify, to the maximum extent possible, each
agency's contribution to the common outcome
expectations and whether those contributions were
unique or common to other organizations. The
organizations said they also took other
organizations' efforts into consideration when
making resource plans. According to the
organizations, working with other agencies also
helped the organizations to leverage resources by
combining project efforts. For example:

ï¿½    NHTSA said it developed a beneficial
relationship in situations where another agency
could regulate something NHTSA could not. For
example, the draw strings on children's jackets
were getting caught on school bus doors, and NHTSA
worked with the Consumer Product Safety
Commission, which provided guidelines on
children's clothing.
ï¿½    The Coast Guard said the agency evaluated
where it fit in a crosscutting program logic model
(described in practice 6) and the level of agency
outputs or outcomes that might be appropriate. In
one case, the Coast Guard said the National
Oceanic and Atmospheric Administration (NOAA)
determined fishing regions and was interested in
the intended outcome of protecting fish habitats.
To contribute to NOAA's intended outcome, the
Coast Guard produced an output-patrolling miles of
ocean. The Coast Guard, said the agency, was not
responsible for protecting fish habitats, but
could support NOAA in doing so.

Establish a Leadership
Role to Coordinate
Cooperative Efforts
The organizations said they recognized that
successfully implementing crosscutting efforts
among "equal" partners could be difficult without
specific leadership. To remedy this, the
organizations said they established one lead
organization or a specific leadership role among
the partners. For example:

ï¿½    The Customs Service said it and the
Immigration and Naturalization Service (INS)
developed the Border Coordination Initiative (BCI)
to increase cooperation on the southwest border
for the interdiction of drugs, illegal aliens, and
other contraband. The agency said the BCI was
intended to be a comprehensive, integrated border
management system at the southwest border,
achieving the mission of both the Customs Service
and INS. According to the Customs Service, the
agencies established a joint Office of Border
Coordination, comprising two border coordinators
representing INS and the Customs Service, that
managed the BCI and was responsible for overseeing
border operations.
ï¿½    FSIS said its strategic plan focused on
national and international food safety and was
targeted at all food safety efforts at the federal
and state levels. Relying on crosscutting efforts,
FSIS said it was working with federal and state
officials to optimize the food safety resources
available at all levels. The agency planned to
exercise leadership through memorandums of
understanding and cooperative agreements.

Use Tools to Facilitate
Common Data Sharing
The organizations said that to assist in
crosscutting efforts, the partner organizations
increased the usefulness of their common data
sharing by establishing common data definitions
and information systems. Common data definitions
were to help ensure that data used for common
purposes would be consistently defined, collected,
calculated, and interpreted. In addition, the
organizations said they (1) identified existing
information systems within each organization that
might serve common interests and (2) shared
information with their partner organizations. For
example:

ï¿½    NHTSA said it recognized the need to share
information with other agencies and establish
information systems to do so. For example, the
agency said that the Department of Health and
Human Services asked NHTSA for information
regarding the use of drugs and driving. According
to the agency, information sharing such as this
bolstered NHTSA's requests for additional
performance data funding.
ï¿½    FDA said the agency's focus on outcomes was
tied to building information systems linked with
other agencies, such as the Center for Disease
Control, with the same goal, thereby making it
easier to integrate and share performance
information with these agencies.
ï¿½    The Customs Service said that it and INS,
collaborating on the previously mentioned BCI
initiative, worked to develop common data
definitions and data collection procedures.

Appendix III
Practice Category 3: Refine Performance Goals,
Measures, and Targets
Page 44         GAO/GGD-00-10 Managing for Results
Summary
In the third category, the organizations said they
refined their performance goals, measures, and
targets to better translate activities into
results. The practices included the following:

6.   Establish a rationale of how individual
  programs delivered results.
  
7.   Select meaningful goals and measures
  consistent with the rationale.
  
8.   Set appropriate targets for performance
  goals.
  
Practice 6: Establish
a Rationale of
How the Program
Delivers Results
Under the sixth practice, the organizations said
they sought to establish a rationale of how their
individual programs delivered results. Specific
practices included the following:

ï¿½    Take a holistic or "systems" approach.
ï¿½    Build a program logic model that described
how activities translated to outcomes.
ï¿½    Expand program assessments and evaluation
efforts to validate model linkages and rationale.

Look at Programs Using
a Holistic or "Systems" Approach
The organizations said they recognized that
performance management and measurement efforts
could not be viewed from a limited perspective,
such as that of an individual program, but must be
seen in terms of the operation as a whole. The
organizations said they concentrated on the
relationships and interactions of whole systems,
as opposed to managing parts of a system. The
organizations said they took a holistic approach
to identify and evaluate factors that affected
their outcomes, determine appropriate strategic
goals, and assess how a change in one goal might
affect another.  For example:

ï¿½    OSHA said that the agency's strategic
planning process recognized that all of the
agency's strategic goals would be interrelated,
and that working in one goal area would affect the
work in another area.
ï¿½    FDA said that the agency was improving its
efforts to describe total program relationships
and systems. For instance, FDA said it was moving
from setting only measurable performance goals to
setting goals whose measures were important as
strategic points in an overall system to respond
to the agency's mission.
ï¿½    MPCA said that the agency was dealing with
pollution problems in a more holistic way. For
example, air pollution resulting from mobile
sources-cars, trains, airplanes, and boats-touched
a number of MPCA program areas. In the past, the
agency said it focused on specific media, such as
air and water. The holistic approach, using
environmental impact data, was designed to help
decisionmakers focus on the environment as a
whole.
ï¿½    OSFI said that it used the image of a scale
to demonstrate the interplay of its five strategic
objectives. On one side of the balance was
"safeguarding from undue loss" (representing the
agency's identification of specific risks and
trends and intervention in a timely manner) and
"public confidence" (evaluation of systemwide
risks and promotion of the adoption of sound
business and financial practices). On the other
side was "competition" (the agency's due regard
for the regulated institutions' need to compete
effectively) and "cost effectiveness" (maintaining
full and open dialogue with stakeholders on the
costs and benefits of the agency's work). The
balance's fulcrum was "quality," representing
OSFI's objective of carrying out its mission with
quality people, processes, and technology. OSFI
said that it recognized some objectives would
counterbalance others. For example, if greater
emphasis was placed on "safeguarding from undue
loss" by introducing tougher rules for financial
institutions, there might be an adverse impact on
the ability of financial institutions to be
innovative and competitive.

Develop Basic Program Logic Models
The organizations said they sought to develop a
better understanding of how their programs worked
so that they could select appropriate performance
goals and measures. To do so, they said they
described the logic or rationale of how individual
programs used inputs, such as resources and
workload, in program components, such as
activities and processes, to produce outputs. In
turn, those outputs were connected to intermediate
and final outcomes. These descriptions, often
called program logic models, were not necessarily
the more extensive models that might be used in
more comprehensive program evaluations, but they
were concise descriptions of the basic flow from
inputs to outcomes. The organizations said they
did not necessarily intend to describe causality,
but to develop a description of a reasonable
correlation or association showing how inputs were
converted to outputs and outcomes. According to
the organizations, the exercise of developing
logic models can help internal and external
stakeholders (1) see the progression from outputs
to end outcomes, (2) see how changes in program
components and outputs might better impact
outcomes, and (3) better understand their
contributions to desired results. For example:

ï¿½    NHTSA said the agency was using program
impact models that showed linkages between inputs,
outputs or process, intermediate outcomes, and end
outcomes. NHTSA's annual performance plan
described the linkages to support the presentation
of outcome and intermediate goals and measures in
the body of the plan. Output measures, which were
included in the plan's appendix, were to be used
internally for departmental budget justifications
and management decisions. A program logic model
using components from NHTSA's annual performance
plan is shown in figure III.1.

Figure III.1:  Constructing National Highway
Traffic Safety Administration's Program Logic
Model

Source: NHTSA.

ï¿½    The Coast Guard said the agency's Marine
Safety area's business plan followed the program
logic model from inputs to outputs to outcomes. In
addition, the agency said that in its strategic
planning efforts, the agency used a basic model of
internal processes leading to activities, which
led to outcomes. Management goals reflected
internal processes, while activities reflected
performance goals. In this model, the Coast Guard
noted that its performance outcomes were largely
affected by where the Coast Guard decided to
intervene in a problem, by its particular
strategies and activities, and by its internal
management goals.
ï¿½    OSHA said it was also using formal program
logic models in developing performance measures.
The agency viewed measures of activities, such as
the number of inspections, as intermediate
measures leading to the desired outcome of fewer
workplace accidents. The agency said that OSHA's
model flowed from performance goals to outcome
goals to strategic goals to Department of Labor
outcome goals and indicated the basic assumptions
and strategies that were to be used to achieve
desired outcomes. For example, implementing safety
and health program promotional strategies would
lead to employers' having an effective safety and
health program; leading to better safety and
health programs; leading to changing the workplace
culture to increase awareness and commitment of
safety and health; and, finally, leading to
reducing workplace injuries, illnesses, and
fatalities.

Expand Program Assessment and Evaluation Efforts
The organizations said they expanded their
evaluation efforts to explore and confirm the
linkages and rationale within their program logic
models. These efforts included questioning and
testing the assumptions in the program logic
model, the processes and activities selected to
implement the models, and the way in which program
impacts might be separated from external factors.

ï¿½    The Coast Guard said it planned to use
program evaluation to examine the agency's
activities of interaction, influence, and impact
after the agency developed a program policy. The
agency's view was that program evaluation should
serve performance management in validating the
program logic model and its rationale, and help
determine if performance-based management was
worthwhile.
ï¿½    FDA emphasized that a program evaluation
function was needed to encourage program managers
to think from inputs to the end of the logic model
continuum.

Practice 7: Select Goals
and Measures Consistent With the Program
Rationale
Under the seventh practice, the organizations said
they sought to select vital program and support
goals, as well as performance measures, that were
consistent with the program rationale they had
developed. Specific practices included the
following:

ï¿½    Select vital program and support goals
directly or indirectly from a program logic model.
ï¿½    Use rigorous criteria to assess and select
the most important measures.
ï¿½    Develop a comprehensive suite of measures
representing total program performance.

Select Vital Program and Support Goals From
Program Logic Models
According to the organizations, program logic
models formed a foundation for selecting vital
program performance goals, including management
and support goals. The organizations selected
outcome goals, measures, and targets that
reflected the influence of external factors and
the agency's varying levels of influence and
control at different points in the program logic
model. For example:

ï¿½    The Florida Game and Freshwater Fish
Commission (FGFFC) said the agency tried to use
only intermediate measures where the link to
desired outcomes was established. As an example,
FGFFC said it measured the number of cases
successfully closed-referred to the state's
attorney-instead of reduction in crime rates
because the agency (1) had more control over the
number of cases closed and (2) believed, from
experience, that closed cases generally resulted
in conviction and therefore a reduction in the
crime rate.
ï¿½    MPCA said that in cases where the agency
could not develop or obtain data on an
environmental outcome measure, it tracked changes
in behavior it believed led to the desired
outcome. For example, tracking mercury reduction
required a long time frame, so the agency said it
would track behaviors that lead to decreases in
the release of mercury into the environment.
ï¿½    Florida's Office of Program, Policy Analysis
and Government Accountability (OPPAGA) stated that
agencies could consider, as an intermediate
measure, detailing the characteristics of a
healthy regulated population, then explain how the
agency contributed to the development and
maintenance of a healthy population.

The organizations said they considered
administrative or management support goals as
supportive of, and thus subordinate to, strategic
program goals. For example:

ï¿½    NRC said it selected strategic goals on the
basis of the importance to the agency's mission
areas and made support efforts subordinate to the
strategic areas.
ï¿½    FAA said the agency distinguished between
goals that were directly aimed at achieving the
agency's mission and other goals, such as
administrative or management support goals, that
enabled or supported the agency in achieving its
mission.

Use Rigorous Criteria to Select the Most Important
Measures
The organizations said they used rigorous criteria
to assess and select the actual measures. The
selection criteria-such as availability, accuracy,
validity, potential adverse consequences, balance,
and relevance-recognized that meaningful
performance-based management required the use of a
manageable number of useful measures. According to
the organizations, tracking more measures resulted
in an increased data collection burden.
Organizations also said that not carefully
screening measures resulted in measures that were
similar to others or that might be irrelevant to
program results and operational needs. According
to the organizations, the result might be a large
volume of measures that would overwhelm those
measures considered truly important for
decisionmaking and guiding organizational
operations.

In addition, the organizations said they used the
criteria to regularly review and modify the
measures over time. The selection criteria also
recognized the importance of selecting a suite of
measures, as discussed later, reflecting a balance
of measures across the logic model and for
different decisionmaking tiers. The organizations
said the logic models also allowed them to define
what activity information-workload and
process-would best support movement toward the
ultimate outcomes.

The following are examples of the organizations
using criteria to select the most important
measures:

ï¿½    FSIS said its strategic plan focused
primarily on its fundamental public health mission
by limiting the number of goals and accompanying
measures and tying them to the primary goal of
protecting public health.
ï¿½    FGFFC said it selected only measures that
were easily explainable to the legislature and did
a "reality check" with all measures to ensure they
would be useful for both internal and external
decisionmaking.
ï¿½    Several organizations said they took a
customer-based focus. MDES recommended identifying
customers for each measure, ensuring that the
measures were clear, understandable, meaningful,
and measurable. MPCA and MDOT said they primarily
focused on outcomes and measures from the citizen
or customer point of view, and they secondarily
focused on outputs, such as the number of tests
conducted.
ï¿½    OSFI said that before implementing a measure,
the agency emphasized having a full understanding
of the impact of the measure on stakeholders as
well as the potential repercussions of
communicating the measure.
ï¿½    The Customs Service said the agency's fiscal
year 2000 annual performance plan had 31
performance measures, down from 61 in 1999, and
focused on the most meaningful and vital few
measures. At one time, the agency said it used the
number of inspections to measure the performance
of its inspectors, but dropped the measure because
it forced inspectors into "body counts." Other
measures, such as the number of indictments, were
dropped because the Customs Service had no control
over them.

Develop a Comprehensive
Suite of Measures
Representing Total
Program Performance
The organizations said they recognized that
developing a suite of measures across the logic
models and addressing different decisionmaking
tiers within the organization, best measured
program and overall organizational performance.
Thus, the organizations chose performance measures
from the different stages in the model, covering
inputs, outputs, outcomes, and activities or
processes. According to the organizations, the
suite of measures, representing the
interrelationship of multiple measures, helped the
organizations take a broader perspective of
program performance. For example, EPA said it
categorized its measures into three types. These
types included the following: (1) program output
measures that represented actions taken by the
agency, such as the number of permits
issued/revised or the number of enforcement
actions; (2) program outcome measures that
reflected the direct results that lead to
environmental improvements, such as the decreased
use of higher risk pesticides or reduced emissions
of toxics from manufacturing facilities; and (3)
core environmental indicators that represented
ultimate results, such as increases in the number
of rivers and lakes in healthy ecological
condition or decreases in the number of people
with air quality-related illnesses/deaths.

In addition, the organizations said that
developing a suite of measures allowed them to
select a subset of measures targeted at the
different needs of specific audiences. For
example, the measures could meet the information
needs of both internal managers, who needed
information for operational control from the
bottom of the organization to the top, and
external policymakers, who were interested in
measures evaluating the program's success. Figure
III.2 is a diagram highlighting the types of
information needed to make decisions at different
levels within an organization and how the
granularity increases from the top to the bottom.
That is, the most aggregated information is at the
top level, and the most detailed information is at
the bottom. Figure III.2 also describes the timing
and focus of information at these different
levels.

Figure III.2:  Decisionmaking Tiers

Source: GAO summary of decisionmaking tiers.

The organizations said they used the
decisionmaking information to select measures and,
in situations where final outcome information was
not available or would be delayed, to provide
justification for the use of output or
intermediate outcome information. For example:

ï¿½    For each of its five strategic objectives,
OSFI said the agency was developing a suite of
performance measures designed to evaluate progress
in reaching its objectives. For the strategic
objective of safeguarding depositors from undue
loss, the agency established the following
measures: (1) the Level of Intervention Index,
which measured the level of OSFI intervention and
tracked individual financial institutions as their
financial condition changed; (2) the Loss Recovery
Index, which measured the amounts depositors of
liquidated institutions could expect to receive;
(3) the Risk Exposure Index, which was a composite
measure of OSFI's assessment of the level of risk
facing the financial industry at a given time; and
(4) the Intervention Effectiveness Measure, which
measured OSFI's effectiveness in identifying
problem institutions and intervening in a timely
way to address regulatory concerns. In each of
these areas, OSFI said it identified the measures
as in either an early or advanced stage of
development or maturity. For example, the agency
considered its employee satisfaction level
measuring quality as advanced. However, another
quality measure, the extent to which OSFI staff
met identified core competencies, was less mature
and was in an earlier stage of development.
ï¿½    For each of its agencywide goals, MDOR said
it used several different measures to evaluate the
agency's success in achieving its goals. For
example, one of its goals was that "everyone is
paying what is owed, no more, no less." To
evaluate progress toward achieving this goal, MDOR
said it measured voluntary compliance rates,
nonfiler discovery rates, tax filing accuracy, use
and sales tax compliance levels, and the number of
corporate audits completed. The agency said this
mix of measures provided a complete performance
picture.
ï¿½    The Coast Guard said the agency's
aeronautical engineers identified critical
dimensions, or key issues, concerning Coast Guard
aircraft. These dimensions were reliability,
maintainability, supportability, and
affordability. Using these dimensions, the agency
said it created a system of Measures of
Effectiveness (MOE) to track performance against
aviation system goals. Daily, all air stations
were to document when an aircraft was available to
fly a mission. The MOE produced index information
on (1) the percentage of time that aircraft at
Coast Guard air stations were available to perform
missions, (2) how often air stations had needed
aircraft parts, (3) how much unit effort it took
to generate each flight hour, and (4) an overall
maintenance effort index. The agency said that a
suite of measures could be evaluated to help
identify common causes and trends, and were used
to determine workforce levels, program flight
hours, maintenance costs, and budgetary
considerations.

Practice 8: Set Appropriate Targets for
Performance Goals
Under the eighth practice, the organizations said
they used different methods to set appropriate
targets for their performance goals. Specific
practices included the following:

ï¿½    Use different types of performance
comparisons to match performance goals.
ï¿½    Provide multiyear and subgoal performance
targets.
ï¿½    Use targets for distinct populations or
comparison categories that are meaningful to the
organization.
ï¿½    Use baselines to set realistic, but
challenging targets.

Use Different Types of Comparisons
The organizations said they used a diversity of
performance comparisons, depending on the goal, to
set performance targets. The comparisons included
(1) predefined performance specifications, (2)
future performance levels or changes in levels to
be achieved at a later date, (3) best practice
benchmarks from other organizations, and (4)
program implementation milestones. For example,
the Coast Guard said it based its performance
targets on historical performance, trend analysis,
and improvements currently under way. In addition,
some targets were based on a defined performance
level, such as an absolute readiness index score
required by the Department of Defense. Figure
III.3 provides examples of several types of
performance comparisons.

Figure III.3:  Examples of Performance Comparisons

Sources: FDA, the Coast Guard, NRC, FAA, and OSHA.

The organizations said they also recognized that
because of the nature of some goals, they could
not always set absolute targets. For example, in
some cases, the organizations said they did not
have a baseline or benchmark to set a target, such
as when a measure was new, a baseline had never
been established, or benchmarks were not readily
available. When this occurred, the organizations
said they either set a preliminary target or
directional target, or stated that a baseline
would be set with the initial collection of data.
For example:

ï¿½    FDA said that the targets for its strategic
investment goals were a series of milestones for
achieving the desired capability, such as
developing modeling techniques to assess human
exposure and dose response to certain foodborne
pathogens. For program result or outcome goals,
the agency said targets were quantitative or
productivity goals. For example, one target was to
have 80 percent of the domestic seafood industry
operating preventative controls for safety.
ï¿½    FAA said it gathered data to set a baseline
for assessing future needs and desired
accomplishments, such as FAA's efforts to reduce
delays, accidents, and incidents.
ï¿½    The Customs Service said the agency often set
targets as a certain percentage increase in a
measure, rather than setting a specific number as
a target. For example, instead of setting a target
of 91 percent, the agency said the target would be
set as a certain percentage change over the
previous year. Once baseline information was
available, the agency would set specific target
numbers.
ï¿½    NHTSA and MPCA said that they did not set
specific targets in some cases. For example, NHTSA
said it had a difficult time deciding on
appropriate targets in the face of unpredictable
trends and many external factors, and ended up
agreeing on a target of "no increase." For some
measures, MPCA said the agency did not set a
specific target, but only indicated whether it
wanted the measure to increase or decrease.

Use Multiyear and Subgoal Targets
The organizations said they faced situations where
they could not identify specific annual targets.
In these cases, the organizations established
multiyear goals and targets and conducted annual
progress checks. In addition, they set targets at
the subgoal level that were to cumulatively reach
an overarching goal over time. For example:

ï¿½    FAA said it used 3-year or even 10-year
baselines in such areas as commercial aviation
fatal accidents because there were so few of them
in any given year. An increase or decrease of even
one or two accidents could skew the data.
ï¿½    NHTSA said it set a long-term goal of
reducing highway fatalities and injuries by 20
percent by the year 2008 and tied in annual and
multiyear efforts. The target was set by using an
analysis of factors the agency could influence,
and how it could influence them, to estimate the
cumulative effect of reaching goals with various
interventions (intermediate outcomes). The agency
said it developed subgoal targets that were
expected to produce the 20-percent reductions by
the year 2008. The agency thus worked on
intermediate outcomes at the subgoal level and
tied that to an overall outcome goal. NHTSA also
said that the agency tried to avoid straight-line
annual targets, preferring floating targets that
the agency could reassess each year, depending on
progress towards the final outcome goal.
ï¿½    FAA said that the agency also used varying
targets. For instance, FAA's strategic plan
contained a safety mission goal of reducing the
United States aviation fatal accident rates by 80
percent from 1996 levels by the year 2007.
According to the agency, the fiscal year 2000
performance plan had several performance goals to
achieve this overall goal with either annual or
multiyear targets. For example, performance goals
for the safety mission goal included (1) reducing
the fatal aviation accident rate for commercial
air carriers from a 1994-96 baseline of 0.037
fatal accidents per 100,000 flight hours; (2)
decreasing the rate of air shipment hazardous
materials incidents by the year 2000 from a 1998
base; (3) by the year 2007, reducing (by a to be
determined percentage from baseline levels) the
rate of airport accidents/incidents that result in
injury to persons or damage to aircraft; and (4)
reducing the rate of operational errors and
deviations by 10 percent from 1994 baselines. The
agency said it had or was developing a fiscal year
2000 target for each performance goal, such as
reducing the fatal accident rate for commercial
air carriers to 0.033 per 100,000 flight hours.

Use Distinct Populations or Comparison Categories
The organizations, with input from stakeholders,
said they carefully selected performance
categories down to the lowest, meaningful level of
disaggregation of data and used these categories
to set appropriate targets. For example, the
categories included geographical areas, workload
or customer groups, or types of services. The
organizations said they sought to set targets at
the lowest, most disaggregated level so that they
would be meaningful to managers and staff at the
activity level within the organization. In
addition, the organizations said that setting
goals at the appropriate level of disaggregation
aided decisionmakers in evaluating how successful
a program was working with respect to different
categories. For example:

ï¿½    OSHA and FDA said that their agencies set
performance targets for different industries. For
example, OSHA said the agency set targets for
nursing homes, logging, food processing, and
shipyards to focus agency efforts on the most
hazardous industries and workplaces. FDA said it
set targets for such industries as the domestic
seafood industry and foreign food establishments.
ï¿½    MDOR said it segmented its client population
into those with minor offenses and those who were
considered repeat offenders, and the agency
developed measures for each category.
ï¿½    MDOT noted that it had the agency's divisions
categorize or segregate roads by type and develop
measures specific for each type of road. The
agency also set targets for different geographical
areas within the state.

Use Baselines to Set Realistic But Challenging
Targets
The organizations said they used baselines to set
performance targets that were realistic, but they
expected the baselines to challenge the
organization to continually improve. The
organizations said they used the baselines as the
lowest acceptable performance expectation and set
targets at higher levels. For example:

ï¿½    OSHA recommended selecting a "typical" year
and developing averages, rolling averages, or
other statistical measures to set the challenging
targets.
ï¿½    FDA said it planned to reduce the percentage
of food and color additive petitions under review
for more than 360 days to 20 percent in fiscal
year 2000. The baseline data were 44 percent in
fiscal year 1997.
ï¿½    The Coast Guard said the agency used actual
information to develop a trend line and set a
target that was based on planned strategies. For
example, figure III.4 shows a graphic from the
Coast Guard's fiscal year 2000 annual performance
plan illustrating a goal and target to reduce the
passenger vessel casualty rate.

Figure III.4:  Examples of Targets Set Using
Baseline Data-U.S. Coast Guard

Source: U.S. Coast Guard.

Appendix IV
Practice Category 4: Strengthen Analytical
Capabilities and Techniques
Page 49         GAO/GGD-00-10 Managing for Results
Summary
In this practice category, the organizations said
they built on and strengthened their analytical
capabilities and techniques to better meet
performance management information needs. The
practices included the following:

9.   Ensure that data resources and analytical
  capabilities were sufficient to provide
  performance management information.
  
10.  Target analysis at regulatory intervention.
  
11.  Account for external and contextual factors.
  
Practice 9: Ensure Adequate Data Resources and
Analytical Capabilities
In the ninth practice, the organizations said they
sought to ensure that their data resources and
analytical capabilities were sufficient to provide
information necessary for formulating and
assessing strategies. Specific practices included
the following:

ï¿½    Ensure staff analytical capabilities met
performance management needs.
ï¿½    Develop a data infrastructure and information
systems to generate useful performance data.
ï¿½    Use specific tools to support the use of
performance information.
ï¿½    Ensure the quality, timeliness, and
continuity of performance information.

Ensure Staff Capabilities
Meet Performance
Management Needs
The organizations said they assessed their
internal analytical capacity to deal with
performance management-both in terms of staff
available to do analysis and the types of
analytical skills possessed by the staff. They
recognized that analytical capabilities were
critical in assessing risks, aggregating and
disaggregating performance information in a
meaningful fashion, and allocating appropriate
resources to activities linked to strategic goals.
For example:

ï¿½    FAA found that it was critical to have
statistical and financial analysis experts
available to provide data needed by managers to
change practices and make decisions. According to
the agency, FAA analysis units also provided
information to other FAA units for performance
measurement.
ï¿½    The Customs Service said it hired a
statistician to oversee sampling efforts related
to trade compliance measurement and assessment.
According to the agency, compliance measurement
and assessment helped the Customs Service identify
problems and make a specific evaluation of the
risk of noncompliance posed by an individual
company and the industry overall.

Develop Data Resource Infrastructures and
Supporting Information Systems
The organizations said they did not firmly
establish data sources and data collection
processes until they had identified their program
rationale and accompanying goals, measures, and
targets. Once they knew what data they would need,
they said they either used existing data resources
and information systems or upgraded or built new
systems to collect the necessary performance
information and conduct quicker and better data
analyses. For example:

ï¿½    NHTSA said that even though the agency had
tracked performance data for several years-even
producing a data book to support agency
decisionmaking and performance management-the
agency was reassessing its existing data. NHTSA
said it wanted to look at existing data and
evaluate which factors lead to desirable outcomes
and then develop interventions to achieve
intermediate outcomes.
ï¿½    NRC said it initiated the development of an
agencywide, integrated financial and resource
management system called STARFIRE, which was to
serve as the single authoritative source for
financial and resource information and support the
alignment of agency resources with program
outputs, strategies, and strategic goals.
ï¿½    MDOC said it was using the development of a
new data system as an opportunity to include
routine collection of more useful performance
measures and not just a new means to collect the
same information that has always been collected.
MDOC said the system was intended to increase its
ability to provide accurate, descriptive
statistics and to support program evaluations.
ï¿½    MDOT said it was integrating different
management systems within the agency to ensure
consistency in measurement through common
definitions and measurement timing.

Use Specific Tools to Support the Use of
Performance Information
To enhance their performance measurement efforts,
the organizations said they used different tools
to help support the understanding, communication,
tracking, and reporting of performance
information. They said they used performance
measure definitions to identify-for both external
and internal audiences and partners-the measure
being used, its purpose, its source, how it would
be calculated, how it would be verified and
validated, who was responsible for its collection
and analysis, and any limitations it might have.
According to the organizations, they also used
specific definitions to remove ambiguity regarding
what was being measured and to ensure that
measurements could be replicated. For example:

ï¿½    MDOC said it developed a glossary with
performance measurement terms and agency acronyms
that accompanied its performance report and used
extensive footnoting to better explain measures
and data to the reader.
ï¿½    The Coast Guard said that for each of its
performance goals, it included a description of
why the agency acts, key factors, strategies,
coordination, analysis and evaluation-including a
graphic of the target, information for the past
several years, the trend line, and key
initiatives.
ï¿½    Florida state agencies said they were
required to provide specific performance measure
information as part of their performance
management. The measures and their definitions are
shown in table IV.1.

Table IV.1: Florida Measure Definitions
Measure information   Measure definition
Measure number        Unique alpha/numeric
                      identifier that is assigned
                      to each measure
Type of measure       Input, output, or outcome
Name of responsible   Individual responsible for
individual(s)         creating/developing and
                      defining the measure
Purpose of measure    Brief description of what
                      information the measure
                      will provide
Definition of measure Thorough description of the
                      measure that includes
                      defining each word in the
                      measure and what will be
                      included
Formula description   If used to calculate the
                      measure, the formula is
                      written out and each aspect
                      of the formula is defined
Measurement period    Time period that will be
                      covered by the measure
Relationship to       How the measure relates to
mission               the unit's overall mission
Output/Outcome        If applicable, the
relationship          relationship between output
                      measures and outcome
                      measures; identification of
                      any associated measures
                      that relate to the measure
Reporting             Who the measure is reported
requirements          to and the frequency of
                      reporting (e.g., monthly,
                      quarterly, etc.)
Source: Florida Agency for Health Care
Administration.

In addition, the organizations said they used
performance measure databases to track
performance. For example:

ï¿½    FDA and EPA said they collected information
in a performance database. FDA said the agency
maintained a performance goal database showing
inputs and outputs, dollars associated with
specific goal clusters, and the part of the agency
associated with the goals. FDA said it also used a
database warehouse to track goals and resources
over time and link FDA goals and measures to other
legislative requirements, such as FDA's
modernization act and the Chief Financial Officers
Act. According to FDA, this approach was designed
to allow the agency to maintain just one set of
performance information, instead of several sets.
EPA said its database was to be used by EPA goal
teams and national program managers.
ï¿½    OSHA said it was developing a database for
strategic plan measures and, when completed, the
database would be available to all OSHA employees
through its Intranet.
ï¿½    MOEA and MDES were using data matrices as
part of their performance management efforts. MOEA
said it used a set of matrices that described the
linkages between the agency's environmental
outcomes and measures and the agency's primary and
secondary staff responsible for collecting and
evaluating data and information for the measures.
The agency said the matrices had been a useful
tool at all levels in the organization to
understand staff accountability. MDES said the
agency's data matrix listed the customer(s), the
customer needs, what the agency would measure, the
type of measurement, and how data would be
collected for each service that the agency
provided.
ï¿½    MDOR said the agency set up a performance
measure database that included the measure
definition, the purpose of the measure, the
measure owner, and up-to-date trend data.
According to the agency, this standardized format
allowed the agency to use one set of data to
generate reports for each program area. MDOR also
said the agency used this database to generate an
executive summary of measures that listed
agencywide performance measures for each of the
agency's four goals.

Ensure the Quality,
Timeliness, and Continuity
of Performance Information
The organizations said they supported their data
analysis capabilities with accessible, high-
quality and timely data. They said they made a
senior manager or team responsible for data
resources and used data quality evaluative or
auditing functions to assess data integrity and
its sufficiency for supporting the program logic
models. According to the organizations, the
evaluative and auditing functions assessed the
choice of measures, the collection and processing
of data, the quality of information, the
interpretation and explanation of results, and the
relevance and adequacy for decisionmaking. For
example:

ï¿½    The Customs Service said that when field
staff realized management was using performance
data to make decisions, they began providing
accurate information and explanations for any
incorrect data. The agency said it required each
office to establish a data quality function,
responsible for verification and validation, that
would be inspected annually. The Customs Service
also said it established specific measure owners
who were responsible for handling measures in
their own process areas.
ï¿½    FGFFC said it had a key person in each
division or functional area responsible for
developing measures, educating staff, and
gathering performance information.

Practice 10: Target
Analysis at Regulatory Intervention
Under the tenth practice, the organizations said
they targeted analytical techniques for their
regulatory intervention efforts. Specific
practices included the following:

ï¿½    Use risk management techniques to target
resources for the maximum results.
ï¿½    Take a balanced approach to regulatory
efforts by targeting actions before, during, and
after a problem event.

Use Risk Management Techniques to Target
Resources for Maximum
Results
The regulatory organizations said they developed
risk management techniques to target the largest
regulatory problems, attempting to achieve maximum
results for their resource allocation. For
example:

ï¿½    The Coast Guard, OSHA, and NRC said they were
targeting risk areas. The Coast Guard said the
agency used risk analysis to define risk groups
and levels of risk and then put investments toward
prevention in high-risk areas. OSHA noted that the
agency changed its enforcement strategy to target
high-hazard workplaces, focusing on preventing
accidents. NRC said it was developing
recommendations for improving NRC's inspection,
assessment, and enforcement processes to focus on
the most important safety issues.
ï¿½    The Customs Service said it formalized a
Trade Compliance Risk Management Process that
collected data and information, analyzed and
assessed risk, prescribed action, and tracked and
reported information. The agency said that it then
used this data to analyze historical compliance
data and trends for various industries, specific
commodities, and certain importers and, by
applying definitions of significance and
materiality, to focus on areas with the greatest
potential risk. For example, identifying whether
the source of risk was an importer's lack of
knowledge, complex trade laws, or willful
disregard for importer laws would result in
different, specific action plans to assign
resources and address the risk.
ï¿½    OSFI said it developed a risk exposure index
as a composite measure of the agency's assessment
of the level of risk facing the financial industry
at a given time, weighted by the condition of the
institution, the value of assets, and the type of
institution. Over time, OSFI said it planned to
compare and track the impact of major events on
overall system risk.

Take a Balanced Approach
to Regulatory Efforts
The organizations said they carefully balanced
their regulatory efforts to address both
prevention and mitigation. According to the
agencies, they did this by analyzing how they
influenced and directly impacted the entities they
regulated and by targeting regulatory actions
before, during, and after a potential or actual
problem event. The organizations said they also
supplemented their goals and measures related to
mitigation with goals and measures designed to
track and reduce risk factors, thus seeking to
anticipate and prevent problem events where
possible. For example:

ï¿½    NHTSA said the agency sought to achieve
progress in the following two intermediate outcome
areas: (1) reducing the occurrence of crashes and
(2) mitigating the consequences of crashes. NHTSA
said programs used performance measures to help
achieve the intermediate outcomes, which, in turn,
influenced the outcomes. The agency said it used a
matrix, shown in figure IV.1, that addressed three
time phases-precrash, crash, and postcrash, and
where each of its programs had an impact. The
matrix was a tool to use in defining problems and
posing strategies. For example, the agency said
NHTSA's National Advanced Driving Simulator was a
specific effort to conduct research on driver
performance and behavior during the precrash
sequence of events. For postcrash events, the
Intelligent Vehicle Initiative was targeted at
understanding the causes of highway collisions,
and the Emergency Medical Services effort was
designed to enhance the comprehensive emergency
medical service systems to care for victims of
crashes.

Figure IV.1:  National Highway Traffic Safety
Administration Matrix to Define Problems and
Strategies

Source: NHTSA.

ï¿½    The Coast Guard said the agency targeted its
activities to (1) improve operational methods for
a quicker response-mitigation-and (2) prevent
problems from occurring-prevention. In the Coast
Guard, the agency said the focus on prevention was
preceded by a careful analysis of cause and effect
and what the agency could influence. For example,
the Coast Guard said it developed an analysis that
recommended preventing Alaskan commercial fishing
industry disasters before they occurred, as well
as preparing to react to them if they should
occur. The analysis described the fatal events and
recommended strategies to prevent or reduce the
seriousness of these events. The prevention
efforts revolved around critical factors, such as
vessel stability and hull integrity, skipper and
crew training and licensing, avoidance of harsh
sea and weather conditions, preventing falls
overboard, and safer diving practices. Instead of
saving lives after a vessel casualty, the Coast
Guard said it could take action in these areas to
improve the prevention of fatalities.
ï¿½    The Customs Service said the agency found
that it could no longer depend on enforcement and
interdiction efforts because the volume of
transactions required an ever-increasing amount of
resources. Realizing that most importer errors
were not deliberate, but the result of trying to
follow complex requirements, the agency said it
began to focus on accountability management, where
the agency had an obligation to inform importers
about compliance, and thus prevent compliance
problems. However, when importers continued to
violate requirements, the agency said it pursued
appropriate enforcement actions. In addition, the
agency said it examined measures that could
provide some data on the effectiveness of its
border interdiction efforts, such as drug
transport costs. The assumption was that if the
agency could disrupt and dismantle drug
transportation attempts, then drug suppliers would
find it too costly to operate. In this case, the
agency said measures of disruption would
compliment measures of drug seizures.

Practice 11: Account for External and Contextual
Factors
Under the eleventh practice, the organizations
said they sought to account for factors beyond
their control that might have an impact on their
efforts to achieve outcomes. Specific practices
included the following:

ï¿½    Identify and track external and contextual
factors to explain their influence on program
results.
ï¿½    Use smaller units of analysis to better
understand program effects.
ï¿½    Use statistical techniques and program
evaluation to adjust for and isolate the influence
of external factors.

Identify and Track External
and Internal Contextual
Factors
The organizations said they grappled with the
issues of responsibility and accountability for
performance in those areas affected by external
factors, such as the state of the economy or
internal changes in program operations and
technological support. They said they identified
these factors and tracked them over time,
analyzing their impact on specific performance
goals and targets. The organizations said they
used this information to help internal and
external stakeholders understand the influence of
these factors on program results. In addition,
this information was used to refine the rationale
of the program logic models. For example:

ï¿½    NHTSA said it found that discussing the
influence of external factors in the performance
plan helped overcome agency fears of being held
accountable for measures over which the agency had
limited control. The agency said it examined the
effect of external factors using internal and
external resources, such as local law enforcement
studies. In addition, tracking external factors
lent credibility to other data, making it easier
to explain the impact of the agency. The external
factors were discussed in detail in the agency's
strategic plan.
ï¿½    The Florida Department of Insurance (FDI)
said it identified the contributing factors that
could create a demand for services, such as
insurance or fire marshal services. These factors,
such as arson, defective products, acts of
violence, and inadequately trained emergency
personnel, provided the fire marshal with a
starting point for identifying strategies for
influencing or controlling specific factors.
ï¿½    The Coast Guard said it analyzed key factors
related to its goals. For example, the Coast Guard
said it identified several key factors that
increased the difficulty of successfully
responding to mariners in distress, such as
untimely distress notification, severe weather,
poor communication, and poor information about the
distress. The agency said its strategies were
targeted at preventing the distress, but also at
maximizing the survival chances by addressing
these key factors.
ï¿½    FAA said the agency determined that weather
was a factor in 40 percent of aviation accidents
and 50 percent of aviation fatalities; therefore,
FAA focused on influencing the impact of the
weather conditions. For example, the agency said
it had strategies to invest in an integrated
terminal weather system and a weather and radar
processor. These would give air traffic
controllers instant access to current weather
data. FAA said the agency was also implementing
and improving existing weather sensors, targeting
a weather research program to demonstrate storm
growth, and strengthening aviation delay
forecasting technology.
ï¿½    FGFFC said the agency tracked external
factors related to boating accidents, such as
alcohol use, to be able to make a thoughtful
analysis of the factors contributing to an
accident, even though the factors were beyond the
agency's control.
ï¿½    TDOB said it tracked explanatory measures,
such as the number of state-chartered banks in
Texas and the total assets they represent, which
provided the agency with the opportunity to
explain the influence of external factors on
agency performance measures. The agency said it
also included a separate section in the strategic
plan that discussed relevant external and internal
factors.

Use Smaller Units of Analysis
The organizations said they recognized that
program effects were often difficult to evaluate
if analyzed on a national or statewide basis. In
some cases, they said they used a smaller unit of
analysis that was more clearly defined and in
which program efforts and impacts could be more
clearly identified and understood. For example:

ï¿½    Florida's OPPAGA recommended that agencies
segment the population they regulate into smaller
segments and evaluate their efforts on that
segment, where they were better able to control
for external factors.
ï¿½    FGFFC said it conducted detailed observations
on a smaller population, such as a portion of a
river, attempting to identify outputs that
contribute to outcomes in a habitat. The agency
said it was able to carefully measure its efforts,
detailing all of the inputs and outputs, such as
number and location of officers, weather
conditions, accidents, and violations. In doing
this, the agency said it could take a snapshot of
the agency's impact and use it for future
planning.

Use Statistical Techniques
and Program Evaluation
The organizations said they used statistical
techniques, such as normalization, ratios, or
trend analysis, to better identify the influence
of external factors. For example, using
calculations that normalized information as a
standard rate, such as "per capita," provided a
standard that could be compared from year to year.
In addition, the organizations said they used
program evaluations to examine very long-term
outcomes and to isolate program effects from other
factors. For example:

ï¿½    FAA said it accounted for the growth of the
airline industry by normalizing its measures,
stating them in terms of "per flight hours." In
addition, FAA said the agency planned to use logic
models to capture a uniform picture of external
factors and share them with stakeholders.
ï¿½    OSHA said it found that the decline in
occupational injury and illness rates in the early
to mid-1990s was attributable to legislative
reforms motivated by increases in workers'
compensation payments and a growing awareness of
workplace hazards among unions, employers, and the
insurance industry. According to OSHA, it found
such factors as employment shifts into low-hazard
industries and underreporting of injury and
illness rates were not contributory. OSHA said its
reform efforts during this period affected the
agency's inspection strategy and resulted in a
renewed emphasis on outreach, partnering, and
working cooperatively with employers to address
workplace hazards. According to the agency, the
new approach complemented market influences
affecting industry, namely, escalating costs for
workers' compensation programs and the dawning
realization that corrective action was needed to
reduce workplace accidents. OSHA said its reforms
reinforced and supported industry initiatives and
contributed to the decline in occupational injury
and illness rates.
ï¿½    MOEA said the agency contracted with a
consultant to do an evaluation of the effect of
municipal solid waste management on resource
conservation and greenhouse gas emissions to get
some idea of the impact of the agency's efforts on
its outcome goals. The agency said it also
conducted an internal study of the economic
benefits of recycling.
ï¿½    MPCA said the agency discovered that its
"intuitive" linkages regarding mercury
contamination were incorrect when tested by an
independent institute. MPCA adjusted the agency's
strategies accordingly.

Appendix V
Practice Category 5: Continue Improving
Performance-Based Management
Page 56         GAO/GGD-00-10 Managing for Results
Summary
In this category, the organizations said they
continuously assessed their performance-based
management efforts and results to identify areas
for improvement. The specific practice in this
area was as follows:

12. Continuously assess and strengthen performance-
based management
      efforts.

Practice 12: Continuously Assess and Strengthen
Performance-Based Management
The organizations said they continuously assessed
their performance-based management approach and
results, and modified their approach as necessary.
In addition, they said they recognized that the
evolution of an organization's performance
management efforts over time involved changes
across the other four practice areas. For example,
as previously discussed, the leading organizations
said they regularly reviewed their measures,
goals, and targets and adjusted them as necessary.

ï¿½    FHCA started with key measures that were
simple and easy to understand. Once performance
measurement was accepted in the agency and
processes were well established, the agency then
could move to experiment with more sophisticated
measures.
ï¿½    FDA said the agency changed its performance
plan presentation to show the integration between
initiatives and performance goals, more clearly
stating how FDA planned to close the gap between
strategic priorities and current performance. FDA
also said it provided additional information on
baselines and contexts for the agency's
performance goals.

The organizations said they also sought to improve
their management approach by moving on to more
sophisticated methodologies, such as the balanced
scorecard, once initial experience was gained.1
For example:

ï¿½    FDBF said the agency used simple program
logic chains at the beginning of its performance
management efforts. Later, the agency considered
other methodologies, such as the balanced
scorecard. Over time, FDBF said it looked to
further develop and reinforce performance
management capabilities.
ï¿½    The Customs Service, NRC, MDES, OSHA, MDOT,
and MDOR said they were either using, or planning
to use, all or part of a balanced scorecard
approach, building on their past performance
management efforts to create a balanced set of
measures.

In many cases, the organizations cautioned that
they were just at the beginning of a journey to
successfully implement performance-based
management. For example:

ï¿½    MDES said that agencies must be willing to
change many management and operational systems to
better align them for performance. Changing those
systems might take up to 10 years to accomplish.
ï¿½    MDOT said the agency spent 5 years developing
a process for establishing meaningful measures and
then continuously improving them. In the agency's
early performance management efforts, it began
focusing on what the agency could achieve when it
had direct control over events. Later, MDOT said
it adopted more extensive in-depth analysis of
cause-and-effect linkages and the influence of
external factors.

Overall, the organizations' experiences indicated
that a strong performance-based management
approach was under constant review and refinement.
The end result was not just intended to be a
written strategic or performance plan, but a
results-oriented culture within the organization
and among stakeholders.

_______________________________
1The balanced scorecard approach is summarized in
Robert S. Kaplan and David P. Norton, The Balanced
Scorecard: Translating Strategy Into Action,
Harvard Business School Press, Boston, 1996. The
scorecard emphasizes the use of a balanced set of
measures across the four categories of financial
performance, customer knowledge, internal business
processes, and learning and growth.

Bibliography
Page 61         GAO/GGD-00-10 Managing for Results
Aristigueta, Maria P. "Fine-Tuning Performance
Measurement as a Management Strategy: Insights
from the States." 1998 National Public Sector
Productivity On-Line Conference, Nov. 1998.

Atkinson, Anthony A., Waterhouse, John H., and
Wells, Robert B. "A Stakeholder Approach to
Strategic Performance Measurement," Sloan
Management Review. Spring 1997, pp. 25-37.

Australian National Audit Office. Better Practice
Principles for Performance Information. Internet
address: http://www.anao.gov.au.

Berman, Evan M. "Measuring Productivity,"
Productivity in Public and Nonprofit
Organizations. Sage, Thousand Oaks, CA, pp. 51-76,
1998.

Berman, Evan M. and West, Jonathan P.
"Productivity Enhancement Efforts in Public and
Nonprofit Organizations," Public Productivity &
Management Review. Vol. 22, No. 2, Dec. 1998, pp.
207-219.

Brizius, Jack A. and Campbell, Michael D. Getting
Results. Council of Governors' Policy Advisors,
Washington, D.C., 1991.

Buckman, James F. and Holter, Suzanne W.
"Alignment: The Path to Performance Improvement,"
Journal of Strategic Performance Measurement. Vol.
2, No. 6, Dec. 1998, pp. 40-46.

Campbell, Wilson and Fountain, Jay. "Service
Efforts and Accomplishments-Lessons Learned." 1998
National Public Sector Productivity On-Line
Conference, Nov. 1998.

Center for Accountability and Performance.
Performance Measurement: Concepts and Techniques,
American Society for Public Administration.
Washington, D.C., undated.

Crookall, Paul and Ingstrup, Ole. "The Citizen as
Customer: An International Survey of Well-
Performing Public Service Organizations." 1998
National Public Sector Productivity On-Line
Conference, Nov. 1998.

Ernst & Young Center for Business Innovation,
Perspectives on Business Innovation. Issue 2,
Measuring Business Performance, Ernst & Young,
Center for Business Innovation, Cambridge, MA,
1998.

Fischer, Richard J. "An Overview of Performance
Measurement," Benchmarks of Performance, Special
Section, reprint of Sept. 1994 issue of Public
Management, pp. S2-S8.

Frigo, Mark L. and Krumwiede, Kip R. "Balanced
Scorecards: A Rising Trend in Strategic
Performance Measurement," Journal of Strategic
Performance Measurement. Vol. 3, No. 1, Feb.-Mar.
1999, pp. 42-48.

Glover, Mark. A Practical Guide for Measuring
Program Efficiency and Effectiveness in Local
Government, The Innovation Groups. Tampa, FL,
1994.

Grifel, Stuart S. "Organizational Culture: Its
Importance in Performance Measurement," Benchmarks
of Performance. Special Section, reprint of Sept.
1994 issue of Public Management, pp. S19-S20.

Hacker, Maria E. and Brotherton, Paul A.
"Designing and Installing Effective Performance
Measurement Systems." Internet address:
http:/www.balancedscorecard.com.

Hatry, Harry P. "Where the Rubber Meets the Road:
Performance Measurement for State and Local Public
Agencies," Using Performance Measurement to
Improve Public and Nonprofit Programs. Kathryn E.
Newcomer, Editor, New Directions for Evaluation,
Number 75, Fall 1997, Jossey-Bass Publishers, San
Francisco, pp. 31-44.

Hatry, Harry P. Draft manuscript, Performance
Measurement for Results-Focused Agencies, Aug.
1998.

Hatry, Harry, Gerhart, Craig, and Marshall,
Martha. "Eleven Ways to Make Performance
Measurement More Useful to Public Managers,"
Benchmarks of Performance. Special Section,
reprint of Sept. 1994 issue of Public Management,
pp. S15-S18.

Julnes, Patria de Lancer. "Lessons Learned About
Performance Measurement: Rebuilding Procrustes'
Bed." 1998 National Public Sector Productivity On-
Line Conference, Nov. 1998.

Kaplan, Robert S. and Norton, David P. The
Balanced Scorecard. Harvard Business School Press,
Boston, 1996.

Madigan, James M. "Measures Matrix Chart: A
Holistic Approach to Understanding Operations,"
Quality Management Journal. Oct. 1993, pp. 77-86.

National Academy of Public Administration. Center
for Improving Government Performance, Helpful
Practices in Implementing Government Performance.
Internet address:
http://www.napawash.org/NAPA/NewNAPAHome.

National Academy of Public Administration. Center
for Improving Government Performance, Improving
Performance Across Programs: Thinking About the
Issue-Taking the First Steps. National Academy of
Public Administration,Washington, D.C., undated.

National Academy of Public Administration. Center
for Improving Government Performance, Planning for
Results. National Academy of Public
Administration, Washington, D.C., undated.

Newcomer, Kathryn E. "Using Performance
Measurement to Improve Programs," Using
Performance Measurement to Improve Public and
Nonprofit Programs. Kathryn E. Newcomer, Editor,
New Directions for Evaluation, Number 75, Fall
1997, Jossey-Bass Publishers, San Francisco, pp. 5-
14.

Organisation for Economic Co-Operation and
Development. Performance Measurement in
Government: Issues and Illustrations. Organisation
for Economic Co-Operation and Development, Paris,
1994.

Organisation for Economic Co-Operation and
Development. Performance Management in Government:
Performance Measurement and Results-Oriented
Management. Organisation for Economic Co-Operation
and Development, Paris, 1994.

Organisation for Economic Co-Operation and
Development. In Search of Results: Performance
Management Practices. Organisation for Economic Co-
Operation and Development, Paris, 1997.

Osborne, David and Gaebler, Ted. "The Art of
Performance Measurement," Accountability for
Performance. David Ammons, Editor, 1995, ICMA,
Washington, D.C., pp. 33-43.

Patton, Michael Q. Utilization-Focused Evaluation.
3rd Edition, Sage Publications, Thousand Oaks, CA,
1997.

Pautke, Robert W. and Redman, Thomas C. "Creating
and Aligning High-Quality Data Resources: What
Senior Leaders Need to Know," Journal of Strategic
Performance Measurement. Vol. 2, No. 6, Dec. 1998,
pp. 33-38.

Plantz, Margaret C., Greenway, Martha T., and
Hendricks, Michael. "Outcome Measurement: Showing
Results in the Nonprofit Sector," Using
Performance Measurement to Improve Public and
Nonprofit Programs. Kathryn E. Newcomer, Editor,
New Directions for Evaluation, Number 75, Fall
1997, Jossey-Bass Publishers, San Francisco, pp.
15-30.

Poister, Theodore H. "Productivity Monitoring:
Systems, Indicators, and Analysis," Accountability
for Performance. David Ammons, Editor, 1995, ICMA,
Washington, D.C., pp. 98-120.

Poister, Theodore H. and Strieb, Gregory D.
"Strategic Management in the Public Sector,"
Public Productivity and Management Review. Vol.
22, No. 3, Mar. 1999, pp. 308-325.

Rossi, Peter H. and Freeman, Howard E. Evaluation:
A Systematic Approach. 5th Edition, Sage
Publications, Newbury Park, CA, 1993.

Scheirer, Mary Ann. "Designing and Using Process
Evaluation," Handbook of Practical Program
Evaluation. Joseph S. Wholey, Harry P. Hatry, and
Kathryn E. Newcomer, Editors, 1994, Jossey-Bass
Publishers, San Francisco, pp. 40-68.

Schneiderman, Arthur M. "Why Balanced Scorecards
Fail," Journal of Strategic Performance
Measurement. Special Edition, Jan. 1999, pp. 6-11.

Sparrow, Malcolm K. Imposing Duties. Praeger
Publishers, Westport, CT, 1994.

State of Arizona, 1998 Strategic Planning and
Performance Measurement Handbook, 1998. Internet
address:
http://www.state.az.us/ospb/StatPlan.html.

Stenzel, Catherine and Stenzel, Joe. "From the
Editors," Journal of Strategic Performance
Measurement. Special Edition, Jan. 1999, pp. 3-5.

Swiss, James E. "Performance Monitoring Systems,"
Accountability for Performance. David Ammons,
Editor, 1995, ICMA, Washington, D.C., pp. 67-97.

Texas State Auditor's Office, Legislative Budget
Board, Governor's Office of Budget and Planning.
Guide to Performance Measurement for State
Agencies, Universities, and Health-Related
Institutions. Austin, TX, 1995.

Truitt, Bruce. "Measuring the Unmeasurable." Paper
presented at the Managing for Results conference,
Austin, TX, Nov. 1-3, 1995.

Van Wart, Montgomery and Berman, Evan.
"Contemporary Public Sector Productivity Values,"
Public Productivity and Management Review. Vol.
22, No. 3, Mar. 1999, pp. 326-347.

Waller, Carl. "Emerging Issues in Performance
Management," Journal of Strategic Performance
Measurement. Special Edition, Jan. 1999, pp. 37-
40.

Walters, Jonathan. Measuring Up. Governing Books,
Washington, D.C., 1998.

Weiss, Carol H. Evaluation Research. Prentice-
Hall, Englewood Cliffs, NJ, 1972.

Wholey, Joseph S. "Assessing the Feasibility and
Likely Usefulness of Evaluation," Handbook of
Practical Program Evaluation. Joseph S. Wholey,
Harry P. Hatry, and Kathryn E. Newcomer, Editors,
1994, Jossey-Bass Publishers, San Francisco, pp.
15-39.

Wholey, Joseph S. "Trends in Performance
Measurement: Challenges for Evaluators,"
Evaluation for the 21st Century. Eleanor Chelimsky
and William R. Shadish, Editors, 1997, Sage:
Thousand Oaks, CA, pp. 124-133.

Wholey, Joseph S. "Performance-Based Management:
Responding to the Challenges," Public Productivity
and Management Review. Vol. 22, No. 3, Mar. 1999,
pp. 288-307.

Related GAO Products
Page 63         GAO/GGD-00-10 Managing for Results
Performance Plans: Selected Approaches for
Verification and Validation of Agency Performance
Information (GAO/GGD-99-139, July 30, 1999).

Managing for Results: Opportunities for Continued
Improvements in Agencies' Performance Plans
(GAO/GGD/AIMD-99-215, July 30, 1999).

Agency Performance Plans: Examples of Practices
That Can Improve Usefulness to Decisionmakers
(GAO/GGD/AIMD-99-69, Feb. 26, 1999).

Managing for Results: Measuring Program Results
That Are Under Limited Federal Control (GAO/GGD-99-
16, Dec. 11, 1998).

Managing for Results: An Agenda to Improve the
Usefulness of Agencies' Annual Performance Plans
(GAO/AIMD/GGD-98-228, Sept. 8, 1998).

The Results Act: Assessment of the Governmentwide
Performance Plan for Fiscal Year 1999
(GAO/AIMD/GGD-98-159, Sept. 8, 1998).

Performance Management: Aligning Employee
Performance With Agency Goals at Six Results Act
Pilots (GAO/GGD-98-162, Sept. 4, 1998).

Program Evaluation: Agencies Challenged by New
Demand for Information on Program Results (GAO/GGD-
98-53, Apr. 24, 1998).

The Results Act: An Evaluator's Guide to Assessing
Agency Annual Performance Plans, Version 1
(GAO/GGD-10.1.20, Apr. 1998).

Executive Guide: Measuring Performance and
Demonstrating Results of Information Technology
Investments (GAO/AIMD-98-89, Mar. 1998).

Agencies' Annual Performance Plans Under the
Results Act: An Assessment Guide to Facilitate
Congressional Decisionmaking, Version 1
(GAO/GGD/AIMD-10.1.18, Feb. 1998).

Managing for Results: Critical Issues for
Improving Federal Agencies' Strategic Plans
(GAO/GGD-97-180, Sept. 16, 1997).

Managing for Results: Using the Results Act to
Address Mission Fragmentation and Program Overlap
(GAO/AIMD-97-146, Aug. 29, 1997).

Managing for Results: Regulatory Agencies
Identified Significant Barriers to Focusing on
Results (GAO/GGD-97-83, June 24, 1997).

The Government Performance and Results Act: 1997
Governmentwide Implementation Will Be Uneven
(GAO/GGD-97-109, June 2, 1997).

Managing for Results: Analytic Challenges in
Measuring Performance (GAO/HEHS/GGD-97-138, May
30, 1997).

Performance Budgeting: Past Initiatives Offer
Insights for GPRA Implementation (GAO/AIMD-97-46,
Mar. 27, 1997).

Measuring Performance: Strengths and Limitations
of Research Indicators (GAO/RCED-97-91, Mar. 21,
1997).

Executive Guide: Effectively Implementing the
Government Performance and Results Act (GAO/GGD-96-
118, June 1996).

*** End of Document ***