Managing for Results: Regulatory Agencies Identified Significant Barriers
to Focusing on Results (Letter Report, 06/24/97, GAO/GGD-97-83).
GAO reviewed five regulatory agencies' efforts to implement the
provisions of the Government Performance and Results Act (GPRA) of 1993,
focusing on the: (1) five agencies' strategic goals and related program
performance measures as well as selected employee performance standards
as of January 1, 1997; (2) extent to which agency officials and GAO
believed that these goals, program performance measures, and employee
performance standards focused on results; and (3) aids and barriers that
officials in the agencies said they faced in attempting to focus on
results.
GAO noted that: (1) as would be expected in the early stages of
implementing a new and difficult initiative, GAO observed that some of
the five regulatory agencies were further along in the development of
strategic goals, program performance measures, and employee performance
standards than others; (2) the agencies also varied in the degree to
which their goals, associated sets of program performance measures, and
employee performance standards that were in use as of January 1, 1997,
focused on results as judged by both agency officials and GAO; (3) in
this regard, it is important to note that although GPRA was intended to
encourage agencies to focus their goals and measures on results, the Act
does not require that all of an agency's goals or performance measures
be explicitly results-oriented; (4) similarly, the President's directive
to orient frontline regulators' performance standards toward results
does not appear to require that every standard be results-oriented; (5)
there were differences in the extent to which agency officials
characterized their goals, program performance measures, and employee
performance standards as "results-oriented" and the extent to which GAO
did; (6) in general, agencies frequently concluded that their goals,
measures, and standards were more results-oriented than GAO did; (7) at
the broader and more conceptual level of strategic goals, there were
relatively few differences between agency officials' assessments of the
extent of results-orientation and GAO's; (8) in enacting GPRA, Congress
realized that the transition to results-oriented management would not be
easy; (9) for that reason, the Act provided for a phased approach to
implementation, during which time agencies have been able to identify
the obstacles that need to be overcome and some factors they found
helpful; (10) the factor that agency officials most commonly said aided
the establishment of a results-orientation in the agencies was the
enactment of GPRA; (11) although agency officials identified some aids
to focusing their agencies on results, they cited numerous barriers to
their efforts to establish results oriented goals and measures; (12)
these barriers included significant problems in identifying and
collecting the data they needed to demonstrate their agencies' results;
(13) agencies also cited as a barrier the fact that the diverse and
complex factors affect agencies' results, and their lack of control or *
--------------------------- Indexing Terms -----------------------------
REPORTNUM: GGD-97-83
TITLE: Managing for Results: Regulatory Agencies Identified
Significant Barriers to Focusing on Results
DATE: 06/24/97
SUBJECT: Agency missions
Strategic planning
Federal agency reorganization
Public administration
Productivity
Regulatory agencies
Data collection
Work measurement standards
Personnel management
**************************************************************************
* This file contains an ASCII representation of the text of a GAO *
* report. Delineations within the text indicating chapter titles, *
* headings, and bullets are preserved. Major divisions and subdivisions *
* of the text, such as Chapters, Sections, and Appendixes, are *
* identified by double and single lines. The numbers on the right end *
* of these lines indicate the position of each of the subsections in the *
* document outline. These numbers do NOT correspond with the page *
* numbers of the printed product. *
* *
* No attempt has been made to display graphic images, although figure *
* captions are reproduced. Tables are included, but may not resemble *
* those in the printed version. *
* *
* A printed copy of this report may be obtained from the GAO Document *
* Distribution Facility by calling (202) 512-6000, by faxing your *
* request to (301) 258-4066, or by writing to P.O. Box 6015, *
* Gaithersburg, MD 20884-6015. We are unable to accept electronic orders *
* for printed documents at this time. *
**************************************************************************
Cover
================================================================ COVER
Report to Committee on Governmental Affairs, U.S. Senate, and
Committee on Government Reform and Oversight, U.S. House of
Representatives
June 1997
MANAGING FOR RESULTS - REGULATORY
AGENCIES IDENTIFIED SIGNIFICANT
BARRIERS TO FOCUSING ON RESULTS
GAO/GGD-97-83
Barriers to Managing for Results
(410043)
Abbreviations
=============================================================== ABBREV
CID - Criminal Investigation Division
CSHO - Compliance Safety and Health Officer
CSR - customer service representative
ELA - establishment licensing agreement
EPA - Environmental Protection Agency
FAA - Federal Aviation Administration
FDA - Food and Drug Administration
FTD - federal tax deposits
FTE - full time equivalent
GAO - General Accounting Office
GPRA - Government Performance and Results Act
HACCP - Hazard Assessment Critical Control Points
IDRS - Integrated Data Retrieval System
IRC - Internal Revenue Code
IRS - Internal Revenue Service
MQSA - Mammography Quality Standards Act
NDA - new drug approval
OMB - Office of Management and Budget
OSHA - Occupational Safety and Health Administration
PDUFA - Prescription Drug User Fee Act
PEP - performance evaluation plan
PLA - product licensing agreement
Letter
=============================================================== LETTER
B-275122
June 24, 1997
The Honorable Fred D. Thompson
Chairman
The Honorable John Glenn
Ranking Minority Member
Committee on Governmental Affairs
United States Senate
The Honorable Dan Burton
Chairman
The Honorable Henry A. Waxman
Ranking Minority Member
Committee on Government Reform and Oversight
House of Representatives
The Government Performance and Results Act of 1993 was enacted to
improve the efficiency and effectiveness of federal programs by
establishing a system to set goals for program performance and to
measure results. In recognition of the potential difficulties
inherent in establishing results-oriented goals and performance
measures, the Act, which is referred to as "GPRA" or "the Results
Act," provided for a phased implementation period. Beginning in
fiscal year 1994 and extending over several years, agencies are to
develop strategic goals tied to agency missions, identify performance
measures associated with those goals, and by fiscal year 1999
implement annual results-oriented performance reports linked to
budget requests. In addition, in a March 4, 1995, memorandum, the
President specifically directed the heads of regulatory agencies to
change the way they measure both their agencies' performance and the
performance of their regulatory staff so as to focus on results.
The Results Act required us to report to Congress on the prospects
for the Act's implementation governmentwide, and our recent
report--The Government Performance and Results Act: 1997
Governmentwide Implementation Will Be Uneven (GAO/GGD-97-109, June 2,
1997)--was developed in response to that requirement. In support of
that broader assessment, we initiated an examination of the early
progress toward results-oriented goal setting and performance
measurement in selected federal regulatory agencies as they prepared
to meet the first statutorily required reporting date of September
30, 1997. We considered "regulatory agencies" to include any agency
(as defined in 5 U.S.C. 551[1]) that issues rules (as defined in 5
U.S.C. 551[4]). We examined regulatory agencies in part because of
the administration's initiative to better focus the performance of
regulatory agencies on achieving results through more
results-oriented identification of organizational objectives and
individual performance standards. The administration's initiative
differs from the Results Act by adding the requirement that agencies
change the way they measure the performance of frontline regulators
to focus on results. In addition, because the President required
regulatory agencies to start focusing on results in March 1995, these
agencies were expected to adopt a results orientation sooner than the
rest of the federal government.
The regulatory agencies we focused on were the Occupational Safety
and Health Administration (OSHA), the Federal Aviation Administration
(FAA), the Food and Drug Administration (FDA), the Internal Revenue
Service (IRS), and the Environmental Protection Agency (EPA).
We developed this report to describe the status of the five agencies'
attempts as of January 1, 1997, to focus on results and to ascertain
what had assisted or impeded these attempts. Our specific objectives
were to describe (1) the five agencies' strategic goals and related
program performance measures as well as selected employee performance
standards as of January 1, 1997; (2) the extent to which agency
officials and we believed that these goals, program performance
measures, and employee performance standards focused on results; and
(3) the aids and barriers that officials in the agencies said they
faced in attempting to focus on results.
Focusing on results represents a new approach for both federal
agencies and Congress. Both agency management and congressional
oversight have traditionally focused on inputs (e.g., the number of
dollars or staff allotted to an agency); compliance with legislative
or agency-developed rules; and the accomplishment of outputs (e.g.,
the number of inspections completed). Although a results orientation
does not ignore these other factors, it focuses attention on what
outcomes agencies--including regulatory agencies--are trying to
accomplish.
Because a focus on results is new for federal agencies, and criteria
for definitively characterizing goals, measures, and standards as
"results oriented" were not yet developed, we adopted an approach
that was intended to stimulate discussion of the characteristics of
results-oriented goals, measures, and standards. We hypothesized
that, if there were differences between our characterizations and
those of the agencies, these differences and the explanations for
them might be useful to the agencies by illustrating how, for
example, a goal that appears to be clearly results oriented and
linked to an agency's mission in the agency's view may not appear the
same way to someone from outside the agency. By comparing our
characterizations with those of agency officials, we intended to
foster discussion and contribute to agencies' efforts to develop
results-oriented goals, measures, and standards.
RESULTS IN BRIEF
------------------------------------------------------------ Letter :1
As would be expected in the early stages of implementing new and
difficult initiatives, and consistent with our broader assessment of
the prospects for implementing the Results Act, we observed that some
of the five regulatory agencies were further along in the development
of strategic goals, program performance measures, and employee
performance standards than others. The agencies also varied in the
degree to which their goals, associated sets of program performance
measures, and employee performance standards that were in use as of
January 1, 1997, focused on results as judged by both agency
officials and by us. In this regard, it is important to note that
although the Results Act was intended to encourage agencies to focus
their goals and measures on results, the Act does not require that
all of an agency's goals or performance measures be explicitly
results oriented. Moreover, agencies were not required by the Act to
actually have strategic goals and related performance measures at
that point in time.\1 Similarly, the President's directive to orient
frontline regulators' performance standards toward results does not
explicitly require that every standard be results oriented.
There were differences in the extent to which agency officials
characterized their goals, program performance measures, and employee
performance standards as "results oriented," and the extent to which
we did. In general, agencies frequently concluded that their goals,
measures, and standards were more results oriented than we did. At
the broader and more conceptual level of strategic goals, there were
relatively few differences between agency officials' assessments of
the extent of results orientation and ours. As we and they
considered the more specific program performance measures,
differences between our views and the agencies were greater, although
both we and agency officials characterized the agencies' performance
measures as less results oriented than the strategic goals. The
differences between the agencies' assessments of the extent of
results orientation and ours were greatest in considering the more
detailed performance standards for regulatory personnel. (See apps.
I through V for a detailed description of the extent to which we and
agency officials characterized the results orientation of agencies'
goals, measures, and standards.) Since January 1997, the agencies
have continued their efforts to develop strategic goals and
performance measures to meet the requirements of the Results Act. In
commenting on a draft of this report, Office of Management and Budget
(OMB) officials noted that most of the agencies' goals had changed
since our review and some of the revised goals and measures were more
results oriented. As we stated previously, it was our intent for
this work to contribute to such an outcome.
In enacting the Results Act, Congress realized that the transition to
results-oriented management would not be easy. For that reason, the
Act provided for a phased approach to implementation, during which
time agencies have been able to identify obstacles that need to be
overcome and some factors they found helpful. The factor that agency
officials most commonly said aided the establishment of a results
orientation in their agencies was the enactment of the Results Act.
Specifically, they said that GPRA-related requirements for
outcome-oriented strategic goals and performance measures by
September 1997 forced them to focus their attention on results more
than they had prior to the passage of the Act. Other factors that
agency officials said aided their focus on results included
issue-specific legislation that cited results-oriented goals or
measures, feedback from stakeholders including Congress and other
interested parties, and top management leadership focused on
achieving results in agency performance.
While agency officials identified some aids to focusing their
agencies on results, they cited numerous barriers to their efforts to
establish results-oriented goals and measures. These barriers
included significant problems in identifying and collecting the data
they needed to demonstrate their agencies' results. For example, IRS
officials said that there are significant difficulties involved in
measuring the fair amount of total taxes due, so it is hard to
determine how much IRS should be collecting. Other data-related
barriers included the difficulty of determining the causes of
illnesses with long latency periods and problems with data collection
efforts, such as under- or inconsistent reporting. Agencies also
cited as a barrier the fact that diverse and complex factors affect
agencies' results (e.g., business cycles, technological innovations,
or the need to deliver federal program initiatives and thus achieve
results through third parties), and their lack of control or
influence over those factors. For example, although FDA regulations
attempt to ensure that food is safe, FDA cannot control whether
consumers ultimately ensure that food is stored and prepared
properly. Finally, agency officials observed that the long time
period needed to see results in some areas of federal regulation was
a barrier to identifying and managing toward those results in the
framework of annual performance plans and budgets. The impact of
some agencies' regulatory actions, such as limiting exposure to a
hazardous chemical, may not be evident for years. These barriers
suggest that the implementation of the Results Act in a regulatory
environment may prove more difficult in some cases than in others.
Although we did not determine the validity of their claims or the
extent to which the relative lack of results orientation in the
performance measures was directly traceable to these barriers, the
issues the agency officials raised should not be summarily
discounted. Yet, while solutions may not be readily apparent, the
potential long-term benefits merit efforts to develop better
measures. For example, although long-term results may be difficult
to use in an annual process, interim measures indicating the degree
of progress toward the achievement of those results can sometimes be
used to determine whether current policies are having the desired
effects. Likewise, output measures such as the number of inspections
conducted can be used when studies exist to demonstrate their
relationship to the results that agencies are attempting to
accomplish. Finally, agencies may be able to develop analytic
techniques to enable them to control for extraneous factors and
isolate the impact the agencies are having on such outcomes as injury
rates or clean water.\2
For GPRA to work in a regulatory environment, and for regulatory
agencies to meet their September 1997 deadlines, it is important that
those agencies begin to develop solutions to these measurement
problems. Some agencies have already begun to do so. For example,
IRS has been working for some time on improving its estimates of
taxes owed, which might ultimately be used to determine the
effectiveness of the agency's compliance programs. Agencies will
also need to learn from the experiences of other agencies that have
previously confronted and addressed similar issues. Our Executive
Guide, for example, identified key steps that agencies need to take
to implement the Results Act and provided case illustrations where
federal agencies had put these practices to use.\3
However, while we conceptually agree with the Act's emphasis on
managing for results, we also recognize that the development and
practical application of results-oriented program performance
measures in regulatory agencies will not always be done quickly or
easily. For example, in order for an agency to isolate and measure
its unique contribution to a result that is affected by a variety of
factors outside of the agency, the agency must control for the
effects of all of those nonagency factors. It is yet unclear whether
that degree of analytical precision will always be possible.
--------------------
\1 Under GPRA, agencies are to submit strategic plans identifying
strategic goals tied to agency missions to Congress by September 30,
1997, and annual performance plans related to the strategic plans are
to be submitted to Congress after the President's budget is submitted
in February 1998.
\2 Although Congress will probably require agencies to provide
information on their unique contributions to outcomes as part of the
oversight process, GPRA does not technically require agencies'
performance measures to provide this kind of information.
\3 Executive Guide: Effectively Implementing the Government
Performance and Results Act (GAO/GGD-96-118, June 1996).
BACKGROUND
------------------------------------------------------------ Letter :2
Federal regulations affect the lives of virtually every American, and
have requirements that are designed to ensure, among other things,
that workplaces, air travel, foods, and drugs are safe; that the
nation's air, water, and land are not polluted; and that the
appropriate amount of taxes are collected. They are the means by
which statutory public policy goals are put into effect to achieve
desired economic and social benefits. Regulations also have billions
of dollars of impact on businesses, state and local governments, and
the economy as a whole. Therefore, it is important that
decisionmakers and taxpayers understand what specific results federal
regulatory agencies are attempting to achieve and how those agencies
are measuring their progress toward achieving those results.
Two efforts--one by Congress and the other by the Clinton
administration--have required regulatory agencies to become more
results oriented. Congress passed the Results Act in July 1993 with
the intent of improving the efficiency and effectiveness of federal
programs. Under the Act, federal agencies are to develop a strategic
plan by September 1997, including general goals for the major
functions and operations of the agency and a description of how the
goals are to be achieved. Agencies are to submit annual performance
plans to Congress after the President's budget submission in early
1998. Beginning in fiscal year 1999, the Act requires that each
agency report annually to the President and to Congress on its
performance and its progress toward meeting its goals. Agencies'
plans are to be used to support the President's budget, and agencies'
annual performance reports are intended to provide important
information to agency managers, policymakers, and the public on what
each agency accomplished with the resources it was given.
Agencies' strategic plans are to cover a period of at least 5 years
and, as shown in figure 1, should contain the agency's mission
statement, a set of strategic goals (including outcome-related
goals), a description of how the agency intends to achieve these
goals, and a description of how the agency's performance goals will
be related to its strategic goals. Strategic plans are intended to
be the starting point for each agency's performance measurement
efforts. The annual performance plans are to use program performance
measurement to reinforce the connection between the long-term
strategic goals outlined in their strategic plans and the day-to-day
activities of their managers and staff. The annual plans are to
include performance goals for an agency's program activities as
listed in the budget, a summary of the necessary resources to achieve
these goals, and the performance indicators that will be used to
measure performance. Although these requirements are intended to
encourage agencies to move toward results-oriented management, the
Act recognizes that not all goals and measures will necessarily be
results oriented.
Figure 1: Linking Results Act
Requirements to Day-to-Day
Activities
(See figure in printed
edition.)
Source: GPRA.
Getting regulatory agencies to focus on achieving desired results was
also one aspect of the administration's regulatory reinvention
initiative. In Executive Order 12866, which was issued on September
30, 1993, the President set forth his philosophy for reforming the
regulatory process. One of the "principles of regulation" in that
order is that each regulatory agency should, to the extent feasible,
"specify performance objectives, rather than the behavior or manner
of compliance that regulated entities must adopt." In a March 4,
1995, memorandum, the President followed up on this executive order
by directing the heads of all federal regulatory agencies to, among
other things, "change the way you measure the performance of both
your agency and your frontline regulators so as to focus on results,
not process and punishment."\4 Although the President's requirement
that agencies measure their performance in terms of results achieved
is consistent with GPRA's emphasis, measuring employees' performance
in that manner is unique to the administration's regulatory
reinvention initiative. As previously noted, the President's
initiative does not explicitly require that all performance standards
be results oriented.
The five regulatory agencies have each been attempting to develop a
results orientation for some time. EPA launched a National Goals
Project in 1992 to develop measurable long-range environmental goals
and, more recently, attempted to increase the agency's focus on
results in response to an April 1995 National Academy of Public
Administration report.\5 OSHA and IRS began agencywide GPRA pilot
projects in 1993.\6 Although FAA and FDA started their agencywide
efforts more recently, they had GPRA pilot efforts under way in some
of their units. All five of the agencies said they are currently
consulting with Congress and preparing for agencywide compliance as
part of the Act's governmentwide implementation beginning in
September 1997. In addition, these five agencies are subject to the
President's March 4, 1995, memorandum directing them to focus on
results.
These agencies also have implemented various reforms in response to
the administration's regulatory reinvention initiatives. In June
1995, each agency reported on its actions regarding four tasks the
President assigned to regulatory agencies in March 1995: eliminating
or improving regulations, rewarding results, creating grassroots
partnerships, and negotiating rules. The "rewarding results"
sections of the reports focused on the status of agency program
performance measures and steps taken to ensure that frontline
regulators were evaluated on results rather than process or
punishment measures. For example, EPA noted in its report that it
had shifted its employee performance standards from specific
process-driven numeric goals to definitions of performance in terms
of the quality of the output.
--------------------
\4 The President's memorandum did not define "frontline regulators."
We defined "frontline regulators" as regulatory staff who work
directly with the regulated community by enforcing regulations.
\5 Setting Priorities, Getting Results: A New Direction for the
Environmental Protection Agency, a National Academy of Public
Administration Report to Congress, April 1995.
\6 GPRA Performance Reports (GAO/GGD-96-66R, Feb. 14, 1996)
summarized our initial observations on selected agencies' first
attempts to report on their performance in the pilot phase of GPRA.
SCOPE AND METHODOLOGY
------------------------------------------------------------ Letter :3
We selected the five regulatory agencies in this review--OSHA, FAA,
FDA, IRS, and EPA--on the basis of their size (i.e., budget and
personnel); the number of significant regulations they planned to
finalize or complete from October 1, 1995, to October 1, 1996; and
the opinions and recommendations of experts in the field of federal
regulation.\7 The agencies selected were those that appeared most
frequently across all of these factors, although each agency selected
may not have been at the top of any particular factor. For example,
OSHA is a relatively small agency that typically issues few rules
each year, but it was listed by most of the experts as a major
regulatory agency.
To describe the agencies' strategic goals and program performance
measures, we first obtained copies of the goals and measures the
agencies had developed as of January 1, 1997. To describe the
employee performance standards for selected regulator positions at
these agencies, we first asked officials in each agency to identify
frontline regulator positions that they felt would fairly represent
the actions of their agencies' regulators. Using our definition of
"frontline regulator"--regulatory staff who work directly with the
regulated community by enforcing regulations--the agencies designated
which positions they considered "frontline regulator" positions. We
then obtained copies of the performance standards being used as of
January 1, 1997, for the selected positions. Because the Results Act
does not cover employee performance standards, we judged each set of
performance standards according to the expectation set by the
President in his March 4, 1995, memorandum that these standards
should "focus on results, not process and punishment."
To describe the extent to which we and agency officials believed that
the goals, measures, and performance standards were results oriented,
we developed and provided to each agency guidelines for agency
officials to use if they chose in characterizing the agencies' goals,
program performance measures, and employee performance standards as
results oriented to a great extent, to some extent, or to little or
no extent. We used a similar framework to characterize the agencies'
goals, performance measures, and employee performance standards.
(See app. VI for a more extensive discussion of the methodology we
used to determine the extent to which their goals, measures, and
standards were results oriented.)
To respond to our third objective, we asked officials in each agency
to describe the aids and barriers that their agencies faced in
attempting to focus on results. We also reviewed related documents
that described how agencies developed results-oriented goals and
program performance measures and any problems or successes agencies
had encountered. We did not attempt to determine the validity of the
factors that agency officials said aided or hindered their efforts to
becoming results oriented.
We did our work between April 1996 and March 1997 in Washington,
D.C., in accordance with generally accepted government auditing
standards. We sent a draft of this report to the heads of each of
the five agencies included in this review and to the Director of OMB.
Officials from each of these agencies provided oral comments, which
are summarized at the end of this report and discussed in more detail
in the appendix relating to that agency. In addition, two of the
agencies (OSHA and IRS) commented in writing. These comments are
reprinted in their entirety in appendix VII.
--------------------
\7 We contacted experts in the field of federal regulation from the
following organizations and asked them to identify which agencies
they considered to be the major federal regulatory agencies: the
American Enterprise Institute, the U.S. Chamber of Commerce, the
National Federation of Independent Businesses, the Center for the
Study of American Business, Citizens for a Sound Economy, the
National Performance Review, and the Small Business Administration.
VIEWS ON THE RESULTS
ORIENTATION OF STRATEGIC GOALS,
PERFORMANCE MEASURES, AND
PERFORMANCE STANDARDS
------------------------------------------------------------ Letter :4
There were differences in the extent to which agency officials
characterized their goals, program performance measures, and employee
performance standards as "results oriented," and the extent to which
we did. At the broader and more conceptual level of strategic goals,
there were relatively few differences between agency officials'
assessments of the extent of results orientation and ours. However,
as we and they considered the more specific program performance
measures, greater differences existed between our views and those of
the agencies. The differences between the agencies' assessments of
the extent of results orientation and ours were greatest in
considering the more detailed performance standards for selected
frontline regulator positions. (App. VI contains a detailed
description of our scope and the methods that we used. Apps. I
through V contain detailed information regarding each of the five
regulatory agencies' strategic goals, program performance measures,
and employee performance standards.)
We and agency officials most frequently agreed in our
characterizations of strategic goals. We both said that most of the
agencies' goals were at least somewhat results oriented. Agency
officials considered almost 70 percent of the strategic goals to be
greatly results oriented, and considered the remaining goals to be
results oriented to some extent. We characterized about one-half of
the agencies' goals as greatly results oriented and about one-third
as somewhat results oriented. We considered the remaining goals to
be results oriented to little or no extent. An example of one of
IRS' strategic goals was as follows:
"Increase compliance: Our objective is to encourage and assist
taxpayers to voluntarily file timely and accurate returns and to
pay on time, and, if taxpayers do not comply, to take
appropriate compliance actions. We do this to ensure we collect
the proper amount of tax at the least cost."
Both we and IRS characterized this goal as results oriented to a
great extent because it was directly related to IRS' mission of
collecting tax revenue, provided a clear definition of the results
expected, and permitted assessment of the agency's programs.
As we and agency officials considered the more specific program
performance measures, differences between our views and the agencies'
were greater, although both we and agency officials characterized the
agencies' performance measures as less results oriented than the
strategic goals. However, our characterizations of the performance
measures were not as positive as those of the agencies. Agency
officials said they believed that most of their sets of program
performance measures were somewhat results oriented, and
characterized almost 40 percent of the measures as greatly results
oriented. However, we did not consider any of the sets of program
performance measures to be greatly results oriented. We agreed with
agency officials that most of the sets of measures were somewhat
results oriented, but we viewed over 40 percent of the measures as
results oriented to little or no extent. GPRA requires agencies to
establish performance indicators to be used in measuring or assessing
the relevant outputs, service levels, and outcomes of each program
activity as part of agencies' performance plans.
For example, OSHA's sole performance measure for its goal to
"implement a common sense strategy developed in partnership with
stakeholders, for rulemaking and alternative approaches to emerging
and priority safety and health issues" was "the percentage of
rulemaking activity on the regulatory agenda undertaken as negotiated
rulemaking." OSHA officials considered this measure somewhat results
oriented because they believed that using the negotiated rulemaking
process would improve the regulated entities' acceptance of rules,
which would lead to improved compliance and fewer hazards, thereby
leading to safer workplaces. However, we said this measure was
results oriented to little or no extent because (1) it was not clear
how use of the negotiated rulemaking process would make workplaces
safer; (2) the measure simply tracked the percentage of cases in
which this rulemaking process was used and, therefore, OSHA did not
have a set of measures that provided relevant information to gauge
progress toward the goal/mission; and (3) this measure alone did not
address other aspects of the goal, such as the use of alternatives to
direct regulation.
We and agency officials did not often agree about how results
oriented the agencies' employee performance standards were. Agency
officials characterized over 80 percent of their sets of performance
standards as greatly results oriented, and considered the rest
somewhat results oriented. In contrast, we viewed 40 percent of the
sets of standards as results oriented to little or no extent. We
characterized over one-third of the standards as somewhat results
oriented, and thought that less than one-fourth of the performance
standards were greatly results oriented.
For example, we generally disagreed with EPA officials'
characterization of the agency's performance standards for the
criminal investigator and environmental scientist positions. They
said that all but two of the nine sets of performance standards for
these positions were greatly results oriented and that those two were
somewhat results oriented. However, we characterized eight of these
sets of performance standards as results oriented to little or no
extent because they were not linked with any expected results. For
example, EPA officials said that the "professional attributes"
standards for EPA's criminal investigator position were results
oriented to a great extent because "presenting a positive
well-balanced presence when conducting investigations" resulted in
good internal and external working relationships. EPA officials said
these professional attributes were especially important in risky
situations where the investigator must be tactful in enforcing the
law so that the situation does not get out of hand. However, we
characterized this set of standards as results oriented to little or
no extent because it was unclear how maintaining "a good level of
physical conditioning" or displaying "an appearance and demeanor
which is consistent with the position" were linked to any of EPA's
strategic goals or its overall mission of environmental protection.
AGENCY OFFICIALS IDENTIFIED
SOME AIDS BUT NUMEROUS BARRIERS
TO FOCUSING ON RESULTS
------------------------------------------------------------ Letter :5
Agency officials identified relatively few aids to focusing their
agencies on results, but said they faced significant barriers in
these efforts. Although a few of the officials' comments described
aids to establishing results-oriented goals and employee performance
standards, most of their comments focused on barriers to developing
results-oriented program performance measures.
AIDS TO FOCUSING ON RESULTS
---------------------------------------------------------- Letter :5.1
The factor that agency officials most commonly said aided the
establishment of a results orientation in their agencies was the
enactment of GPRA. Specifically, they said that GPRA-related
requirements for outcome-oriented strategic goals and performance
measures by September 1997 forced their agencies to pay more
attention to results than they had prior to passage of the Act. The
officials also said that congressional interest in GPRA's
implementation was an impetus for a stronger focus on results.
Some agency officials cited other issue-specific legislation that
they said had also encouraged their agencies to increase their
results orientation. For example, agency officials said that the
Aviation Security Act of 1990 and the Prescription Drug User Fee Act
of 1992 contained specific results-oriented goals and performance
measures for FAA and FDA, respectively.
Other aids that some agency officials also mentioned to focusing
agencies on results included: training provided to staff on how to
develop program performance measures; feedback from stakeholders
(including Congress, the public, industry, other federal agencies,
state and local governments, and interest groups); and top agency
management leadership. For example, both OSHA and FDA officials said
that the heads of their agencies strongly emphasized using
results-oriented performance measures to focus on results.
BARRIERS TO FOCUSING ON
RESULTS
---------------------------------------------------------- Letter :5.2
Agency officials cited numerous barriers that their agencies faced in
becoming results oriented, some of which reportedly hindered them
from establishing results-oriented strategic goals or employee
performance standards. For example, some agency officials said that
the conflicting views of diverse stakeholders and inconsistent
statutory requirements were barriers to reaching agreement on
agencies' goals. In addition, officials from IRS said that the
agency is prevented by federal law from using performance standards
that evaluate its staff on the basis of enforcement results.\8
Most of the barriers that the officials mentioned focused on
difficulties they experienced in developing results-oriented program
performance measures. The barriers that they mentioned generally
fell into three broad categories: (1) problems in collecting program
performance data, (2) the diverse and complex factors that affect
agencies' results and the lack of control agencies have over some of
those factors, and (3) the long time frames sometimes needed to see
the results of agencies' actions.
--------------------
\8 The Technical and Miscellaneous Revenue Act of 1988 contains the
Omnibus Taxpayer Bill of Rights, which prohibits IRS from evaluating
employees on the basis of tax enforcement results.
AGENCY OFFICIALS IDENTIFIED
PROBLEMS IN COLLECTING
PROGRAM PERFORMANCE DATA
---------------------------------------------------------- Letter :5.3
Officials from all five agencies said that their agencies found it
difficult to establish program performance measures because of
various problems they experienced in collecting necessary data. Some
of those problems involved identifying the data they needed to
collect in order to demonstrate their agencies' results. For
example, officials at several agencies said that they sometimes were
put in the difficult position of trying to gather data to "prove a
negative," such as the number of aircraft accidents that were avoided
by FAA's efforts to improve airport runways or the number of people
who did not get sick because FDA approved a new drug or kept an
unsafe product off the market. FAA officials said that they may
improve runways after a runway condition was identified as
contributing to an accident. However, because each aircraft accident
results from a unique combination of factors, it is difficult to
determine the results from the improvements made. EPA officials said
that in some cases they did not know how to measure environmental
outcomes. For example, EPA officials said that scientific
uncertainty about the effects of exposures to various levels of
pollution and pesticides made it difficult to establish appropriate
measures for aquatic environments. IRS officials said that there are
significant difficulties in attempting to measure the fair amount of
tax due, so it is hard to determine how much IRS should be
collecting.\9 IRS officials also said it was difficult to determine
the impact of customer services on tax compliance levels.
Officials from FDA and OSHA pointed out that uncertainties regarding
the causes of illnesses with long latency periods made it more
difficult for their agencies to collect accurate data on mortality
and morbidity. OSHA's Deputy Director of Policy said it was
particularly difficult to determine whether an illness resulted from
conditions in the workplace or from other factors. For example, he
said it is difficult to determine whether a worker developed cancer
after inhaling asbestos fibers at the workplace for 20 years, or from
the cigarettes the worker smoked during that period. FDA officials
said that an ongoing study attempting to monitor and define the
sources of foodborne diseases is expected to take years to
complete.\10
Agency officials noted other problems that made obtaining
results-oriented performance data difficult. For example, OSHA
officials said that it was difficult to collect data and track trends
in the number of accidents and injuries that occurred in the
construction industry because construction site locations frequently
change. Other agency officials said that some outcome-oriented
performance data must be collected through voluntary self-reporting
methods, which are often susceptible to underreporting. For example,
FDA officials said that people who become ill from food poisoning may
not recognize it as such, or even if they do, they may not report it
to a health official. Consequently, data on the number of food
poisoning incidents may severely undercount such occurrences.\11
Similarly, one FAA official said that the data the agency collected
on flight "incidents" were self-reported, and so were likely to be an
understatement of the true number of such incidents.
Agency officials cited still other data-related problems. EPA
officials said that some states were not measuring certain
environmental conditions and, where data were being gathered, the
states sometimes differed in the measures they used. As a result,
they said, EPA lacked nationwide measures for some environmental
conditions. FAA officials said that the small number of aircraft
accidents each year made analysis of the data statistically
unreliable. Therefore, they said that it was difficult to identify
trends in the factors contributing to aviation accidents.
--------------------
\9 IRS has been working to improve estimates of taxes owed by
conducting research projects and building databases to allow
researchers access to current data. Taxpayer Compliance: Analyzing
the Nature of the Income Tax Gap (GAO/T-GGD-97-35, Jan. 9, 1997).
\10 This study, known as the Foodborne Diseases Active Surveillance
Network, or "FoodNet," is a collaborative project among the Centers
for Disease Control and Prevention, seven state health departments,
the Department of Agriculture's Food Safety Inspection Service, and
FDA.
\11 See Food Safety: Information on Foodborne Illnesses
(GAO/RCED-96-96, May 8, 1996), where we found that most cases of
foodborne illness go unreported and that existing data therefore
understate the extent of the problem.
COMPLEXITY AND LACK OF
CONTROL INCREASED THE
DIFFICULTY OF MEASURING
RESULTS
---------------------------------------------------------- Letter :5.4
Officials in four of the five agencies we reviewed said it was often
difficult to sort out the interaction of diverse factors that
affected the results that the agencies were trying to achieve. They
said that this complexity, combined with their agencies' lack of
control over many of these factors, made it difficult to establish
meaningful performance measures that focused solely on their
agencies' contributions toward achieving their goals. For example,
one of EPA's strategic goals was to achieve clean and safe water,
which is affected by numerous factors that are outside of EPA's
control (e.g., natural causes including weather conditions). In
addition, EPA shares governmental responsibility to regulate water
cleanliness and safety with other federal agencies and with state and
local governments. As a result, EPA officials said it is difficult
to establish performance measures that gauge only the agency's
contribution to "clean and safe water."
Similarly, OSHA officials said that business cycles, the development
of new and safer technologies, changes in laws or regulations, and
other factors over which OSHA has no control affect the number of
workplace accidents and injuries occurring each year more than OSHA's
actions. For example, OSHA's Deputy Director of Policy said that
recent declines in accident and injury rates in Maine were a function
of at least three factors: (1) the partial closure of a major
industrial facility, (2) a tightening of the state's worker
compensation laws, and (3) OSHA's programs in that state. He also
said that worker safety data are sometimes influenced by catastrophic
events, such as bombings or aircraft crashes, over which OSHA has no
control. Finally, OSHA officials pointed out that the 2,000 OSHA and
state inspectors can only visit a small fraction of the 6 million
workplaces in the nation. Therefore, they said that they did not
think it would be appropriate to hold OSHA accountable for increases
in nationwide workplace injury or fatality rates.
FAA officials said that the diversity of factors in the aviation
system that contribute to aircraft accidents presented a strong
barrier to establishing results-based performance measures. Although
they and academic researchers have studied aircraft safety issues for
years, no techniques have been developed that can isolate any one
factor or group of factors that can predict an accident. Because a
combination of unique factors generally causes aircraft accidents, an
FAA official said that it is "virtually impossible" to scientifically
measure FAA's effect on aircraft safety. FAA officials also said
that they could not control many of the factors that contribute to
aircraft accidents, such as how alert aircraft pilots feel during a
flight or the weather conditions. Therefore, they said that FAA can
only "influence" aviation safety.
Similarly, FDA officials said that many factors affect whether food
and drugs are safe, some of which are outside of the agency's
control. For example, although FDA enforces regulations at various
stages of the production process to ensure that food is safe and
drugs are safe and effective, FDA officials said that consumers must
ultimately ensure that food is stored properly and drugs are taken
only as prescribed. Although FDA also regulates the provision of
consumer information to ensure the public is aware of such
considerations, FDA officials said that these and other public health
issues are primarily influenced by individual behaviors that FDA
cannot control.
AGENCY OFFICIALS SAID LONG
TIME FRAMES ARE NEEDED TO
JUDGE RESULTS
---------------------------------------------------------- Letter :5.5
Officials from several of the agencies said that it was difficult to
develop annual performance measures and track their agencies' yearly
progress when the impact of some regulatory actions cannot be seen
for years. For example, OSHA's Deputy Director of Policy said that
the latency period between exposure to a hazardous substance, such as
asbestos, and a resulting illness can be 20 years or more.
Therefore, the results of any actions OSHA takes to reduce exposures
to those substances may be equally long in coming. EPA also said
that a significant lag usually exists between the elimination of a
chemical hazard and any corresponding change in illness rates.
Similarly, FAA officials said that it takes several years to
determine the effect that new equipment and facilities have on the
air system's safety. They also told us that FAA's impact on safety
is only evident when examining data over a number of years. The
officials said that an increase in the number of accidents in 1 year
does not necessarily mean that FAA's performance was worse, because
the small number of accidents is affected by a variety of factors
that the agency does not control, such as weather. Likewise, an IRS
official said that focusing on yearly progress may result in solving
immediate problems but may divert attention and reduce resources
available for accomplishing longer term goals. For example, he said
that a focus on current compliance levels and enforcement efforts
could lessen the agency's long-term efforts to increase taxpayer
compliance through education and outreach efforts.
AGENCY OFFICIALS IDENTIFIED
VARIOUS OTHER BARRIERS TO
RESULTS ORIENTATION
---------------------------------------------------------- Letter :5.6
Agency officials cited a variety of other barriers that they said
hampered their agencies' efforts to focus on results. For example,
some agency officials said that other agency priorities, some of
which are also presidential initiatives, were inconsistent with a
results orientation because they focused on processes rather than
results. For example, an EPA official said that the President's call
to reduce reporting burdens on the public appeared to be somewhat
contrary to the emphasis on gathering data to measure progress toward
results. OSHA officials said that their agency measured the number
of times the negotiated rulemaking process was used because the OSHA
Administrator wanted to emphasize the importance of responding to a
presidential initiative to use this process.
In addition, some agency officials said resource constraints were a
barrier to a results orientation in their agencies. For example, an
OSHA official said that they could only do one or two retrospective
studies each year to determine the effects of prior regulatory
initiatives on workplace safety and health. OSHA officials planned
to use the retrospective studies to provide baseline information on
the results of prior regulatory actions. (See app. I for an example
of one of OSHA's retrospective studies.) FDA officials said that
resource constraints made it difficult to divert resources from high
priority risks to establish a statistically valid baseline for
measuring industry compliance.
Also, some agency officials said that their agencies found it
difficult to focus on results when they did not perceive that
Congress shared that orientation. They said some congressional
attention continued to focus on regulatory activities unrelated to
results or that the congressional committees they work with had not
yet shown any interest in GPRA.
AGENCY COMMENTS AND OUR
EVALUATION
------------------------------------------------------------ Letter :6
We obtained comments from each of the five regulatory agencies in
this review as well as from OMB. The agencies generally agreed with
the draft report's overall message that regulatory agencies face
difficulties in focusing on results. Officials from FDA, for
example, said that the report's discussion of the aids and barriers
was fair, essentially complete, and consistent with their
perceptions. However, officials from three of the agencies (OSHA,
FDA, and IRS) commented that the "snapshot" approach we used did not
adequately capture the dynamic nature of the process of establishing
results-oriented goals, measures, and standards or the progress the
agencies had made in improving their results orientation.
We recognized in the draft report that our approach provided an
interim assessment at an early stage of implementing new and
difficult initiatives (i.e., GPRA and regulatory reinvention) and
that the agencies' goals, program performance measures, and some of
their employee performance standards were changing. We did not have
historical information that would have allowed us to determine the
extent to which goals and performance measures that were in effect as
of January 1, 1997, represented advances over past goals and
measures. However, we have expanded the discussion of our
methodology to further clarify our purpose and to describe the
related constraints in our scope.
The following section summarizes the comments that we received from
OMB. The comments from each of the agencies are summarized, and our
response included, at the end of each of the relevant appendixes
(apps. I through V).
OMB'S COMMENTS AND OUR
EVALUATION
---------------------------------------------------------- Letter :6.1
We met with OMB officials, including the Deputy Administrator of the
Office of Information and Regulatory Affairs, on May 5, 1997, to
obtain their comments on the draft report. In general, the OMB
officials thought that our characterizations of the extent of results
orientation of agencies' strategic goals, program performance
measures, and employee performance standards were fair and
reasonable. OMB officials also said that they agreed with the draft
report's overall message that these regulatory agencies face
significant barriers in focusing on results. They said that the
discussion of barriers was important and noted that meeting GPRA
requirements will probably take a number of iterations and
considerable time. The officials said that given the difficulties
agencies face, it was not surprising that there were gaps in
agencies' results orientation.
The officials said that most of the agencies' goals had changed since
our review. For example, they said that all of OSHA's goals have
changed and that OSHA's new goals are more results oriented than
those we examined. They said that OSHA continues to grapple with the
question that OSHA officials identified in our report about how to
measure OSHA's effects on national illness, injury, and fatality
rates. As stated in our draft report, we described the agencies'
goals, measures, and standards as of January 1, 1997, to provide an
early indication of the extent to which these agencies were
implementing new and difficult initiatives. The draft report
recognized that many of the agencies were making changes to their
goals and measures and some of their performance standards at the
time of our review.
OMB officials recommended that our draft report state more clearly
that the expectation that employee performance standards for
frontline regulators focus on results was directed by the President's
regulatory reinvention initiative and not by the Results Act. The
officials pointed out that there are many factors interspersed
between the two end points we reviewed (i.e., between the agencies'
goals and measures and the standards that frontline regulators are
held accountable for as they implement the agencies' regulatory
programs). These factors include statutory provisions, agencies'
cultures, and the level of individual performance. We agree that it
is important to clearly communicate that the administration's
regulatory reinvention initiative, not GPRA, set the expectation for
results-oriented employee performance standards. We placed greater
emphasis on this difference throughout the final report by clearly
stating in each section on performance standards that the
administration's regulatory reinvention initiative set the
expectation for results-oriented performance standards.
---------------------------------------------------------- Letter :6.2
We are sending copies of this report to the heads of OSHA, FAA, FDA,
IRS, and EPA as well as to the Director of OMB, the Chairmen and
Ranking Minority Members of the House and Senate Appropriations and
other interested congressional committees, and others who may have an
interest in this matter. Copies will also be made available to
others upon request.
The major contributors to this report are listed in appendix VIII.
If you have any questions, please contact me on (202) 512-8676.
L. Nye Stevens
Director
Federal Management and Workforce Issues
OSHA'S MISSION, STRATEGIC GOALS,
PERFORMANCE MEASURES, AND
PERFORMANCE STANDARDS FOR SELECTED
FRONTLINE REGULATOR POSITIONS
=========================================================== Appendix I
OSHA's mission is "to assure so far as possible every working man and
woman in the nation safe and healthful working conditions." OSHA
administers portions of the Occupational Safety and Health Act of
1970, and is responsible for promulgating federal occupational safety
and health regulations, enforcing occupational safety and health
standards, and for approving plans for the development and
enforcement of standards by state governments.
We assessed the results orientation of OSHA's strategic goals,
performance measures, and employee performance standards according to
the criteria presented in the scope and methodology appendix of this
report. Because we characterized the extent to which each agency
goal and set of performance measures was results oriented, the
application of these criteria represented a higher standard of
results orientation than required by the Government Performance and
Results Act (referred to as "GPRA" or "the Results Act"). GPRA does
not require that all of an agency's goals or measures be results
oriented. Also, the Results Act does not cover employee performance
standards. We judged each set of performance standards according to
the expectation set by the President in his March 4, 1995, memorandum
that these standards should "focus on results, not process and
punishment." Similar to the way in which GPRA views agencies' goals
and performance measures, we do not believe that each such standard
must be results oriented to achieve the President's objective.
We applied our criteria to the agencies' written strategic goals,
program performance measures, and employee performance standards and
did not consider any supplementary or contextual information. In
some cases, an agency's goals, measures, and standards appeared
results oriented on the surface, but closer examination and strict
application of our criteria resulted in a lower characterization. We
recognize that other information outside of the written statements
and outside of our review may have existed that could have resulted
in our characterizing certain goals, measures, or standards
differently.
Both we and OSHA officials characterized the extent of results
orientation of OSHA's strategic goals, program performance measures,
and employee performance standards for selected frontline regulator
positions. As shown in table I.1, there were differences in how we
and OSHA officials viewed the agency's goals, measures, and
standards. We did not agree with agency officials that most of
OSHA's goals were somewhat results oriented. There was more
disagreement between us and OSHA regarding the extent of results
orientation of OSHA's performance measures, in that OSHA
characterized them all as somewhat results oriented, but we
characterized them all as results oriented to little or no extent.
We agreed with OSHA officials, however, regarding the results
orientation of the performance standards for Compliance Safety and
Health Officers (CSHOs).
Table I.1
Differences Between Our and OSHA
Officials' Characterizations
Little Little
A great Some or no A great Some or no
Total extent extent extent extent extent extent
---------- -------- -------- -------- -------- -------- -------- --------
Strategic 3 0 3 0 0 1 2
goals
Performanc 3 0 3 0 0 0 3
e
measures
Performanc 3 2 1 0 2 1 0
e
standards
--------------------------------------------------------------------------------
Source: Agency and GAO analysis.
VIEWS ON THE RESULTS
ORIENTATION OF OSHA'S STRATEGIC
GOALS
--------------------------------------------------------- Appendix I:1
As of January 1, 1997, OSHA had three strategic goals:\1
1. "Eliminate hazards through offering partnerships or traditional
enforcement;
2. Implement a common sense strategy, developed in partnership with
stakeholders, for rulemaking and alternative approaches to emerging
and priority safety and health issues; and
3. Focus OSHA programs and service delivery systems using internal
and external partnerships to achieve results."
Although OSHA characterized each of these goals as results oriented
to some extent, we believed that only the first goal met that
standard. We considered the other two goals to be results oriented
to little or no extent.
OSHA's Deputy Director for Policy said that the agency's first goal
was somewhat results oriented because hazard elimination should lead
to a reduction in workplace fatalities, injuries, and illnesses--the
outcomes or results OSHA was trying to achieve. He described this
goal as a surrogate for these results, and said that OSHA did not
focus directly on reducing workplace fatalities, injuries, and
illnesses because of the difficulty created by other factors such as
business cycles and technological advances which also affected these
outcomes.\2 We agreed that this goal was somewhat results oriented
because the goal, although interim in nature, was plausibly related
to the results OSHA was trying to achieve. However, it was not clear
what results OSHA was expecting (i.e., whether the elimination of a
few hazards was sufficient) or how progress would be assessed.
OSHA officials said that the agency's second goal regarding
implementation of a "common sense strategy" and "alternative
approaches" was somewhat results oriented because, as figure I.1
shows, they believed that building partnerships into the rulemaking
process would increase the acceptability of regulations, which would
in turn improve regulatory compliance by both industry and employee
groups, and ultimately make workplaces safer.\3
Figure I.1: Assumptions Linked
Regulatory Approach to Results
(See figure in printed
edition.)
Source: Information from agency officials.
However, we considered this goal to be results oriented to little or
no extent because the statement was a strategy rather than an outcome
that the agency was attempting to achieve.\4 OSHA has not clearly
established this goal's link to OSHA's mission and the goal did not
define what results OSHA hoped to achieve, and therefore did not
permit assessment of the agency's progress. OSHA could not provide
any evidence that these approaches actually result in increased
compliance or contribute to the agency's desired results of reducing
fatalities, injuries, and illnesses in the workplace.
OSHA officials said that their third goal--to "focus OSHA programs
and service delivery systems using internal and external partnerships
to achieve results"--was established in reaction to the
administration's desire for OSHA to improve its customer service by
becoming more responsive to its customers. The officials said that
they considered the goal somewhat results oriented because they
believed that hazards would be addressed more quickly through
partnerships and hazard reduction would ultimately result in fewer
workplace fatalities, injuries, and illnesses. However, we believed
that this goal was results oriented to little or no extent because it
was unclear ("focus OSHA programs and ... systems") and it utilized
a strategy ("partnerships") to achieve unspecified "results."
Therefore, it was also impossible to assess OSHA's progress toward
achieving any results.
--------------------
\1 These goals were actually developed for fiscal year 1996. OSHA
officials said that they were revising the agency's goals and
performance measures and that we should use these goals to reflect
their position as of January 1, 1997.
\2 Similarly, OSHA's fiscal year 1995 report on performance
measurement said that increased hazard identification and abatement
was an "intermediate outcome," capturing the short-term results of
activities that reflect desirable changes in the workplace.
\3 OSHA's Deputy Director for Policy said that this goal may also
help OSHA change its regulatory framework. For example, he said that
in some instances regulations might not be the best way to achieve a
desired result, so OSHA might issue guidelines instead of
regulations.
\4 Similarly, in Managing For Results: Achieving GPRA's Objectives
Requires Strong Congressional Role (GAO/T-GGD-96-79, Mar. 6, 1996),
we said that OSHA's goal to "focus resources on achieving workplace
hazard abatement through strong enforcement and innovative incentive
programs" represented a strategy rather than a measurable goal.
VIEWS ON THE RESULTS
ORIENTATION OF OSHA'S PROGRAM
PERFORMANCE MEASURES
--------------------------------------------------------- Appendix I:2
OSHA officials said that the sets of program performance measures
OSHA had adopted for the three strategic goals were each results
oriented to some extent. However, we believed that each of these
sets of performance measures was results oriented to little or no
extent.
VIEWS ON THE RESULTS
ORIENTATION OF MEASURES
ASSOCIATED WITH OSHA'S
STRATEGIC GOAL TO "ELIMINATE
HAZARDS"
------------------------------------------------------- Appendix I:2.1
As of January 1, 1997, the performance measures that OSHA used to
assess progress toward its first strategic goal, "to eliminate
hazards through offering partnerships or traditional enforcement,"
were:
"Percentage of programmed inspections that result in the
identification of significant hazards;
Percentage of initial consultation visits that result in the
identification of significant hazards;
Median time for hazard abatement for inspections;
Median time for hazard abatement for consultation visits;
Median time to respond to requests from smaller employers in high
hazard industries and operations;
Average cost per person trained through the targeted training grant
program; and
Average evaluation score on courses offered by the OSHA Training
Institute and Education Centers."
OSHA officials characterized this set of program performance measures
as somewhat results oriented because they believed that all of the
measures in the set were related to the goal of eliminating hazards.
However, OSHA's Deputy Director for Policy also said that OSHA is
working to make the measures even more results oriented, and has
taken other actions to measure the agency's effects and focus on
achieving particular outcomes. For example, he said that OSHA has
done some retrospective analyses to identify the results of their
efforts. In one such study OSHA found that within 5 years of the
promulgation of OSHA's general industry lead standard in 1978, the
number of workers in the lead smelting and battery manufacturing
industries who had high-lead concentrations in their blood dropped by
66 percent, from about 19,000 to about 6,500. The Deputy Director
also told us that OSHA is in the process of redesigning its field
offices, and that each redesigned office has a local problem-solving
component with outcome measures. One of these redesigned field
offices is using data from the state registry on lead poisoning to
measure OSHA's impact on the health and safety of construction
workers who paint bridges (which involves the removal of lead paint).
The field office is conducting a training program, and next year they
will determine whether there are significantly fewer reports of lead
poisoning in the state registry.
We believed that the set of program performance measures associated
with OSHA's goal of eliminating hazards was results oriented to
little or no extent because, although they initially appeared to be
results oriented, the measures were not clearly and unambiguously
related either to the goal or to the agency's desired results of
reducing workplace fatalities, injuries, and illnesses. The set of
measures did not address all aspects of the goal and also did not
provide relevant information for assessing progress toward the goal
or the agency's mission.
For example, the first two measures--the percentage of inspections or
consultations that resulted in the identification of significant
hazards--did not reflect the actual number of significant hazards
found, only how often inspections or consultations revealed at least
one such hazard. One inspection that identified 50 significant
hazards would be recorded the same as another inspection that
identified only 1 significant hazard. The measures did not reflect
the number of hazards that were eliminated in those workplaces or
whether the OSHA inspections and consultations resulted in any
reduction in fatalities, injuries, or illnesses. Furthermore, it was
unclear whether changes in these measures would indicate that
workplaces are becoming more or less hazardous. For example, if 30
percent of the inspections conducted in 1995 resulted in the
identification of a significant hazard, and that figure rose to 50
percent in 1996, it would not be clear whether the 20 percentage
point increase reflected increased diligence and effectiveness on the
part of OSHA inspectors, better targeting of OSHA inspections, or
that workplaces had become more hazardous.
The third and fourth measures--"median time for hazard abatement" for
inspections and consultation visits--were also not as results
oriented as they appeared. These measures reflected the time OSHA
gave companies to abate a hazard, not the actual amount of time it
took until the hazard was abated or even whether the hazard ever was
abated.\5
Similarly, the fifth measure gauged the time it took OSHA to respond
to requests from smaller employers in high hazard industries and
operations and did not measure whether hazards were eliminated or
whether workplace fatalities, injuries, or illnesses were reduced as
a result of their response. Also, OSHA's 1995 performance report
showed that smaller employers in the manufacturing sector had the
lowest percentage of inspections with significant hazards of any size
group. Therefore, this measure appears to have focused on the group
of employers with the fewest significant hazards.
We also did not believe that the training-related measures that OSHA
associated with this goal were clearly related to hazard abatement.
For example, although the average cost per person trained through the
targeted training grant program may be a useful measure in other
respects (e.g., a cost-effectiveness analysis), the resultant dollar
figure may bear little relationship to the elimination of hazards or
reduction of workplace fatalities, injuries, or illnesses.
Similarly, average evaluation scores on OSHA training may have had
relevance in terms of selecting one course or instructor over
another, but those scores may be totally unrelated to the agency's
goal or desired results.\6
--------------------
\5 OSHA's 1995 performance report indicated that OSHA might move to
measure actual time to abate hazards rather than the amount of time
assigned for abatement.
\6 However, the provision of relevant training (not the cost per
person trained or average evaluation scores) can be a legitimate and
results-oriented measure for hazard abatement or even the reduction
of fatalities, injuries, and illnesses when the linkage between
training and the result is demonstrated. For example, OSHA's fiscal
year 1995 Performance Measurement Report cited a study examining the
effects of OSHA-funded training on logging safety provided to 250
companies in Idaho and Washington during fiscal year 1993 and the
first quarter of fiscal year 1994 that discovered that accident
claims and lost work days per year declined after the safety training
was provided.
VIEWS ON THE RESULTS
ORIENTATION OF THE MEASURE
FOR OSHA'S SECOND GOAL ON
RULEMAKING AND ALTERNATIVE
APPROACHES
------------------------------------------------------- Appendix I:2.2
OSHA used one performance measure, "the percentage of rulemaking
activity on the regulatory agenda undertaken as negotiated
rulemaking," to assess the agency's progress in meeting its second
strategic goal, to "implement a common sense strategy, developed in
partnership with stakeholders, for rulemaking and alternative
approaches to emerging and priority safety and health issues."
A negotiated rulemaking brings together interested parties who draft
a proposed rule that then goes through notice and comment
procedures.\7 OSHA officials characterized this measure as results
oriented to some extent because, as noted previously and depicted in
figure I.1, they believed that using partnership techniques such as
negotiated rulemaking would ultimately make workplaces safer.
However, OSHA officials could provide no evidence to support this
assumption. Therefore, we considered this measure to be results
oriented to little or no extent because it was not clearly and
unambiguously related to OSHA's mission and would not provide
relevant information to permit assessment of the agency's programs.
Also, according to OSHA's fiscal year 1995 Performance Measurement
Report, the negotiated rulemaking process was applicable to only a
portion of proposed rules, and therefore this measure could not be
used for all of OSHA's regulatory activities. Finally, the measure
did not address all aspects of the goal in that it did not gauge any
"alternative approaches" to regulation.
--------------------
\7 See the Negotiated Rulemaking Act of 1990, 5 U.S.C. 561-570.
VIEWS ON THE RESULTS
ORIENTATION OF THE MEASURE
FOR OSHA'S THIRD GOAL TO
FOCUS OSHA PROGRAMS AND
SERVICE DELIVERY SYSTEMS
------------------------------------------------------- Appendix I:2.3
OSHA used one performance measure, "the median time to respond to
complaints," to measure progress toward its third goal, to "focus
OSHA programs and service delivery systems using internal and
external partnerships to achieve results." OSHA officials
characterized this measure as results oriented to some extent because
they said it reflected one of OSHA's responses to the
administration's reinvention initiatives to improve its customer
service.\8 We characterized the measure as results oriented to little
or no extent because it did not clearly relate to OSHA's goal or
mission and did not provide relevant information regarding progress
toward the goal. The measure focused on how long it took OSHA to
respond to a complaint, not how long it took to abate the hazard that
originated the complaint or even whether the hazard was eliminated.
Also, there were no other measures regarding the contributions of
internal or external partnerships to any results achieved, so the
goal was incompletely addressed.
--------------------
\8 OSHA has changed its policy so that when it receives a complaint,
it determines whether to immediately inspect or investigate the
alleged hazards. If OSHA determines an investigation is appropriate,
the agency will telephone the employer, describe the alleged hazards,
and follow up with a fax or a letter. The employer can respond in
kind, identifying any problems found and noting corrective actions
taken or planned. An adequate response generally negates the need
for an inspection. According to OSHA officials, lower priority
complaints can be resolved through the phone/fax procedure, therefore
allowing a more effective use of OSHA inspection resources.
VIEWS ON THE RESULTS
ORIENTATION OF OSHA'S
PERFORMANCE STANDARDS FOR
SELECTED FRONTLINE REGULATOR
POSITIONS
--------------------------------------------------------- Appendix I:3
In contrast to strategic goals and program performance measures, the
administration's regulatory reinvention initiative, not GPRA, set the
expectation for agencies to focus on results when measuring the
performance of their frontline regulators. Two of OSHA's frontline
regulator positions were safety inspectors and health inspectors.\9
Although both positions were covered by the same set CSHO standards,
the standards varied somewhat by OSHA regional office. OSHA
officials said we could select the standards used in any of its
regions, so we used the San Francisco regional office's CSHO
standards in this study.
As of January 1, 1997, OSHA had three CSHO job elements, with one
descriptive performance standard for each element. They were:
1. "Building Partnerships and Improving the Agency's Effectiveness.
Participates in planning and executing safety and health program
activity that serves our customer and improves OSHA's mission of
assuring worker protection.
2. On Site Inspection. Performs inspections in accordance with
agency policy and organizational goals. The scope and focus of the
inspection is based on the type of inspection, the hazards found and
the effectiveness of the employer's safety and health program.
3. Case Management. Plans, prepares, organizes and documents
enforcement and non-enforcement cases in accordance with agency
policy and organizational goals."
Each standard also had a set of "performance measurement
considerations" that elaborated the standard. For example, the
considerations for the first standard included "promotes voluntary
compliance efforts" and "maintains and improves safety/health
knowledge and skills."
OSHA officials characterized the performance standard for the
"building partnerships" and "on-site inspection" job elements as
results oriented to a great extent because they believed they were
directly linked with OSHA's mission to assure worker protection.
OSHA officials characterized the performance standard for the "case
management" element as somewhat results oriented because it was less
directly linked to the agency's mission.
We agreed with OSHA's characterizations of these three performance
standards. We viewed the first two performance standards as greatly
results oriented because they explicitly referred to OSHA's mission
and goals, and a majority of the associated performance measurement
considerations addressed results. For example, in "building
partnerships" the CSHOs were expected to identify and design
intervention tools "that will effect the greatest reduction of
illness, injury, and death." We agreed with OSHA officials that the
third performance standard regarding case management was somewhat
results oriented because the standard and the associated performance
measurement considerations were only indirectly related to results.
For example, both the standard and the considerations were primarily
focused on internal procedures such as documenting cases,
demonstrating knowledge of OSHA standards, and applying professional
judgment--actions that may or may not lead to safer workplaces.
--------------------
\9 Although the President's March 4, 1995, memorandum on his
regulatory reinvention initiative did not define "frontline
regulators," we defined them as regulatory staff who work directly
with the regulated community by enforcing regulations. OSHA
officials identified these two frontline regulator positions for this
review.
OSHA'S COMMENTS AND OUR
EVALUATION
--------------------------------------------------------- Appendix I:4
We met with OSHA officials, including the Acting Director of Policy,
on April 25, 1997, to receive their comments. OSHA officials said
that they generally agreed with the draft report's overall message
regarding the difficulties regulatory agencies have experienced in
becoming results oriented. They provided one technical correction,
which was included in the final report.
Both in our meeting with OSHA officials and in OSHA's written
comments (see app. VII), OSHA emphasized that its approach to
focusing on results began with the development of processes and data
systems, not the development of results-oriented goals. They said
that agencies can legitimately be in different stages of results
orientation because of differences in the approaches agencies took to
focusing on results. Therefore, the officials said they believed
that any comparison of the information presented about the agencies
would be misleading and inappropriate. They also recommended that
the final report more prominently reflect that not all goals need to
be results oriented.
We agree that agencies can use different approaches to focus on
results and that they can legitimately vary in the degree to which
they are results oriented at a particular point in time. As
reflected in the draft report, we expected such variance in the early
stages of implementing GPRA and the regulatory reinvention initiative
and explicitly recognized the interim nature of the characterizations
we made as agencies continued to change their goals, measures, and
standards. We removed tables in the draft report that were intended
only to summarize information across agencies because OSHA's comments
led us to realize that the tables could have the unintended effect of
leading a reader to compare the agencies with one another and to
assume that all goals, measures, and standards are required to be
results oriented.
OSHA officials also said that the discussion in the draft report
about the barriers agencies face in focusing on results should be
highlighted more prominently in the results-in-brief section. In
response, we included more information in the results-in-brief
section of the final report describing the barriers agencies
identified and also summarized the information about the agencies'
goals, measures, and standards in a more general manner.
In its comment letter, OSHA said that its fiscal year 1995
Performance Measurement System Framework had long-term outcome
measures of injuries, illnesses, and fatalities prevented that were
not recognized in the draft report. However, the letter also said
that "these long term outcomes may not be immediately evident and
therefore were not included in OSHA's fiscal year 1996 goals and
objectives." Therefore, OSHA apologized "for any misunderstanding or
miscommunication of information related to our measures." The scope
of this review included only OSHA's fiscal year 1996 goals and
measures, not those for fiscal year 1995. When asked to confirm that
our information on OSHA's fiscal year 1996 goals and program
performance measures was accurate and complete, both before doing our
analysis and again in January 1997, OSHA officials did not identify
any additional measures for our review.
FAA'S MISSION, STRATEGIC GOALS,
PERFORMANCE MEASURES, AND
PERFORMANCE STANDARDS FOR SELECTED
FRONTLINE REGULATOR POSITIONS
========================================================== Appendix II
FAA's mission is to provide "a safe, secure, and efficient global
aerospace system that contributes to the national security and the
efficiency of U.S. aerospace" and be "responsive to the dynamic
nature of customer needs, economic conditions, and environmental
concerns."
We assessed the results orientation of FAA's strategic goals,
performance measures, and employee performance standards according to
the criteria presented in the scope and methodology appendix of this
report. Because we characterized the extent to which each agency
goal and set of performance measures was results oriented, the
application of these criteria represented a higher standard of
results orientation than required by the Results Act. The Act does
not require that all of an agency's goals or measures be results
oriented. Also, GPRA does not cover employee performance standards.
We judged each set of performance standards according to the
expectation set by the President in his March 4, 1995, memorandum
that these standards should "focus on results, not process and
punishment." Similar to the way in which the Act views agencies'
goals and performance measures, we do not believe that each such
standard must be results oriented to achieve the President's
objective.
We applied our criteria to the agencies' written strategic goals,
program performance measures, and employee performance standards and
did not consider any supplementary or contextual information. In
some cases an agency's goals, measures, and standards appeared
results oriented on the surface, but closer examination and strict
application of our criteria resulted in a lower characterization. We
recognize that other information outside of the written statements
and outside of our review may have existed that could have resulted
in our characterizing certain goals or sets of measures and standards
differently.
Both we and FAA officials characterized the extent of results
orientation of FAA's strategic goals, program performance measures,
and employee performance standards for selected frontline regulator
positions. As shown in table II.1, there were differences in how we
and FAA officials viewed the agency's goals, measures, and standards.
Generally, we agreed that most of FAA's strategic goals were results
oriented. There was some disagreement between us and FAA regarding
the extent of results orientation of FAA's performance measures,
although we both characterized the measures as less results oriented
than the goals. There was a sharp disagreement in ours and the
officials' characterizations of FAA's performance standards. FAA
officials said that five of the six sets of performance standards
were greatly results oriented. However, we characterized one set as
greatly results oriented, one set as somewhat results oriented, and
the remaining four sets as results oriented to little or no extent.
Table II.1
Differences Between Our and FAA
Officials' Characterizations
Little Little
A great Some or no A great Some or no
Total extent extent extent extent extent extent
---------- -------- -------- -------- -------- -------- -------- --------
Strategic 9 3 6 0 3 5 1
goals
Performanc 6 4 2 0 0 4 2
e
measures\
a
Performanc 6 5 1 0 1 1 4
e
standards
--------------------------------------------------------------------------------
\a Three of FAA's strategic goals had no program performance
measures.
Source: Agency and GAO analysis.
VIEWS ON THE RESULTS
ORIENTATION OF FAA'S STRATEGIC
GOALS
-------------------------------------------------------- Appendix II:1
We and FAA officials reviewed 9 of FAA's 10 strategic goals in place
as of January 1, 1997:\1
1. "System safety--zero accidents. Eliminate accidents\2 and
incidents\3 in aviation and protect public safety and property in
space transportation systems by targeting the most critical areas.
2. Security--zero incidents. Eliminate security incidents in the
aviation system.
3. Human factors safety--zero accidents. Eliminate human factors as
a causal factor in accidents and incidents.
4. System capacity. Meet the system capacity needs for air and
space transportation safely and efficiently through near-term actions
targeted at specific problems and a long-term comprehensive program
of research, planning, and investment matching user needs.
5. Achieve safe, secure, and efficient global air and space
transportation systems by promoting international cooperation and
harmonization.
6. Provide strong leadership regarding the environmental impacts of
aviation and commercial space transportation.
7. Manage resources and investments to control costs and increase
productivity, efficiency, and effectiveness, while promoting safety
and customer service.
8. Transform FAA into the model Federal workplace.
9. Establish and maintain mutual lines of communications with the
public and with employees to promote understanding, awareness, and
cooperation, and to serve the interests of the traveling public."
FAA officials characterized the first three goals as results oriented
to a great extent and the six remaining goals as somewhat results
oriented. We considered the first three goals to be results oriented
to a great extent, five goals somewhat results oriented, and one goal
results oriented to little or no extent.
According to FAA officials, the agency's first strategic goal was
results oriented to a great extent because it established a clear
definition of the results expected ("zero accidents"). An FAA
official said the agency recognized that, although it was possible to
achieve zero accidents for commercial aviation during 1 year, in the
long term an accident would likely occur. However, the official said
that this goal was created in response to the administration's desire
to ensure aviation safety and express no tolerance for aviation
accidents. He also said that FAA management did not want to imply
that having accidents would be acceptable or that anything other than
achieving zero accidents was desirable. Finally, the official said
that the goal reflected a quality management perspective of setting
high expectations and striving for the best results that could be
achieved. We agreed that the goal was greatly results oriented
because it was directly linked to FAA's mission of providing a safe
and secure aerospace system, contained a clear definition of expected
results, and permitted assessment of the agency's progress toward the
goal.
FAA officials said the agency's second strategic goal of eliminating
security incidents (i.e., no hijackings or explosions aboard
aircraft) was also results oriented to a great extent because, like
the first goal, FAA deliberately established a clear outcome-related
goal of eliminating security incidents in the aviation system. We
agreed that this goal was greatly results oriented because it
provided a clear definition of what outcomes were expected, was
measurable, and was unambiguously linked to FAA's mission.
FAA's third strategic goal, "human factors safety--zero accidents,"
was a subset of the first goal, but FAA established it as a separate
goal to enable the agency to isolate human factors from other causes
of accidents. FAA officials characterized this goal as results
oriented to a great extent. We agreed with that assessment because
the goal was directly linked to FAA's mission, was clear in the
results expected, and permitted assessment of the agency's progress
toward the result.
FAA officials characterized the fourth strategic goal ("to meet
system capacity") as somewhat results oriented because of the
difficulties they experienced in defining aviation system "capacity."
An FAA official said that defining system capacity solely in terms of
the increased number of aircraft traveling through the nation's
airspace or the number of additional airport runways would not be
sufficiently inclusive. He said that any goal relating to system
capacity would need to reflect other considerations as well, such as
delays, capital expenditures for aviation facilities, and the needs
of the aviation community. We agreed with FAA that the fourth
strategic goal was somewhat results oriented because, although it was
clearly linked to FAA's mission, it did not clearly identify the
results FAA was trying to achieve regarding system capacity, nor did
it provide a means of assessing the agency's progress in achieving
the results. We also agreed with FAA officials' characterization
that the fifth strategic goal, to "achieve safe, secure, and
efficient global air and space transportation systems," was somewhat
results oriented. Although the goal was linked to the agency's
mission, it did not clearly define the results expected and did not
provide a means of assessing those results.
FAA officials characterized FAA's sixth strategic goal, to "provide
strong leadership regarding the environmental impacts of aviation and
commercial space transportation," as results oriented to some extent.
However, they noted that the term "strong leadership" was subject to
a variety of interpretations by interest groups and other
stakeholders, such as aircraft manufacturers, airline operators, and
the general public. We characterized this goal as somewhat results
oriented because, although it was linked to FAA's mission, it did not
provide any description of what would result from the "strong
leadership" that FAA was to provide or how to assess the effects of
this leadership.
FAA characterized the seventh ("managing resources"), eighth
("transforming FAA into a model Federal workplace"), and ninth
("establishing and maintaining lines of communications") strategic
goals as somewhat results oriented because they were written in broad
terms and not easily measurable. We agreed with FAA officials that
all three strategic goals were not measurable, although two of the
goals ("managing resources" and "establishing and maintaining lines
of communications") were linked to the agency's mission. As such, we
agreed with FAA officials' characterization that these two goals were
somewhat results oriented. We considered the goal of transforming
FAA into a model federal workplace as results oriented to little or
no extent because it was not linked to FAA's mission and did not
provide a clear definition of what results were expected or permit
assessment of the agency's progress.
--------------------
\1 FAA officials said that one of the agency's strategic
goals--"promote active, vital, and efficient aviation and space
transportation industries"--would soon be eliminated because of a
change in the agency's mission contained in the Federal Aviation
Reauthorization Act of 1996. This act changed FAA's mission by
eliminating the dual mandate of promoting both the "development" and
"safety" of U.S. air commerce. Therefore, we decided to eliminate
it from our review.
\2 An aircraft "accident" is defined by regulation as "an occurrence
associated with the operation of an aircraft which takes place
between the time any person boards the aircraft with the intention of
flight until all such persons have disembarked, and in which any
person suffers death or serious injury as a result of being in or
upon the aircraft or by direct contact with the aircraft or anything
attached thereto, or in which the aircraft receives substantial
damage."
\3 An aviation "incident" is defined by regulation as "an occurrence
other than an accident associated with the operation of an aircraft
which affects or could affect the safety of operations."
VIEWS ON THE RESULTS
ORIENTATION OF FAA'S PROGRAM
PERFORMANCE MEASURES
-------------------------------------------------------- Appendix II:2
FAA officials said that their agency's program performance measures
were evolving and they planned to modify these measures to better
focus on results before submitting them to the Office of Management
and Budget (OMB) for review in September 1997 as part of GPRA
implementation. However, as of January 1, 1997, FAA had sets of
program performance measures for six of its nine strategic goals.
FAA officials said that FAA was still developing measures for the
agency's seventh and ninth goals. FAA officials did not identify any
performance measures for the agency's fifth goal.
FAA officials said that the sets of program performance measures for
the first four strategic goals were results oriented to a great
extent and the sets of measures for the sixth and eighth strategic
goals were somewhat results oriented. We considered three of the
first four sets of measures as somewhat results oriented and the sets
of measures for the third and eighth strategic goals to be results
oriented to little or no extent. We agreed with FAA officials that
the set of measures for the sixth strategic goal was somewhat results
oriented.
VIEWS ON THE RESULTS
ORIENTATION OF THE MEASURES
FOR FAA'S FIRST AND THIRD
GOALS
------------------------------------------------------ Appendix II:2.1
The program performance measures FAA used for both its first ("system
safety--zero accidents") and third ("human factors safety--zero
accidents") strategic goals were:
"percent reduction in accident rate;
number of fatalities per million passenger miles;
percent reduction in operational error\4 rates;
percent reduction of near mid-air collisions;\5 and
percent reduction in runway incursions."\6
FAA officials said they characterized this set of program performance
measures as greatly results oriented because the measures enabled the
agency to track accidents and fatalities and were directly related to
the FAA's first and third strategic goals. We considered this set of
measures to be somewhat results oriented with regard to FAA's first
goal because it did not address all aspects of the goal. For
example, although part of the goal was the elimination of all
aviation incidents, the incident-related measures focused on only
certain types of incidents (operational errors, midair collisions,
and runway incursions) and not other types (e.g., weather-related
incidents). Also, the accident rate measure (the number of accidents
per 100,000 flight hours) was not clearly related to the goal of
"zero accidents." For example, the absolute number of accidents could
actually increase while the percentage of accidents per 100,000
flight hours decreased because of a large increase of flight hours.
We considered this set of measures to be results oriented to little
or no extent with regard to FAA's third goal--human factors safety.
The measures were not clearly linked to the goal of eliminating human
factors in accidents and incidents because none of them isolated
human factors in the results they were measuring. For example, the
number of fatalities could be a function of factors totally unrelated
to human factors.
--------------------
\4 An operational error is an occurrence attributable to an element
of the air traffic control system that results in less than the
applicable separation minima between two or more aircraft, or between
an aircraft and terrain or obstacles and obstructions. Obstacles
include vehicles, equipment, and/or personnel on runways.
\5 A near midair collision is an incident associated with the
operation of an aircraft in which a possibility of collision occurs
as a result of proximity of less than 500 feet to another aircraft,
or a report is received from a pilot or flight crew member stating
that a collision hazard existed between two or more aircraft.
\6 A runway incursion is any occurrence at an airport involving an
aircraft, vehicle, person, or object on the ground that creates a
collision hazard or results in loss of separation with an aircraft
taking off, intending to take off, landing, or intending to land.
VIEWS ON THE RESULTS
ORIENTATION OF THE MEASURES
FOR FAA'S SECURITY GOAL
------------------------------------------------------ Appendix II:2.2
FAA's program performance measures for FAA's second strategic goal
("security--zero incidents") were:
"Percent improvement in detection of complex improvised explosive
devices through all threat vectors;\7
percent increase in the system's ability to sustain 100 percent
compliance with contingency measures;\8 and
percent decrease in vulnerability to existing or new threats."\9
FAA officials said they characterized this set of performance
measures as results oriented to a great extent because the measures
were quantitative and reflected improvements made to the security
system. Although we agree that this set of measures was related to
the agency's security system goal, we considered it somewhat results
oriented because it did not measure all aspects of system security.
For example, an FAA official said that the set's first measure will
initially measure only the detection performance of "threat vectors"
with screening equipment (e.g., x-ray machines) and may not include
other threat sources. Also, the set's first measure refers only to
explosive devices and not other weapons such as handguns.
--------------------
\7 This measure refers to an air carrier or airport security system's
ability to detect explosive devices attempting to reach a target
through different approaches, such as within a checked bag, cargo, or
with a person.
\8 "Contingency measures" refers to security countermeasures to be
taken by air carriers and aviation industry personnel as directed by
FAA through its emergency authority. These countermeasures are
previously agreed-upon actions based on assessments made by law
enforcement and intelligence agencies. FAA assesses a security
system's implementation of a countermeasure through surveillance,
interviews, audits, and testing.
\9 This measure refers to FAA's airport-by-airport vulnerability
assessments and use of risk assessment methodologies and tools to
determine the reduction of risk when mitigating or closing
vulnerabilities in airports' security systems.
VIEWS ON THE RESULTS
ORIENTATION OF THE MEASURES
FOR FAA'S SYSTEM CAPACITY
GOAL
------------------------------------------------------ Appendix II:2.3
The program performance measures for FAA's fourth strategic goal
regarding system capacity were:
"Percent increase in Instrument Flight Rule/Visual Flight Rule
capacity ratio;\10
percent increase in number of Global Positioning System
non-precision and precision approaches;\11
percent reduction in ground movement time;\12
percent reduction in system induced delays per number of
operations;
percent reduction in equipment caused delays per number of
operations; and
percent reduction in weather induced delays per number of
operations."
FAA officials characterized this set of program performance measures
as greatly results oriented because it focused on how the end user
actually experiences the aviation system. However, FAA officials
also told us that the agency recognized that this set of measures did
not include indicators for some capacity issues that were of concern
to the aviation system's users, including the system's flexibility,
accessibility, and predictability.\13 We concluded that the measures
do not provide a means of assessing how FAA will "meet the system
capacity needs" in the near and long term as described in the system
capacity strategic goal. Therefore, we considered this set of
measures to be somewhat results oriented.
--------------------
\10 The Instrument Flight Rule/Visual Flight Rule capacity ratio
estimates an airport's capacity to handle air traffic during good and
poor weather conditions.
\11 The Global Positioning System is a satellite-based navigation
system which can provide aircraft with extremely precise position
information.
\12 The "percent reduction in ground movement time" measures the
amount of time aircraft taxi in and out of the airport, a means of
measuring a delay in the aviation system.
\13 FAA has recently developed a new set of measures for system
capacity. An FAA official said that this new set of measures will be
included in FAA's Air Traffic Services' Fiscal Year 1999 Business
Performance Plan.
VIEWS ON THE RESULTS
ORIENTATION OF MEASURES FOR
FAA'S ENVIRONMENTAL
RESPONSIBILITY GOAL
------------------------------------------------------ Appendix II:2.4
The program performance measures for FAA's sixth strategic
goal--"provide strong leadership regarding environmental impacts of
aviation and commercial space transportation"--were as follows:
For industry operations:
"Percent reduction in number of residents exposed to significant
airport noise levels."
For FAA operations:
"Percent of facilities which have received safety inspections,
percent of facilities which have received environmental
inspections,
number of fuel tanks brought into compliance with EPA requirements,
and
number of facilities audited for energy conservation."
FAA officials characterized this set of program performance measures
as somewhat results oriented because they believed that several of
the measures assessed FAA's compliance with environmental laws rather
than being focused on results. However, because FAA's environmental
goal lacked a clear definition of the outcomes the agency was trying
to achieve, it was difficult for us to determine whether this set of
measures was clearly linked to the goal. Although some of the
measures may relate to the goal (e.g., "number of fuel tanks brought
into compliance with EPA requirements"), we considered this set of
measures somewhat results oriented because many of the measures did
not permit assessment of progress toward the agency's goal. For
example, the percent of facilities that have received safety or
environmental inspections may bear no relation to whether those
facilities are in compliance with OSHA or EPA requirements.
VIEWS ON THE RESULTS
ORIENTATION OF THE MEASURES
ASSOCIATED WITH THE GOAL ON
THE FAA WORKPLACE
------------------------------------------------------ Appendix II:2.5
The program performance measures for FAA's eighth strategic
goal--"transform FAA into the model Federal workplace"--were:
"Conduct an employee attitude survey of FAA employees and register
a 70 percent positive response on overall job satisfaction;
implement national actions to help manage diversity and transform
the FAA into a model Federal workplace; and
through demonstrated actions and accomplishments, the FAA's
workforce demographics will reflect positive trends or increases
with regard to workforce representation for minorities, women,
and people with disabilities."
FAA officials characterized this set of measures as somewhat results
oriented because it did not provide a "measurable outcome" of what
FAA was trying to achieve. We agreed with FAA officials that the set
did not provide a measurable outcome. We also believe that the set
did not address all aspects of the strategic goal, such as the amount
of support for innovation or risk taking within the agency. Perhaps
most importantly, though, the set of performance measures and its
related strategic goal were not linked to the agency's mission.
Therefore, we considered this set of measures to be results oriented
to little or no extent.
VIEWS ON THE RESULTS
ORIENTATION OF FAA'S
PERFORMANCE STANDARDS FOR
SELECTED FRONTLINE REGULATOR
POSITIONS
-------------------------------------------------------- Appendix II:3
In contrast to strategic goals and program performance measures, the
administration's regulatory reinvention initiative, not GPRA, set the
expectation for agencies to focus on results when measuring the
performance of their frontline regulators. Two of FAA's frontline
regulator positions were air traffic control specialists and aviation
safety inspectors.\14 Air traffic control specialists are responsible
for issuing control instructions and advice to pilots in the vicinity
of airports as well as guiding aircraft operations en route along the
airways. Aviation safety inspectors administer and enforce
regulations concerning the airworthiness of aircraft and aircraft
systems and the competence of pilots, mechanics, and other aviation
personnel.
FAA used six sets of performance standards for these positions. FAA
officials characterized five sets as results oriented to a great
extent and one set as somewhat results oriented. They said the
standards were linked to outcomes specified in FAA's strategic goals,
but that the linkage was through several steps and not always clearly
apparent. We characterized only one set of standards as results
oriented to a great extent, one set as somewhat results oriented, and
four sets as results oriented to little or no extent.
--------------------
\14 Although the President's March 4, 1995, memorandum on his
regulatory reinvention initiative did not define "frontline
regulators," we defined them as regulatory staff who work directly
with the regulated community by enforcing regulations. We selected
these two positions for review from the frontline regulator positions
FAA officials identified.
VIEWS ON THE RESULTS
ORIENTATION OF PERFORMANCE
STANDARDS FOR AIR TRAFFIC
CONTROL SPECIALISTS
------------------------------------------------------ Appendix II:3.1
As of January 1, 1997, the air traffic control specialist position in
en route centers and terminals had three job elements and associated
sets of performance standards: "operating methods and procedures,"
"communications," and "training." An FAA official characterized the
first two sets of performance standards as results oriented to a
great extent because following proper air traffic procedures and
effective communication with pilots were essential in providing safe
aircraft landings and departures. We agreed with FAA's
characterization of the first set of standards because the standards
were clearly related to the agency's mission, described the
position's purpose (provide for "the safe, orderly and expeditious
flow of air traffic"), and provided criteria for evaluating
performance ("in accordance with procedures, directives and letters
of agreement"). The second set of standards on "communications"
required incumbents to be "cooperative, courteous and tactful," use
"prescribed phraseology," and listen "effectively," but did not
clearly link these standards to the agency's safety goal or state
what results would occur as a result of these behaviors. Therefore,
we considered this set of standards results oriented to little or no
extent.
The "training" standards required employees taking instruction to
complete classes "in a timely manner," and said that employees
providing instruction should do so "objectively." FAA officials said
that this set of standards was somewhat results oriented because
training was essential to support the duties of the position. We
considered the set of training standards to be results oriented to
little or no extent because it was not clearly related to the
agency's goals or mission and did not provide a description of the
expected results from the training or criteria for evaluating
results.
VIEWS ON THE RESULTS
ORIENTATION OF THE
PERFORMANCE STANDARDS FOR
AVIATION SAFETY INSPECTORS
------------------------------------------------------ Appendix II:3.2
As of January 1, 1997, the aviation safety inspector position (for
operations/maintenance/avionics with responsibility for general
aviation aircraft) had three job elements and associated sets of
performance standards: "technical administration," "certification,"
and "surveillance." FAA officials characterized all three sets of
performance standards as results oriented to a great extent.
However, we considered the first two sets of standards results
oriented to little or no extent because they were not clearly linked
to FAA's mission or any of its strategic goals. For example, the
technical administration standards required inspectors to conduct
investigations "thoroughly" and "in a timely manner," but did not say
what mission-related results would follow from those reviews. We
considered the set of performance standards for "surveillance" to be
somewhat results oriented because it described actions that were more
clearly linked to FAA's mission. For example, inspectors were
required to identify and communicate safety issues that require
special emphasis to the appropriate management levels. However, the
set of standards was unclear as to what outcomes incumbents should
achieve as a result of those actions.
FAA'S COMMENTS AND OUR
EVALUATION
-------------------------------------------------------- Appendix II:4
We met with FAA officials, including several program managers, on
April 29, 1997, to receive their comments. FAA officials generally
agreed with the draft report's overall message regarding the barriers
regulatory agencies face in focusing on results and particularly
noted how difficult it was to determine the unique effect that FAA
had on aviation system safety. They indicated that the agency's
goals and performance measures had changed since the time of our
review. They also said that differences between their and our
characterizations of the goals, measures, and standards in use as of
January 1, 1997, were partially due to the fact that we did not use
the same criteria to determine the extent of result orientation. FAA
officials also provided some technical corrections, which were
included in the final report.
The FAA officials said that FAA had a set of performance goals that
connected the agency's fiscal year 1996 strategic goals to its fiscal
year 1996 program performance measures. They contended that there
was a closer linkage between FAA's program performance measures and
its performance goals than existed between the measures and FAA's
strategic goals (the subject of our review). They also said that
FAA's strategic goals were broad goals for the aviation system as a
whole, whereas its performance goals related more specifically to the
agency's performance. We reviewed FAA's strategic goals and the
program performance measures FAA officials identified that related to
these goals. In order to be consistent in reviewing the five
regulatory agencies, we reviewed the strategic goals of each of these
agencies.\15
FAA officials pointed out that the performance standards for the
aviation safety inspector frontline regulator position were written
before the President's March 4, 1995, memorandum calling for
performance standards to be results oriented. They also believed
that individual regulators understood the relationship of their
performance standards to FAA's mission. Finally, the FAA officials
said they thought it was unrealistic to expect the standards, by
themselves, to reflect a results orientation when there were other
supporting documents that linked the activities of the regulators to
the agency's mission.
The President's memorandum directed the heads of all federal
regulatory agencies to change the way they measure the performance of
their frontline regulators so as to focus on results, not process and
punishment. Therefore, a group of performance standards for a
frontline regulator position that does not in some way focus on
results would not comply with the memorandum, regardless of when the
standards were written. Also, Office of Personnel Management
guidance states that goal setting is the first component of effective
performance management.\16 According to this guidance, research has
shown that specific, hard goals result in higher performance than
vague or abstract goals, or no goals at all. Therefore, the
President's initiative focusing standards on results could improve
regulators' performance in obtaining desired results. Although other
supporting documents may have linked regulators' performance
standards to the agency's mission, we focused solely on agencies'
performance standards to ensure a fair and balanced approach across
all of the agencies.
--------------------
\15 The only performance goals included in this report were those of
FDA because these goals served as the agency's program performance
measures.
\16 "Improving Customer Service Through Effective Performance
Management," Office of Personnel Management Performance Management
Practitioner Series PMD-04, May 1996.
FDA'S MISSION, STRATEGIC AND
PERFORMANCE GOALS, AND ITS
PERFORMANCE EVALUATION PLAN
========================================================= Appendix III
FDA's mission is to "protect, promote and enhance the health of the
American people." FDA seeks to achieve this mission by ensuring that:
"Foods are safe, wholesome, and sanitary; human and veterinary
drugs, biological products, and medical devices are safe and
effective; cosmetics are safe; and electronic products that emit
radiation are safe;
regulated products are honestly, accurately, and informatively
represented; and
products are in compliance with the law and FDA regulations;
noncompliance is identified and corrected; and any unsafe or
unlawful products are removed from the marketplace."
According to the fiscal year 1996 FDA Almanac, the agency supports
its mission through the enforcement of 13 major laws enacted
throughout the century, from the Food and Drugs Act of 1906 to the
Dietary Supplement Health and Education Act of 1994.
We assessed the results orientation of FDA's strategic and
performance goals according to the criteria presented in the scope
and methodology appendix of this report.\1 Because we characterized
the extent to which each strategic goal and related sets of
performance goals were results oriented, the application of these
criteria represented a higher standard of results orientation than
required by GPRA. The Results Act does not require that all of an
agency's goals or measures be results oriented.
We applied our criteria to FDA's written strategic and performance
goals and did not consider any supplementary or contextual
information. In some cases, the agency's goals appeared results
oriented on the surface, but closer examination and strict
application of our criteria resulted in a lower characterization. We
recognize that other information outside of the written statements
and outside of our review may have existed that could have resulted
in our characterizing certain goals differently.
Both we and FDA officials characterized the extent of results
orientation of FDA's strategic and performance goals. We did not
assess any employee performance standards because, as of January 1,
1997, FDA was using generic performance standards which varied from
employee to employee. Therefore, no group of standards applicable to
a frontline regulator position existed. As shown in table III.1, we
and FDA officials differed somewhat in how we characterized the
results orientation of FDA's strategic and performance goals.
Generally, we thought that FDA's goals were less results oriented
than the FDA officials did. However, we agreed with FDA officials
that most of their performance goals were somewhat results oriented.
Table III.1
Differences Between Our and FDA
Officials' Characterizations
Little Little
A great Some or no A great Some or no
Total extent extent extent extent extent extent
---------- -------- -------- -------- -------- -------- -------- --------
Strategic 3 3 0 0 0 3 0
goals
Performanc 9 2 7 0 0 5 4
e goals\a
--------------------------------------------------------------------------------
\a FDA had nine sets of performance goals that it used to assess
progress toward its three strategic goals, with different sets of
measures for different FDA programs. These performance goals
contained the program performance measure in the goal statement. In
addition, performance standards were not included because, as of
January 1, 1997, the agency was using performance standards that were
customized for individual employees.
Source: Agency and GAO analysis.
--------------------
\1 Instead of using performance measures as the means of measuring
their progress on the agency's strategic goals, FDA used sets of
"performance goals." These "performance goals," as defined by GPRA,
are the target level of performance (either output or outcome)
expressed as a tangible, measurable objective, against which actual
achievement will be compared.
VIEWS ON THE RESULTS
ORIENTATION OF FDA'S STRATEGIC
GOALS
------------------------------------------------------- Appendix III:1
As of January 1, 1997, FDA had three strategic goals:
1. Pre-market: "Make timely and cost effective pre-market review
decisions while assuring product safety and efficacy;"
2. Post-market: "Strengthen assurance that products on the market
or about to enter the market are safe;" and
3. Capacity: "Strengthen the nation's capacity to manage health
risks associated with FDA-regulated products."
Although FDA characterized all three strategic goals as results
oriented to a great extent, we viewed all three goals as somewhat
results oriented.
FDA officials said that the agency's first strategic goal regarding
pre-market review decisions was results oriented to a great extent.
By establishing a review process, they said that FDA can ensure the
safety of FDA-regulated products and, therefore, promote and protect
the health of the American public. Although we agreed that "assuring
product safety and efficacy" is related to the agency's mission to
promote and protect public health, we considered the goal to be
somewhat results oriented because it did not establish a clear
definition of the results expected through its pre-market review
decisions (e.g., what a "timely" or "cost effective" decision was)
and, therefore, did not permit assessment of the agency's progress
toward achieving these results.\2 Also, the major emphasis in the
goal is on making timely and cost-effective decisions, which is not
part of the agency's stated mission.\3
FDA officials said the agency's second strategic goal regarding the
safety of products on the market or about to enter the market was
greatly results oriented because the agency's inspection process
provided a safeguard to prevent the distribution of potentially
harmful food, drugs, and other FDA-regulated products. They also
said that by preventing the distribution of these harmful products,
FDA protected the health of the American public. We again
characterized this goal as somewhat results oriented because it did
not provide a clear definition of its intended results or permit
assessment of the agency's progress toward achieving the results.
For example, the goal did not identify how or the extent to which FDA
expects to "strengthen assurance" that products are safe, or even
whose "assurance" would be strengthened.
Finally, FDA officials said the agency's third strategic goal--to
"strengthen the Nation's capacity to manage health risks associated
with FDA-regulated products"--was results oriented to a great extent
because it referred to the agency's internal processes to maintain
the science capacity of its staff. Through its science capacity, FDA
officials said the agency can be assured that it has valid and
reliable science to support its review processes and inspections--the
activities by which FDA protects and promotes the public's health.
However, we characterized this goal as somewhat results oriented
because, although it was tenuously linked to FDA's mission, it was
unclear whose "capacity" would be strengthened, what "capacity"
means, or what "manage health risks" means. If, as FDA officials
said, the agency's capacity was to be strengthened through
performance of this goal, it was unclear how and the degree to which
FDA would strengthen its capacity or the effect that strengthening
would have on the nation's or the agency's ability to manage health
risks.
--------------------
\2 In October 1992, Congress passed the Prescription Drug User Fee
Act (PDUFA) to provide FDA with additional resources to expedite the
review of human drug applications. PDUFA incorporated by reference
specific goals and time frames for FDA to review and act on drug
applications set forth in letters from the FDA Commissioner to the
Chairmen of the House Committee on Energy and Commerce and the Senate
Committee on Labor and Human Resources. However, FDA did not link
its strategic goal with these time frames or any other criteria
defining what would be considered timely.
\3 PDUFA did not make any specific alterations to FDA's mission.
VIEWS ON THE RESULTS
ORIENTATION OF FDA'S
PERFORMANCE GOALS
------------------------------------------------------- Appendix III:2
As of January 1, 1997, FDA was using nine sets of performance goals
for its three strategic goals. As table III.2 shows, different sets
of performance goals were used not only for the different goals but
also for many of the different FDA programs. Post-market performance
goals were not applicable to two of the agency's programs. However,
the set of performance goals for the capacity strategic goal was
applicable to all of the agency's programs except for toxicological
research. FDA officials characterized two of these sets of
performance goals as results oriented to a great extent and seven
sets as somewhat results oriented. However, we considered five sets
of FDA's performance goals as somewhat results oriented and four sets
as results oriented to little or no extent.
Figure III.1: Sets of FDA
Performance Goals Varied By
Strategic Goal and FDA Program
(See figure in printed
edition.)
VIEWS ON THE RESULTS
ORIENTATION OF THE SETS OF
PERFORMANCE GOALS FOR THE
PRE-MARKET STRATEGIC GOAL
----------------------------------------------------- Appendix III:2.1
FDA officials characterized four sets of performance goals related to
the agency's pre-market strategic goal as somewhat results oriented
and one set as results oriented to a great extent. We considered all
five sets of performance goals as somewhat results oriented.
FDA officials characterized three sets of pre-market performance
goals (biologics/drugs, food safety and nutrition, and devices and
radiological health) as somewhat results oriented. The officials
classified these sets of performance goals as such because, although
the sets measured the number or percentage of applications FDA
completed within a certain period of time, these sets did not include
measures for the length of time needed to approve an application.
For example, the performance goals related to biologics and drugs
were:
"Review 90 percent of complete product licensing agreements (PLA),
establishment licensing agreements (ELA), and new drug approvals
(NDA) for priority applications within 6 months after the
submission date;
review 90 percent of complete PLAs, ELAs, and NDAs for standard
applications, within 12 months after the submission date;
review 90 percent of priority supplements to complete PLAs, ELAs,
and NDAs within 6 months after the submission date;
review 90 percent of standard supplements to complete PLAs, ELAs,
and NDAs that require review of clinical data (efficacy
supplements) within 12 months after the submission date;
review 90 percent of standard supplements to PLAs, ELAs, and NDAs
that do not require review of clinical data (efficacy
supplements) within 6 months after the submission date;
review 90 percent of complete PLA, ELA, and NDA applications
resubmitted following receipt of a non-approvable letter, within
6 months after the resubmission date; and
review and act upon 55 percent of fileable generic drug
applications received during Fiscal Year 1997 within 6 months
after submission."
According to an FDA official, completed applications include those
approved by FDA or those requiring additional information from the
applicant.\4 When FDA requests additional information, the applicant
may respond to FDA's request or withdraw the application from
consideration. The FDA official said that because applicants are
never denied the opportunity to return to the application process,
FDA rarely issues a final negative response to applications.
However, some FDA programs do have a "refuse to file" policy that
prevents some incomplete applications from entering FDA's review
process.
An FDA official said that if the performance goals had measured the
timeliness of approvals they would be more results oriented because
timeliness would translate into new products being made available for
public use, potentially improving the health of the American public.
We also considered these sets of performance goals somewhat results
oriented because they did not cover all aspects of the strategic
goal. None of the goals addressed the cost-effectiveness of these
reviews or whether product safety and efficacy were being assured.
FDA officials said the set of performance goals for veterinary
medicine was results oriented to a great extent because, unlike the
first three sets of performance goals, it measured the length of time
to approve an application.\5 For example, one performance goal was to
"reduce the average number of months required for approval of a new
animal drug from 41 months in fiscal year 1996 to 40 months in fiscal
year 1997." FDA officials said that approving applications for animal
drugs and feeds assisted in maintaining the health of livestock and
limited the spread of contaminated foods, and therefore protected
public health. However, we considered this set of performance goals
somewhat results oriented because, like the previous sets of goals,
it did not address all aspects of the strategic goal. None of the
performance goals measured the cost-effectiveness of the reviews or
whether product safety and efficacy were being assured.
FDA had three performance goals relating to toxicological research:
1. "Complete the inlife portions of the mechanistic and tumor
studies on fumonisin and chloral hydrate, which will provide
information on specific agent toxicity and a new methodology for
future assessments;
2. develop and evaluate a prototype predictive screening process to
streamline the review of new compounds and to more accurately predict
human risk; and
3. construct a prototype `knowledge base' (expert systems to
evaluate and predict risk associated with toxicity by estrogenic
compounds) for selected reviewer evaluation."
FDA officials characterized this set of goals as somewhat results
oriented because it focused on the development of a good "knowledge
base," which they said enabled FDA to make valid and reliable
decisions during the approval process.
We agreed that developing a knowledge base is important for FDA's
review process and that the measures were generally related to the
strategic goal of making "timely and cost-effective pre-market
decisions." However, the set of goals focused on developing processes
(i.e., developing "a new methodology for future assessments," "a
prototype predictive screening process," or "prototype `knowledge
base") did not provide a clear means to assess progress toward
achieving the pre-market strategic goal. In addition, the set of
goals did not address the elements of timeliness or
cost-effectiveness of the review process, and did not assess whether
product safety or efficacy were being assured. Therefore, we
considered this set of performance goals to be somewhat results
oriented.
--------------------
\4 FDA notifies applicants as to the disposition of their application
by using one of three "action letters." Applicants may receive either
an "approved letter," an "approvable letter" (an approval of the
application noting minor deficiencies that could be corrected by the
applicant), or a "not approvable letter" (which provides the
applicant the option to resubmit a modified application).
\5 Similar to other FDA programs, completed applications for
veterinary medicine could be either approved by FDA or a request
could be made to the applicant for additional information.
VIEWS ON THE RESULTS
ORIENTATION OF THE SETS OF
POST-MARKET PERFORMANCE
GOALS
----------------------------------------------------- Appendix III:2.2
FDA had three sets of performance goals for its post-market strategic
goal to "strengthen assurance that products on the market or about to
enter the market are safe." FDA officials characterized one set of
performance goals as results oriented to a great extent and the
remaining two sets as somewhat results oriented. We considered all
three sets of performance goals to be results oriented to little or
no extent.
The first set of post-market performance goals, related to food
safety and nutrition, were:
"By December 31, 1996, develop a training program on how to conduct
Hazard Assessment Critical Control Points (HACCP) inspections;
by April 1, 1997, develop a model Federal/State partnership
agreement for seafood HACCP to implement HACCP-based inspections
of seafood processors by the states;
by May 31, 1997, provide training to FDA inspectors who will
conduct verification inspections, and appropriate headquarters
staff of core HACCP curriculum;
by June 30, 1997, revise the FDA Inspection Operations Manual for
HACCP-based inspections of seafood processors;
by June 30, 1997, revise compliance program criteria for
HACCP-based inspections of seafood processors;
by September 30, 1997, determine utility of establishing a
reference/baseline of industry preventive controls based on an
assessment of past data;
by September 30, 1997, establish a Seafood HACCP database to record
inspection results and facilitate evaluation of the system; and
increase Threshold of Regulation Policy Actions Completed from 50
in 1996 to 55 in 1997."
FDA officials characterized the set of performance goals related to
food safety and nutrition as results oriented to some extent. They
said that the HACCP inspections indicated a new strategic direction
for FDA by emphasizing the detection and prevention of
microbiological contamination of food products and extending the
responsibility for food safety to the food processing industry. An
FDA official said that this set of performance goals illustrated how
FDA was implementing HACCP (e.g., developing partnership agreements,
training inspectors, and revising an inspection manual and compliance
program criteria). However, we characterized this set of performance
goals as results oriented to little or no extent because none of the
goals clearly related to or measured the strategic goal or the
agency's mission. It was not clear to us that any of the actions
contained in these measures (e.g., develop a training program, revise
a manual, or establish a database) would, by themselves, strengthen
assurance that products are safe.
The second set of post-market performance goals, related to devices
and radiological health, were:
"Process 95 percent of the medical device reports received, and
maintain a zero backlog;\6
process 5,400 reports under the MedWatch Voluntary Reporting
Program;\7
train and certify approximately 250 State and Federal inspectors to
conduct annual and follow-up inspections at the approximately
10,000 U.S. mammography facilities;
conduct 10,000 annual and 200 follow-up inspections of U.S.
mammography facilities under the Mammography Quality Standards
Act (MQSA); and
conduct 3,000 U.S. mammography facility recertifications under
MQSA."
FDA officials characterized this set of performance goals as greatly
results oriented because three of the goals related to the MQSA,
which authorized FDA to accredit and inspect mammography facilities.
An FDA official said that FDA enhances the performance of equipment
and radiologists through its MQSA program, thereby producing improved
patient care through a better screening process and diagnosis.
However, we characterized this set of performance goals as results
oriented to little or no extent because it did not clearly relate to
or provide relevant information for assessing progress toward the
strategic goal of strengthening assurance that products are safe.
For example, the fourth goal only measured the number of MQSA
inspections conducted and did not measure how many facilities were in
compliance with mammography standards or assess the impact or effect
of these inspections.\8 Similarly, the performance goals regarding
medical device reports and the MedWatch Voluntary Reporting Program
only measured the processing of reports, and did not gauge FDA's
impact on the safety of medical devices.\9
The third set of post-market performance goals, related to veterinary
medicine, were (1) "conduct 700 establishment inspections" and (2)
"conduct, by States under FDA contract, 550 feed mill inspections."
FDA officials said they considered this set of goals somewhat results
oriented because inspections can improve the quality of animal drugs
and feeds on the market. However, similar to the performance goals
on MQSA inspections, this set did not measure how many establishments
or feed mills were in compliance with FDA regulations. As such we
considered this set of performance goals as results oriented to
little or no extent because it did not clearly relate to or provide
relevant information for assessing progress toward the strategic goal
of strengthening assurance that products are safe.
--------------------
\6 Under the 1976 Medical Device Amendments to the Federal Food,
Drug, and Cosmetic Act, manufacturers of medical devices are required
to report to FDA--and maintain records on--device-related deaths,
serious injuries, and malfunctions that, should they recur, would be
likely to result in death or serious injury.
\7 The MedWatch Voluntary Reporting Program (MedWatch) was
established by FDA to encourage health professionals to report
problems associated with medical devices to FDA. MedWatch and
medical device reports are intended to gather information to enable
FDA and the medical device industry to take corrective action on
device problems and, when appropriate, to alert the public to
potentially hazardous devices.
\8 In our report entitled FDA's Mammography Inspections: While Some
Problems Need Attention, Facility Compliance Is Growing
(GAO/HEHS-97-25, Jan. 27, 1997), we found that although "FDA has
made progress in bringing facilities into compliance with mammography
standards, it lacks procedures to enforce timely correction of all
deficiencies found during inspections."
\9 In our report entitled Medical Device Reporting: Improvements
Needed in FDA's System for Monitoring Problems With Approved Devices
(GAO/HEHS-97-21, Jan. 29, 1997), we found that FDA had not
"routinely documented the final corrective actions taken by
manufacturers and FDA to resolve the problems identified in these
reports." The report concluded that "without adequate documentation,
FDA has little assurance that the many reports received are, in fact,
useful and result in better protection of the public health."
VIEWS ON THE RESULTS
ORIENTATION OF THE SET OF
CAPACITY PERFORMANCE GOALS
----------------------------------------------------- Appendix III:2.3
FDA had three performance goals related to its strategic goal of
strengthening the nation's capacity to manage health risks:
1. "Complete design on improved version of full Operational and
Administrative System for Import Support;
2. allow for a range of between 62 percent and 65 percent automated
`may proceeds' on a monthly basis; and
3. maintain a foreign inspection program with a planned total of
1,020 foreign inspections in fiscal year 1997."
FDA officials characterized this set of performance goals as somewhat
results oriented because they were all output measures of FDA's
internal processes. An FDA official said that the first goal was a
process improvement goal to automate FDA's system of handling
imported foods and other products. The second goal measured the
percentage of "may proceed" decisions being automated as part of
FDA's import entry clearance process, indicating a more efficient
process. The third goal referred to the number of inspections FDA
expected to complete of foreign manufacturers of drugs and devices,
which FDA officials said was another example of how FDA protects
public health. However, we characterized this set of performance
goals as results oriented to little or no extent because the actions
contained in these goals (designing systems, automating processes, or
maintaining an inspection program) were not clearly and unambiguously
related to FDA's mission of promoting and protecting the health of
the American people. Also, the set did not provide relevant
information to assess progress toward that mission, such as the
number of foreign establishments in compliance with FDA regulations.
FDA REPLACED PERFORMANCE
STANDARDS WITH A CUSTOMIZED
PERFORMANCE EVALUATION PLAN
------------------------------------------------------- Appendix III:3
On September 1, 1996, FDA adopted a new employee performance
evaluation program that replaced most of FDA's employee performance
standards.\10 FDA officials said that the new program includes a
"performance evaluation plan" (PEP)--an evaluation form that lists
five generic job elements on "individual work management," "technical
competency," "teamwork," "innovation," and "customer service." PEP
also contains blank spaces for supervisors to write in new job
elements tailored to the employee's position. FDA documentation
stated that each employee and his or her supervisor should discuss
the employee's official duties and responsibilities and use PEP's
checklist format to customize a set of performance standards for the
employee. PEP requires that nonsupervisors be evaluated on at least
one job element, but supervisors have the final decision on the
content of the PEP form.
FDA officials said that most of the positions they considered
"frontline regulators" were using PEP as of January 1, 1997, and,
therefore had unique sets of performance standards. As a result, we
did not include any performance standards for FDA in our review
because FDA officials told us that the performance standards for any
of their frontline regulator positions would vary from employee to
employee.
--------------------
\10 FDA officials said that the majority of FDA staff were under the
new performance evaluation system with the exception of some union
employees remaining under the prior system.
FDA'S COMMENTS AND OUR
EVALUATION
------------------------------------------------------- Appendix III:4
We met with FDA officials, including the Director for Planning and
Management Communications Staff, on April 22, 1997. FDA officials
said they generally agreed with the report's overall message
regarding the difficulties that regulatory agencies have experienced
in becoming results oriented. They also said that they would have
reached conclusions similar to ours regarding the results orientation
of their strategic and performance goals had they used the same
criteria we used. The FDA officials also provided several technical
corrections, which were included in the final report.
FDA officials said that (1) the progress made by FDA's field offices
was not reflected in the report's focus on FDA's agencywide goals;
(2) the performance goals FDA is developing for the fiscal year 1999
budget cycle will be more results oriented than the January 1, 1997,
goals assessed in this report; and (3) many of the interim goals that
FDA had (e.g., developing a training program for conducting HACCP
verification inspections), while not results oriented themselves,
would ultimately achieve outcome-oriented results.\11
We agree that focusing on agencywide goals did not capture
information on progress made in field office pilot programs. In the
draft report, we recognized that the goals, measures, and some of the
standards were changing at the time of our review. FDA's efforts to
establish results-oriented systems, such as HACCPs, may allow FDA to
achieve outcome-oriented results in the future. However, our
objective was to describe the agency's degree of results orientation
as of January 1, 1997.
--------------------
\11 The HACCP system is designed to identify the critical points in
food processing and establish controls to prevent adulteration caused
by microbiological, chemical, or physical hazards in order to produce
safer food products.
IRS' MISSION, STRATEGIC
OBJECTIVES, KEY PERFORMANCE
INDICATORS, AND PERFORMANCE
STANDARDS FOR SELECTED FRONTLINE
REGULATOR POSITIONS
========================================================== Appendix IV
IRS' mission is to "collect the proper amount of tax revenue at the
least cost; serve the public by continually improving the quality of
our products and services; and perform in a manner warranting the
highest degree of public confidence in our integrity, efficiency, and
fairness." IRS also has developed a business vision "to be a
state-of-the-art financial services company that eliminates most
paper in the quest for quick, efficient customer service. The new
IRS will be handling less paper and collecting taxes in new and more
efficient ways."
We assessed the results orientation of IRS' strategic objectives,
performance indicators, and employee performance standards according
to the criteria presented in the scope and methodology appendix of
this report.\1 Because we characterized the extent to which each
agency goal and set of performance measures was results oriented, the
application of these criteria represented a higher standard of
results orientation than required by the Results Act. The Act does
not require that all of an agency's goals or measures be results
oriented. Also, GPRA does not cover employee performance standards.
We judged each set of performance standards according to the
expectation set by the President in his March 4, 1995, memorandum
that these standards should "focus on results, not process and
punishment." Similar to the way in which GPRA views agencies' goals
and performance measures, we do not believe that each such standard
must be results oriented to achieve the President's objective.
We applied our criteria to the agencies' written strategic goals,
program performance measures, and employee performance standards and
did not consider any supplementary or contextual information. In
some cases an agency's goals, measures, and standards appeared
results oriented on the surface, but closer examination and strict
application of our criteria resulted in a lower characterization. We
recognize that other information outside of the written statements
and outside of our review may have existed that could have resulted
in our characterizing certain goals or sets of measures and standards
differently.
Both we and IRS officials characterized the extent of results
orientation of IRS' strategic objectives, key performance indicators,
and employee performance standards for selected frontline regulator
positions. As shown in table IV.1, there were differences in how we
and IRS officials viewed the agency's objectives, indicators, and
standards. We and IRS officials agreed that IRS' strategic
objectives were greatly results oriented. There was somewhat more
disagreement regarding IRS' performance indicators in that we
characterized them as somewhat less results oriented than did the IRS
officials. The differences in our characterizations were greatest
with regard to the employee performance standards. IRS officials
said that most of the standards were greatly results oriented, but we
characterized most as somewhat results oriented.
Table IV.1
Differences Between Our and IRS
Officials' Characterizations
Little Little
A great Some or no A great Some or no
Total extent extent extent extent extent extent
---------- -------- -------- -------- -------- -------- -------- --------
Strategic 3 3 0 0 3 0 0
objective
s\a
Key 3 2 1 0 0 3 0
performan
ce
indicator
s\b
Performanc 10 9 1 0 3 7 0
e
standards
--------------------------------------------------------------------------------
\a IRS used the term "strategic objective" instead of "strategic
goal."
\b IRS used the term "key performance indicators" instead of "program
performance measures."
Source: Agency and GAO analysis.
--------------------
\1 IRS used the term "strategic objective" instead of "strategic
goal" and the term "key performance indicators" instead of "program
performance measures."
VIEWS ON THE RESULTS
ORIENTATION OF IRS' STRATEGIC
OBJECTIVES
-------------------------------------------------------- Appendix IV:1
As of January 1, 1997, IRS had established three "strategic
objectives":
1. "Increase compliance: Our objective is to encourage and assist
taxpayers to voluntarily file timely and accurate returns and to pay
on time, and, if taxpayers do not comply, to take appropriate
compliance actions. We do this to ensure we collect the proper
amount of tax at the least cost.
2. Improve customer service: Our objective is to reduce the time
and expense experienced by taxpayers, tax professionals and others in
complying with the tax laws, while increasing their satisfaction with
tax administration. We want to ensure that filing materials are
readily available, and to reduce the data required to be reported by
taxpayers and captured by IRS to only the most critical. We do this
to help collect the proper amount of tax at the least cost, serve the
public, and increase public confidence in tax administration.
3. Increase productivity: Our objective is to continually improve
our operations and the quality of products and services we provide to
our customers, through strategic use of reengineering and systems
management tools and the development of a highly skilled work force.
We do this to improve compliance, provide improved customer service
and reduce the cost of tax administration."
IRS officials said that these three objectives were results oriented
to a great extent because they related directly to IRS' mission. We
agreed with that assessment with regard to each of the strategic
objectives. IRS' first objective, to increase taxpayer compliance,
was clearly related to IRS' mission to collect the proper amount of
tax revenue, provided a clear definition of the results expected, and
permitted assessment of the agency's programs. Similarly, IRS'
second objective, to improve customer service, directly related to
IRS' mission to "serve the public," and IRS' third objective of
increasing productivity was also linked to IRS' mission to collect
revenue "at the least cost."
VIEWS ON THE RESULTS
ORIENTATION OF IRS' KEY
PERFORMANCE INDICATORS
-------------------------------------------------------- Appendix IV:2
As of January 1, 1997, IRS was using three sets of servicewide "key
performance indicators" for its strategic objectives. IRS officials
said that two of the three sets of indicators were results oriented
to a great extent, and one was results oriented to some extent. We
characterized the three sets of indicators as somewhat results
oriented.
VIEWS ON THE RESULTS
ORIENTATION OF THE
INDICATORS ASSOCIATED WITH
IRS' STRATEGIC OBJECTIVE TO
"INCREASE COMPLIANCE"
------------------------------------------------------ Appendix IV:2.1
The indicators that IRS used to assess progress toward its first
strategic objective, to "increase compliance," were:
"Total collection percentage": A ratio of IRS' total collections
to the total estimated true tax liability.
"Total net revenue collected": All revenue collected by the IRS.
"Servicewide enforcement revenue collected/protected": All revenue
collected through enforcement actions and all revenue prevented
from being released from the Treasury in response to fraud,
disallowed claims, and other issues.\2
"Servicewide audit coverage": The percentage of returns audited
divided by the total number of returns filed during the prior
calendar year.
IRS officials said that this set of indicators was results oriented
to a great extent because they believed that the first three
indicators were clear indications of IRS' collection results.
However, we characterized this set of indicators as somewhat results
oriented because, although the "total collection percentage" was
clearly and unambiguously related to the strategic objective of
increasing compliance, the set did not address all aspects of the
strategic objective.\3 For example, the set did not contain
indicators for IRS efforts to encourage and assist taxpayers in
filing accurate and timely returns.
We did not view the remaining indicators in this set as clearly
linked to the strategic objective to "increase compliance." The
second indicator, "total net revenue collected," is already contained
in the ratio of the "total collection percentage" and is ambiguous in
its relation to the strategic objective. An increase or decrease in
revenue collected may simply reflect a change in economic activity or
tax laws rather than any change in the level of taxpayer compliance.
Similarly, the amount collected/protected through enforcement actions
and the percentage of returns audited may not provide a valid
indication of the level of taxpayer compliance. For example,
enforcement revenues collected could increase while voluntary
revenues decreased, resulting in no change (or even a decrease) in
the overall level of taxpayer compliance.
--------------------
\2 According to IRS, enforcement revenue protected and enforcement
revenue collected will be expressed as separate indicators in fiscal
year 1998.
\3 Although the "total collection percentage" information is
relevant, it may not be accurate. Our reports have shown that IRS'
estimate of the true tax liability lacks statistical precision at the
national level. Without such an estimate, IRS will not be able to
accurately determine the "total collection percentage" nor judge its
progress toward increasing compliance. See Taxpayer Compliance:
Analyzing the Nature of the Income Tax Gap (GAO/T-GGD-97-35, Jan. 9,
1997).
VIEWS ON THE RESULTS
ORIENTATION OF THE
INDICATORS ASSOCIATED WITH
IRS' STRATEGIC OBJECTIVE TO
"IMPROVE CUSTOMER SERVICE"
------------------------------------------------------ Appendix IV:2.2
IRS used two indicators to assess the agency's progress toward its
second strategic objective, to "improve customer service":
1. "Taxpayer burden cost (in dollars) for IRS to collect $100": A
ratio that measured taxpayers' paperwork burden costs compared to the
cost for IRS to collect $100 in net tax revenue.\4
2. "Initial contact resolution rate": The rate at which taxpayer
issues were resolved during the first contact with IRS.\5
IRS officials characterized this set of indicators as results
oriented to some extent, and said they were working to improve IRS'
customer service measures. We also characterized the set of indices
as somewhat results oriented because the indicators did not address
all aspects of the customer service objective.\6 For example, IRS did
not measure "public confidence in tax administration" even though the
agency regularly conducts customer service surveys. IRS also did not
include any indicators of how they would "ensure that filing
materials are readily available."
--------------------
\4 Paperwork burden costs were based on the estimated time individual
and business taxpayers spend keeping tax records, learning about tax
laws, preparing tax forms, and sending tax forms to the IRS.
Taxpayer paperwork burden was converted from time to dollars by
multiplying total time by $20 per hour.
\5 As of January 1, 1997, IRS was revising the definition of this
rate.
\6 The paperwork burden data upon which the first indicator is based,
although relevant to assessing progress toward the strategic
objective, is generally considered to be inaccurately measured. Our
reviews of tax compliance burden have shown that a reliable estimate
of the overall costs of tax compliance is not currently available,
and would be costly and burdensome to obtain. See Tax System:
Issues in Tax Compliance Burden (GAO/T-GGD-96-100, Apr. 3, 1996);
Tax System Burden: Tax Compliance Burden Faced by Business Taxpayers
(GAO/T-GGD-95-42, Dec. 9, 1994); and Tax Administration:
Opportunities to Reduce the Burden of Filing and Processing Tax
Returns (GAO/T-GGD-92-41, May 13, 1992).
VIEWS ON THE RESULTS
ORIENTATION OF INDICATORS
ASSOCIATED WITH IRS'
STRATEGIC OBJECTIVE TO
"INCREASE PRODUCTIVITY"
------------------------------------------------------ Appendix IV:2.3
IRS' indicators for the strategic objective to "increase
productivity" were:
"Budget cost to collect $100 in revenue."
"Percent of returns filed electronically": The number of
electronically filed tax returns, divided by the total returns
filed.
"Percent of dollars received electronically."\7
"Percent of dollars received via third party processors": The
dollar amount of federal tax deposits (FTD) processed by
commercial banks under contract to the Financial Management
Service as a percentage of the total FTD dollars received (paper
and electronic).
IRS officials said that this set of indicators was results oriented
to a great extent mainly because the first indicator provided an
overall indicator of how efficiently IRS collects revenues. However,
we characterized the set as somewhat results oriented because,
although the indicator "budget cost to collect $100 in revenue" is a
good overall productivity measure, the set of indicators did not
address all aspects of the objective. For example, none of the
indicators measured the "quality of the products and services" IRS
provides.
--------------------
\7 As of January 1, 1997, IRS was revising this indicator.
VIEWS ON THE RESULTS
ORIENTATION OF IRS' PERFORMANCE
STANDARDS FOR SELECTED
FRONTLINE REGULATOR POSITIONS
-------------------------------------------------------- Appendix IV:3
In contrast to strategic objectives and performance indicators, the
administration's regulatory reinvention initiative, not GPRA, set the
expectation for agencies to focus on results when measuring the
performance of their frontline regulators. IRS identified two
frontline regulator positions for this review: customer service
representatives (CSR) and revenue officers.\8 Employees in the CSR
position work as technicians in a district office or service center
to provide information and direction on tax issues to the public.
Revenue officers collect delinquent taxes and secure delinquent
returns. Revenue officers' work requires application of a knowledge
of (a) general or specialized business practices; (b) pertinent tax
laws, regulations, procedures, and precedents; (c) judicial
processes, laws of evidence, and the interrelationship between
federal and state laws with respect to collection and assessment
processes; and (d) investigative techniques and methods.
As of January 1, 1997, the CSR and revenue officer positions had a
total of 12 job elements and we and IRS officials reviewed 10 of the
12 sets of associated performance standards.\9 IRS officials
characterized nine of the sets of standards as results oriented to a
great extent and one set as somewhat results oriented. We considered
three of the sets of standards to be greatly results oriented and the
remaining seven sets as somewhat results oriented.
--------------------
\8 Although the President's March 4, 1995, memorandum on his
regulatory reinvention initiative did not define "frontline
regulators," we defined them as regulatory staff who work directly
with the regulated community by enforcing regulations.
\9 We did not review the sets of standards for the CSR or revenue
officer positions' job elements for "other duties and assignments"
because they were not specific and, although they may have related to
results in specific instances, reviewing these standards would not
have provided any consistent perspective on the extent to which these
standards focused on results.
VIEWS ON THE RESULTS
ORIENTATION OF IRS' CUSTOMER
SERVICE REPRESENTATIVES'
PERFORMANCE STANDARDS
------------------------------------------------------ Appendix IV:3.1
We and IRS officials reviewed the sets of performance standards
related to five job elements for the customer service representative
position:
1. "Customer relations: Achieves quality customer service by
communicating professionally with internal and external customers in
all oral and written contacts.
2. Workload management: Plans and utilizes time based on priorities
in accordance with Service policy and managerial direction. Arranges
activities to minimize unproductive time and unnecessary delays in
servicing the customer.
3. Procedures and notices: Applies procedural instructions issued
by National, Regional, District and local offices to maximize the
efficient flow of work in meeting program objectives. Correctly
interprets notices to resolve inquiries appropriately.
4. Integrated Data Retrieval System (IDRS) inquiries and on-line
resolution: Resolves notice inquiries, refund and other account
related issues through accessing IDRS. Researches and analyzes or
performs on-line adjustment to accounts by utilizing appropriate
reference materials.
5. Tax law: Encourages voluntary compliance and improves the
public's confidence in the Service by correctly interpreting and
applying tax laws, researching technical reference materials, and
analyzing and evaluating the customer's needs; informs customer of
options and consequences as they relate to tax obligations."
The performance standards for each element described what the CSRs
were expected to do for that element. For example, the performance
standards for the "tax law" job element were:
"Interpretation and application: Generally interprets and applies
tax laws appropriately resulting in technically correct
responses to customer inquiries.
Analysis and evaluation: Generally evaluates and analyzes the
customer's circumstances and advises of tax computation and
other benefits, i.e., Earned Income Credit, Child Care Credit,
estimated tax payments, reasonable cause, etc.
Research: Generally researches in appropriate and mandatory
reference materials such as the Probe and Response Guide, IRS
Publications, Tax Services, Internal Revenue Code (IRC), etc.,
to provide correct and complete response; generally does not
seek assistance from others even on complex and non-routine
issues. Increases customer confidence by citing specific
references and offering tax publications."
IRS officials characterized all five sets of performance standards as
results oriented to a great extent because they said each set was
related to IRS' strategic objectives and/or its mission. We agreed
with IRS officials' assessment for the sets of standards for the
"workload management" and "tax law" job elements. For example, the
set of standards related to the "tax law" job element included the
expectation that CSRs would provide "technically correct responses to
customer inquiries" and "increase customer confidence by citing
specific references and offering tax publications." Therefore, these
standards appeared directly related to IRS' mission of encouraging
voluntary compliance and warranting public confidence in the agency,
described the results expected, and established criteria to evaluate
performance. However, we characterized the remaining three sets of
performance standards as somewhat results oriented, because they did
not clearly describe the results CSRs were expected to work toward
and/or did not establish criteria for evaluating performance directly
associated with an expected result. For example, the set of
standards for the "IDRS inquiries and on-line resolution" job element
called for CSRs to "perform adjustments correctly" and "recognize
when related account data is pertinent and handle appropriately," but
these actions were not clearly linked to an expected result.
VIEWS ON THE RESULTS
ORIENTATION OF THE
PERFORMANCE STANDARDS FOR
IRS' REVENUE OFFICER
POSITION
------------------------------------------------------ Appendix IV:3.2
We and IRS officials reviewed the sets of employee performance
standards related to five job elements for the revenue officer
position: "investigation and analysis," "case decisions," "time and
workload management," "customer relations," and "accounts
maintenance." The standards described the types of actions revenue
officers were expected to take for that element. For example, the
performance standards for the "case decisions" job element were:
"Collectibility: Based on the available facts, determines the
proper case direction. Decisions to pursue full payment,
immediate or deferred, or suspend collection activity are based
on full consideration of the overall benefit to the government
and taxpayers' rights.
Collection tools: Based on case history, determines when levy,
seizure, or summons action is appropriate. Considers other
collection tools such as transferee, alter-ego, nominee, and
Offer-in-Compromise. May need occasional assistance making
final decision on use of transferee, alter-ego, etc.
Protection of the government's interest: Based on case history,
determines when actions are necessary to protect the
government's interest, including collection and assessment
statutes. Decides on the proper action to take, including but
not limited to filing and/or refiling liens, recommending suits,
100 percent penalty, prompts, quick, and IRC 6020(b)
assessments, fraud, or other appropriate referrals and waivers."
IRS officials characterized four of the sets of performance standards
as results oriented to a great extent, and said the set of standards
related to "customer relations" was somewhat results oriented. The
officials said that all five of the sets of standards were primarily
process measures, but said the standards were related to IRS' goals.
For example, they said that the standards for the "investigation and
analysis" job element allow IRS to collect the proper amount of tax
and increase productivity. In addition, the IRS officials pointed
out that the Technical and Miscellaneous Revenue Act of 1988
prohibits the IRS from evaluating employees on the basis of tax
enforcement results. Section 6231 of this act states that IRS cannot
use records of tax enforcement results to evaluate employees directly
involved in collection activities and cannot impose production quotas
or goals on individual employees. We characterized one set of
performance standards as results oriented to a great extent and the
remaining four sets as somewhat results oriented.
We agreed with IRS that the standards for the "case decisions" job
element were greatly results oriented because they were directly
related to IRS' mission to collect the proper amount of tax, clearly
described the results the revenue officers were expected to work
toward, and established criteria for evaluating the officers'
performance directly associated with the desired results. For
example, the standards stated that "decisions to pursue full payment,
immediate or deferred, or suspend collection activity are based on
full consideration of the overall benefit to the government and
taxpayers' rights." We characterized the remaining four sets of
performance standards as somewhat results oriented because they were
indirectly linked with IRS' mission or one of its three strategic
objectives, but did not clearly specify the results revenue officers
were expected to work toward or establish criteria for evaluating
performance in relation to those results. For example, the set of
standards for the "time and workload management" job element were
unclear about the results expected and did not establish criteria for
evaluating performance in relation to those results. Revenue
officers were to take "timely" initial, follow-up, and closing
actions, but the standards did not define what "timely" meant nor how
it was related to results.
IRS' COMMENTS AND OUR
EVALUATION
-------------------------------------------------------- Appendix IV:4
We met with IRS officials, including the National Director of the
Strategic Planning Division, on April 29, 1997. IRS also provided a
comment letter, which is reprinted in appendix VII. Generally, the
IRS officials said they were pleased that we characterized IRS' goals
and measures as results oriented.
In both the meeting and in written comments, IRS officials said that
we did not give the agency sufficient credit for doing a major
overhaul of its measures. However, one of the objectives of our
review was to describe each agency's performance measures as of
January 1, 1997, not to describe the specific efforts each agency had
taken in developing these measures. Furthermore, any efforts IRS had
made to overhaul its measures prior to January 1, 1997, should have
been reflected in the measures we reviewed.
IRS' written comments also state that IRS was concerned that we
created our own set of criteria in lieu of using GPRA criteria to
evaluate the agencies' goals and program performance measures. Other
than the fact that GPRA does not require all of an agency's goals or
program performance measures to be results oriented, which we
emphasize throughout the report, we do not believe that our criteria
differ substantially from GPRA requirements. For example, the report
of the Senate Committee on Governmental Affairs on GPRA states that
strategic or general goals should elaborate on the mission statement,
should constitute a specific set of objectives for the agency's
programs and operations, and must be expressed in a manner that
allows a future assessment of whether the goal is being achieved.\10
This is very similar to our criteria that greatly results-oriented
strategic goals (1) were linked closely to the agency's mission, (2)
provided a clear definition of what results were expected, and (3)
permitted assessment of progress in achieving these results.
IRS officials also did not believe that the draft report placed
enough emphasis on the fact that the Technical and Miscellaneous
Revenue Act of 1988 prohibits IRS from evaluating employees on the
basis of tax enforcement results. We agree that the Technical and
Miscellaneous Revenue Act of 1988 prohibits IRS from evaluating
employees on the basis of tax enforcement results and included this
information in this appendix.
--------------------
\10 Senate Report No. 103-58, June 16, 1993, page 24.
EPA'S MISSION, STRATEGIC GOALS,
AND PERFORMANCE STANDARDS FOR
SELECTED FRONTLINE REGULATOR
POSITIONS
=========================================================== Appendix V
EPA's mission is "to protect public health and to safeguard and
improve the natural environment--air, water, and land--upon which
human life depends." In doing so, EPA ensures that:
"Federal environmental laws are implemented and enforced fairly,
effectively, and cost effectively;
environmental protection is an integral consideration in U.S.
policies concerning economic growth, energy, transportation,
agriculture, industry, international trade, and natural
resources;
national efforts to reduce environmental risk are based on the best
available scientific information; and
all parts of society--business, state and local governments,
communities, citizens--have full access to information so that
they can become full participants in preventing pollution and
protecting human health and the environment."
We assessed the results orientation of EPA's strategic goals and
employee performance standards according to the criteria presented in
the scope and methodology appendix of this report. Because we
characterized the extent to which each agency goal was results
oriented, the application of these criteria represented a higher
standard of results orientation than required by the Results Act.
The Act does not require that all of an agency's goals be results
oriented. Also, GPRA does not cover employee performance standards.
We judged each set of performance standards according to the
expectation set by the President in his March 4, 1995, memorandum
that these standards should "focus on results, not process and
punishment." Similar to the way in which GPRA views agencies' goals
and performance measures, we do not believe that each such standard
must be results oriented to achieve the President's objective.
We applied our criteria to the agencies' written strategic goals and
employee performance standards and did not consider any supplementary
or contextual information. In some cases an agency's goals and
standards appeared results oriented on the surface, but closer
examination and strict application of our criteria resulted in a
lower characterization. We recognize that other information outside
of the written statements and outside of our review may have existed
that could have resulted in our characterizing certain goals or sets
of standards differently.
Both we and EPA officials characterized the extent of results
orientation of EPA's strategic goals and employee performance
standards for selected frontline regulator positions. As shown in
table V.1, there were differences in how we and EPA officials viewed
the goals and performance standards. Generally, we agreed that most
of EPA's strategic goals were greatly results oriented. However, we
sharply disagreed regarding the extent to which the performance
standards were results oriented. EPA officials characterized most of
the 11 sets of performance standards as greatly results oriented, but
we characterized most of them as results oriented to little or no
extent.
Table V.1
Differences Between Our and EPA
Officials' Characterizations
Little Little
A great Some or no A great Some or no
Total extent extent extent extent extent extent
---------- -------- -------- -------- -------- -------- -------- --------
Strategic 10 10 0 0 8 1 1
goals\a
Performanc 11 9 2 0 1 2 8
e
standards
--------------------------------------------------------------------------------
\a EPA did not have program performance measures as of January 1,
1997.
Source: Agency and GAO analysis.
VIEWS ON THE RESULTS
ORIENTATION OF EPA'S STRATEGIC
GOALS
--------------------------------------------------------- Appendix V:1
As of January 1, 1997, EPA had 10 strategic goals, which were:
1. "Clean air: Every American city and community will be free of
air pollutants at levels that cause cancer or respiratory and other
health problems. The air will be clearer in many areas and life in
damaged forests and polluted waters will rebound as acid rain is
reduced.
2. Clean and safe water: America's rivers, lakes and coastal waters
will support healthy communities of fish, plants and other aquatic
life and will support uses such as fishing and swimming. Wetlands
will be protected and rehabilitated to provide wildlife habitat,
reduce floods and improve water quality. Ground waters will be
uncontaminated and every American public water system will provide
water that is safe to drink all the time.
3. Safe food: All foods Americans produce or consume will continue
to be safe for all people to eat.
4. Toxic free communities and workplaces: By relying on pollution
prevention in the way we produce, consume, reuse and recycle
materials, all Americans will live and work in safe and healthy
communities.
5. Safe waste management: Wastes produced by every person and
business in America will be stored, treated and disposed of in ways
that prevent harm to people and other living things. Places in
America currently contaminated by hazardous or radioactive materials
will not endanger public health nor the natural environment and will
be restored to uses desired by surrounding communities.
6. Reducing global and transboundary environmental risks: The
United States and other nations will eliminate significant risks to
human health and ecosystems arising from climate change,
stratospheric ozone depletion and other environmental problems of
global concern.
7. Empowering people with information and education and expanding
their right to know: Information on the state of the environment
will be user-friendly and widely accessible, increasing opportunities
for Americans to learn about the environment and natural resources
and fostering environmental stewardship through information exchange
within and among communities and across economic sectors.
8. Develop tools for the future: EPA will develop innovative tools
and approaches that will permit environmental protection programs to
transcend the simple additivity of individual media-based programs
and move the Agency towards truly integrated and continually
improving environmental protection programs. The EPA will enhance
its science and research program in order to better anticipate
environmental issues coming over the horizon and to find early
solutions.
9. Provide a credible deterrent: The EPA will continue to implement
programs and monitor compliance to ensure that environmental gains
that have been achieved are not lost.
10. Effective management: EPA will establish a management
infrastructure that will set and implement the highest quality
standards for effective internal management and fiscal
responsibility."
EPA officials characterized each of these goals as results oriented
to a great extent because the goals described the results the agency
was trying to achieve. For example, EPA officials said that the goal
regarding "clean and safe water" was greatly results oriented because
it addressed the environmental result of improving or maintaining
water quality so that aquatic communities will be healthy, human uses
will be supported, drinking water will be safe, and ground water will
be uncontaminated. EPA officials said that the goal to "provide a
credible deterrent" was greatly results focused in two ways. First,
they said that the enforcement program targets industries and
facilities for attention based in part on the expected degree of
environmental benefit and risk reduction. Second, they cited the
deterrent effect of ensuring that the regulated community perceives
that compliance will be strongly pursued by EPA, so that overall
environmental results are enhanced due to voluntary compliance.
Finally, EPA officials said that the goal for "effective management"
was greatly results oriented because it complied with GPRA's
requirement to measure performance in all areas, and that the results
of the management area were primarily outputs focused on customer
service.
We agreed with EPA officials that 8 of the 10 goals were results
oriented to a great extent. In addition to the reasons cited by EPA
officials, we also viewed the eight goals as relating to EPA's
mission and permitting assessment of the agency's progress in
achieving desired results. However, we characterized the goal
"developing tools for the future" as results oriented to some extent
because it did not provide a clear definition of the results expected
and it was unclear how EPA would assess progress toward achieving
this goal. We characterized the remaining goal for "effective
management" as results oriented to little or no extent because it was
focused internally and therefore did not relate to the agency's
mission.
EPA HAD NO PROGRAM PERFORMANCE
MEASURES IN PLACE AT THE TIME
OF OUR REVIEW
--------------------------------------------------------- Appendix V:2
As of January 1, 1997, EPA had not established sets of program
performance measures to assess progress in meeting the agency's
goals. According to EPA officials, the agency was establishing
quantitative "objectives" that describe what EPA wants to achieve by
an agreed-upon target date. EPA plans to have a full set of program
performance measures relating to these objectives by July 1997.
Although EPA had no official program performance measures as of
January 1, 1997, the agency has been working since 1992 to develop
long-range national environmental goals with results-oriented
performance goals and measures. EPA officials said that some of the
performance measures for the national goals program may be used as
measures for the agency's strategic goals.
VIEWS ON THE RESULTS
ORIENTATION OF EPA'S
PERFORMANCE STANDARDS FOR
SELECTED FRONTLINE REGULATOR
POSITIONS
--------------------------------------------------------- Appendix V:3
In contrast to strategic goals and program performance measures, the
administration's regulatory reinvention initiative, not GPRA, set the
expectation for agencies to focus on results when measuring the
performance of their frontline regulators. EPA identified two
frontline regulator positions for this review, criminal investigators
and environmental scientists.\1 As of January 1, 1997, there were the
equivalent of about 225 full-time criminal investigators at EPA, and
the equivalent of about 1,225 full-time environmental scientists
working out of EPA's regional offices.\2 EPA criminal investigators
investigate allegations or suspected violations of criminal
environmental laws, and environmental scientists implement and
enforce environmental regulations.
EPA officials characterized 9 of the 11 sets of performance standards
reviewed for these 2 positions as results oriented to a great extent,
and 2 sets of standards as somewhat results oriented. They believed
that most of the sets of standards clearly described the results that
should occur through performance of the standards. However, we
characterized eight of the sets of employee performance standards we
reviewed as results oriented to little or no extent, two as somewhat
results oriented, and one as results oriented to a great extent.
--------------------
\1 Although the President's March 4, 1995, memorandum on his
regulatory reinvention initiative did not define "frontline
regulators," we defined them as regulatory staff who work directly
with the regulated community by enforcing regulations.
\2 EPA also has environmental scientists working in its headquarters
offices, but EPA officials said that these environmental scientists
were not frontline regulators.
VIEWS ON THE RESULTS
ORIENTATION OF EPA'S
CRIMINAL INVESTIGATOR
PERFORMANCE STANDARDS
------------------------------------------------------- Appendix V:3.1
We and agency officials reviewed five job elements for the criminal
investigator position:\3
1. "Professional attributes: In the performance of duties, displays
and possesses professional attributes and conduct which are a benefit
to the agency and which foster good working relationships within and
outside the office. Such professional attributes include physical
conditioning, appearance and demeanor, proficiency with weapons and
equipment, work attitude, and self-improvement measures.
2. Communication (oral and written): Prepares and presents written
and oral communication and reports as needed to the public; Regional
officials; Criminal Investigation Division (CID) Headquarters;
supervisors, the Director, CID; prosecuting attorneys and other
officials.
3. Knowledge of directives, laws, regulations, and procedures:
Maintains and demonstrates knowledge of programs, missions, and
operations within EPA; CID policy; and pertinent environmental laws
and regulations.
4. Case investigation: Conducts complex investigations within an
assigned area of responsibility.
5. Liaison: Identifies, establishes and maintains contacts with
other federal, state, and local law enforcement and regulatory
agencies, EPA Regional officials, public institutions, the private
sector, and other CID offices."
EPA officials classified all but one of the sets of performance
standards related to these elements as results oriented to a great
extent. For example, EPA officials said that the set of standards
related to the "communication" job element was greatly results
oriented because presentations that were thorough, easily understood,
and appropriately targeted would result in an accurate understanding
of the message being delivered. EPA officials said the set of
standards for the "liaison" job element was somewhat results oriented
because regular communications with a broad base of external contacts
provided the intermediate result of a highly efficient and timely
resource for the agency.
However, we considered four of the sets of performance standards (for
the "professional attributes," "communication," "knowledge of
directives," and "liaison" job elements) as results oriented to
little or no extent because these standards were not tied to any
expected results and therefore had no criteria for evaluating
performance associated with expected results. For example, the
standards for the "professional attributes" job element included
maintaining "a good level of physical conditioning," displaying "an
appearance and demeanor which is consistent with the position," and
displaying "a positive attitude towards the job." We did not believe
that these standards clearly described the results that criminal
investigators were expected to work toward or were linked to any
environmental results that EPA was trying to achieve. We viewed the
remaining set of performance standards for the "case investigation"
job element as somewhat results oriented because although it was
linked to EPA's goal to "provide a credible deterrent," the set of
standards did not clearly describe the results the investigator was
expected to work toward. For example, investigators were expected to
"identify and preserve physical evidence in accordance with currently
acceptable standards," but the result expected from following the
standards for evidence was not specified.
--------------------
\3 We did not include the "special projects" job element in our
review because of the general nature of these standards and their
variability between criminal investigators.
VIEWS ON THE RESULTS
ORIENTATION OF EPA'S
PERFORMANCE STANDARDS FOR
THE ENVIRONMENTAL SCIENTIST
POSITION
------------------------------------------------------- Appendix V:3.2
The environmental scientist position had six job elements:
1. "Customer service: To meet customer expectations with respect to
products or services provided.
2. Communication: To communicate effectively with internal and
external customers in accomplishing the organization's work.
3. Teamwork: To contribute individual talents and energies toward
achieving team goals and objectives.
4. Grants/cooperative agreements: To coordinate the negotiation and
award of grants, cooperative agreements and/or interagency
agreements, and/or monitor work plans, progress, and activities
performed under these arrangements for acceptability and achievement
of objectives.
5. Enforcement/compliance: To ensure compliance with asbestos
regulations under the Clean Air Act, National Emissions Standards for
Hazardous Air Pollutants, Toxic Substances Control Act, and related
legislation and regulations through effective management of
compliance assistance and enforcement activities while meeting
programmatic goals and objectives.
6. Inspections: To conduct compliance inspections consistent with
Agency priorities and programmatic goals and objectives."
EPA officials classified all but one of the sets of performance
standards for these elements as results oriented to a great extent.
For example, they said that the set of standards for the "customer
service" job element covered all the major indicators of providing
highly efficient customer services, including mutual agreement
between the customer and service provider on product delivery dates,
frequent feedback to customers, and checking to determine whether
customer needs were met. EPA officials said the set of standards
related to the "communication" job element was somewhat results
oriented because the set of standards did not cover whether the
communications were helpful to customers or could have been improved,
leaving a somewhat incomplete set regarding whether effective two-way
communication had been achieved.
We agreed that the set of standards associated with the environmental
scientist "enforcement" job element was greatly results oriented
because it was directly related to EPA's mission and strategic goals,
clearly described the results scientists were expected to work
toward, and established criteria for evaluating their performance
directly associated with the result. However, we characterized the
sets of standards for four of the job elements ("customer service,"
"communication," "teamwork," and "grants/cooperative agreements") as
results oriented to little or no extent because they did not
establish what results would be achieved through performance of the
standards and did not establish results-oriented criteria for
evaluating performance. For example, the standards for the
"communication" job element included "conducts internal and external
communications through appropriate channels;" "writes clearly and
concisely, in compliance with established formats, and neat in
appearance;" and "promptly handles and responds to phone, local area
network, or voice mail messages." It was not evident what
environmental results were anticipated by performing these
communications in accordance with the standards. We classified the
remaining set of standards for the "inspections" job element as
somewhat results oriented. This set was somewhat linked to EPA's
mission of protecting public health, but the results scientists were
expected to work toward and the criteria EPA would use to evaluate
performance toward achieving these results were unclear.
EPA'S COMMENTS AND OUR
EVALUATION
--------------------------------------------------------- Appendix V:4
We met with EPA officials, including the Acting Deputy Director of
the Office of Planning, Accountability, and Analysis and the Acting
Director of the Strategic Planning and Policy Services Division, on
April 28, 1997, to obtain their comments. The EPA officials agreed
with the draft report's overall message that regulatory agencies face
significant barriers in becoming results oriented and noted that EPA
faces a challenge in trying to determine what portions of changes in
environmental conditions can realistically be considered a result of
EPA's programs. In fact, the officials said they believed the draft
report should more strongly reflect the barriers that agencies face
in order to emphasize that focusing on results will take time and, in
some situations, may be difficult to do. We agree that managing for
results may be difficult in some cases and, therefore, included more
information in the results-in-brief section of the final report
describing the barriers agencies identified.
EPA officials said that some EPA programs were upgrading their
ability to report on environmental results. They said that other EPA
programs have long had in place the environmental monitoring and
reporting systems needed to measure results and that the prospect of
meeting GPRA reporting requirements would not be an insurmountable
challenge for these programs. For example, the officials said that
the "criteria air pollutants program" has consistent, quality-assured
data on both environmental quality and emissions from many of the
major sources of air pollution. They said that these emissions data
link EPA's regulatory actions to environmental results. However,
they said that some EPA programs (those that provide services) will
have more difficulty determining outcome-related results, as opposed
to output-related (i.e., activity) results. The EPA officials also
said that the agency has made significant progress in revising its
goals and developing performance measures since the time of our
review in early January 1997.
EPA officials said that although it is appropriate to hold frontline
regulators accountable for organizational results, we should not have
expected agencies to have results-oriented performance standards that
met our criteria. They said that the Office of Personnel Management
and the chief financial officers and human resource communities have
not decided how to translate the mission of an agency into measurable
standards that are results oriented at the individual level. They
also said that we based our characterizations on a definition that
could not be found in any law, regulation, or guidance.
Although there may not yet be a consensus among the relevant
stakeholders or a formal definition regarding how the measurement of
employee performance should be linked to agencies' results, the
President's memorandum nevertheless requires that agencies change the
way they measure both agency and frontline regulators' performance
"so as to focus on results, not process and punishment." In addition,
in a June 15, 1995, report to the President, EPA stated that it
reviewed its existing employee performance standards and said that it
had made some changes to those standards and had plans for further
changes. Nowhere in the June 1995 report did EPA assert, as it does
now, that it is inappropriate to expect the agency's employee
performance standards to focus on results. To develop our criteria
for employee performance standards, we considered Office of Personnel
Management guidance on establishing employee performance standards in
conjunction with GPRA criteria for goals and performance measures as
a model.
The officials also said that one of the reasons we may have differed
with agencies' conclusions about the results orientation of their
employee performance standards was the fact that we did not consider
additional contextual information when we reviewed the standards. As
stated in the draft report, we applied our criteria to the agencies'
performance standards without other contextual information. We
believe that in order to characterize each agency's performance
standards using a fair and balanced approach, it was necessary to
limit our scope to the standards themselves. Otherwise, we could
have had additional information about one aspect of an agency's
performance standard but not another, which would have resulted in an
inconsistent application of our criteria. In addition, Office of
Personnel Management guidance states that research has shown that
specific, hard goals result in higher performance than vague or
abstract goals, or no goals at all. Therefore, the President's
initiative focusing standards on results could improve regulators'
performance in obtaining desired results.
SCOPE AND METHODOLOGY
========================================================== Appendix VI
To describe the agencies' strategic goals and program performance
measures, we first obtained copies of the goals and measures the
agencies were using as of January 1, 1997. To describe the employee
performance standards for selected regulator positions at these
agencies, we first asked officials in each agency to identify
frontline regulator positions that they felt would fairly represent
the actions of their agencies' regulators.\1 We then obtained copies
of the performance standards being used as of January 1, 1997, for
the selected positions.
To describe the extent to which agency officials and we believed that
these goals, program performance measures, and employee performance
standards focused on results, we developed a standardized data
collection framework. We provided the initial framework to agency
officials and told them that the criteria (as shown in table VI.1)
were under development and would be further refined and revised as we
progressed through the assignment. We left it up to the agency
officials to decide whether they wanted to use these criteria,
because we were primarily interested in obtaining their perspectives
about the extent to which and, more importantly, why they considered
their agencies' goals, measures, and standards to be results
oriented. At a minimum, however, we asked agencies to characterize
their goals, program performance measures, and employee performance
standards as results oriented to a great extent, some extent, or
little or no extent.
Table VI.1
Initial Framework Provided to Agency
Officials
To be considered results oriented to a great
extent:
------------------ --------------------------------------------------
Strategic goals Must provide a clear definition of what results,
effects, or impact (i.e., outcomes) are expected
from the regulatory program, and permit assessment
of progress in meeting the goal.
Program Should (1) assess the actual results, effects, or
performance impact of a program activity compared with its
measures intended purpose; (2) be linked to a goal; (3)
permit comparison of organizational performance
over time; and (4) if appropriate, include a range
of measures such as quality, quantity, cost, and
timeliness. (A range of measures would be
appropriate when the goal is multipart and not
self-measuring, or when the agency has set forth
several measures and we can specify gaps in the
measures used.)
Employee Should describe the objective or overall results
performance which the frontline regulators are expected to
standards accomplish, and set forth tangible or demonstrable
criteria for evaluating each performance element
directly associated with desired results. (It
should enable a "cold reader" to clearly
understand how an individual would have to perform
to meet the standard.)
----------------------------------------------------------------------
In our analysis, we also characterized the agencies' goals, program
performance measures, and employee performance standards as results
oriented to a great extent, some extent, or little or no extent. As
we carried out these judgments, we expanded our initial framework
somewhat as we learned more about how agencies' goals or measures
were structured. As a result, the framework we used was in the end
somewhat more detailed than the initial framework we provided to
agency officials. We did not believe these differences materially
affected the outcome of our analysis. Specifically,
We characterized an agency's strategic goal as results oriented to
a great extent if it was clearly linked to the agency's mission,
provided a clear definition of what results were expected, and
permitted assessment of progress in achieving these results.
(For example, IRS' goal to increase taxpayer compliance was
clearly related to IRS' mission to collect the proper amount of
tax revenue. It provided a clear definition of the expected
results--"to encourage and assist taxpayers to voluntarily file
timely and accurate returns and to pay on time, and, if
taxpayers do not comply, to take appropriate compliance
actions." Finally, it is possible to assess progress toward
increasing compliance.) If a goal was not linked to an agency's
mission, did not clearly define desired results, and did not
permit assessment of progress toward achieving these results, we
considered it results oriented to little or no extent. We
classified goals that fell between these two categories as
results oriented to some extent.
We characterized an agency's set of program performance measures as
results oriented to a great extent if the measures (1) were
clearly and unambiguously linked to the associated goal (or the
agency's mission, if the goal was not results oriented); (2)
provided relevant information for assessing progress toward
achieving the goal; and (3) addressed all aspects of the goal.
We classified sets of program performance measures that were not
linked to a goal or mission, that would not provide relevant
data for an agency to assess progress toward a result, and did
not address all aspects of the goal as results oriented to
little or no extent. We considered sets of measures that fell
between these categories as results oriented to some extent.
We considered a position's performance standards to be greatly
results oriented if they clearly described the results that the
frontline regulators were expected to work toward, these results
were clearly and unambiguously linked to the agency's mission or
goals, and the standards established criteria for evaluating
performance directly associated with the desired results. We
classified performance standards as having little or no results
orientation if they were unclear about the results that the
frontline regulators were expected to accomplish and did not
establish criteria for evaluating performance related to
expected results. We considered performance standards that fell
between these two categories as results oriented to some extent.
We did not include performance standards associated with "other
duties as assigned" or "special projects" in this review because
of the general nature of these standards.
There are several aspects of our work which need to be emphasized to
clarify the significance of our results. First, our review was not
intended to determine agencies' compliance with GPRA; compliance was
not required at the time of our review. Furthermore, because we
characterized the extent to which each agency goal and set of
performance measures was results oriented, the application of these
criteria represented a higher standard of results orientation than
will be required by GPRA. As noted previously, the Act does not
require that all of an agencies' goals or measures be results
oriented and the President's March 4, 1995, memorandum does not
explicitly require that each employee performance standard be results
oriented to achieve the President's objective.
In applying our criteria for results orientation to the goals,
performance measures, and performance standards, we limited our
analysis to the documents describing those goals, measures, and
standards as of January 1, 1997, as provided by the agencies. As a
result, we did not consider additional contextual information that
may have existed and could have resulted in our reaching different
conclusions about results orientation if it had been a part of our
review. For example, it is possible that we might have considered
the goals or measures for an agency's training or inspection program
as more results oriented if we had obtained information about how
they ultimately produced mission-oriented results.
We did not judge whether the agencies' goals and program performance
measures were appropriate or achievable, only whether they appeared
to be results oriented. We also did not assess whether better
program performance measures existed that the agencies could have
used. Because many of these agencies were making changes to those
goals and measures and some of the performance standards at the time
of our review, our and the agencies' assessments should be viewed as
a snapshot at a particular point in time.
Because we reviewed the performance standards for only a few
frontline regulator positions at each agency, the information we
obtained is not representative of all such positions. Rather, the
discussion of these standards should be viewed as examples, or case
studies, of performance standards used by the agencies as of January
1, 1997. Each of the positions that were assessed contained several
job elements (e.g., oral and written communication or customer
service) that, in combination, constituted the major responsibilities
for the position. Each job element, in turn, had a set of
performance standards that described the level of performance that
the employee needed to meet to be considered successful in that
position. Table VI.2 shows the seven frontline regulator positions
that we focused on in assessing employee performance standards. The
positions had a total of 30 job elements, each accompanied by a set
of performance standards associated with that element.
Table VI.2
Frontline Regulator Positions and Number
of Job Elements as of January 1, 1997
Position (and Number
number of of job
frontline elemen
Agency\a regulators) Description of position ts\b
---------- ---------------- -------------------------------- ------
OSHA Compliance Conduct workplace safety and 3
safety and health inspections.
health officer
(930)\c
FAA Air traffic Regulate air traffic throughout
control the nation. 3
specialist
(12,900)
Aviation safety Inspect and certify aircraft for
inspector safe operations. 3
(1,525)
IRS Customer service Provide information and
representative direction on tax issues to the 5
(5,750) public.
Revenue officer Collect delinquent taxes and 5
(7,360) secure delinquent returns.
EPA Criminal Investigate allegations or
investigator suspected violations of criminal 5
(225)\d laws.
Environmental Implement and enforce
scientist environmental regulations. 6
(1,225)\c
======================================================================
Total \e 30
----------------------------------------------------------------------
\a FDA is not included because, as of January 1, 1997, the agency was
using performance standards that were customized for individual
employees.
\b We reviewed the sets of performance standards that were associated
with each of these job elements.
\c Includes both safety and health inspector positions.
\d The number of EPA frontline regulators (for the criminal
investigator and environmental scientist positions) were full time
equivalent (FTE) staffing levels, not number of frontline regulators.
In addition, the number of FTEs in the environmental scientist
position includes only environmental scientists in EPA's regional
offices. EPA officials said that they would not classify
environmental scientists who work in headquarters as frontline
regulators.
\e Not applicable because the numbers listed include both number of
staff and FTEs and therefore these amounts cannot be added to a
meaningful total.
Source: Agency information.
To describe the aids and barriers that officials in the agencies said
they faced in attempting to focus on results, we asked officials in
each agency to describe the aids and barriers they had experienced.
We also reviewed related documents that described any problems or
successes agencies had encountered.
(See figure in printed edition.)Appendix VII
--------------------
\1 Using our definition of "frontline regulators"--regulatory staff
who work directly with the regulated community by enforcing
regulations--the agencies designated which positions they considered
"frontline regulator" positions.
COMMENTS FROM OSHA AND IRS
========================================================== Appendix VI
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
MAJOR CONTRIBUTORS TO THIS REPORT
======================================================== Appendix VIII
GENERAL GOVERNMENT DIVISION
Curtis Copeland, Assistant Director, Federal Management and Workforce
Issues
Susan Ragland, Evaluator-in-Charge
Steven Lozano, Senior Evaluator
Thomas Beall, Technical Analyst
Kiki Theodoropoulos, Communications Analyst
Joseph S. Wholey, Senior Advisor for Evaluation Methodology
*** End of document. ***