Performance Management Systems: IRS's Systems for Frontline	 
Employees and Managers Align with Strategic Goals but		 
Improvements Can Be Made (12-JUL-02, GAO-02-804).		 
                                                                 
The Internal Revenue Service (IRS) established critical job	 
responsibilities for group managers and frontline employees that 
align with each of the agency's three strategic goals. In the	 
group managers' performance management system, all of the	 
supporting behaviors clearly align with the critical job	 
responsibilities. However, for frontline employees, the 	 
supporting behaviors do not always align with IRS's description  
of the critical job responsibilities. Misalignments occur when	 
(1)  supporting behaviors reflect concerns not expressed in the  
description, (2) supporting behaviors that relate to a		 
responsibility are located under other responsibilities, or (3)  
no supporting behavior exists to support the description. Raters 
provided feedback on the three critical job responsibilities	 
related to IRS's organizational goals in 90 percent or more of	 
evaluations and provided feedback on leadership and equal	 
employment opportunity in 70 percent of the evaluations. IRS	 
senior executives have no firm plans to monitor how well the	 
group managers' and frontline employees' systems are being	 
implemented or to assess whether changes need to be made, even	 
though IRS's management processes call for obtaining data on how 
well programs are achieving their goals.			 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-02-804 					        
    ACCNO:   A03908						        
  TITLE:     Performance Management Systems: IRS's Systems for	      
Frontline Employees and Managers Align with Strategic Goals but  
Improvements Can Be Made					 
     DATE:   07/12/2002 
  SUBJECT:   Agency missions					 
	     Performance measures				 
	     Strategic planning 				 
	     Human resources utilization			 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-02-804

Report to Congressional Requesters

United States General Accounting Office

GAO

July 2002 PERFORMANCE MANAGEMENT SYSTEMS

IRS*s Systems for Frontline Employees and Managers Align with Strategic
Goals but Improvements Can Be Made

GAO- 02- 804

Page i GAO- 02- 804 Performance Management Systems Letter 1

Results in Brief 3 Background 5 Critical Job Responsibilities Align with
Strategic Goals but Do Not

Always Align with Other Elements of the Frontline Employee Performance
Management System 8 Enforcement Group Managers* Commitments Align with
IRS*s

Strategic Goals but Are Not Always Written So that Raters Can Hold
Managers Accountable 12 Raters Generally Provided Feedback to Enforcement
Group

Managers on Meeting Critical Job Responsibilities and Commitments but
Provided More Extensive Feedback on Some Critical Job Responsibilities
Than Others 17 IRS Has Made Progress but Could Further Enhance Its New

Managers* and Frontline Employees* Performance Management Systems 21
Conclusions 25 Recommendations for Executive Action 26 Agency Comments and
Our Evaluation 26

Appendix I Objectives, Scope, and Methodology 28 Objectives 28 Scope and
Methodology 28

Appendix II GAO Sampling Methodology 31 Sample Designs 31 Sample
Disposition 32 Sampling Error 33 Nonsampling Error 33

Appendix III Samples of Manager and Frontline Employee Evaluation Forms 35

Appendix IV Examples of Evaluations Addressing Group Managers*
Responsibilities 43 Contents

Page ii GAO- 02- 804 Performance Management Systems Appendix V
Distribution of Raters* Comments for Each Supporting

Behavior 44

Appendix VI Examples of Group Managers* Commitments 46 Commitments that
Were Specific, Outcome or Output Oriented, or

Easily Monitored 46 Commitments that Were Not Specific, Outcome or Output

Oriented, or Easily Monitored 47

Appendix VII Comments from the Internal Revenue Service 49

Appendix VIII GAO Contacts and Staff Acknowledgments 51 GAO Contacts 51
Acknowledgments 51

Tables

Table 1: Examples of Enforcement Group Managers* Commitments that Align
with IRS*s Strategic Goals 13 Table 2: Percentage of Commitments
Reflecting Critical Job

Responsibilities for Fiscal Years 2000 and 2001 14 Table 3: Percentage of
Commitments Not Specific,

Outcome/ Output Oriented, or Easily Monitored for Fiscal Years 2000 and
2001 15 Table 4: Estimated Percentage of Fiscal Years 2000 and 2001

Written Evaluations in Which Raters Assessed Critical Job Responsibilities
by Number of Responsibilities 18 Table 5: Estimated Percentage of
Enforcement Group Managers

Reporting Verbal Feedback by Number of Critical Job Responsibilities for
Fiscal Year 2001 19 Table 6: Estimated Percentage of Fiscal Year 2000 and
2001

Evaluations Assessing Enforcement Group Managers* Performance in Each
Critical Job Responsibility 19 Table 7: Estimated Percentage of
Enforcement Group Managers

Reporting Feedback to a Great or Very Great Extent for the Critical Job
Responsibilities for Fiscal Year 2001 20

Page iii GAO- 02- 804 Performance Management Systems

Table 8: Estimated Percentage of Enforcement Group Managers Reporting that
the Respective Critical Job Responsibilities Are Very Important or
Important to Their Raters for Fiscal Year 2001 21 Table 9: Disposition of
Sample Cases for Our Review of Group

Manager Evaluations 32 Table 10: Disposition of Sample Cases for Our
Survey of Group

Managers* Perceptions of Raters* Feedback 33 Table 11: Examples of
Critical Job Responsibilities, Supporting

Behaviors, and Evaluation Narratives for Group Managers 43 Table 12:
Distribution of Raters* Comments 44

Figures

Figure 1: Alignment of Strategic Goals, Critical Job Responsibilities, and
Supporting Behaviors for Enforcement Group Managers 9 Figure 2:
Misalignment of Critical Job Responsibility, Narrative

Description, and Supporting Behaviors for Revenue Agents 11

Abbreviations

CQMS Collections Quality Management System IDPs Individual Development
Plans IRM Internal Revenue Manual IRS Internal Revenue Service NTEU
National Treasury Employee Union RRA 98 Restructuring and Reform Act of
1998 TFRP Trust Fund Recovery Penalty

Page 1 GAO- 02- 804 Performance Management Systems

July 12, 2002 The Honorable Bill Thomas Chairman, Committee on Ways and
Means House of Representatives

The Honorable Amo Houghton Chairman, Subcommittee on Oversight Committee
on Ways and Means House of Representatives

Performance management systems can be powerful tools in helping an agency
achieve its mission and ensuring employees at every level of the
organization are working toward common ends. Performance management
systems should help employees understand their responsibilities and how
their day- to- day work contributes toward meeting their agency*s
strategic goals as well as providing a mechanism for giving employees
candid, specific feedback on how well they are meeting their rater*s
expectations. For agencies like the Internal Revenue Service (IRS) that
are undergoing a cultural change, performance management systems help
reinforce behaviors and actions that support the agency*s mission. 1

In February 2000, IRS implemented a new performance management system for
its executives and managers and in October 2001 implemented a new
performance management system for frontline employees. These systems were
built upon IRS*s three strategic organizational goals** topquality service
to each taxpayer in every interaction,* *top- quality service to all
taxpayers through the fair and uniform application of the law,* and
*productivity through a quality work environment** and the corresponding
balanced performance measures of customer satisfaction, business results
(quality and quantity), and employee satisfaction that are applied to all
organizational units, from IRS- wide down to the group level. IRS
executives and managers are evaluated on five critical job
responsibilities (customer satisfaction, business results, employee
satisfaction, leadership, and equal employment opportunity) and on

1 The mission of IRS is to *provide American taxpayers top quality service
by helping them understand and meet their tax responsibilities, and by
applying the tax laws with integrity and fairness to all.*

United States General Accounting Office Washington, DC 20548

Page 2 GAO- 02- 804 Performance Management Systems

written commitments they develop on the specific actions they will take to
support IRS*s strategic goals over the course of the evaluation period. 2
Frontline employees are evaluated against five critical job
responsibilities, (customer satisfaction* knowledge, customer
satisfaction* application, business results* quality, and business
results* efficiency, and employee satisfaction* employee contribution).

Because of the challenging nature of implementing a new performance
management system, you asked us to assess how well IRS is aligning its
performance management systems with its strategic goals and is using these
systems to hold managers and employees accountable for meeting those
goals. Specifically, as agreed with your office, our objectives were to
determine (1) the extent to which group managers* and frontline employees*
critical job responsibilities and other elements of the employee
performance management systems are structurally aligned with IRS*s
strategic goals and organizational unit performance measures, (2) the
extent to which group managers* commitments align with IRS*s strategic
goals and are written so that raters can hold managers accountable for
meeting their commitments, (3) the extent to which raters provide feedback
to enforcement group managers on their performance in meeting critical job
responsibilities and commitments, and (4) whether IRS has adequate plans
to monitor and assess the implementation of its performance management
systems.

To address these objectives, we reviewed documents related to IRS*s
performance management systems and discussed the systems with officials
responsible for developing and implementing them. We performed qualitative
analyses of the extent to which the critical job responsibilities and
supporting behaviors for enforcement group managers and frontline
employees 3 align with IRS*s strategic goals and organizational unit
performance measures. We performed a content analysis of a representative
sample of the written evaluations for 126 enforcement group managers for
fiscal years 2000 and 2001. We also surveyed the enforcement group
managers about their perceptions of the verbal and

2 See Appendix IV for examples of evaluations addressing group managers*
critical job responsibilities. 3 Enforcement group managers are those
managers in IRS*s field offices that supervise frontline employees, such
as revenue agents who directly deal with taxpayers when auditing tax
returns and revenue officers who deal with taxpayers when collecting
unpaid taxes.

Page 3 GAO- 02- 804 Performance Management Systems

written feedback they received on their performance. 4 We used information
contained in IRS documents and guidance given to group managers and
frontline employees as criteria for assessing those systems.

We did our work between April 2001 and July 2002 in accordance with
generally accepted government auditing standards. A more detailed
description of our methodology is provided in appendix I and information
on our sampling methodology is provided in appendix II.

IRS has established critical job responsibilities for group managers and
frontline employees that align with each of the agency*s three strategic
goals. In the group managers* performance management system, all the
supporting behaviors* those actions and competencies that the enforcement
group manager is expected to demonstrate during the performance cycle*
clearly align with the critical job responsibilities.

However, for frontline employees, the supporting behaviors do not always
align with IRS*s description of the critical job responsibilities.
Misalignments occur when supporting behaviors reflect concerns not
expressed in the description, when supporting behaviors that relate to a
responsibility are located under other responsibilities, or when no
supporting behavior exists to support the description. For example, the
description of the customer satisfaction* application responsibility for
revenue agents states that employees should communicate with taxpayers in
a clear, user- friendly manner. However, the supporting behaviors focus
more on IRS*s procedural requirements, such as the quality of internal
work products, than on whether taxpayers received understandable audit
reports. In addition to the misalignments in evaluating supporting
behaviors, IRS has not fully aligned its frontline employee performance
management system with its organizational unit performance measures. These
misalignments can reduce the new system*s ability to promote frontline
employees behaviors that would help IRS achieve its mission.

4 We express the precision of the results of our analysis of group
managers* commitments and evaluations, and the survey (that is, the
sampling errors associated with these estimates) as 95 percent confidence
intervals. All percentage estimates dealing with our review of evaluations
have sampling errors of +/- 11 percentage points or less, unless otherwise
stated. All percentage estimates dealing with the group manager survey
have sampling errors of +/- 12 percentage points or less, unless otherwise
stated. Results in Brief

Page 4 GAO- 02- 804 Performance Management Systems

During both fiscal years 2000 and 2001, almost all of the enforcement
group managers* commitments that we reviewed were aligned with IRS*s
strategic goals. However, commitments were not always well written. About
85 percent of the commitments did not meet IRS*s criteria of being
specific, outcome or output oriented, or easily monitored, although almost
all met IRS*s criteria of being clear and achievable. Thus, even though
these commitments may encourage desired behaviors, raters may have
difficulty holding managers accountable for meeting the majority of their
commitments because of the way they were written.

Our review of enforcement group managers* evaluations showed that raters
generally provided feedback to managers on their performance in meeting
critical job responsibilities and commitments. For example, during fiscal
year 2000 and 2001, we estimate that more than 80 percent of the written
performance evaluations assessed managers* performance in meeting four or
more of the five critical job responsibilities. In addition, about 76
percent of managers* commitments were addressed in the written
evaluations. In our survey of enforcement group managers, more than an
estimated 60 percent of managers reported receiving verbal feedback in all
five of the critical job responsibilities, and about half of managers said
they received verbal feedback on how well they met most or all of their
commitments.

According to our analysis of written evaluations, raters provided feedback
on the three critical job responsibilities related to IRS*s organizational
goals in 90 percent or more of evaluations and provided feedback on
leadership and equal employment opportunity in about 70 percent of the
evaluations. In our survey, enforcement group managers reported that
raters addressed business results more often than employee satisfaction or
customer satisfaction during both verbal and written feedback. IRS senior
executives said the relative emphasis raters were placing on the critical
job responsibilities was in line with their expectations, and noted that a
heavy emphasis on a subset of critical job responsibilities could lead
managers away from the balanced behaviors IRS seeks to achieve.

IRS senior executives have no firm plans to monitor how well the group
managers* and frontline employees* systems are being implemented or to
assess whether changes need to be made, even though IRS*s management
processes call for obtaining data on how well programs are achieving their
goals. As IRS continues to implement its performance management systems,
IRS can ensure the systems continue to meet its needs by monitoring how
well the systems provide useful feedback to managers and employees about
their performance and assessing whether the systems

Page 5 GAO- 02- 804 Performance Management Systems

encourage activities and behaviors consistent with its goals. Although IRS
senior executives are currently satisfied with the implementation of the
new systems, if the difference in raters* emphasis on critical job
responsibilities grows over time, IRS could find that enforcement group
managers are receiving a skewed perception of the behaviors that IRS
values. In addition, IRS can begin planning for a more thorough assessment
of how well the systems are achieving their overall objectives, such as
improving communications, encouraging performance that actually leads to
organizational goals, and ensuring fair and consistent performance
evaluations.

We are making recommendations to the Commissioner of Internal Revenue that
IRS improve the alignment between strategic goals and elements of the
frontline employee performance management system and ensure the new
performance management systems are working as intended. In a July 5, 2002
letter, the Commissioner agreed with the recommendations contained in this
report. (See agency comments and our evaluation section and app. VII.)

The enactment of the IRS Restructuring and Reform Act of 1998 (RRA 98)
signaled congressional concern that the IRS had been emphasizing revenue
production at the expense of fairness and consideration of taxpayers. In
response, the Commissioner of Internal Revenue sought to transform the
agency*s culture to one that more equally embraced responding to
legitimate taxpayer needs, ensuring compliance with tax laws, and
considering employee needs as core organizational values. IRS has
undertaken a number of long- range initiatives to make this transformation
a reality, including

 establishing three strategic goals of *top- quality service to each
taxpayer in every interaction,* *top- quality service to all taxpayers
through the fair and uniform application of the law,* and *productivity
through a quality work environment;*  developing a strategic planning and
budgeting process that provides a

framework for operating units to develop goals and action- oriented
business plans that support the strategic goals;  using balanced
performance measures corresponding to each of the three

strategic goals* customer satisfaction, business results in terms of
quality and quantity, and employee satisfaction* to measure the
performance of every organizational unit in achieving those strategic
goals; and Background

Page 6 GAO- 02- 804 Performance Management Systems

 establishing new performance management systems 5 designed to promote
employee behaviors and actions that support IRS*s strategic goals and the
operational business plans that lay out each operating division*s strategy
for achieving IRS*s strategic goals.

IRS*s goal in redesigning, or modernizing, its performance management
systems was to make the process more meaningful for all participants. The
new managers* and frontline employee performance management systems were
required to include goals and objectives at the organizational, group, or
individual level, and a retention standard 6 to ensure that managers and
frontline employees administer the tax laws fairly and equitably, protect
taxpayers* rights, and treat taxpayers with honesty, integrity, and
respect. The systems were intended to provide a framework for managers and
employees to improve communications, coordinate planning activities, align
individual performance to organizational performance, and ensure fair and
consistent performance evaluations. Further, the performance management
systems were intended to accurately reflect employees* performance,
facilitate their development, and improve and enhance their work.

As part of the new process, IRS establishes performance expectations at
the beginning of the performance cycle that serve as the basis for the
group managers* and frontline employees* annual performance evaluations.
Beginning in fiscal year 2000, IRS began implementing its new performance
management system for executives and managers. The key components of the
new system for group managers are the critical job responsibilities
(behaviors) and commitments (actions). The critical job responsibilities,
which represent the new core values of IRS, replaced the general standards
for group managers that had been in effect since 1984. They apply to all
IRS executives and managers and are further defined by supporting
behaviors* broad actions and competencies that IRS expects its

5 Office of Personnel Management regulations define performance management
as the integrated processes agencies use to (1) communicate and clarify
organizational goals, (2) identify accountability for accomplishing
organizational goals, (3) identify and address developmental needs, (4)
assess and improve performance, (5) measure performance for recognizing
and rewarding accomplishments, and (6) prepare appraisals.

6 In July 1999 IRS incorporated into the evaluation process a new
retention standard relating to the fair and equitable treatment of
taxpayers that managers and frontline employees must meet at a passing
level to retain their jobs. When assessing managers and employees, raters
are to first determine whether employees met the standard and, if the
employees did, then proceed to prepare a written evaluation of how the
employees met their critical job responsibilities.

Page 7 GAO- 02- 804 Performance Management Systems

executives and managers to demonstrate during the performance cycle. The
commitments, which are specific to and developed by each manager, are
distinct actions or desired results to be achieved during the performance
period. Each commitment should relate to, and support, one or more of the
critical job responsibilities.

For group managers, the performance management process includes an annual
performance evaluation* an assessment of performance supported by written
narrative commentary. While there is no current IRS requirement that
raters address managers* performance in each critical job responsibility
or each commitment individually in the written annual evaluation, raters
are expected to provide a summary of managers* performance that accurately
describes how well managers performed in meeting both their critical job
responsibilities and commitments.

IRS began implementing the performance management system for frontline
employees in fiscal year 2001. The frontline employees* system is
conceptually similar to that for executives and managers but there are
some differences. As with managers, frontline employees are evaluated on
how well they met their critical job responsibilities. However, the
responsibilities are tailored to reflect the activities and behaviors that
employees with direct contact with taxpayers are expected to display. Each
responsibility consists of a brief description of the key behaviors
related to the responsibility and supporting behaviors, which IRS calls
performance aspects, that further define the responsibility and
demonstrate the expected level of performance. In addition, frontline
employees are not required to develop written commitments about the
specific actions they will take to support IRS*s strategic goals.

IRS has not completed the redesign of its performance management systems.
For example, IRS envisions that its performance management system for
frontline employees will eventually include a requirement to develop
commitments, mirroring the system for executives and managers. In
addition, IRS expects to integrate the new performance management systems
with its overall human resource systems. This integration would, for
example, link evaluations to decisions about developmental needs, rewards
and recognition, and compensation. IRS expects the complete redesign and
implementation of both managers* and frontline employees* performance
management systems to take about five years.

Page 8 GAO- 02- 804 Performance Management Systems

IRS has established critical job responsibilities for managers and
frontline employees that align with each of the agency*s three strategic
goals. However, supporting behaviors described in the frontline employees*
performance management system do not always align with critical job
responsibilities or with their organizational unit*s performance measures.

IRS has established critical job responsibilities and supporting behaviors
for group managers that align with the agency*s strategic goals. These
responsibilities and supporting behaviors, described on the evaluation
form as shown in appendix III, provide managers with a consistent message
about how their daily activities are to reflect the organization*s core
values. IRS*s performance management system for group managers assesses
performance in five critical job responsibilities: customer satisfaction,
business results, employee satisfaction, leadership, and equal employment
opportunity. Of these five responsibilities, three align directly to IRS*s
strategic goals, as shown in figure 1. For example, by establishing a
critical job responsibility and supporting behaviors in customer
satisfaction, IRS aligns managers* performance to its strategic goal of
*topquality service to each taxpayer in every interaction.* Critical Job

Responsibilities Align with Strategic Goals but Do Not Always Align with
Other Elements of the Frontline Employee Performance Management System

Enforcement Group Managers* Critical Job Responsibilities and Supporting
Behaviors Align with Strategic Goals

Page 9 GAO- 02- 804 Performance Management Systems

Figure 1: Alignment of Strategic Goals, Critical Job Responsibilities, and
Supporting Behaviors for Enforcement Group Managers

Source: GAO analysis of IRS*s group manager performance management system.

The other two critical job responsibilities, leadership and equal
employment opportunity, support the agency*s strategic goals not because
they directly address IRS*s strategic goals but because they reinforce
behaviors that IRS considers necessary for organizational change and an
open and fair work environment.

As with enforcement group managers, the critical job responsibilities for
frontline employees, shown on the evaluation form in appendix III, align
with the agency*s strategic goals. IRS evaluates frontline employees
against five critical job responsibilities: customer satisfaction*
knowledge, customer satisfaction* application, business results* quality,
business results* efficiency, and employee satisfaction* employee
contribution. All five responsibilities directly align with IRS*s
strategic goals. For example, the two critical job responsibilities
addressing customer satisfaction (customer satisfaction* knowledge and
customer Frontline Employees*

Critical Job Responsibilities Align with Strategic Goals but Not Always
with Supporting Behaviors and Organizational Unit Performance Measures

Page 10 GAO- 02- 804 Performance Management Systems

satisfaction* application) align with IRS*s strategic goal of *top-
quality service to each taxpayer in every interaction.* Similarly, the two
critical job responsibilities addressing business results (business
results* quality and business results* efficiency) align with IRS*s
strategic goal of *topquality service to all taxpayers through the fair
and uniform application of the law.* Finally, the critical job
responsibility addressing employee satisfaction* employee contribution
aligns with IRS*s strategic goal of *productivity through a quality work
environment.*

Although the framework for IRS*s frontline employee performance management
system supports the agency*s strategic goals, the description of the
critical job responsibility given to IRS employees and the accompanying
supporting behaviors do not always align with one another. This
misalignment occurs when supporting behaviors reflect concerns not
expressed in the description, when supporting behaviors that relate to a
responsibility are located under other responsibilities, or when no
supporting behavior exists to support the description.

For example, under the customer satisfaction* application responsibility
for revenue agents, IRS*s description states that employees should
communicate with taxpayers in a clear, user- friendly manner. However, the
related supporting behaviors focus more on IRS*s own procedural
requirements and internal work products, such as the quality of audit
workpapers, than on whether taxpayers received understandable reports of
their audit, as shown in figure 2. These supporting behaviors would better
align with the business results* quality critical job responsibility,
which covers how well employees accomplish tasks within IRS*s procedures
and established guidelines.

Page 11 GAO- 02- 804 Performance Management Systems

Figure 2: Misalignment of Critical Job Responsibility, Narrative
Description, and Supporting Behaviors for Revenue Agents

Source: GAO analysis of IRS*s frontline employee performance management
system.

Also, for the customer satisfaction* knowledge responsibility for both
revenue agents and revenue officers, the description states that *correct
interpretation of laws, rules, and regulations* is a key component of the
critical job responsibility. However, for revenue agents, the supporting
behaviors that address *correct interpretation of laws* can also be found
under two other responsibilities, customer satisfaction* application and
business results* quality, even though correctly interpreting laws is not
explicitly described as a key component of those responsibilities. For
revenue officers, correctly interpreting laws is not directly addressed
under any of the critical job responsibilities.

In addition to assessing frontline employees individually using its
performance management system, IRS uses organizational unit

Page 12 GAO- 02- 804 Performance Management Systems

performance measures to assess how well frontline employees are
collectively achieving the agency*s strategic goals. Unit performance
measures provide aggregated feedback at different organizational levels on
how well employees are achieving IRS*s strategic goals. For example,
customer satisfaction surveys measure the quality of service employees
provided to taxpayers and quality measurement systems assess whether
employees followed IRS standards and procedures. In many cases, the
activities addressed on these organizational unit performance measures
align with the behaviors assessed in the frontline employee performance
management system. However, in some cases, the unit performance measures
and the frontline performance management system*s supporting behaviors do
not align. We found seven activities relating to informing taxpayers of
their rights, explaining the audit process, and being courteous that were
addressed in organizational unit performance measures but not in the
performance management system. For example, IRS does not explicitly
evaluate revenue agents on whether they explained the audit process to
taxpayers, an activity addressed on its examination customer satisfaction
survey. During discussions with IRS senior executives, they agreed that
the frontline employee performance management system should better align
with organizational unit performance measures.

According to our analysis of a sample of enforcement group managers*
evaluations during both fiscal years 2000 and 2001, almost all of the
commitments managers developed aligned with IRS*s strategic goals but
these commitments were not always written so that managers could be held
accountable for meeting them because they were not specific, outcome or
output oriented, or easily monitored.

Almost all of the commitments written by our sample of enforcement group
managers aligned with their critical job responsibilities and thus
ultimately with IRS*s strategic goals. Table 1 provides examples of
commitments that align with strategic goals. Enforcement Group

Managers* Commitments Align with IRS*s Strategic Goals but Are Not Always
Written So that Raters Can Hold Managers Accountable

Commitments Align with Strategic Goals

Page 13 GAO- 02- 804 Performance Management Systems

Table 1: Examples of Enforcement Group Managers* Commitments that Align
with IRS*s Strategic Goals

Strategic goals Commitments

Top- quality service to each taxpayer in every interaction *I will ensure
employees are prompt and professional

in their dealings with taxpayers by being directly involved in
examinations through observations and visits.* (improves customer
satisfaction) Top- quality service to all taxpayers through the fair and
uniform application of the law

*I will review quality measurement system results to identify deficient
areas, address deficiencies at group meetings and take corrective actions
as needed.* (improves business results) Productivity through a quality
work environment *I will perform evaluative reviews, such as case

reviews, on- the- job visits and workload reviews, on each employee and
provide immediate feedback on their performance.* (improves employee
satisfaction)

Source: GAO analysis of fiscal year 2000 and 2001 enforcement group
managers* commitments in the written performance evaluation.

Although commitments align with IRS*s strategic goals, our analysis of
fiscal year 2000 and 2001 written performance evaluations shows that
enforcement group managers* commitments were not evenly distributed among
the five critical job responsibilities. 7 Employee satisfaction and
business results were each addressed in more than a third of commitments
each year, while a quarter addressed customer satisfaction. Leadership and
equal employment opportunity were each addressed by fewer than 10 percent
of the commitments in each year. Although these critical job
responsibilities do not directly align with IRS*s strategic goals, IRS
senior executives believe that effective leadership is an important
component of changing organizational culture and supporting equal
employment opportunity principles is important to establishing a fair work
environment. Table 2 shows the percentage of commitments that focused on
each responsibility.

7 In our analysis, we only counted the number of commitments that aligned
with each of the critical job responsibilities. We did not try to assess
the significance or relative importance of each commitment.

Page 14 GAO- 02- 804 Performance Management Systems

Table 2: Percentage of Commitments Reflecting Critical Job
Responsibilities for Fiscal Years 2000 and 2001

Percentage of commitments Critical job responsibility FY 2000 FY 2001

Customer satisfaction 24 25 Business results 34 42 Employee satisfaction
37 40 Leadership 7 6 Equal employment opportunity 7 4

Note: Because managers* commitments might apply to more than critical job
responsibility, percentages do not add to 100.

Source: GAO analysis of fiscal year 2000 and 2001 enforcement group
managers* commitments in the written performance evaluation.

IRS senior executives were generally pleased with the distribution of
enforcement group managers* commitments among the critical job
responsibilities. They said that the distribution of commitments among the
three responsibilities aligned most directly with IRS*s strategic goals
was relatively equal and that it was more difficult for managers to
develop commitments for the other two responsibilities.

In an attempt to assist managers in developing their commitments and hold
them accountable for meeting commitments, IRS issued guidance in November,
2000 regarding criteria for well- constructed commitments. According to
the guidance, well- constructed commitments should be clear, specific,
achievable, outcome or output oriented, and easily monitored. Appendix I
provides an explanation of how we used these criteria to assess
enforcement group managers* individual commitments.

In keeping with the guidance for well- written commitments, we found that
almost all of the enforcement group managers* commitments in our sample of
evaluations met two of IRS*s five criteria* being clear and achievable.
For example, we concluded that the following commitments would be
understood by both the manager and the rater, did not include jargon
unfamiliar to IRS employees, and described actions that could
realistically be accomplished within the normal work environment.

*I will provide resources where necessary to ensure an effective filing
season.* Raters May Have Difficulty

Holding Managers Accountable

Page 15 GAO- 02- 804 Performance Management Systems

*I will encourage and support the development of employees through the use
of individual development plans, as submitted, and the assignment of
appropriate details whenever possible.*

*I will work with my group members to ensure that our jointly developed FY
2000 action plan objectives are met or exceeded.*

In contrast, our analysis showed that about 85 percent of the enforcement
group managers* commitments, for both rating years, did not meet the
remaining three IRS criteria for well- written commitments of being
specific, outcome or output oriented, or easily monitored. We found that
the majority of commitments were broad, vague statements that did not give
a relatively clear indication of the action that would be taken, and did
not include relevant information such as the frequency of the action or
what would be accomplished. Moreover, there was little focus on expected
outcomes, such as the potential impact of actions on employees or
taxpayers, or outputs, such as the number of activities to be
accomplished, for which the manager*s performance in meeting the
commitment could be assessed or measured. In IRS*s performance management
system, raters and managers share responsibility for ensuring that
commitments meet the criteria for well- written commitments* managers
initially develop the commitments and raters review and, if necessary,
revise them. The majority of group managers reported in our survey that
their raters were actively involved in revising their commitments.

Table 3: Percentage of Commitments Not Specific, Outcome/ Output Oriented,
or Easily Monitored for Fiscal Years 2000 and 2001

Percentage of commitments not meeting criteria Criteria FY 2000 FY 2001

Specific 84 85 Outcome/ output oriented 84 85 Easily monitored 85 86

Source: GAO analysis of fiscal year 2000 and 2001 enforcement group
managers* commitments in the written performance evaluation.

The commitments that were not specific, outcome or output oriented, or
easily monitored could, nevertheless, be encouraging desired behaviors.
However, because so many commitments did not meet these criteria, raters
may have difficulty holding enforcement group managers accountable for
meeting the majority of their commitments because of the

Page 16 GAO- 02- 804 Performance Management Systems

way they were written. The following two commitments illustrate a vague
commitment and a specific, outcome/ output oriented and easily monitored
commitment:

 *I will provide leadership by increased managerial involvement in my
employees* casework.*  *I will conduct quarterly reviews of casework
focusing on IRM (Internal

Revenue Manual) requirements and CQMS (Collections Quality Management
System) to ensure timely closing of cases.*

It would be difficult to definitively judge whether a manager met the
first commitment because the statement is vague about what the manager
intends to do. In contrast, the second commitment is explicit about the
type of managerial involvement, the frequency of the reviews, the
standards to be applied during the reviews, and the potential impact of
the manager*s involvement. A rater could monitor and assess whether the
manager conducted these types of reviews as often as promised, and use
timeliness statistics to assess the manager*s performance in actually
closing cases in less time. Although performance management systems are
inherently subjective to some degree because raters have to assess how
well employees carry out their duties, raters also must make subjective
judgments about whether commitments are met when those commitments are
vague. When commitments are more explicit, there is a greater likelihood
that both managers and raters will know and agree whether managers have
met their commitments, and that commitments are specific, outcome or
output oriented, and easily monitored. Appendix VI provides additional
examples of commitments.

Page 17 GAO- 02- 804 Performance Management Systems

According to our review of fiscal year 2000 and 2001 written evaluations
and our survey of enforcement group managers, raters generally provided
feedback to managers on how well they met their critical job
responsibilities and commitments by addressing them in written or verbal
feedback. However, raters provided more extensive written and verbal
feedback for some critical job responsibilities than for others. In
addition, for each critical job responsibility, raters provided more
extensive feedback on certain supporting behaviors than on others. In our
survey, enforcement group managers reported receiving more extensive
written and verbal feedback on business results than on the other critical
job responsibilities and because of that they believed that business
results was more important to their raters than the other critical job
responsibilities. Additionally, half of the enforcement group managers we
surveyed reported receiving verbal feedback on how well they met most or
all of their commitments.

Our review of enforcement group managers* performance evaluations showed
that raters provided written and verbal feedback to managers on their
performance in meeting most critical job responsibilities and commitments.
However, raters provided more extensive written and verbal feedback for
some critical job responsibilities and for certain supporting behaviors
than for others. As shown in table 4, our sample of performance
evaluations for IRS employees who were managers in both fiscal years 2000
and 2001 showed that more than 80 percent of the written evaluations
assessed managers* performance in meeting four or more of the five
critical job responsibilities. Raters Generally

Provided Feedback to Enforcement Group Managers on Meeting Critical Job
Responsibilities and Commitments but Provided More Extensive Feedback on
Some Critical Job Responsibilities Than Others

Raters Provided Feedback to Enforcement Group Managers on Meeting Most
Critical Job Responsibilities and Commitments in Evaluations and Verbal
Feedback

Page 18 GAO- 02- 804 Performance Management Systems

Table 4: Estimated Percentage of Fiscal Years 2000 and 2001 Written
Evaluations in Which Raters Assessed Critical Job Responsibilities by
Number of Responsibilities

Percentage of written evaluations Number of critical job responsibilities
assessed FY 2000 FY 2001

5 5447 4 2941 3 or fewer 17 12

Total 100 100

Source: GAO analysis of fiscal year 2000 and 2001 enforcement group
managers* written performance evaluations.

As noted in the previous section, commitments were not always written in a
way that was conducive for raters to hold enforcement group managers
accountable. This fact notwithstanding, raters provided feedback to
managers on how well they met their commitments. Specifically, raters
provided written feedback on managers* performance in meeting most of
their commitments. Our sample of fiscal year 2000 and 2001 written
performance evaluations indicates that raters addressed 76 percent of
commitments in employee evaluations, discussing each commitment
separately. Raters addressed 6 percent of commitments by providing summary
feedback such as *you met or exceeded your commitments* and provided no
feedback on 18 percent of commitments. According to senior executives, IRS
is considering new guidance that would require raters to assess managers*
performance in meeting each commitment, with the intent of holding
managers more accountable for achieving the commitments they make.

In our survey, most enforcement group managers reported receiving verbal
feedback about their performance in meeting critical job responsibilities
and commitments. We estimate that for more than 60 percent of the
managers, raters discussed all five responsibilities in the verbal
feedback that occurs with the delivery of the written performance
evaluation, and in ongoing verbal feedback throughout the year. Table 5
below shows the percentage of enforcement group managers who reported
receiving verbal feedback throughout the year in each of the critical job
responsibilities.

Page 19 GAO- 02- 804 Performance Management Systems

Table 5: Estimated Percentage of Enforcement Group Managers Reporting
Verbal Feedback by Number of Critical Job Responsibilities for Fiscal Year
2001

Number of critical job responsibilities covered during feedback

Verbal feedback with written performance

evaluation Ongoing verbal feedback

5 6373 4 67 3 or fewer 31 20

Total 100 100

Source: GAO survey of IRS enforcement group managers.

In addition, an estimated half of the enforcement group managers in our
survey reported receiving verbal feedback on how well they met most or all
of their commitments.

In our review of enforcement group managers* written evaluations, we found
that raters were more likely to assess a manager*s performance in terms of
customer satisfaction, business results, and employee satisfaction, and
less likely to assess performance in leadership and equal employment
opportunity. As shown in table 6, about 72 percent of written performance
evaluations in fiscal years 2000 and 2001 assessed leadership or equal
employment opportunity responsibilities. The employee satisfaction
responsibility was the only critical job responsibility that was assessed
in all evaluations and in both fiscal years.

Table 6: Estimated Percentage of Fiscal Year 2000 and 2001 Evaluations
Assessing Enforcement Group Managers* Performance in Each Critical Job
Responsibility

Percentage of written evaluations Critical job responsibility FY 2000 FY
2001

Customer satisfaction 92 96 Business results 96 99 Employee satisfaction
100 100 Leadership 72 71 Equal employment opportunity 72 71

Source: GAO analysis of fiscal year 2000 and 2001 enforcement group
managers* written performance evaluations.

Furthermore, for each critical job responsibility, certain key supporting
behaviors were emphasized more than others in the written performance
evaluations, as shown in appendix V. For example, IRS has identified
Raters Provide More

Extensive Feedback on Some Critical Job Responsibilities and Supporting
Behaviors Than on Others

Page 20 GAO- 02- 804 Performance Management Systems

certain key behaviors that support the critical job responsibility of
employee satisfaction. Of these behaviors, we found that *effectively uses
ongoing feedback, coaching, and timely evaluations of performance to
promote cooperation, teamwork, knowledge/ skill sharing and goal
accomplishment* was a key behavior that raters discussed in about 80
percent of written performance evaluations. In contrast, our sample
indicated that a key supporting behavior that was discussed in less than
40 percent of evaluations was *creates an environment for continuous
learning, pursuing development opportunities for self and others, with the
intent to increase individual and organizational effectiveness.* During
our discussions, IRS*s senior executives said they were comfortable with
the relative emphasis placed on each of the supporting behaviors.

As shown in Table 7, the evidence from our study is that enforcement group
managers believed that their raters emphasized business results the most
and emphasized equal employment opportunity the least in the written
performance evaluation and ongoing verbal discussions. In addition, we
estimate that equal employment opportunity was emphasized the least in the
verbal feedback occurring when the written evaluation was delivered.

Table 7: Estimated Percentage of Enforcement Group Managers Reporting
Feedback to a Great or Very Great Extent for the Critical Job
Responsibilities for Fiscal Year 2001

Type of feedback Critical job responsibility Written

evaluation Verbal feedback

with written performance

evaluation Ongoing

verbal feedback

Customer satisfaction 51 31 31 Business results 60 36 43 Employee
satisfaction 51 33 33 Leadership 49 34 30 Equal employment opportunity 31
23 13

Source: GAO survey of IRS enforcement group managers.

As shown in table 8, we found that managers believed raters conveyed that
business results was the most important to them and equal employment
opportunity the least, when they took all forms of feedback into
consideration. Group Managers Report

Raters* Feedback Focused More on Business Results

Page 21 GAO- 02- 804 Performance Management Systems

Table 8: Estimated Percentage of Enforcement Group Managers Reporting that
the Respective Critical Job Responsibilities Are Very Important or
Important to Their Raters for Fiscal Year 2001

Critical job responsibility Perception of which respective critical

job responsibilities are very important or important to their raters

Customer satisfaction 69 Business results 87 Employee satisfaction 68
Leadership 74 Equal employment opportunity 58

Source: GAO survey of IRS enforcement group managers.

IRS officials explained that while they do not intend that equal emphasis
be always given to all five critical job responsibilities, or to all
commitments, they believe that an over emphasis on one or just a few of
the job responsibilities would be inappropriate. Overall, they were
satisfied with the emphasis given to each of the five critical job
responsibilities in both the evaluations and the commitments. However,
they noted that a heavy emphasis on some critical job responsibilities
could lead managers away from the balanced behaviors that IRS seeks to
achieve through its new performance management system.

IRS has made progress in the challenging task of redesigning its
performance management systems to help reinforce behaviors and actions
that support the agency*s mission. IRS senior executives were generally
satisfied with the implementation of the new performance management
systems but expect that the systems will continue to evolve as the agency
gains more experience in implementing the new culture and processes.
However, IRS has not established mechanisms to monitor how well the
systems provide useful feedback to group managers and frontline employees
about their performance and whether feedback aligns with IRS*s strategic
goals. For example, IRS does not have plans to determine if raters are
over- emphasizing one critical job responsibility during written and
verbal feedback and when managers develop commitments. Further, IRS does
not have plans to assess at an appropriate time whether the new
performance management systems are achieving their objectives. Such an
assessment would be consistent with IRS*s guidance for implementing
strategic initiatives. IRS Has Made

Progress but Could Further Enhance Its New Managers* and Frontline
Employees* Performance Management Systems

Page 22 GAO- 02- 804 Performance Management Systems

IRS has implemented a number of initiatives to support its performance
management systems, and set the stage for the reform of its entire
performance management system over the coming years. According to
officials, in response to RRA 98 and as part of its overall modernization
efforts, IRS realized it needed to redesign its performance management
process to better communicate the behaviors constituting customer
satisfaction, business results, and employee satisfaction, and to ensure
managers and employees adopt the newly desired behaviors in their dayto-
day activities.

In an attempt to make managers more familiar with the new system, IRS has
undertaken several initiatives, with similar actions also planned for its
frontline employees, such as

 providing interim guidance and templates of sample commitments,
selfassessments, and summary evaluations;  distributing computer discs
with user- friendly access to information on

performance management and added performance management information to the
IRS intranet website, where it is constantly being updated;  conducting
an interactive videoconference for all managers on the new

performance management system. Further, IRS has started its implementation
of a pay- for- performance system, which emphasizes performance instead of
longevity in determining compensation. IRS is also experimenting to see
whether the manager evaluation forms might also be suitable for making
decisions about developmental needs, rewards and recognitions, and
compensation. IRS has already placed senior managers in a senior pay-
band, and expects that other managers, including those at the group
manager level, will be placed in pay- bands by October 2002. Ultimately,
IRS plans to rollout the pay- for- performance system to include all
managers as well as frontline employees. IRS Has Made Progress in

Making Its Managers* and Frontline Employees* Performance Management
Systems an Effective Management Tool

Page 23 GAO- 02- 804 Performance Management Systems

IRS recognizes that revamping its employee performance management systems
is a major effort that presents significant implementation challenges. IRS
senior executives were generally satisfied with the implementation of the
managers* performance management system given that it has only been in
place for two years. They expect that implementation of both systems will
improve as raters gain more experience in implementing the new culture and
processes. For example, they believe it will take some time for raters of
group managers and frontline employees to fully interpret the new critical
job responsibilities and become comfortable with how to apply them.

However, senior executives believe that IRS*s plans to expand its pay-
forperformance system underscore the need to ensure verbal and written
feedback are consistent with the message IRS wants to deliver to its
managers and employees about the activities and behaviors it values. They
believe the validity of pay and bonus decisions will depend heavily on how
well the performance management systems clearly communicate what managers
and employees are expected to do as well as provide comprehensive feedback
on how well they performed in carrying out their responsibilities and
meeting their commitments. IRS senior executives are concerned that if
misunderstandings about performance expectations creep into the system,
for example because commitments are vaguely worded, managers and employees
will begin to discredit the performance management system and the pay-
for- performance decisions. IRS senior executives believe that raters have
to be clear about their expectations so that managers know in advance what
they need to do to be successful and receive bonuses or salary increases.

IRS has not established a monitoring mechanism to ascertain whether raters
are implementing the new employee performance management systems as
intended and has no plans to assess whether the new systems are achieving
their objectives. IRS senior executives said they have done a limited
assessment of the evaluations for executives and senior managers because
they are the only group currently under the pay- for- performance system.
IRS senior executives said they have not established a monitoring
mechanism or a plan to assess the managers* or frontline employees*
systems due to resource constraints.

A monitoring mechanism could provide useful information on whether IRS*s
processes for evaluating employees are being followed. A monitoring system
could include, for example, an employee survey to obtain information on
whether raters are involved in the development of their IRS Officials
Generally

Satisfied with Implementation of Managers* and Frontline Employees*
Performance Management Systems but Realize Implementation Challenges
Remain

Monitoring and Assessment Mechanisms Would Help Ensure IRS*s Employee
Performance Management Systems Are Implemented as Intended

Page 24 GAO- 02- 804 Performance Management Systems

subordinates* commitments, giving appropriate emphasis to each of the
critical job responsibilities and supporting behaviors, and are providing
useful feedback. Also, content analyses of commitments and written
performance evaluations could help senior executives monitor whether the
systems are encouraging the kinds of behaviors and activities that support
IRS*s strategic goals. Active monitoring could give IRS senior executives
a sense of how the systems are working in practice and whether any
modifications are needed to provide more useful feedback to managers and
employees about their performance and better align the systems with IRS*s
strategic goals.

Although we have not estimated the costs to perform such monitoring, costs
likely could be minimal. Relatively small statistical samples of
performance evaluations and of employees to survey likely would be
sufficient, and once a monitoring methodology is developed it could be
used for several years. Further, to the extent such monitoring could
prevent potential problems, IRS would avoid the costs associated with
employee performance that is not well- balanced and aligned with IRS*s
goals.

In addition to active monitoring of the employee performance management
systems, a more thorough future assessment of whether the systems are
achieving their various goals and objectives could provide IRS assurance
that the systems are properly designed and implemented, or identify
whether changes are needed. For example, IRS could assess whether the
systems are providing a framework for improving communications between
raters and their subordinates, coordinating planning activities, aligning
individual performance to organizational performance, and ensuring fair
and consistent performance evaluations. Although it would be premature to
perform an assessment now* since the systems are not fully implemented*
planning now for such an assessment could better ensure that IRS will have
the necessary data to assess the performance management systems*
performance in the future. Such upfront planning is, for instance, part of
IRS*s guidance for planning and implementing reorganizations. That
guidance requires that a plan for assessing the results of a
reorganization be developed at the same time that the reorganization is
proposed.

Monitoring and assessing IRS*s new performance management systems also
would be in line with the expectations IRS has established for the
strategic initiatives being carried out by its operating divisions. In
March 2000, IRS implemented a new strategic planning, budgeting, and
performance management process intended to provide a more structured

Page 25 GAO- 02- 804 Performance Management Systems

format for establishing strategic direction, identifying the resources
needed to accomplish the operational activities supporting the strategy,
and assessing performance results. 8 IRS recognizes that conducting
assessments of operational activities at all levels of the organization is
essential to ensuring its programs are achieving their goals. As IRS*s
strategic plan points out, regular, structured feedback on program
performance is essential if managers are to recognize and react to
successes or failures and ensure programs are adhering to the agency*s
strategic intent.

Although IRS would incur costs to assess whether its employee performance
management systems achieve their objectives, IRS*s strategic planning,
budgeting and performance management guidance implicitly recognizes that
such costs are properly part of a sound management system. Further,
planning ahead for such assessments can help ensure that costs are
minimized. Finally, as with monitoring, to the extent that an assessment
results in further improvements to IRS*s performance management systems,
costs due to unbalanced or misaligned employee behaviors would be avoided.

IRS is in the midst of a cultural change that requires group managers and
frontline employees to better balance taxpayer needs, business results,
and employee needs in carrying out their day- to- day responsibilities. By
aligning its new performance management systems with its strategic and
operational goals, IRS has made progress in supporting this cultural
change and providing guidance to managers and employees on how to
prioritize their activities and carry out their responsibilities. In
addition, most enforcement group managers received written or verbal
feedback on how well they met their critical job responsibilities and
commitments. However, we identified several opportunities to enhance IRS*s
performance management systems for managers and employees. IRS could
better align employee performance with strategic goals by establishing
greater consistency within the frontline employee performance management
system. When misalignment exists between the unit performance measures and
the frontline employee performance management system, IRS is emphasizing
different behaviors at the unit

8 IRS*s strategic guidance calls for periodic evaluations. For the
purposes of this report, we are using the term assessment. Conclusions

Page 26 GAO- 02- 804 Performance Management Systems

level than at the employee level. As a result, IRS is not consistently
communicating its strategic goals to employees.

Commitments could be developed to be more specific, outcome or output
oriented, and easily monitored so that group managers can be held
accountable for actions they intend to take to support IRS* goals. IRS is
considering new guidance that would require raters to assess managers*
performance in meeting each commitment, with the intent of holding
managers more accountable. Such guidance on providing more comprehensive
feedback would further reinforce use of IRS*s new performance management
system for managers to encourage behaviors and actions that support IRS*s
strategic goals.

Further, if unequal emphases in written and verbal feedback grow over
time, the performance management systems might not adequately align with
strategic goals and provide the intended message about how managers and
employees should conduct their daily activities. If so, managers could
receive a skewed perception of the behaviors that IRS values. Misusing
even a well- designed performance management system can drive
dysfunctional behaviors, such as an overemphasis on certain activities to
the detriment of the balanced approach IRS is trying to achieve and
maintain. Accordingly, it is important for IRS to monitor and assess its
progress in fully implementing the systems and position itself to take
corrective action if and when needed.

To better hold managers accountable for meeting strategic goals and ensure
the new performance management systems are working as intended, we
recommend that the Commissioner of Internal Revenue take steps to:

 improve the linkage between frontline employees* critical job
responsibilities, the supporting behaviors, and organizational unit
performance measures and  develop plans for monitoring and assessing
whether the new employee

performance management systems are operating as intended and take the
necessary actions to resolve any identified problems.

On July 5, 2002, we received written comments on a draft of this report
from the Commissioner of Internal Revenue (see app. VII). The commissioner
agreed with the recommendations contained in this report and said they
would be included as IRS works to continuously improve its Recommendations
for

Executive Action Agency Comments and Our Evaluation

Page 27 GAO- 02- 804 Performance Management Systems

performance management system. In his comments, the Commissioner noted
that IRS currently uses performance review boards to ensure executives*
and senior managers* performance agreements and evaluations align with and
reflect IRS*s strategic goals but this approach is less practical at lower
levels of the organization due to the larger numbers involved. Therefore,
IRS is going to explore other means of ensuring alignment. In addition,
the Commission said IRS will continue to monitor the overall efficacy of
its performance management system.

As agreed with your office, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30 days from
the date of this letter. At that time, we will send copies to the Ranking
Minority Member of the Committee on Ways and Means; the Ranking Minority
Member of the Subcommittee on Oversight; the Secretary of the Treasury;
and the Commissioner of Internal Revenue. We will also make copies
available to others on request. In addition, the report will be available
at no charge on the GAO Web site at http:// www. gao. gov.

Please contact me at (202) 512- 9039, Ralph Block at (415) 904- 2150, or
Jonda Van Pelt at (415) 904- 2186 if you or your staff have any questions.
Key contributors to this report are acknowledged in appendix VIII.

Michael Brostek Director, Tax Issues

Appendix I: Objectives, Scope, and Methodology

Page 28 GAO- 02- 804 Performance Management Systems

Our objectives in this report were to determine (1) the extent to which
group managers* and frontline employees* critical job responsibilities and
other elements of the employee performance management systems are
structurally aligned with IRS*s strategic goals and organizational unit
performance measures, (2) the extent to which group managers* commitments
align with IRS*s strategic goals and are written so that raters can hold
managers accountable for meeting their commitments, (3) the extent to
which raters provide feedback to enforcement group managers on their
performance in meeting critical job responsibilities and commitments, and
(4) whether IRS has adequate plans to monitor and assess the
implementation of its performance management systems.

To determine the extent to which the critical job responsibilities for
group managers and frontline employees align with IRS*s strategic goals,
we performed separate qualitative analyses of both performance management
systems to assess whether the critical job responsibilities align with
IRS*s goals and whether supporting behaviors were consistent with the
critical job responsibilities. Our analysis of the frontline employee
system also included reviewing organizational unit performance measures to
determine if these measures were being reflected in the performance
management system*s supporting behaviors. We only included those
components of the unit measures that deal with employee behaviors and
actions, not those measures that address how well IRS*s procedures and
systems function or that set organizational performance goals.

To determine the extent to which group managers* commitments align with
IRS*s strategic goals and are written so that managers can be held
accountable for meeting those commitments, we analyzed the commitments of
a statistically valid random sample of group managers in IRS*s Small
Business/ Self- Employed Division. 1 We selected this IRS division because
it included managers in key enforcement positions who supervise employees
dealing directly with taxpayers when auditing tax returns or collecting
unpaid taxes. To determine whether commitments aligned with IRS*s
strategic goals, we assessed whether the objective of the commitment was
to improve customer satisfaction, business results,

1 We express the precision of the results of our analysis of group
managers* commitments and evaluations (that is, the sampling errors
associated with these estimates) as 95 percent confidence intervals.
Appendix I: Objectives, Scope, and

Methodology Objectives

Scope and Methodology

Appendix I: Objectives, Scope, and Methodology

Page 29 GAO- 02- 804 Performance Management Systems

employee satisfaction, leadership or equal employment opportunity. 2 To
determine the extent to which managers could be held accountable for
meeting their commitments, we applied IRS*s criteria for formulating
wellconstructed commitments, which included that they are clear, specific,
achievable, outcome or output oriented, and are easily monitored. As we
applied the criteria to individual commitments, we found
interrelationships between certain criteria. For example, commitments that
are specific also tend to be outcome or output oriented and easily
monitored. Similarly, commitments that include explicit outcomes or
outputs tend to be easier to monitor. As agreed with IRS officials, we
addressed the following questions in determining whether commitments met
the criteria:

 Clear: Is the commitment easy to understand? Does it contain jargon that
might make it unfamiliar to managers or their supervisors?

 Specific: Does the commitment include details, such as the specific
actions to be taken, when the actions are to be taken, whether other IRS
employees are involved, the expected result, or a numeric target? Is the
commitment specific enough so that a third party can understand the
actions and timeframes to which the manager is committing?

 Achievable: Can the manager accomplish the task in the normal work
environment? To what extent does the manager control the resources
necessary to accomplish the commitment? Outcome or output oriented: Does
the commitment include an expected result, the type and level of activity
to be accomplished, or a program impact?

 Easily monitored: Does the commitment include a deadline, an expected
result, a numeric target, or some other means of measurement?

To determine the extent to which raters provided feedback to group
managers on how well they met their critical job responsibilities and
commitments, we performed a content analysis of the narrative portion of
the evaluation that discussed managers* performance. To assess the extent
to which managers were given feedback on meeting their critical job
responsibilities, we assessed whether the evaluation narrative described
behaviors that were consistent with the key supporting behaviors for each

2 Our assessment was limited to a count of the number of commitments, not
the significance or the relative challenge of accomplishing them.

Appendix I: Objectives, Scope, and Methodology

Page 30 GAO- 02- 804 Performance Management Systems

critical job responsibility as laid out on the managers evaluation form
(see app. III for a sample of the evaluation.) We also reviewed the
narrative to determine whether the rater assessed how well the managers
met their commitments. In addition, we surveyed the managers for
information they received on their performance during verbal feedback
sessions held with their raters. 3

To determine whether IRS has adequate plans to monitor and assess the
implementation of its employee performance management systems, we talked
to IRS officials about initiatives taken to date and reviewed IRS guidance
for strategic planning.

Our review was subject to some limitations. Our assessment of group
managers* commitments and our content analysis required us to make
judgments that were, in part, subjective. To maximize the objectivity of
our analyses, we had our reviewers conduct separate and independent
assessments of (1) whether commitments met IRS*s criteria and (2) the
narrative discussions of each sampled evaluation. When differences arose
between the two reviewers, a collaborative approach was used to resolve
them.

3 As with our analysis of managers* commitments and performance
evaluations, we express the precision of the results of our survey (that
is, the sampling errors associated with these estimates) as 95 percent
confidence intervals.

Appendix II: GAO Sampling Methodology Page 31 GAO- 02- 804 Performance
Management Systems

This appendix discusses the sampling methodology we used to determine (1)
to what extent group managers* commitments aligned with IRS*s strategic
goals and were written so that raters can hold group managers accountable
for meeting their commitments and (2) to what extent raters provided
feedback to group managers on how well they met their commitments and
critical job responsibilities.

We randomly selected a probability sample of 150 from the study population
of 1,374 group managers with enforcement responsibilities working in IRSs
Small Business/ Self- Employed Division. These managers were identified
from a database of IRS personnel information current as of January 2001.
With this statistically valid probability sample each member of the
population had a nonzero probability of being included, and that
probability could be computed for any member. Each manager selected in the
sample was subsequently weighted in the analysis to account statistically
for all the members of the population, including those that were not
selected.

For the objective of determining to what extent group managers*
commitments aligned with IRS*s strategic goals and were written so that
raters can hold managers accountable for meeting their commitments, we
reviewed fiscal year 2000 and fiscal year 2001 written annual employee
evaluations. For this objective, we found that 126 group managers in our
sample had enforcement responsibilities in fiscal years 2000 and 2001. We
therefore estimate that about 1,154 group managers in the study population
had enforcement responsibilities in fiscal years 2000 and 2001.

For the objective of determining to what extent raters provided feedback
to group managers on how well they met their commitments and critical job
responsibilities, we reviewed fiscal year 2000 and fiscal year 2001
employee evaluations. For this objective, we found that 120 group managers
in our sample had enforcement responsibilities in fiscal year 2002. We
therefore estimate that about 1,099 group managers in the study population
had enforcement responsibilities in fiscal year 2002. We also administered
a web- based survey to obtain information from current, fiscal year 2002,
managers on the verbal feedback that raters provided to them.

As we received evaluations from IRS we found that some of the managers in
our sample were temporary, acting group managers who were not assessed
using the same evaluation form as permanent group managers. Appendix II:
GAO Sampling Methodology

Sample Designs

Appendix II: GAO Sampling Methodology Page 32 GAO- 02- 804 Performance
Management Systems

We were unable to use these evaluations in our analysis and deleted
temporary, acting group managers from our sample.

For our review of written annual performance evaluations, we received 112
usable evaluations (those that indicated the employee was an enforcement
group manager for both, complete fiscal years and included both
commitments and summary narrative) out of 126 eligible respondents for a
response rate of approximately 89 percent. We eliminated 24 ineligible
group managers, including those for whom we did not have two complete
evaluations as a group manager or who functioned in a special capacity
such as training specialist.

For our survey of enforcement group managers* perceptions of the amount
and type of feedback they received from raters, we received 84 usable
responses (those who completed the survey and indicated that they were
currently a group manager with enforcement responsibilities) out of 120
eligible respondents for a response rate of 70 percent. We eliminated 30
ineligible employees, including those who were no longer a group manager
due to job reassignment or retirement.

The disposition for the sampled cases for our review of group manager
evaluations and for our group manager survey is shown in table 9 and table
10, respectively.

Table 9: Disposition of Sample Cases for Our Review of Group Manager
Evaluations

Total number of initially sampled elements 150 Sampled elements outside
the study population (i. e., *ineligibles*)

Not a group manager with enforcement responsibilities for FY 2000 and 2001
24

Subtotal of ineligible elements 24

Sampled elements in the study population (i. e., *eligibles*)
Nonrespondents

Provided unusable evaluation( s) 6 No evaluation( s) available 8
Respondents (provided usable evaluations) 112

Subtotal of eligible elements 126

Evaluation response rate (respondents/ subtotal of eligible elements) 89%
Source: IRS data and GAO sample.

Sample Disposition

Appendix II: GAO Sampling Methodology Page 33 GAO- 02- 804 Performance
Management Systems

Table 10: Disposition of Sample Cases for Our Survey of Group Managers*
Perceptions of Raters* Feedback

Total number of initially sampled elements 150

Sampled elements outside the study population (i. e., *ineligibles*) Not a
current group manager with enforcement responsibilities for FY 2002 30

Subtotal of ineligible elements 30

Sampled elements in the study population (i. e., *eligibles*)
Nonrespondents

Refused 2 Unable to respond 1 No response after contact 26 Unable to
contact due to missing or incorrect contact information 2 Returned
unusable survey 5 Respondents (returned usable surveys) 84

Subtotal of eligible elements 120

Evaluation response rate (respondents/ subtotal of eligible elements) 70%
Source: IRS data and GAO sample.

Because we followed a probability sampling procedure based on random
selections, our sample is only one of a large number of samples that we
might have drawn. Since each sample could have provided different
estimates, we express our confidence in the precision of our particular
sample*s estimates by providing a sampling error (for example, +/- 10
percentage points) with a specified confidence level, for example, the 95-
percent confidence level. The confidence interval (i. e., the estimate
plus or minus its sampling error) would contain the actual population
value for 95 percent of the samples we could have drawn. As a result, we
are 95- percent confident that a confidence interval includes the true
value in the study population. In this report, all percentage estimates
dealing with our review of evaluations have sampling errors of +/- 11
percent or less of the value of those numerical estimates, unless
otherwise stated. All percentage estimates dealing with the group manager
survey have sampling errors of +/- 12 percent or less, unless otherwise
stated.

In addition to the reported sampling errors, the practical difficulties of
conducting any survey may introduce other types or *nonsampling* errors.
For example, differences in how a particular question is interpreted or
the types of individuals that do not respond can introduce unwanted
variability into the survey results. To minimize such nonsampling errors,
we pre- tested the survey with four enforcement group managers and
Sampling Error

Nonsampling Error

Appendix II: GAO Sampling Methodology Page 34 GAO- 02- 804 Performance
Management Systems

incorporated their feedback into the survey design. We did not analyze the
survey non- respondents or employee evaluations non- respondents to
determine whether the non- response was random. We assumed that the survey
and employee evaluations non- respondents had the same characteristics as
group managers who responded to the survey and provided usable
evaluations, respectively.

Appendix III: Samples of Manager and Frontline Employee Evaluation Forms

Page 35 GAO- 02- 804 Performance Management Systems

Appendix III: Samples of Manager and Frontline Employee Evaluation Forms

The following sixpage form is used to evaluate performance for managers.

Appendix III: Samples of Manager and Frontline Employee Evaluation Forms

Page 36 GAO- 02- 804 Performance Management Systems

Appendix III: Samples of Manager and Frontline Employee Evaluation Forms

Page 37 GAO- 02- 804 Performance Management Systems

Appendix III: Samples of Manager and Frontline Employee Evaluation Forms

Page 38 GAO- 02- 804 Performance Management Systems

Appendix III: Samples of Manager and Frontline Employee Evaluation Forms

Page 39 GAO- 02- 804 Performance Management Systems

Appendix III: Samples of Manager and Frontline Employee Evaluation Forms

Page 40 GAO- 02- 804 Performance Management Systems

Appendix III: Samples of Manager and Frontline Employee Evaluation Forms

Page 41 GAO- 02- 804 Performance Management Systems

The following twopage form is used to evaluate performance for IRS
frontline employees, including revenue agents and revenue officers.

Appendix III: Samples of Manager and Frontline Employee Evaluation Forms

Page 42 GAO- 02- 804 Performance Management Systems

Appendix IV: Examples of Evaluations Addressing Group Managers*
Responsibilities

Page 43 GAO- 02- 804 Performance Management Systems

This appendix provides examples of critical job responsibilities,
supporting behaviors, and statements from narratives that we found in our
review of the enforcement group managers* evaluations.

Table 11: Examples of Critical Job Responsibilities, Supporting Behaviors,
and Evaluation Narratives for Group Managers Critical job responsibility
Key supporting behavior Behavior described in evaluation narrative

Leadership Uses sound judgment to make effective and timely decisions. *As
a leader, you have demonstrated sound judgment in

your decisions organizing the second group offer.* Develops, prioritizes
and aligns strategies, objectives and goals, taking into account key
influences on organizational performance.

*He led his employees to develop realistic personal and group goals that
easily meshed into the needs of the service and the general public.*
Customer satisfaction Constantly listens to customers, analyzes their

feedback to identify their needs and expectations, and acts continuously
to improve products and services.

*You take the time to speak with the taxpayers and their representative
about their concerns and try to accommodate their needs.*

Ensures that employees do the same (as immediately above) and that they
are prompt, fair, and responsive to the circumstances of individual
customers to the extent permitted by law and regulation.

*He insists his employees take the same approach with their customers and
that they address each case with the rights of the taxpayers up front.*

Employee satisfaction Effectively uses ongoing feedback, coaching, and
timely evaluations of performance to promote cooperation, teamwork,
knowledge/ skill sharing and goal accomplishment.

*Throughout the year you provided employees with regular, specific
performance feedback geared to improve and enhance performance, in both
verbal and written form.* Acts on employee concerns, values, ideas and
perspectives of people from diverse backgrounds.

*He heartily endorsed Survey 2000 participation and was rewarded by
increased levels of employee satisfaction as reflected in survey results.*
Business results Develops and executes plans to achieve

organizational goals, leveraging available resources (e. g., human,
financial, etc.) to maximize efficiency and produce high- quality results.

*In addition to your responsibilities to the offer program, you have
ensured your employees have supported the walk- in program when needed.*

Learns about current and emerging issues/ developments in own field of
expertise and applies knowledge to make technically sound operational
decisions.

*Because of his recognized technical expertise, he took an active part in
training during the year and his employees and peers relied upon him to
clarify, explain, and reinforce complicated or newly enacted tax law.*
Equal employment opportunity Promptly responds to allegations of

discrimination and/ or harassment and initiates appropriate action to
address the situation.

*You are also sensitive to the work environment and bring any potential
problems or issues to the attention of the appropriate people.* Supports
staff participation in special emphasis programs. *He makes sure that his
employees have the opportunity

to participate in EEO and diversity programs, and demonstrates his
commitment by attending himself.*

Appendix IV: Examples of Evaluations Addressing Group Managers*
Responsibilities

Appendix V: Distribution of Raters* Comments for Each Supporting Behavior

Page 44 GAO- 02- 804 Performance Management Systems

This appendix provides information on the distribution of raters* comments
among the behaviors supporting each critical job responsibility. The
supporting behaviors listed for each critical job responsibility are taken
directly from IRS*s evaluation form, shown in appendix III.

Table 12: Distribution of Raters* Comments Percentage of managers
evaluated

on each supporting behavior Critical job responsibilities and supporting
behaviors FY 2000 FY 2001 Customer satisfaction

Communicates to employees the importance of customer focus as a critical
component of IRS* mission. 25 19 Constantly listens to customers, analyzes
their feedback to identify their needs and expectations, and acts to
continuously improve products and services. 46 56 Insures employees do the
same (as immediately above) and that they are prompt, professional, fair,
and responsive to the circumstances of individual customers to the extent
permitted by law and regulation. 47 47

Identifies and utilizes policies, and economic, political and social
trends in an effort to improve organizational performance. 1 0 Builds
strong alliances, involving stakeholders (for example NTEU, internal
customers, suppliers, etc.) in making decisions, and gaining cooperation
to achieve mutually satisfying solutions. 37 55

Initiates actions and manages risks to develop new products and services
within or outside the organization. 9 7 Shares innovations with others. 0
1

Business results

Pursues business excellence through effective process management and the
application of balanced measures. 78 87 Develops and executes plans to
achieve organizational goals, leveraging available resources (human,
financial, etc.) to maximize efficiency and produce high- quality results.
64 68

Identifies and analyzes problems to resolve individual and organizational
issues in accordance with law, regulation, and IRS policy. 18 15
Continuously seeks to improve business processes, sharing those efforts
with other units to better overall IRS performance. 21 25 Takes steps to
prevent waste, fraud and abuse and instill public trust. 3 7 Learns about
current and emerging issues/ developments in own field of expertise and
applies knowledge to make technically sound operational decisions. 3 2
Understands and uses organizational realities, networks and accepted
practices to achieve desired results. 2 3

Employee satisfaction

Ensures that a safe, healthy work environment is maintained. 8 16
Demonstrates importance of employee satisfaction in accomplishing IRS*
mission. 7 2 Motivates employees to achieve high performance through
empathetic, open and honest communication, by involving them in decision
making, and ensuring that they have the tools and training to perform
their jobs. 82 85

Appendix V: Distribution of Raters* Comments for Each Supporting Behavior

Appendix V: Distribution of Raters* Comments for Each Supporting Behavior

Page 45 GAO- 02- 804 Performance Management Systems

Percentage of managers evaluated on each supporting behavior Critical job
responsibilities and supporting behaviors FY 2000 FY 2001

Effectively uses ongoing feedback, coaching, and timely evaluations of
performance to promote cooperation, teamwork, knowledge/ skill sharing and
goal accomplishment. 78 81 Develops and recognizes employees so that they
realize their full potential.

62 80 Acts on employee concerns, values, ideas and perspectives of people
from diverse backgrounds. 65 63 Ensures all employees are treated in a
fair and equitable manner. 11 12 Creates an environment for continuous
learning, pursuing development opportunities for self and others, with the
intent to increase individual and organizational effectiveness. 38 38

Leadership

Demonstrates integrity and the highest ethical standards of public
service. 11 7 Develops, prioritizes and aligns strategies, objectives and
goals, taking into account key influences on organizational performance.
10 5 Successfully leads organizational change, effectively communicating
IRS* mission, core values, and strategic goals to employees and other
stakeholders and responding creatively to changing circumstances. 52 57

Creates and sustains a positive workplace that inspires others to support
IRS* mission and goals. 15 12 Shows a commitment to public service and
citizenship 3 3 Uses sound judgment to make effective and timely
decisions. 6 3

Equal employment opportunity

Takes steps to implement EEO and affirmative employment goals established
by IRS. 32 28 Supports staff participation in special emphasis programs.
31 34 Promptly responds to allegations of discrimination and/ or
harassment and initiates appropriate action to address situation. 7 5
Cooperates with EEO counselors, investigators, and other officials
responsible for conducting inquires into EEO complaints. 8 9 Assigns work
and makes employment decisions in areas such as hiring, promotion,
training and developmental assignments without regard to sex, race, color,
national origin, religion, age, disability, sexual orientation or prior
participation in EEO process. 26 25

Monitors work environment to prevent instances of prohibited
discrimination and/ or harassment. 18 14

Note: Percentages are rounded to the nearest percent. Source: GAO analysis
of fiscal year 2000 and 2001 enforcement group managers* written
performance evaluations.

Appendix VI : Examples of Group Managers* Commitments

Page 46 GAO- 02- 804 Performance Management Systems

This appendix provides examples of commitments written by enforcement
group managers in our sample. We found that almost all commitments met
IRS*s criteria of being clear and achievable, but about 85 percent did not
meet the criteria of being specific, outcome or output oriented, or easily
monitored. The examples of commitments are grouped into two categories,
those that were specific, outcome or output oriented, and easily
monitored, and those that were not.

We judged that the following commitments were specific, outcome or output
oriented, or easily monitored.

*Use of Balanced Measures Matrix will be demonstrated on one case or
situation each quarter.*

*I will meet on a quarterly basis with the Office Assistant to discuss
issues relevant to his job performance.*

*I will have monthly group meetings to discuss case issues, Survey 99
updates, RRA updates, FY 2000 Operating Plan activities, and group
concerns.*

*I will review and update IDPs (Individual Development Plans) on a semi-
annual basis to monitor accomplishments and modify the existing
development plans.*

*I will give a presentation to Group __ and/ or Branch II managers about
what I have learned about organizing.*

*By 03/ 31/ 2000 I will conduct refresher training/ briefings for my
employees in Group __ on TFRP (Trust Fund Recovery Penalty) procedures.*

*I will continue to conduct monthly meetings with the employees to
communicate changes and any impact.*

*I will conduct quarterly reviews of casework focusing on IRM (Internal
Revenue Manual) requirements and CQMS (Collection Quality Measurement
System) measures to ensure timely closing of cases.*

*To improve employee satisfaction, I will keep employees informed of their
performance by preparing timely mid- year and annual appraisals supported
with sufficient recordation.*

*Walk- in customer service will be provided at four outreach sites; Sioux
City, IA, Storm Lake, IA, O*Neill, NE and Columbus, NE. These sites will
be staffed for one day every other week from January through April 2000.*
Appendix VI : Examples of Group Managers*

Commitments Commitments that Were Specifi c, Outcome or Output Oriented,
or Easily Monitored

Appendix VI : Examples of Group Managers* Commitments

Page 47 GAO- 02- 804 Performance Management Systems

*To improve Employee Satisfaction, I will ensure I provide timely and
constructive feedback to employees. I will do sample case reviews
semimonthly and quarterly bundle reviews and/ or other written recordation
concerning individual performance and timely completion of mid- year and
annual appraisals.*

The following commitments were determined not to be specific, outcome or
output oriented, or easily monitored.

*Support and develop employees in a manner which encourages participation
by all.* *Provide leadership to my group in achieving organizational goals
by the use of effective group meetings, effective review and evaluation
practices, and the Individual Development process.*

*I will work with the municipal governments within the State.
Relationships built here will continue to homogenize State employment tax
administration efforts.*

*I will provide leadership by increased managerial involvement in my
employee*s casework.*

*I will use required group meetings to provide guidance on relevant
issues.* *I will support the partnership with NTEU [National Treasury
Employees Union] so that positive employee relations are maintained.*

*The identification and development of fraud cases will be encouraged in
my work group by creating an environment of fraud awareness.*

*Provide leadership and direction to the field staff to regain appropriate
levels of compliance on a case by case basis by conducting reviews, timely
entity queries and follow ups.*

*I will support the Field Branches as needed.* *I will ensure sufficient
resources and managerial attention to support the walk- in program in my
post of duty.*

*I will continue to have open and honest communications with all
employees.* *I will continue to be supportive of employees* efforts to
advance their careers.* *I will lead Field Group #__ through the
organizational changes demanded by the modernization of the IRS by
providing leadership, planning, and support to effectively Commitments
that

Were Not Specific, Outcome or Output Oriented, or Easily Monitored

Appendix VI : Examples of Group Managers* Commitments

Page 48 GAO- 02- 804 Performance Management Systems

transition the work and human resources of the current district structure
to the new operating units.*

Appendix VII: Comments from the Internal Revenue Service

Page 49 GAO- 02- 804 Performance Management Systems

Appendix VII: Comments from the Internal Revenue Service

Appendix VII: Comments from the Internal Revenue Service

Page 50 GAO- 02- 804 Performance Management Systems

Appendix VIII: GAO Contacts and Staff Acknowledgments

Page 51 GAO- 02- 804 Performance Management Systems

Michael Brostek (202) 512- 9039 Ralph Block (415) 904- 2150 Jonda Van Pelt
(415) 904- 2186

In addition to those named above, Michelle Bowsky, Maya Chakko, Elizabeth
Fan, Tre Forlano, Diana Hu, Ann Lee, and Samuel Scrutchins made key
contributions to this report. Appendix VIII: GAO Contacts and Staff

Acknowledgments GAO Contacts Acknowledgments

(440028)

The General Accounting Office, the investigative arm of Congress, exists
to support Congress in meeting its constitutional responsibilities and to
help improve the performance and accountability of the federal government
for the American people. GAO examines the use of public funds; evaluates
federal programs and policies; and provides analyses, recommendations, and
other assistance to help Congress make informed oversight, policy, and
funding decisions. GAO*s commitment to good government is reflected in its
core values of accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no cost
is through the Internet. GAO*s Web site (www. gao. gov) contains abstracts
and fulltext files of current reports and testimony and an expanding
archive of older products. The Web site features a search engine to help
you locate documents using key words and phrases. You can print these
documents in their entirety, including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as *Today*s Reports,* on its
Web site daily. The list contains links to the full- text document files.
To have GAO e- mail this list to you every afternoon, go to www. gao. gov
and select *Subscribe to daily E- mail alert for newly released products*
under the GAO Reports heading.

The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:

U. S. General Accounting Office 441 G Street NW, Room LM Washington, D. C.
20548

To order by Phone: Voice: (202) 512- 6000 TDD: (202) 512- 2537 Fax: (202)
512- 6061

Contact: Web site: www. gao. gov/ fraudnet/ fraudnet. htm E- mail:
fraudnet@ gao. gov Automated answering system: (800) 424- 5454 or (202)
512- 7470

Jeff Nelligan, managing director, NelliganJ@ gao. gov (202) 512- 4800 U.
S. General Accounting Office, 441 G Street NW, Room 7149 Washington, D. C.
20548 GAO*s Mission

Obtaining Copies of GAO Reports and Testimony

Order by Mail or Phone To Report Fraud, Waste, and Abuse in Federal
Programs

Public Affairs
*** End of document. ***