Tax Administration: IRS Needs Better Strategic Planning and	 
Evaluation of Taxpayer Assistance Training (11-JUL-05,		 
GAO-05-782).							 
                                                                 
Millions of taxpayers ask IRS questions about tax law each year. 
While the accuracy of IRS's answers has improved in some cases,  
it is still not always what taxpayers or Congress expect.	 
Concerns about accuracy have raised questions about the adequacy 
of the training IRS provides to its taxpayer assistance staff.	 
Because of these questions, GAO was asked to assess the extent to
which IRS's planning and evaluation of its taxpayer assistor	 
training conformed to guidance published by GAO and others.	 
Planning and evaluation are part of a feedback loop whereby	 
lessons from one year can be applied to making improvements in	 
future years.							 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-05-782 					        
    ACCNO:   A29597						        
  TITLE:     Tax Administration: IRS Needs Better Strategic Planning  
and Evaluation of Taxpayer Assistance Training			 
     DATE:   07/11/2005 
  SUBJECT:   Customer service					 
	     Data integrity					 
	     Employee training					 
	     Evaluation criteria				 
	     Performance appraisal				 
	     Performance management				 
	     Performance measures				 
	     Strategic planning 				 
	     Tax administration 				 
	     Tax law						 
	     Training utilization				 
	     Taxpayers						 
	     Tax administration systems 			 
	     Systems analysis					 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-05-782

                 United States Government Accountability Office

                     GAO Report to Congressional Requesters

July 2005

TAX ADMINISTRATION

IRS Needs Better Strategic Planning and Evaluation of Taxpayer Assistance
                                    Training

                                       a

GAO-05-782

[IMG]

July 2005

TAX ADMINISTRATION

IRS Needs Better Strategic Planning and Evaluation of Taxpayer Assistance
Training

  What GAO Found

IRS devotes significant resources to training its tax law assistors to
answer questions, by telephone and at walk-in sites, and prepare tax
returns. Although IRS cannot separate the costs of training tax law
assistors from other assistance staff, the thousands of staff devoted to
providing tax law assistance receive training each year. The training
includes classroom and computer-based training on such subjects as tax law
and communication. While IRS has some data on travel and course
development costs associated with training, it does not have data on what
is likely the largest cost component, the value of staff time devoted to
tax law training.

Responsibility for training IRS taxpayer assistance staff is
decentralized. IRS's Human Capital Office provides guidance and sets
policy. The two divisions responsible for tax law assistance each have a
human resources office with technical staff that are assigned to the
various taxpayer assistance programs. Generally speaking, the taxpayer
assistance programs share responsibility for planning training with human
resource staff. The human resource staff are responsible for training
evaluations.

IRS's planning for taxpayer assistor training could be enhanced by a more
strategic approach. To their credit, some taxpayer assistance programs
clearly communicated the importance of training to staff and had knowledge
and skills inventories. All the units had analyses of some of the
individual factors that affect accuracy, such as the quality and use of
their taxpayer assistance guidance. However, none of the programs had
long-term goals for either accuracy or training or had benchmarked their
training efforts against those of other organizations. Nor had they done a
combined analysis of the major factors that affect accuracy in order to
determine their relative importance. Setting long-term goals and analyzing
training needs and relative impacts are important steps in strategic
planning. Goals can provide a yardstick for measuring progress and
benchmarking can help identify best practices. Analyses of relative
impacts can help IRS make informed decisions about a strategy for
improving accuracy, including the importance of training compared to other
factors in that strategy.

The taxpayer assistance programs routinely conducted evaluations of their
training efforts. However, with one exception, the evaluations did not
include analyses of the impact of training on the accuracy of assistance.
Instead, the units conducted less sophisticated analyses of more immediate
impacts, such as trainees' satisfaction. Given the importance of accurate
answers to taxpayers' questions and the resources spent on training, the
four assistance programs would benefit from more sophisticated evaluations
of the effectiveness of training. One program had recognized the potential
value of a more sophisticated evaluation of training and pilot tested an
analysis in 2004. The value of evaluation is that it provides feedback
about the effectiveness of one year's training that can be used to plan
improvements to future training.

United States Government Accountability Office

Contents

  Letter

Results in Brief
Background
Scope and Methodology
Long-term Goals and Analyses of Needs and Impacts Could

Improve IRS's Planning for Training and Development Evaluating the Impact
of Training on Accuracy Could Provide a

Basis for Future Improvements Conclusions Recommendations Agency Comments
and Our Evaluation 1

3 4 7

9

15 19 20 21

Appendix I	What GAO Looked For: Examples of Planning Practices That Would
Conform to Strategic Guidance for Training

Appendix II	What GAO Looked For: Examples of Evaluation Practices That
Would Conform to Strategic Guidance 28

Appendix III	Assessments of Planning: Less Complex Tax Law Telephone
Service by W&I CAS

Appendix IV	Assessments of Planning: More Complex Tax Law Telephone
Service by SB/SE TEC

Appendix V	Assessments of Planning: Tax Law Questions Walk-In Service by
W&I CARE 34

  Appendix VI	Assessments of Planning: Return Preparation Walk-In Service by W&I
  CARE 36

     Appendix VII       Assessments of Evaluation: Less Complex Tax Law 
                               Telephone Service by W&I CARE               38 
     Appendix VIII      Assessments of Evaluation: More Complex Tax Law 
                               Telephone Service by SB/SE TEC           

Appendix IX	Assessments of Evaluation: Tax Law Walk-in Service by W&I CARE

Appendix X	Assessments of Evaluation: Return Preparation Walk-in Service
by W&I CARE

Appendix XI	Information on the Kirkpatrick Model of Training and
Development Evaluation

Appendix XII Comments from IRS

Appendix XIII GAO Contact and Staff Acknowledgments

  Tables

Table 1: Assessments of IRS's Planning Practices in Training and
Developing Staff to Provide Accurate Taxpayer Assistance 10

Table 2: Assessments of IRS Practices in Evaluating Training and
Development of Staff to Provide Accurate Taxpayer Assistance 16

  Figures

Figure 1: Planning and Evaluation as Part of Four Components of the
Training and Development Process 7 Figure 2: Example Agency's Training and
Development Programs Assessed Using Each Level of Evaluation 47

This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.

United States Government Accountability Office Washington, DC 20548

July 11, 2005

The Honorable Max Baucus Ranking Minority Member Committee on Finance
United States Senate

The Honorable Byron L. Dorgan United States Senate

Taxpayers expect timely and accurate assistance from the Internal Revenue
Service (IRS) when they have tax law questions or have tax returns
prepared. The quality of IRS's assistance can reduce the time and
aggravation of preparing tax returns and increase taxpayers' compliance
with the tax laws. In 2004, IRS answered almost 9 million tax law
questions by telephone and prepared almost half a million tax returns for
low income taxpayers.

IRS's past performance has shown that taxpayers cannot always rely on it
to provide accurate information. While the accuracy of IRS's taxpayer
assistance has improved in some cases, it has been inconsistent or below
expectations in others. After 2 years of decline, in the first weeks of
the 2005 filing season IRS telephone assistance accuracy was estimated at
87, percent compared to 76 percent for the same time period in 2004.
Although data on the accuracy of assistance at IRS's walk-in sites are
limited by not being representative, Treasury Inspector General for Tax
Administration (TIGTA) reports have raised concerns about the accuracy of
the returns IRS prepares and that IRS had not met its annual tax law
accuracy goal.1

This performance has raised questions about the adequacy of the training
IRS provides its taxpayer assistance staff. Although a number of factors
can affect the accuracy of assistance IRS provides taxpayers, effective
training and development programs can enhance an organization's ability

1Treasury Inspector General for Tax Administration, Improvements Are
Needed to Ensure

Tax Returns Are Correctly Prepared at Taxpayer Assistance Centers,
Reference No. 2004

40-025 (Washington, D.C.: 2003) and Treasury Inspector General for Tax
Administration,

Customer Service at the Taxpayer Assistance Centers Is Improving but Is
Still Not

Meeting Expectations, Reference No. 2005-40-021 (Washington, D.C.: 2004).

to achieve its mission and goals, such as improving accuracy. At the same
time training and developing staff is costly, making it important that
such investments are targeted strategically and not wasted on efforts that
are irrelevant, duplicative, or ineffective.

Because of your interest in ensuring that taxpayers receive accurate
information when they contact IRS for assistance, and the contribution
that training makes to that end, you asked that we review how IRS plans
and evaluates its primary taxpayer assistance training and development
efforts. We focused on planning and evaluation from a strategic
perspective, that is, how planning and evaluation of training can help
improve accuracy. As discussed with your offices, our objectives were to
assess whether IRS's processes for planning and evaluating the training
and development of taxpayer assistance staff conform to published
guidance.

In conducting our work we developed detailed criteria to assess how IRS
plans and evaluates its training and development of taxpayer assistance
staff based on guidance we published for assessing strategic training and
development efforts in the federal government.2 We developed 27 separate
criteria for planning and evaluation, such as establishing goals,
conducting knowledge and skills needs analyses, benchmarking against other
organizations, systematically collecting data, and comparing benefits and
costs. We shared these criteria with officials in IRS's Human Capital
Office and they said the criteria are appropriate and consistent with
their policy guidance. We assessed training and development for four types
of assistance: less complex tax law questions answered by phone, more
complex tax law questions answered by phone, tax law questions answered at
IRS walk-in sites, and return preparation at IRS walk-in sites. We
collected documents describing IRS's planning and evaluation process for
the four taxpayer assistance programs, interviewed officials in IRS's Wage
and Investment (W&I) and Small Business/Self-employed (SB/SE) divisions to
get more detail, and compared IRS's practices to our criteria. The scope
and methodology section provides more details.

We conducted our work from May 2004 through May 2005 in accordance with
generally accepted government auditing standards.

2GAO, Human Capital: A Guide for Assessing Strategic Training and
Development Efforts in the Federal Government, GAO-04-546G (Washington,
D.C.: March 2004).

  Results in Brief

IRS's planning for taxpayer assistance training could be enhanced by
longterm goals and analyses of the relative importance of the factors that
affect accuracy, other organizations' experiences, and gaps between
longterm needs and existing skills. To their credit, all four taxpayer
assistance programs had efforts that conformed, at least in part, to many
of our planning criteria. They had efforts to involve stakeholders in key
annual planning decisions, communicate the importance of training to
staff, and plan changes to their training to address annual tax law
changes. The programs also had analyses of some of the individual factors
that affect accuracy, such as the quality and use of their taxpayer
assistance guidance. Two programs had annual goals for accuracy. However,
the programs did not conform to other criteria. None of the programs had
long-term accuracy goals, training goals, or measures suitable for an
assessment of the impact of training on accuracy. Nor had they determined
the relative importance of the various factors that impact accuracy,
benchmarked the training practices of other organizations, or conducted
assessments of long-term skill needs. Setting goals and conducting these
analyses could set a direction and provide a more informed basis for
planning improvements to training in order to improve accuracy.

The four taxpayer assistance programs routinely conducted evaluations of
their training efforts, but these efforts did not fully comply with
criteria for strategic evaluations. IRS's Human Capital Office has policy
guidance urging such evaluations. One program met some of the criteria for
evaluating the impact of training on accuracy. Given the importance of
accurate answers to taxpayers' questions and the resources spent on
training, the four assistance programs would benefit from more
sophisticated evaluations of the effectiveness of training. One program
had recognized the potential value of a more sophisticated evaluation of
training and pilot tested such an analysis in 2004 and plans to conduct a
similar analysis in 2005.

We are making recommendations to the Commissioner of Internal Revenue to
improve IRS's planning and evaluations of taxpayer assistance training and
development efforts. The recommendations include establishing long-term
goals, determining the importance of the various factors that affect
accuracy, and conducting long-term skills and knowledge gap analyses.

In commenting on a draft of this report (see app. XII), the Commissioner
of Internal Revenue agreed with five of our eight recommendations and
partially responded to the remaining three recommendations.

Background

IRS provides tax law assistance to taxpayers through IRS's toll-free
telephone lines and tax law and return preparation assistance in person at
IRS taxpayer assistance centers (formerly known as "walk-in sites")
nationwide, in addition to other means. The four taxpayer assistance
programs we reviewed were:

o  	W&I Customer Accounts Services (CAS) general tax law assistance by
telephone. Many taxpayers contact IRS by calling its toll-free telephone
number which is operated and staffed by CAS.3 According to IRS fiscal year
2004 data, 3,420 CAS staff4 handled more than 8.7 million telephone calls
from taxpayers with general, less complex tax law questions. We did not
verify these IRS data or other IRS data on taxpayer assistance programs'
workload and staffing.

o  	SB/SE Taxpayer Education Communication (TEC) complex tax law
assistance by phone. According to IRS officials, during the 2005 filing
season and in previous years, TEC staff supported telephone service by
answering taxpayers' questions on selected, more complex tax topics. When
taxpayers called IRS's toll-free number about these topics, W&I CAS staff
recorded the taxpayers' contact information and questions so that TEC
staff could call the taxpayers back later to provide answers. According to
IRS, in the 2004 filing season, TEC handled about 320,000 telephone calls
for complex tax law assistance. According to IRS, TEC trained
approximately 400 staff and used 272,212 staff hours to provide telephone
assistance in 2004. In 2006 these calls will be handled by W&I CAS staff.

o  	W&I Customer Assistance, Relationships, and Education (CARE) walk-in
tax law assistance. According to IRS, about 1.4 million staff hours were
devoted to providing walk-in assistance in fiscal year 2004. Also
according to IRS, 1,654 staff were trained to provide tax law assistance
in 2004. In 2004, there were about 7.7 million contacts with taxpayers at
IRS's approximately 400 taxpayer assistance centers. IRS did not have
information on how many of these contacts were for tax law assistance.
Many of the contacts were for other services, such as tax forms,
publications, or accounts issues.

o  	W&I Customer Assistance, Relationships, and Education (CARE) walk-in
return preparation assistance. According to IRS data, IRS staff prepared
476,813 tax returns in fiscal year 2004. IRS did not have data on

3Millions of taxpayers use IRS's Web site for taxpayer assistance.

4This is the total number of staff that were assigned to provide tax law
assistance by telephone. The staff also had other duties. IRS could not
provide data on the time staff spent providing telephone assistance.

the number of staff that prepared tax returns at taxpayer assistance
centers.

IRS does not have data on the amount of time its assistors spend annually
being trained on tax law.5 However, all staff providing tax law assistance
receive some training each year. IRS has continuing professional education
and refresher training requirements for all its taxpayer assistance staff.

Tax law assistance staff receive a variety of training. According to an
IRS official, the staff get primarily classroom training but also receive
computer-based training. On-the-job training, managerial coaching, and
workshops are also part of training. Taxpayer assistance staff receive
training on technical tax law topics, how to use IRS systems and guidance
to answer questions, and communication.

Responsibility for training and developing IRS's tax law assistance staff
is decentralized. IRS's Human Capital Office provides guidance and sets
policy and standards on training and development for the agency. W&I and
SB/SE each have a human resources office with Learning and Education (L&E)
staff who are assigned to the taxpayer assistance programs. L&E staff
provide program staff advice and analysis on related policies and issues
and formulate strategies, procedures, and practices to address the
programs' human capital needs, including training. Generally speaking, the
taxpayer assistance program offices identify training needs, L&E staff
work with program staff to develop and fund annual training plans, and the
program offices administer training. According to IRS policy, L&E staff
are responsible for evaluating training.

According to fiscal year 2004 IRS data, IRS invested about $7 million in
training W&I CAS and CARE staff to provide taxpayer assistance, including
such expenses as travel, supplies, contractor fees, and development costs.
IRS could not separate these costs into amounts spent on tax law training
and other topics. According to SB/SE TEC fiscal year 2004 data, training
costs were $325,072 in student and instructor travel. However, the cost
data IRS provided did not include the costs of assistors' time associated
with training. As with the staffing and workload data, we did not verify
the accuracy of IRS's training cost data.

5In commenting on a draft of this report, an IRS official noted that IRS
tracks training time, but does not have time data specific to tax law
training.

In March 2004, we issued an assessment guide that introduced a framework
for evaluating a federal agency's training and development efforts.6 This
assessment guide consists of a set of principles and key questions that
federal agencies can use to ensure that their training and development
investments are targeted strategically and are not wasted on efforts that
are irrelevant, duplicative, or ineffective. As detailed in our assessment
guide, the training and development process can loosely be segmented into
four broad, interrelated components: (1) planning/frontend analysis, (2)
design/development, (3) implementation, and (4) evaluation. Figure 1
depicts the general relationships between the four components, including
the feedback loop between evaluation and planning. Planning and evaluation
are highlighted because they are the focus of this report. Although these
components can be discussed separately, they are not mutually exclusive
and encompass subcomponents that may blend with one another. For instance,
evaluation is part of the planning as organizations should reach agreement
up front on how the success of training and development efforts will be
assessed.

6GAO-04-546G.

Figure 1: Planning and Evaluation as Part of Four Components of the
Training and Development Process

TIGTA is conducting a review that covers some aspects of the design and
implementation of training.

Scope and	Our work examined the training and development of employees who
provide tax law and return preparation assistance to taxpayers over the

Methodology 	telephone and at walk-in centers, and covered both seasonal
and full-time employees in IRS's W&I and SB/SE divisions. Our assessment
of IRS's training and development program for taxpayer assistance
employees was

based on analyses of IRS data and interviews with IRS officials. We also
obtained background information from sources outside IRS such as TIGTA,
the IRS Oversight Board, and the IRS National Taxpayer Advocate.

To assess how IRS plans for and evaluates the training and development of
IRS employees who provide tax law and return preparation assistance to
taxpayers over the telephone and at walk-in centers, we used our guide for
assessing training and development programs in the federal government7 as
a framework.

We used the parts of the GAO guide dealing with the planning and
evaluation components, along with supplemental guidance from the Office of
Personnel Management, to identify the detailed strategic training and
development criteria applicable to IRS and organized them into
chronological phases. We focused our criteria on strategic planning and
evaluation, that is, planning and evaluation intended to help achieve
IRS's accuracy goals. We developed 27 criteria, listed in appendices I and
II. We also developed examples of evidence that would demonstrate
conformance with each of the criteria. For example, for the criterion
"conduct a knowledge and skills inventory to identify employees' current
skills and knowledge," we looked for evidence such as a completed
knowledge and skills surveys and proficiency tests (see app. I for the
planning phase criteria and further evidence examples and app. II for the
same information on evaluation.) We shared the criteria with officials in
IRS's Human Capital Office and the taxpayer assistance programs and
assigned L&E staff. The officials from the Human Capital Office said the
criteria are appropriate and consistent with their policy guidance. Some
program and L&E officials expressed concerns about their need and ability
to satisfy the criteria. Their concerns are discussed in the planning and
evaluation sections of this report.

In applying the criteria to IRS, we collected documents describing IRS's
planning and evaluation processes for the four types of assistance. We
also interviewed officials responsible for the taxpayer assistance
programs in W&I CAS and CARE units and SB/SE TEC unit and associated L&E
staff responsible for training planning and evaluation, to get more
details where needed. We reviewed all the evidence and made a judgment
about the extent to which IRS's practices conformed to the criteria. We
then discussed our initial assessments with IRS officials responsible for

7GAO-04-546G.

planning and evaluating the taxpayer assistance training and development
programs who, for some of our preliminary assessments, provided additional
written evidence for us to consider in making our final assessments. We
then revised our initial assessments based on the evidence they provided.
We made our assessments in two steps. First, an analyst reviewed all the
evidence and made a judgment about the extent to which it conformed to the
criteria. Then a second analyst independently reviewed the assessments.
The evidence supporting our assessments is provided in appendices III
through X.

Much of the information we relied on was descriptive. We determined that
the information was reliable for our purposes. To the extent possible we
corroborated interview evidence with documentation. Where not possible,
the description is attributable to IRS officials. Where relevant we
corroborated that policy guidance, such as Internal Revenue Manual
guidance, was being implemented by collecting documentation and reports
showing implementation. With respect to controls over databases, we
reviewed documentation of the controls but did not assess their adequacy
or test data in the databases.

We conducted our work at the Wage and Investment Division headquarters in
Atlanta, Ga.; the Small Business/Self-Employed Division offices in
Philadelphia, Pa.; and IRS headquarters Human Capital Office in
Washington, D.C. from May 2004 through May 2005 in accordance with
generally accepted government auditing standards.

Summarizing table 1, IRS's planning for taxpayer assistance training
primarily focuses on meeting short-term needs, such as the challenge of
training staff about tax law changes in preparation for the next year's
filing season. Planning could be enhanced by long-term goals and analyses
of the relative importance of the factors that affect accuracy, other
organizations' experiences, and gaps between long-term needs and existing
skills. Table 1 shows our assessments of planning for training by IRS's
four primary taxpayer assistance programs for each phase of the planning
process: goals and priorities, information gathering and assessments,
skills and knowledge analysis, and strategy development and selection. The
evidence supporting our assessments is shown in appendices III through VI.

  Long-term Goals and Analyses of Needs and Impacts Could Improve IRS's Planning
  for Training and Development

  Table 1: Assessments of IRS's Planning Practices in Training and Developing
                 Staff to Provide Accurate Taxpayer Assistance

Extent to which IRS's practices conform to strategic guidance by type of
service

        Telephone tax law Walk-in assistance Planning phase and criteria

General (CAS)

More complex (TEC)

  Tax law (CARE) Return preparation (CARE) Phase: goals and priorities Phase:
                     information gathering and assessments

Establish quantitative long-term accuracy goals that link  0L  0m  0L   0m 
                    to IRS's strategic goals                              
       Involve key stakeholders, including human capital      0L  0L  0L   0L 
      professionals, managers, and employees, in key long-                
                    term planning decisions                               
        Establish goals for, and measures to assess the       0m  0m  0m   0m 
     effectiveness of, training and development on accuracy               
             Affirm, through upper-level management           0L  0L  0M   0M 
         communication, the importance of training and                    
                development to improve accuracy                           

    Analytically determine and track the strategic         0L0L         0L 0L 
                         and                                               
    operational factors, including training, that                          
                        affect                                             
                       accuracy                                            
                                                    See table 2 for our    
Incorporate evaluation results into training and     assessments        
                 development planning                                      
    Benchmark the training and development program         0m0m         0m 0m 
        against high-performing organizations                              

Assess whether human resource information systems It is too early for an
assessment on the human resource information. For provide needed data in a
timely manner details see appendices III through VI.

 Phase: skills and knowledge analysis Phase: strategy development and selection

        Perform a needs assessment to determine the        0L   0L   0L    0L 
     knowledge and skills needed now and in the future                   
    Conduct a knowledge and skills inventory to identify   0M   0m   0M    0M 
          employees' current skills and knowledge                        
    Perform a gap analysis to determine the differences    0L   0m   0L    0L 
      between current and needed skills and knowledge                    

      Develop and apply criteria to determine when to use     0m  0m  0m   0m 
     training and development strategies to fill skills and               
                         knowledge gaps                                   
     Ensure training and development efforts target factors   0L  0L  0L   0L 
    that affect accuracy and are linked to needed skills and              
                           knowledge                                      

Extent to which IRS's practices conform to strategic guidance by type of
service

                      Telephone tax law Walk-in assistance

General (CAS)

More complex (TEC)

      Tax law (CARE) Return preparation (CARE)Planning phase and criteria

        Consider the anticipated costs and benefits of        0m  0m  0m   0m 
     alternative training and development efforts, ways to                
    mitigate associated risks, and the appropriate level of               
            investment for training and development                       

Establish a detailed plan for evaluating training, See table 2 for our
assessments
including performance measures, data, planned
analyses, and how the analyses would be used

Establish process to ensure that strategic and tactical 0L0L0L0Lchanges
can be incorporated into T&D efforts

Source: GAO analysis.
Key:
Conforms to guidance to a great extent .0MConforms to guidance to
some extent 0LConforms to guidance to little or no extent 0m

                              Goals and Priorities

In the goals and priorities phase, the four units were relatively strong
in involving key stakeholders and communicating the importance of
training. The programs involved key stakeholders in annual training
planning decisions, shown by half-circles in table 1. However, they did
not have long-term planning processes in which to involve stakeholders.
All four programs had some upper-level management efforts to communicate
to staff the importance of training and development in improving accuracy.
CARE's was a model of communicating to all levels of staff the importance
of training. For example, CARE's management guidance included managers
responsible, communication vehicles, key dates, and the message to be
conveyed. The other programs communicated with managers, but did not
communicate more widely through the organization.

With respect to goals, IRS does not have long-term goals, as opposed to
annual goals, for accuracy; nor do the four programs have goals for
training and development or measures of the impact of training on
accuracy. This observation about the lack of goals for accuracy is
consistent with other recent reports where we cited a lack of IRS

long-term goals, as well as reports and other assessments by OMB and
TIGTA.8 Two of the programs had annual goals, shown by half-circles.

Some taxpayer assistance officials that we talked with questioned the need
for long-term accuracy goals or training and development goals and
measures. For example, W&I CAS telephone assistance officials said setting
long-term accuracy goals is not necessary because their annual accuracy
goals would move IRS toward improved performance. Program officials also
said that since staff training is considered a part of doing business and
is not managed as a program, training goals are unnecessary. Further, some
said that, given the number of factors that affect accuracy, developing
measures of training effectiveness would not be possible.

However, without long-term accuracy goals, IRS, Congress, and others are
hampered in evaluating whether IRS is making satisfactory progress
improving taxpayers' service. As we said in a prior report on IRS's
telephone assistance program, a long-term, results-oriented goal would
provide a meaningful sense of direction as well as a yardstick for
measuring the results of operations and evaluating the extent of
improvements resulting from changes in resources, new technology, or
management of human capital.9 Similarly, goals and performance measures
for training would provide direction and help measure progress. We
recognize that collecting performance data can sometimes be challenging,
as discussed in the evaluation section. One IRS taxpayer assistance
program has been collecting performance data suitable for determining the
effectiveness of training.

Information Gathering and Assessments

In this phase, the four programs determined and tracked selected factors
that affect accuracy, but would benefit from a more complete analysis as
well as benchmarking. IRS has identified, and tracked on an individual
basis, selected factors that affect accuracy. One nontraining factor that
IRS has tracked and analyzed is the use and quality of the Probe and
Response Guide, a manual provided to taxpayer assistance staff intended

8For example, see GAO, Internal Revenue Service: Assessment of Fiscal Year
2006 Budget Request and Interim Results of the 2005 Filing Season,
GAO-05-416T (Washington D.C.: Apr. 14, 2005); and Treasury Inspector
General For Tax Administration, The Accounts Management Program Has Annual
Performance Goals But Should Develop Long-term Performance Goals,
Reference No. 2005-40-079 (Washington D.C: May 6, 2005).

9U.S. General Accounting Office, IRS Telephone Assistance: Opportunities
to Improve Human Capital Management, GAO-01-144 (Washington, D.C.: Jan.
30, 2001).

to guide them in responding to taxpayers' questions. IRS blamed decreases
in accuracy in 2003 and 2004 on problems with the Probe and Response
Guide. According to IRS officials, they used their analysis of the Probe
and Response Guide to make the guidance more usable by taxpayer assistance
staff. Officials attribute improvements in accuracy in 2005, at least in
part, to this effort.

However, IRS has not conducted an analysis of the factors that affect
accuracy, including training-shown by half-circles in Table 1.
Specifically, IRS has not done an analysis to determine the relative
importance of the various factors, including training, that affect
accuracy. Determining the relative importance of the factors that affect
accuracy is important because there are multiple factors. Some factors
that affect accuracy are strategic and outside of IRS's direct control
such as tax law changes. Other factors are operational and subject to IRS
control such as manuals, information systems, and training. Without an
understanding of the relative importance each factor has on accuracy, it
is difficult for IRS to make informed decisions about a strategy for
improving accuracy, including training's role in that strategy. Because of
the purpose of the analysis and the number of factors involved, the
analysis might not give a precise measure of each factor's impact.
Furthermore, such analyses may be conducted on an occasional basis.

None of the taxpayer assistance programs collected another type of
information-best practices of other organizations learned through
benchmarking. Some officials told us it would not be possible to benchmark
training programs because IRS has a unique mission. However, the telephone
taxpayer assistance programs have benchmarked other aspects of their
operations such as performance measures. Also, as we have stated in
another report, many processes that seem unique to the government actually
have counterparts in the private sector.10 Looking at processes in
dissimilar organizations can lead to rewarding improvements because it
stimulates new thinking about traditional approaches to doing work. For
example, in the report cited above, we noted that Xerox benchmarked L.L.
Bean to improve order fulfillment.

Still another type of information useful for planning can be obtained from
evaluations of training efforts. As we discuss in the background,

10GAO, Tax Administration: Planning for IRS's Enforcement Process Changes
Included Many Key Steps But Can Be Improved, GAO-04-287 (Washington, D.C.:
Jan. 20, 2004).

evaluation can provide useful feedback about the impact of existing
training to help plan improvements to training. Evaluation of IRS's
training efforts is the subject of our second section.

    Skills and Knowledge Analysis

All four assistance taxpayer assistance programs conducted annual needs
assessments. These assessments identified the knowledge and skills
assistors needed for the following year. None of them had longer term
analyses to project future needs. Determining skills and knowledge needed
is challenging for IRS, especially when IRS must react quickly to tax law
changes. For example, Congress passed a law in early January 2005 allowing
taxpayers to deduct on either their 2004 or 2005 tax returns contributions
made during January 2005 for relief of the victims of the December 2004
Indian Ocean Tsunami.11 According to an IRS official, IRS had to react to
this change, which took place after the filing season had already begun,
by quickly alerting staff and providing them the information necessary to
assist taxpayers who had questions about the deduction.

However, none of the four programs had analyses of long-term needs, such
as improved proficiency in doing research, which is the basis for the
halfcircles in table 1. Longer-term analyses of needs could help the four
programs better plan strategies to meet future needs. Such planning could
help ensure that the programs acquire, train, and retain the needed staff.

Three of the four programs had knowledge and skills inventories based on
actual testing to determine employees' proficiency and knowledge levels.
TEC was the exception. TEC assumes the employees have an underlying
technical foundation because of their background in IRS. However, in a
recent TEC survey of 133 employees, the percentage of employees who
self-reported having pre-existing skills or being fully proficient in
several technical categories, such as depreciation, sale of property, and
trusts and fiduciaries, ranged from 31 percent to 62 percent.

Three of the four programs performed gap analyses to determine the
difference between current and needed skills and knowledge. Because of the
above limitations in needs analyses, the gap analysis was necessarily
annual not long term. Again, TEC was the exception. Because TEC had not
done a needs analysis, it could not do a gap analysis.

11Pub. L. No. 109-1 (2005).

    Strategy Development and Selection

  Evaluating the Impact of Training on Accuracy Could Provide a Basis for Future
  Improvements

Weaknesses in analyses, described in the previous sections, hampered the
four programs' ability to develop criteria for when to use training, as
opposed to other strategies, for filling knowledge and skills gaps, such
as hiring and retention efforts. In addition, the four programs lacked
information about the benefits and costs of their training efforts. As
noted in the background section, IRS trains several thousand tax law
assistance staff annually but does not have data on the cost of this
significant effort. Without an understanding of the usefulness of other
strategies, and without an understanding of the benefits and costs of
training, IRS lacked information that would be useful for making resource
allocation decisions. This matters because the resources devoted to
training are significant. With better information to select and develop a
strategy, IRS might be able to improve accuracy and save resources.

Given the importance of accurate answers to taxpayers' questions and the
resources spent on training, the four assistance programs would benefit
from more sophisticated evaluations of the effectiveness of training. The
four programs all conduct some evaluations of their training efforts.
However, only TEC attempted an evaluation of the impact of training on
accuracy. In table 2, the lack of evaluation plans in the other three
programs had a cascading effect that left the programs generally unable to
fully satisfy our assessment criteria in subsequent evaluation phases-data
collection, data analysis, and application of evaluation results.
Appendices VII through X show the evidence supporting our assessments in
table 2.

Table 2: Assessments of IRS Practices in Evaluating Training and
Development of Staff to Provide Accurate Taxpayer Assistance

Extent to which IRS's practice conforms to strategic guidance by type of
service

                      Telephone tax law Walk-in assistance

ReturnGeneral Complex Tax law preparation Evaluation phase and criteria
(CAS) (TEC) (CARE) (CARE)

                 Phase: evaluation plan Phase: data collection

     Establish an overall approach for evaluating the impact of   0L 0M 0L 0L 
                        training on accuracy                               
Systematically analyze the feasibility and cost effectiveness  0m 0m 0m 0m 
     of alternative methodologies for evaluating the impact of             
                training and development on accuracy                       
       Before implementing training, develop analysis plans,      0L 0M 0L 0L 
including data to be collected, analyses to be conducted, and           
                   how the analyses will be used                           

Ensure that data are collected systematically and in a timely 0L0M 0L0L
manner

     Ensure that data are accurate, valid, complete, and reliable 0L0L 0L0L

         Phase: data analysis Phase: application of evaluation results

Analyze data collected to assess the impact of training on  0L  0L  0L  0L 
                            accuracy                                       
           Compare accuracy benefits to training costs         0m  0m  0m  0m 
Benchmark training cost, training outcomes (accuracy), and  0m  0m  0m  0m 
    analytical methods against high-performing organizations               
Analyze internal and external stakeholders' assessments of  0L  0L  0L  0L 
             training to include impact on accuracy                        

        Evaluate training program as a whole, in addition to      0m 0L 0m 0m 
                             individual                                    
            course evaluations, and document the results                   
        Incorporate evaluation results into the training and      0L 0L 0L 0L 
              development program to improve accuracy                      

Source: GAO analysis.
Key:
Conforms to guidance to a great extent .0M
Conforms to guidance to some extent 0L
Conforms to guidance to little or no extent 0m

    Evaluation Plan Phase

IRS adopted a four-level model based on the widely accepted Kirkpatrick
model12 for evaluating training. Under this model, the sophistication of
analysis increases as the numerical level of analysis increases:

o  	Level 1 - Reaction. The goal is to measure participants' reaction to
training, usually through questionnaires.

o  	Level 2 - Learning. The goal is to determine what the training
participants learned through various kinds of tests administered
immediately after the training is completed.

o  	Level 3 - Behavior. The goal is to determine if the job performance of
the training participants changed in the aftermath of the training. The
most common means for making this determination is the administration of a
survey of trained staff and their supervisors 3 months, on average, after
training is complete.

o  	Level 4 - Results. The goal is to determine if the training led to the
desired results, in this case, improved accuracy of taxpayer assistance.

Options for level 4 analyses might include statistical correlations
between training and accuracy and controlled experiments where some staff
receive new training and others do not. Such ROI analyses are not a part
of IRS's evaluation approach. This level is sometimes split into two
levels with the fifth level-often referred to as return on investment
(ROI)- representing a comparison of costs and benefits quantified in
dollars.

Not all training and development programs are suitable for evaluations of
their effect on organizational results. The difficulty and costs of
analyzing the impact of training on accuracy need to be weighed against
the benefits of such an evaluation.

However, the more significant the activity targeted by the training and
development program, the greater the need for level 4 analysis. Factors to
consider when deciding the appropriate level of evaluation include
estimated costs of the training effort, size of the training audience,
management interest, program visibility, and anticipated life span of the
effort. As noted in the background, IRS devotes significant resources to
assisting taxpayers and training assistance staff. Congress has also
expressed great interest in improving taxpayer service, such as accuracy.

12Donald L. Kirkpatrick, Evaluating Training Programs: The Four Levels
(San Francisco: Berrett-Koehler, 1994).

For example, much of the focus of the Internal Revenue Service Reform and
Restructuring Act of 199813 was on in improving taxpayer service.

IRS's Human Capital Office recognizes the need to do level 4 evaluations.
The office has a policy statement stating that level 4 evaluations should
be done for all mission-critical training.

Despite the large investment of resources, significant congressional
attention, and Human Capital Office guidance, IRS officials involved in
three of the four taxpayer assistance training programs had not agreed on
whether to conduct a level 4 evaluation. They conducted analyses only at
lower levels, usually levels 1 or 2. By contrast, officials at L&E and TEC
agreed to conduct a pilot test of a level 4 analysis in 2004 for more
complex tax law questions answered by telephone. Although the pilot test
was not successful in 2004-for reasons discussed below in the data
analysis phase-L&E and TEC intend to use the data collection and analysis
plan created for 2004 to conduct a similar evaluation in 2005.

                             Data Collection Phase

For three of the four taxpayer assistance programs, as shown in table 2,
the lack of level 4 analyses had a cascading effect that left the programs
unable to fully satisfy our assessment criteria in the subsequent
evaluation phases. All four programs had controls in place to help ensure
systematic, timely data collection for level 1 and level 3 evaluations.
TEC also had such controls in place for its level 4 evaluation. However,
the database used to store level 2 data-the Administrative Corporate
Education System (ACES)-is no longer in place, and there is no replacement
system planned. With the exception of TEC, the programs did not attempt to
collect level 4 data. Database controls and data collection plans for
TEC's level 4 pilot evaluation helped ensure the systematic and timely
collection of data.

All four units had controls in place to help ensure that level 1 and level
3 data were accurate, valid, reliable, and complete.14 Because of the lack
of a level 2 database, none of the four units had controls in place for
that data.

13Pub. L. No. 105-206 (1998).

14Our review was to determine if some basic controls were in place to
ensure that data were collected timely and systematically and that data
were accurate, valid, reliable, and complete. We did not assess or test
the adequacy of the controls, data collection methods, or data quality or
completeness.

TEC was concerned that the accuracy data, which were based on test
telephone calls, did not reflect the types of calls that taxpayers
actually made. TEC and IRS's quality review staff are working together to
try to address these concerns.

                              Data Analysis Phase

All four assistance programs conducted some analyses of their training
efforts and analyzed stakeholders' assessments of training to identify
potential improvements to individual courses. A level 1 analysis, usually
surveys of trainees' opinions about a course, was conducted for all
courses. The officials also said level 2 analyses, often testing, are
generally done while level 3 analyses are infrequently done.

None of the four assistance programs successfully completed a level 4
evaluation of the impact of training on accuracy, compared benefits to
costs, benchmarked training evaluation, or analyzed stakeholders'
assessments of the impact of training on accuracy. The 2004 TEC level 4
pilot test was not completed because the level of proficiency of TEC staff
in the five technical categories reviewed was insufficient.

    Application of Results Phase

Conclusions

All the business units applied the results of their data analysis to
individual training courses, not their training efforts as a whole. For
example, they used the results of their level 1 and level 3 analyses to
make improvements in individual training courses and to identify training
needs for the upcoming year. However, because there had been no successful
evaluations of the impact of training on accuracy, evaluation results
could not be used to plan a strategy to improve accuracy.

Training has the potential to improve the service received from IRS by
millions of taxpayers. To its credit, IRS has planning processes in place
to address the challenges of training staff for each year's filing season.
The challenges include training staff to answer questions about annual tax
law changes-changes that are often very complex. Although we do not know
how much training has contributed, IRS's taxpayer assistance accuracy has
improved in recent years. On the other hand, IRS's current level of
accuracy remains a concern, especially the accuracy of walk-in assistance
and return preparation. In addition, IRS devotes significant resources to
training its tax law assistance staff.

A more strategic approach to planning and evaluation would have several
benefits. Strategic planning could help managers better understand the

extent to which training, as opposed to other factors that affect
accuracy, could be used to improve accuracy. Evaluations of training's
impact on accuracy could help managers better understand which specific
training techniques are effective and which are not. While potentially
difficult to design and costly to conduct, IRS's Human Capital Office has
policy guidance urging more such analyses. In addition, one taxpayer
assistance program is pilot testing such an analysis. Gap analyses,
evaluations, and cost-benefit comparisons might also contribute to
providing training at lower cost by distinguishing between effective and
ineffective training.

                                Recommendations

We recommend that the Commissioner of Internal Revenue take appropriate
action to ensure that IRS's planning and evaluations of its taxpayer
assistance training and development efforts better conform to guidance for
strategically planning and evaluating the training efforts.

Specifically, in the area of planning, we recommend that IRS:

o  Establish a long-term goal for the accuracy of taxpayer assistance.

o  	Establish goals and measures for training and development logically
linked to accuracy.

o  	Determine and track the relative importance of the various factors,
including training, that affect accuracy.

o  	Benchmark training and development programs against high-performing
organizations.

o  	Conduct skills and knowledge gap analyses for all taxpayer assistance
programs, to include identifying and comparing current skills to long-term
skill needs.

o  	Consider costs, benefits, ways to mitigate risks, and the appropriate
level of investment for training and development efforts.

In the area of evaluation, we recommend that IRS continue to pursue the
level 4 pilot in TEC and, if that analysis is shown to be feasible,
conduct level 4 evaluations for its other taxpayer assistance training and
development programs. The evaluations should include the following:

o  	an analysis of the feasibility and cost effectiveness of alternative
level 4 methodologies and a data collection and analysis plan,

o  a comparison of the accuracy benefits to the costs of the training,

o  	benchmarking of the analytical methods and the results of the data
analysis against high-performing organizations, and

o  	an analysis of stakeholder assessments of the impact of training on
accuracy.

  Agency Comments
  and Our Evaluation

We also recommend that IRS replace the defunct ACES database, which had
been used to store level 2 data, with another database for this purpose.

The Commissioner of Internal Revenue provided written comments on a draft
of this report in a letter dated July 6, 2005 (see app. XII). He said the
report offers valuable insight, is timely, and has been shared with the
project manager for IRS's recently-initiated effort to reengineer its
learning and education processes. Of the eight recommendations we made,
the Commissioner agreed with five recommendations: (1) establish long-term
accuracy goals, (2) determine and track the relative importance of factors
that affect accuracy, (3) benchmark training and development programs, (4)
conduct level 4 evaluations, and (5) replace IRS's system for storing
level 2 evaluation data.

The Commissioner partially responded to the remaining three
recommendations. In commenting on our recommendation to establish training
goals and measures linked to accuracy, the Commissioner focused on
evaluation. Although related to after-the-fact evaluation, setting clear
training goals and measures to ascertain progress toward those goals-
consistent with agency goals which, in the case of taxpayer assistance,
would include accuracy-is an important up-front planning step by which key
stakeholders should agree on what training success is and how it will be
assessed.

The Commissioner said that IRS recognizes the value of conducting skills
and knowledge gap analyses. He summarized the programs' efforts to
identify short-term skills needs, which we recognized in this report.
However, he did not identify how short-term gap analyses would be
conducted for the program for responding to more complex tax questions.
Nor did the Commissioner discuss gap analyses to identify long-term skills
needed to reach accuracy goals. Failing to conduct gap analyses, including
analyses of strategic changes-such as economic, technological, and
demographic developments-can hinder performance and the development of
strategies that integrate new capabilities and provide flexibility to meet
new challenges and improve service.

In commenting on our recommendation that IRS consider the costs and
benefits of training efforts, the Commissioner's comments did not
specifically mention benefits and costs but did mention unfunded needs.
Given the resources dedicated to training staff and providing taxpayer

assistance, and the impact that assistance accuracy can have on taxpayers,
the taxpayer assistance programs would benefit from analytically-based
assurance that training efforts focus in a cost beneficial way on
achieving accuracy goals.

As we agreed with your offices, unless you publicly announce the contents
of this report earlier, we plan no further distribution of it until 30
days from the date of this letter. We will then send copies of this report
to the Chairman of the Senate Committee on Finance, the Chairman and
Ranking Minority Member of the House Committee on Ways and Means, and the
Chairman and Ranking Minority Member, Subcommittee on Oversight, House
Committee on Ways and Means. We will also send copies to the Secretary of
the Treasury; the Commissioner of Internal Revenue; the Director, Office
of Management and Budget; and other interested parties. We will also make
copies available to others on request. In addition, the report will be
available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact
me at (202) 512-9110 or [email protected]. Contact points for our Offices of
Congressional Relations and Public Affairs may be found on the last page
of this report. GAO staff who made major contributions to this report are
listed in appendix XIII.

James R. White Director, Tax Issues

Appendix I: What GAO Looked For: Examples of Planning Practices That Would
Conform to Strategic Guidance for Training

     Phase           Criterion                  Conforming examples           
Goals and  Establish quantitative   Authoritative quantitative targets for 
priorities long-term accuracy                        the level of accuracy 
              goals that link to IRS's to be achieved at a future point in    
              strategic goals          time across multiple                   
                                                    fiscal years              

A logical link between the accuracy goals and higherlevel strategic goals
e.g., such as to improve customer service

Involve key stakeholders, including human capital professionals, managers,
and employees, in key long-term training planning decisions

Establish goals for, and measures to assess the effectiveness of, training
and development on accuracy Documented management processes to ensure that
stakeholders' views are taken into account in key planning milestones and
related decisions, such as developing the training and development budget
request and what measures will be used to evaluate training and
development

Surveys or interview employees to determine their views and perceptions on
training and development in general and more specifically on competencies
and skills needed for the future

Authoritative quantitative targets related to training and development
program activities to be achieved at a future point in time that are
logically linked to accuracy goals

A documented set of established measures used to assess the impacts of the
training and development programs, such as on accuracy or employee skills
and knowledge

A list of methods, tools, and measures used to assess the impact of
training and development, such as control groups, surveys, or a trend
analysis

Targets and goals in strategic and performance plans that establish how
training and development strategies are expected to contribute to improved
organizational and programmatic results Affirm, through upper-level
management communication, the importance of training and development to
improve accuracy Demonstrated efforts to communicate throughout the
organization the importance that upper-level management attaches to
training and development, such as memos, letters, statements found in
strategic plans, or other authoritative documents that articulate
management emphasis on training and development to improve accuracy

Comments from midlevel mangers and employees on the extent to which
management communicates the importance of training, such as statements
captured in an employee survey

Appendix I: What GAO Looked For: Examples of Planning Practices That Would
Conform to Strategic Guidance for Training

                      Phase Criterion Conforming examples

Information gathering and Analytically determine and track the strategic

assessments 	and operational factors, including training, that affect
accuracy

Management processes to systematically collect and analyze data that will
allow IRS to comprehensively identify and report on the major factors
affecting the level of accuracy achieved such as:

Strategic factors (factors largely outside IRS's immediate control) that
might include:

o  Complexity/ changes of the tax law

o  Levels of funding or budget limitations

o  Economic trends

o  Available pools of staff to select from

Operational factors (factors within IRS's immediate control) that might
may include:

o  Workload placed upon assistors

o  Changes in CSR guidance

o  Available learning tools and job aides

o  	Community of practice & communication of lessons learned

o  Manager/employee relations

o  Incentive/rewards & recognition program

Documentation that outlines the analytical methods used to identify
factors affecting accuracy

Periodic, scheduled, systematic studies to determine the key factors
affecting accuracy

Periodic regular efforts to systemically monitor the impacts strategic and
operational factors have had on accuracy, such as studies of factors over
an extended amount of time, such as 2-3 years, to gauge whether IRS has
experienced increased or decreased levels of accuracy due to these factors
Incorporate evaluation results into training See appendix II and
development planning Benchmark the training and development Periodic
studies conducted of the training and program against high-performing
development programs of high-performing customer organizations service
organizations providing similar services

compared to identify potential improvements, to include

such elements as, for example:

o  Employee skill level

o  Resources invested in training and development

o  Training and development curriculum

o  Geographic and demographic trends

Appendix I: What GAO Looked For: Examples of Planning Practices That Would
Conform to Strategic Guidance for Training

                      Phase Criterion Conforming examples

Assess whether human resource information systems provide needed data in a
timely manner

the future

Conduct a knowledge and skills inventory to identify employees' current
skills and knowledge Management processes to identify the user
requirements for its human resource information system used in planning
training and development, such as:

o  Identifying the user needs

o  	Surveying managers to determine if the information provided met their
needs in a timely manner

Management processes that include steps that require IRS to identify the
critical knowledge and skills its employees need now and in the future to
provide accurate tax law assistance, including evidence such as
documentation that explicitly outlines the knowledge and skills
requirements for taxpayer assistors to reach strategic accuracy goals

Management processes designed to ensure that IRS has determined the
current knowledge and skills of its employees. Methods used to obtain this
information may include:

o  Completed knowledge and skills surveys

o  Proficiency tests

o  Manager assessment Skills and knowledge Perform a needs assessment to
determine analysis the knowledge and skills needed now and in Perform a
gap analysis to determine the differences between skills and knowledge
needed now and in the future

strategies to fill skills and knowledge gaps A documented analysis that
compares IRS's current knowledge and skills to the knowledge and skills
needed to effectively assist taxpayers now and in the future to reach
accuracy goals

As part of its process for determining how to fill skill gaps and address
factors that affect accuracy, or as part of its process for determining
whether to implement a given training and development program, steps to:

o  	Require that approaches other than training be considered to fill the
skill gap or address the factor and

o  	Systematically analyze and consider the relative costs and benefits of
the alternative(s) to training

o  	Develop a documented action plan or similar document that outlines
when training and development interventions should take place once a
knowledge and or skill gap has been identified Strategy development and
Develop and apply criteria to determine selection when to use training and
development

Appendix I: What GAO Looked For: Examples of Planning Practices That Would
Conform to Strategic Guidance for Training

                      Phase Criterion Conforming examples

Ensure training and development efforts target factors that affect
accuracy and are linked to needed skills and knowledge

Consider the anticipated cost and benefits of alternative training and
development efforts, ways to mitigate associated risks, and the
appropriate level of investment for training and development As part of
its process for deciding which training and development programs to
implement, steps to ensure efforts will target needed improvements and
enhance needed skills, which would yield such items as

A logical or explicit link between the training and development programs
offered and (1) the knowledge and skills identified in the skill needs
assessment or (2) the strategic and operational factors identified as
affecting accuracy (to the extent that the needs and the identified
factors could be addressed through training)

A documented audit trail showing that training programs offered now and
planned for the future resulted from identified needed skills and
knowledge

Management processes to establish and apply criteria to select the level
of funding invested for training and development, such a process would
yield such items as:

o  	A comprehensive systematic effort that considers all costs related to
training and development and the associated level of return linked to the
investment

o  	Historical data that show IRS reallocates training and development
resources based on data derived from a cost-benefit analysis

o  	An analysis of training and development investments from prior years
and the outcomes achieved

o  	An analysis documenting various training and development investments
scenarios and the respective expected outcomes

As part of its process for deciding which training and development efforts
to implement, steps to require that the relative costs and benefits of
alternative efforts be compared to current training and development
efforts which would include such components as:

o  	The relative projected impact of the efforts on needed skills, factors
that need to be addressed, and accuracy levels

o  Training and development delivery mechanisms

o  Staff time involved/FTE's

o  Logistical options for staff travel

o  Products used to train and develop employees

As part of its processes for deciding which training and development
efforts to implement, steps to minimize risks by requiring that risks of
alternative investments be identified and that alternative ways of
mitigating those risks are identified and considered, to yield such items
as a documented analysis for each proposed investment describing the
details of the risks and how they could be addressed

Appendix I: What GAO Looked For: Examples of Planning Practices That Would
Conform to Strategic Guidance for Training

                      Phase Criterion Conforming examples

Establish a detailed plan for evaluating See appendix II
training, including performance measures,
data, planned analyses, and how the
analyses will be used

Establish a process to ensure that strategic As part of IRS's process for
planning for training and and tactical changes can be incorporated
development, steps to ensure decision makers into training and development
efforts incorporate a forward looking approach to planning which

lay out steps to address potential strategic and operational changes the
could impact IRS's training and development programs such as:

o  A decrease in the training and education budget

o  Legislative changes affecting the tax law

o  Unexpected staffing shortages

o  Technological innovations

o  Restructuring or a reorganization

                                  Source: GAO.

Appendix II: What GAO Looked For: Examples of Evaluation Practices That Would
Conform to Strategic Guidance

Phase Criterion Conforming examples

Evaluation plan	Establish an overall approach for evaluating the impact of
training on accuracy (a level 4 evaluation)

Systematically analyze the feasibility and cost effectiveness of
alternative methodologies for evaluating the impact of training and
development on accuracy

Before implementing training, develop analysis plans, including data to be
collected, analyses to be conducted, and how the analyses will be used

Data collection 	Ensure that data are collected systematically and in a
timely manner

Ensure that data are accurate, valid, complete, and reliable An
authoritative document, or set of documents, that describes how IRS will
evaluate taxpayer assistance training and development over time, to
include evaluations of the impact of training on accuracy

For each line of effort to evaluate training and development impact, and
for each major training and development program, a documented comparison
of alternative analytical approaches to include comparisons of their
practicality and potential costs and effectiveness

For each line of effort to evaluate training and development impact, and
for each major training and development program, a document that describes
in detail the specific data that will be collected and the analyses that
will be conducted

Documentation of protocols and systematic procedures for data collection
specified in a data collection plan that includes details of methods to
ensure timely collection of data

For each line of effort and for each set of data collected to evaluate
training and development, spot checks for accuracy and completeness,
validity reports, confirmation for each set of data, tests to ensure data
reliability, and/or data collection procedures designed to ensure data
reliability

Data analysis Analyze data collected to Steps to ensure that analyses will 
                         assess the impact         help determine the         
                  of training on accuracy  impact of training and development 
                                                      on accuracy             
                                                  performance, such as        
                                           o  Analyses have considered and/or 
                                                    used an array of          
                                            approaches (both qualitative and  
                                                     quantitative)            
                                           o  Analyses that separate training 
                                                     from other strategic and 
                                           operational factors that might     
                                           affect accuracy (to the            
                                            extent that it is cost-effective  
                                                    and/or feasible)          
                                              o  Analyses enable relating     
                                                processes to outputs and      
                                                        outcomes              
                                            o  Analyses allow for comparisons 
                                                    before and after training 
                                                        is taken              
                                             o  Analyses contain consistent   
                                                qualities, allowing for       
                                                   tracking and comparison of 
                                                   processes and results over 
                                                          time                

Appendix II: What GAO Looked For: Examples of Evaluation Practices That
Would Conform to Strategic Guidance

                      Phase Criterion Conforming examples

Compare accuracy benefits to training costs 	Steps to ensure that there
are documented comparisons of the costs and benefits for each key training
and development program effort to the extent deemed feasible, such as

o  	Comparisons of benefits (including qualitative, estimated, and in some
cases monetized benefits) to the costs of the training and development
program

o  	Use of forward-looking analytical approaches, such as forecasting and
trend analysis, to aid in estimating and comparing future performance with
and without the training intervention

Benchmark training cost, training outcomes Efforts to compare IRS's
approach to that of high(accuracy), and analytical methods against
performing organizations in such areas as training for high-performing
organizations customer assistance, including telephone and live

assistance, spending on training, ways to analyze the

impact of training, and the effectiveness of training

Analyze internal and external stakeholders' Efforts to analyze and
consider stakeholder feedback, assessments of training to include impact
on including feedback from internal and external obtained from accuracy
such methods as:

o  	survey results used as data in various analytical reports on training
to improve accuracy

o  	Panel studies, task forces, etc. aimed at accuracyspecific training
guidance

o  	Regular employee satisfaction surveys and focus groups focusing
directly on training for accuracy

Application of Evaluate training program as a whole, in

evaluation results 	addition to individual course evaluations, and
document the results

Incorporate evaluation results into the training and development program
to improve accuracy Reports detailing the results of analyses performed to
evaluate accuracy training and development with assessments of
quantitative and qualitative data pulled together in a comprehensive way
to integrate conclusions from each set of data

Authoritative documents stating how evaluation results will be used to
inform, modify, and improve planning, such as

o  	Annual performance reports comparing actual to target performance

Memos, minutes, etc. from budget and training planning sessions that
describe decision making based on evaluations of training initiatives
aimed at specific accuracy goals

Indications that the agency is making fact-based determinations of the
impact of its training and development programs by using these assessments
to refine or redesign training and development efforts as needed

                                  Source: GAO.

Appendix III: Assessments of Planning: Less Complex Tax Law Telephone Service by
W&I CAS

Planning phase and criteria Extent Evidence summary Phase: goals and priorities

Establish quantitative long-term accuracy  
0L  CAS reported an annual accuracy goal   
linked to IRS's strategic goal of          
goals that link to IRS's strategic goals   
improving taxpayer service. The accuracy   
goal was not long-term. CAS                
                                              had accuracy projections        
                                              through 2010, but according to  
                                              IRS officials,                  
                                                  these projections were only 
                                              internal and subject to change. 
                                                                 In addition, 

1

OMB stated that IRS did not have long-term goals.

Involve key stakeholders, including human 0L CAS involved stakeholders in
its annual planning decisions. For example, capital professionals,
managers, and CAS sought input from employees and managers on training
priorities and employees, in key long-term training planning needed
changes to training. However, CAS did not have a long-term decisions
training planning process for involving key stakeholders in such strategic
issues as establishing training program goals and long-term employee
developmental needs.

Establish goals for, and measures to assess 0m CAS had no such goals or
measures.
the effectiveness of, training and
development on accuracy

     Affirm, through upper-level    0L CAS management had efforts that        
              management               conveyed the importance of training,   
communication, the importance of    such as an annual letter sent to field 
               training                office directors that referred to the  
      and development to improve        need for specific types of training.  
               accuracy                     However, management did not       
                                       communicate more widely throughout the 
                                             organization the strategic value 
                                       of training and development and its    
                                       importance in achieving long-term      
                                                  accuracy goals.             

                  Phase: information gathering and assessments

Analytically determine and track 0L  CAS collected and analyzed accuracy   
            the strategic                  data by using a quality review     
       and operational factors,        system to identify operational factors 
         including training,              affecting accuracy, such as the     
                                                proper use of IRS guidance to 
         that affect accuracy              assistors, and the use of a pareto 
                                                                  analysis to 
                                       determine the correlation between the  
                                       causes and effects of key errors.      
                                       However, CAS did not analyze and track 
                                       the strategic factors such as tax      
                                       law changes. As a result, CAS did not  
                                       have information on the impact of      
                                        training on accuracy, while holding   
                                        other factors such as the quality of  
                                           assistors' guidance or tax law     
                                                 changes, constant.           

Incorporate evaluation results into training and development planning

__ 	For our assessment, see "Application of Evaluation Results" phase in
appendix VII.

 Benchmark the training and development 0m CAS had not benchmarked its training
                        and development program. IRS had

program against high-performing organizations

benchmarked other nontraining practices related to customer service such
as performance measures.

Assess whether human resource        It was too early for an assessment    
            information           __     because the Enterprise Learning      
systems provide needed data in    Management System (ELMS), which would be 
              a timely                              CAS's primary system used 
                                     for providing human resource             
               manner                information, is being implemented in     
                                     fiscal                                   
                                                    year 2005.                

1As reported in GAO-05-416T.

Appendix III: Assessments of Planning: Less Complex Tax Law Telephone
Service by W&I CAS

              Planning phase and criteria Extent Evidence summary

Phase: skills and knowledge analysis

Perform a needs assessment to determine 0L CAS identified the knowledge
and skills assistors needed for the following the knowledge and skills
needed now and in year. For example, CAS projected the number of questions
expected by tax the future topic and took into account recent tax law
changes. However, CAS had not

identified the specific knowledge and skills needed to achieve long-term

accuracy goals.

Conduct a knowledge and skills inventory to 0M CAS had an inventory of
assistors' current knowledge and skills based on
identify employees' current skills and testing.
knowledge

Perform a gap analysis to determine the 0L CAS did an annual gap analysis,
but because CAS had not performed a differences between skills and
knowledge needs assessment for a period that extended beyond the next
year, CAS needed now and in the future was unable to determine the
differences between current skills and

            knowledge and skills and knowledge needed in the future.

Phase: strategy development and selection

Develop and apply criteria to determine when

  0m CAS had no criteria to guide decisions on using training and development

to use training and development strategies to fill skills and knowledge
gaps

strategies, as opposed to other strategies, such as improving guidance or
hiring practices, to address skills and knowledge gaps.

Ensure training and development 0L CAS had practices such as employee      
               efforts                testing, and studies conducted to       
     target factors that affect       identify the most frequent errors made  
          accuracy and are            on an annual basis, that allowed        
     linked to needed skills and      CAS to link training efforts to         
              knowledge               identified short-term skills and        
                                      knowledge                               
                                        gaps. However, because of limitations 
                                           in long-term gap analyses, CAS had 
                                      limited ability to ensure training and  
                                      development efforts were linked to      
                                      long-term needed skills and knowledge.  

Consider the anticipated cost and benefits   0m CAS did not have these 
of                                                           analyses. 
alternative training and development            
efforts,                                        
ways to mitigate associated risks, and the      
appropriate level of investment for training    
and development                                 

Establish a detailed plan for evaluating training, including performance
measures, data, planned analyses, and how the analyses will be used

__ For our assessment, see "Evaluation Plan" phase in appendix VII. Establish a
 process to ensure that strategic 0L CAS had efforts to anticipate and react to
                       potential short-term changes that

and tactical changes can be incorporated into training and development
efforts

could affect training, such as tax law changes or lack of experienced
assistors due to staff turnover. However, CAS did not have a training
planning process to identify potential long-term changes such as
technological innovations or changes in the economy that might impact
training and development.

Source: GAO analysis.
Key:
Conforms to guidance to a great extent .0M
Conforms to guidance to some extent 0L
Conforms to guidance to little or no extent 0m

Appendix IV: Assessments of Planning: More Complex Tax Law Telephone Service by
SB/SE11TEC

Planning phase and criteria Extent Evidence summary

                          Phase: goals and priorities

Establish quantitative long-term accuracy goals that 0m TEC did not have
such goals. link to IRS's strategic goals

     Involve key stakeholders,   0L TEC involved stakeholders in its annual   
      including human capital       training planning. For                    
professionals, managers, and     example, TEC surveyed employees to obtain 
         employees, in key          input on training                         
long-term planning decisions     needs for the upcoming year. However, TEC 
                                                         did not have a long- 
                                    term training planning process for        
                                    involving key stakeholders in             
                                        such strategic issues as establishing 
                                                   training program goals and 
                                     long-term employee developmental needs.  

Establish goals for, and measures to assess the 0m TEC had no such goals
or measures.
effectiveness of, training and development on
accuracy

     Affirm, through upper-level    0L TEC management had efforts that        
              management               conveyed the importance of             
communication, the importance of    training, such as an annual letter     
             training and              sent to field office directors that    
       development on accuracy           referred to the need for specific    
                                            types of training. However,       
                                       management did not communicate more    
                                       widely throughout the                  
                                          organization the strategic value of 
                                             training and development and its 
                                         importance in achieving long-term    
                                                  accuracy goals.             

Phase: information gathering and assessments

Analytically determine and track the strategic and 0L TEC collected and
analyzed accuracy data to identify factors, such operational factors,
including training, that affect as an analysis of quality review system
data. However, TEC did not accuracy analyze and track the strategic
factors such as changes to tax law.

As a result, TEC did not have information on the impact of training on
accuracy, while holding other factors such as the quality of assistors'
guidance or tax law changes, constant.

Incorporate evaluation results into training and ___ For our assessment,
see "Application of Evaluation Results" phase development planning in
appendix VIII.

Benchmark the training and development program 0m TEC had not benchmarked
its training and development program. against high-performing
organizations

    Assess whether human resource       It was too early for an assessment    
         information systems       __         because the Enterprise          
provide needed data in a timely    Learning Management System (ELMS),      
               manner                 which would be TEC's                    
                                      primary system used for providing human 
                                                     resource information, is 
                                      being implemented in fiscal year 2005.  

                      Phase: skills and knowledge analysis

    Perform a needs assessment to  0L Annually, TEC identified the types of   
            determine the             tax law topics it would be              
knowledge and skills needed now      handling and identified the knowledge 
          and in the future                      and skills assistors needed. 
                                      This process included taking account of 
                                      recent tax law changes.                 
                                          However, TEC had not identified the 
                                                specific knowledge and skills 
                                       needed to achieve long-term accuracy   
                                                      goals.                  

1In 2006, W&I will be handling these calls and training their staff to
provide answers to these more complex tax law questions.

Appendix IV: Assessments of Planning: More Complex Tax Law Telephone
Service by SB/SE1 TEC

Planning phase and criteria Extent Evidence summary
Conduct a knowledge and skills inventory to identify 0m TEC did not
conduct an inventory of its employees' current skills

                    employees' current skills and knowledge

and knowledge. Instead, TEC assumed that its employees had an underlying
technical foundation because of their prior IRS experience, and the annual
training they received was to supplement previously acquired skills.

Perform a gap analysis to determine the differences 0m Because TEC had not
done a knowledge and skills inventory, a
between current and needed skills and knowledge gap analysis could not be
done.
now and in the future

Phase: strategy development and selection

Develop and apply criteria to determine when to use training and
development strategies to fill skills and knowledge gaps

0m	TEC had no criteria to guide decisions on using training and
development strategies, as opposed to other strategies such as improving
guidance or hiring practices, to address skills and knowledge gaps.

 Ensure training and development efforts target factors 0L In 2004, TEC matched
                 its training courses to the general skills and

that affect accuracy and are linked to needed skills and knowledge

knowledge assistors needed. However, because of limitations in long-term
gap analyses, TEC had limited ability to ensure training and development
efforts were linked to long-term needed skills and knowledge.

Consider the anticipated cost and benefits of alternative training and
development efforts, ways to mitigate associated risks, and the
appropriate level of investment for training and development

  0m TEC did not have these analyses. Establish a detailed plan for evaluating
 training, __ For our assessment, see "Evaluation Plan" phase in appendix VIIl.

including performance measures, data, planned analyses, and how the
analyses will be used

Establish a process to ensure that strategic and 0L TEC had informal
processes for anticipating and reacting to tactical changes can be
incorporated into training and potential short-term changes that could
affect training. However, development efforts TEC did not have a training
planning process to identify potential

long-term changes such as technological innovations or changes in

the economy that might impact training and development.

Source: GAO analysis.
Key:
Conforms to guidance to a great extent 0M
Conforms to guidance to some extent 0L
Conforms to guidance to little or no extent 0m

Appendix V: Assessments of Planning: Tax Law Questions Walk-In Service by W&I
CARE

              Planning phase and criteria Extent Evidence summary

Phase: goals and priorities

Establish quantitative long-term

0L CARE reported an annual accuracy goal linked to IRS's strategic goal of
improving

accuracy goals that link to IRS's strategic goals

taxpayer service. The goal was not long-term. CARE had accuracy
projections through 2010, but according to IRS officials, these
projections were internal and subject to change. In addition, OMB stated
that IRS did not have long-term goals.

      Involve key stakeholders,       CARE involved stakeholders in its       
              including            0L annual planning. For example, human     
                                      capital                                 
                                        training staff developed an annual    
    human capital professionals,      training plan based on needs identified 
                                                        by                    
managers, and employees, in key       program staff. However, CARE did not 
                long-                      have a long-term training planning 
                                                                      process 
                                          in which to involve stakeholders in 
       term planning decisions           such strategic issues as identifying 
                                                                  measures to 
                                      assess training progress or identifying 
                                      long-term employee developmental needs. 

Establish goals for, and measures to assess the effectiveness of, training
and development on accuracy

                     0m CARE had no such goals or measures.

Affirm, through upper-level management communication, the importance of
training and development to improve accuracy

0M	CARE implemented a plan for communicating to various levels of
staff, including front-line employees, management's vision for training.
The plan included the vehicle and product to be used to communicate this
message, along with key dates activities should occur to deliver the
message of how training and development plays a role in providing quality
taxpayer service and enhancing employee development to achieve future
goals.

Phase: information gathering and assessments
Analytically determine and track the 0L CARE collected and analyzed data,
including quality review system data, to identify

strategic and operational factors, including training, that affect
accuracy operational factors affecting accuracy to identify the most
frequent errors. CARE also had studies to determine why those errors
occurred, such as why staff did not properly use IRS guidance to answer
questions. However, CARE did not analyze and track the strategic factors
such as changes to the tax law or attrition. As a result, CARE did not
have information on the impact these factors had on accuracy holding other
factors such as changes in the tax law or change in guidance constant.

Incorporate evaluation results     For our assessment see the "Application 
                into              ___     of evaluation results" phase in     
      training and development                     appendix IX.               
              planning                
                                  0m   CARE had not benchmarked its training  
     Benchmark the training and          and development program. IRS had     
    development program against       benchmarked other nontraining practices 
               high-                    related to customer service such as   
      performing organizations                 performance measures.          
                                        It was too early for an assessment    
Assess whether human resource          because the Enterprise Learning     
                                  ___               Management                
    information systems provide       System (ELMS), which would be CARE's    
               needed                 primary system used for providing human 
      data in a timely manner             resource information, is being      
                                         implemented in fiscal year 2005.     

1As reported in GAO-05-416T .

Appendix V: Assessments of Planning: Tax Law Questions Walk-In Service by
W&I CARE

Planning phase and criteria Extent Evidence summary Phase: skills and knowledge
                                    analysis

Perform a needs assessment to 0L CARE had identified the knowledge and     
                                    skills assistors needed for the following 
    determine the knowledge and            year. One example is ensuring that 
              skills                  assistors have the skills and knowledge 
                                                                       needed 
needed now and in the future     to respond to questions about recent tax  
                                    law changes. However, CARE had not        
                                    identified the specific knowledge and     
                                    skills needed to achieve long-term        
                                    accuracy                                  
                                                     goals.                   

Conduct a knowledge and skills inventory to identify employees' current
skills and knowledge

0M	CARE had an inventory of assistors' current knowledge and skills based
on a prescreening testing process.

  Perform a gap analysis to determine 0L CARE did an annual gap analysis, but
                     because CARE had not performed a needs

the differences between current and needed skills and knowledge now and in
the future assessment for a period that extended beyond the next year, it
had not performed a gap analysis to determine the differences between
current skills and knowledge and skills and knowledge needed in the
future.

                   Phase: strategy development and selection

Develop and apply criteria to determine when to use training and
development strategies to fill skills and knowledge gaps

0m	CARE had no criteria to guide decisions on using training and
development strategies, as opposed to other strategies, such as improving
guidance, or hiring practices to address skill gaps.

Ensure training and development 0L CARE has practices such as employee testing,
                       and studies conducted to identify

efforts target factors that affect accuracy and are linked to needed
skills and knowledge the most frequent errors made on an annual basis,
that allow CARE to link training efforts to identified short-term skills
and knowledge gaps.

However because of limitations in long term gap analyses, staff have
limited ability to ensure training and development efforts are linked to
long term needed skills and knowledge.

Consider the anticipated costs    0m      CARE did not have these 
and                                                     analyses. 
benefits of alternative training     
and                                  
development efforts, ways to         
mitigate                             
associated risks, and the            
appropriate                          
level of investment for training     
and                                  
development                          

Establish a detailed plan for evaluating training, including performance
measures, data, planned analyses, and how the analyses will be used

 __ For our assessment, see "Evaluation Plan" phase in appendix IX. Establish a
process to ensure that 0L CARE had efforts to anticipate and react to potential
                         short-term changes that could

strategic and tactical changes can be incorporated into training and
development efforts affect training, such as tax law changes or lack of
experienced assistors due to staff turnover. However, CARE did not have a
training planning process to identify potential long-term changes, such as
technological innovations or changes in the economy, that might impact
training and development.

Source: GAO analysis.
Key:
Conforms to guidance to a great extent 0M
Conforms to guidance to some extent 0L
Conforms to guidance to little or no extent 0m

Appendix VI: Assessments of Planning: Return Preparation Walk-In Service by W&I
CARE

              Planning phase and criteria Extent Evidence summary

Phase: goals and priorities

Establish quantitative long-term accuracy goals that link to IRS's
strategic goals

0m	CARE did not have quantitative long-term accuracy goals for return
preparation. Fiscal year 2005 will be the baseline year for a new measure
of tax preparation accuracy, so therefore there are no annual or long-term
goals.

Involve key stakeholders, including human 0L CARE involves stakeholders in
                  annual planning. For example, human capital

capital professionals, managers, and employees, in key long-term planning
decisions training staff developed an annual training plan based on needs
identified by program staff. However, CARE did not have a long-term
training planning process in which to involve stakeholders in such
strategic issues as identifying measures to assess training progress or
identifying long-term employee developmental needs.

Establish goals for, and measures to assess 0m CARE had no such goals or
measures.
the effectiveness of, training and
development on accuracy

     Affirm, through upper-level            CARE implemented a plan for       
              management            0M   communicating to various levels of   
                                                       staff,                 
communication, the importance of    including front-line employees,        
               training                management's vision for training. The  
                                       plan                                   
      and development to improve       included the vehicle and product to be 
               accuracy                used to communicate this message,      
                                       along with key dates activities should 
                                       occur to deliver the message of how    
                                         training and development play a role 
                                        in providing quality taxpayer service 
                                                                          and 
                                         enhancing employee development to    
                                                achieve future goals          

Phase: information gathering and assessments
Analytically determine and track the strategic 0L CARE collected and
analyzed accuracy data, including quality review system

and operational factors, including training, that affect accuracy

data, to identify operational factors affecting accuracy to identify the
most frequent errors. CARE also had studies to determine why those errors
occurred, such as why staff did not properly use IRS guidance to answer
questions. However, CARE did not analyze and track the strategic factors
such as changes to the tax law or attrition. As a result, CARE did not
have information on the impact these factors will have on accuracy holding
other factors such as changes in the tax law or change in guidance
constant.

Incorporate evaluation results into training and development planning

__ 	For our assessment see the "Application of evaluation results" phase
in appendix X.

Benchmark of the training and development 0m CARE had not benchmarked its
                   training and development program. IRS had

program against high- performing organizations

benchmarked other nontraining practices related to customer service such
as performance measures.

Assess whether human resource        It was too early for an assessment    
            information           __     because the Enterprise Learning      
systems provide needed data in    Management System (ELMS), which would be 
              a timely                             CARE's primary system used 
                                                 for providing human resource 
               manner                    information, is being implemented in 
                                                                  fiscal year 
                                                      2005.                   

Phase: skills and knowledge analysis

Perform a needs assessment to determine

 0L CARE identified the knowledge and skills assistors needed for the following

the knowledge and skills needed now and in the future

year. One example is ensuring that assistors have the skills and knowledge
needed to respond to questions about recent tax law changes. However, CARE
had not identified the specific knowledge and skills enhancements needed
to achieve long-term accuracy goals.

Appendix VI: Assessments of Planning: Return Preparation Walk-In Service
by W&I CARE

Planning phase and criteria Extent Evidence summary
Conduct a knowledge and skills inventory to 0M CARE had an inventory of
assistors' current knowledge and skills based on a

identify employees' current skills and knowledge

 prescreening testing process. Perform a gap analysis to determine the 0L CARE
        did an annual gap analysis, but because CARE had not performed a

differences between current and needed skills and knowledge now and in the
future needs assessment for a period that extended beyond the next year,
it had not performed a gap analysis to determine the differences between
current skills and knowledge and skills and knowledge needed in the
future.

Phase: strategy development and selection

Develop and apply criteria to determine

  0m CARE had no criteria to guide decisions on using training and development

when to use training and development strategies to fill skills and
knowledge gaps strategies, as opposed to other strategies, such as
improving guidance, or hiring practices to address skill gaps. Establish a
detailed plan for evaluating training, including performance measures,
data, planned analyses, and how the analyses will be used

Ensure training and development 0L CARE had practices such as employee     
               efforts                testing, and studies conducted to       
     target factors that affect       identify the most frequent errors made  
          accuracy and are               on an annual basis, that allowed     
     linked to needed skills and         CARE to link training efforts to     
              knowledge                  identified short-term skills and     
                                                     knowledge                
                                                       gaps.                  
                                           However, because of limitations in 
                                             long-term gap analyses, CARE had 
                                                                      limited 
                                      ability to ensure training and          
                                      development efforts were linked to      
                                      long-term                               
                                           needed skills and knowledge.       

Consider the anticipated costs and benefits 0m CARE did not have these 
                                                                analyses. 
of alternative training and development        
efforts, ways to mitigate associated risks,    
and the appropriate level of investment for    
training and development efforts               

 __ For our assessment, see "Evaluation Plan" phase in appendix X. Establish a
process to ensure that strategic 0L CARE had efforts to anticipate and react to
                       potential short-term changes that

and tactical changes can be incorporated into training and development
efforts

could affect training, such as tax law changes or lack of experienced
assistors due to staff turnover. However, CARE did not have a training
process to identify potential long-term changes such as technological
innovations or changes in the economy that might impact training and
development.

Source: GAO analysis.
Key:
Conforms to guidance to a great extent .0M
Conforms to guidance to some extent 0L
Conforms to guidance to little or no extent 0m

Appendix VII: Assessments of Evaluation: Less Complex Tax Law Telephone Service
by W&I CARE

  Evaluation phase and criteria Extent Evidence summary Phase: evaluation plan
                  Phase: data collection Phase: data analysis

Establish an overall approach 0L  L&E had adopted a four-level model for   
        for evaluating the                  evaluating training, the          
       impact of training on         Evaluation Monitoring System-Integrated  
        accuracy (a level 4                  Training Evaluation and          
            evaluation)               Measurement Services (EMS-ITEMS), which 
                                                     was based on the widely- 
                                    accepted Kirkpatrick model. IRS's Human   
                                    Capital Office L&E officials              
                                      concluded that level 4 evaluations were 
                                                appropriate. However, CAS and 
                                      L&E officials had not agreed on whether 
                                              to conduct level 4 evaluations. 
                                    In addition, L&E officials had not        
                                    documented an analysis of what level      
                                         of evaluation was appropriate.       

      Systematically analyze the     0m Because L&E and CAS officials had not 
         feasibility and cost                  agreed to do a level 4         
     effectiveness of alternative        evaluation, L&E had not analyzed the 
           methodologies for               feasibility and cost effectiveness 
evaluating the impact of training    of alternative level 4 methodologies. 
            and development             
              on accuracy               
     Before implementing training,   0L Because L&E and CAS officials had not 
           develop analysis                    agreed to do a level 4         
      plans, including data to be       evaluation, L&E did not have a level  
        collected, analyses to          4 data analysis plan. However,        
       be conducted, and how the            EMS-ITEMS did provide general     
         analyses will be used            guidance for different levels of    
                                                     evaluation.              

Ensure that data are collected 0L  EMS-ITEMS had controls to help ensure   
         systematically and                   systematic and timely           
         in a timely manner            collection of data for level 1 and 3   
                                            evaluations. EMS-ITEMS had        
                                     guidance on the collection of data       
                                     including responsible parties and        
                                       timing. However, the Administrative    
                                            Corporate Education System        
                                     (ACES) database used to collect level 2  
                                     data was no longer in place,             
                                     and as a result, there was no system to  
                                     consistently collect and store           
                                       level 2 data. In addition, because L&E 
                                             and CAS officials had not agreed 
                                      to do a level 4 evaluation, L&E had not 
                                              collected the data necessary to 
                                             do a level 4 evaluation.         

Ensure that data are accurate, 0L EMS-ITEMS had level 1 and some level 3   
          valid, complete,           data and the system had                  
            and reliable              controls in place to help ensure that   
                                          data were accurate, valid, and      
                                     reliable. However, because L&E and CAS   
                                     had not agreed to do a level             
                                      4 evaluation, L&E did not collect level 
                                               4 data. The ACES database used 
                                     to collect level 2 data no longer exists 
                                           and there is no replacement        
                                                 system planned.              

Analyze data collected to 0L     L&E conducted some level 1, 2, and 3      
     assess the impact of               evaluations of participants'          
     training on accuracy              opinions, learning, and subsequent job 
                                                performance. For example, L&E 
                                analyzed course evaluations, tests, and       
                                supervisory evaluations after                 
                                employees completed coursework to identify    
                                needed improvements to                        
                                 training. However, L&E had done no level 4   
                                                evaluations.                  

      Compare accuracy benefits to    0m   Because L&E and CAS had not agreed 
             training costs                   to do a level 4 evaluation, L&E 
                                         did not do a comparison of benefits  
                                                       to cost.               
Benchmark training cost, training  0m                None.                 
                outcomes                 
(accuracy), and analytical methods    
             against high-               
        performing organizations         

Appendix VII: Assessments of Evaluation: Less Complex Tax Law Telephone
Service by W&I CARE

Evaluation phase and criteria Extent Evidence summary
Analyze internal and external stakeholders' 0L L&E and CAS analyzed
stakeholder assessments of training to identify

assessments of training to include impact on accuracy

needed changes. For example, management councils and focus groups were
used to determine if changes were needed to improve course material,
training environment, timing, and objectives of training. However, because
L&E and CAS had not agreed to do a level 4 evaluation, there were no
efforts to collect and analyze assessments from internal and external
stakeholders to assess training in terms of its impact on accuracy.

Phase: application of evaluation results

Evaluate training program as a whole, in addition

  0m Evaluations were for individual courses, not for the program as a whole.

to individual course evaluations, and document the results

Incorporate evaluation results 0L L&E and CAS reported using evaluation    
       into the training and         results in planning to make              
development program to improve    improvements to training courses or to   
              accuracy               identify training needs for the          
                                     upcoming year. L&E and CAS used the      
                                     available level 1 through 3              
                                     evaluations to make planned improvements 
                                     on individual courses.                   
                                       However, because L&E and CAS had not   
                                              agreed to do a level 4          
                                      evaluation or cost-benefit comparisons, 
                                                L&E and CAS's ability to make 
                                     informed decisions on improving training 
                                     to improve accuracy was                  
                                                     limited.                 

Source: GAO analysis.
Key:
Conforms to guidance to a great extent .0M
Conforms to guidance to some extent 0L
Conforms to guidance to little or no extent 0m

Appendix VIII: Assessments of Evaluation: More Complex Tax Law Telephone Service
by SB/SE TEC

             Evaluation phase and practice Extent Evidence summary

Phase: evaluation plan

Establish an overall approach for evaluating

0M L&E had adopted a four-level model for evaluating training (EMS-ITEMS)

the impact of training on accuracy (a level 4 evaluation)

that was based on the widely accepted Kirkpatrick model. In 2004, L&E and
TEC planned a pilot test of a level 4 evaluation. L&E and TEC officials
stated in an official document that they intend to conduct a level 4
evaluation in 2005.

     Systematically analyze the   0m L&E had not analyzed the feasibility and 
        feasibility and cost                cost effectiveness of alternative 
    effectiveness of alternative                  methodologies.              
         methodologies for           
      evaluating the impact of       
            training and             
      development on accuracy        
Before implementing training,  0M L&E had a plan to conduct a level 4      
              develop                evaluation in 2004. L&E planned to       
analysis plans, including data        conduct a level 4 evaluation in 2005 
          to be collected,            similar to that done in 2004. Officials 
analyses to be conducted, and     said they would use the 2004 plan as the 
              how the                basis for the 2005 evaluation            
       analyses will be used                         effort.                  

                             Phase: data collection

Ensure that data are collected 0M L&E had controls to ensure data were     
           systematically            collected systematically and in a        
       and in a timely manner        timely manner. Data collection practices 
                                     for all four levels of evaluation in     
                                            the level 4 pilot test evaluation 
                                                included specific steps to be 
                                                                accomplished, 
                                            status updates, and due dates. In 
                                        addition, for the 2005 pilot, data on 
                                                                          TEC 
                                        accuracy in five technical categories 
                                     and staff proficiency in those technical 
                                     categories were available before the end 
                                                     of 2004.                 

    Ensure that data are   0L In 2004, TEC was concerned that the accuracy    
      accurate, valid,        data in the five technical                      
complete, and reliable     categories, based on test calls, did not        
                              reflect the types of calls taxpayers            
                                actually made. TEC and IRS's quality review   
                                         staff subsequently worked            
                              together in an effort to fix the problem.       
                              EMS-ITEMS had level 1 and some                  
                              level 3 data, and the system had controls in    
                              place to ensure that data were                  
                                  accurate, valid, and complete. However, the 
                                             database used to collect level 2 
                                 data no longer existed, and there was no     
                                        replacement system planned.           

Phase: data analysis

Analyze data collected to assess the impact

 0L In the 2004 pilot level 4 evaluation, L&E did an analysis but was unable to
                            of training on accuracy

assess the impact of training on accuracy, because the level of
proficiency of TEC staff in the five technical categories (capital gains,
depreciation and sale of business property, rentals, trust and
fiduciaries, and international and alien) was insufficient. L&E decided
not to measure the impact of training on accuracy until 80 percent of the
employees were proficient in the five technical categories. In a survey of
133 TEC employees, the percent reported having preexisting skills or being
fully proficient in the five technical categories ranged from 31 to 62
percent.

     Compare accuracy benefits to          Comparing the value of training to 
            training costs           0m   the costs of training was to be the 
                                                                        third 
                                         phase of the uncompleted 2004 pilot  
                                                        test.                 
Benchmark training cost, training 0m                 None.                 
               outcomes                 
      (accuracy), and analytical        
            methods against             
     high-performing organizations      

Appendix VIII: Assessments of Evaluation: More Complex Tax Law Telephone
Service by SB/SE TEC

             Evaluation phase and practice Extent Evidence summary

Analyze internal and external 0L L&E and TEC analyzed stakeholder          
           stakeholders'            assessments of training to identify       
    assessments of training to      needed changes. For example, TEC used     
         include impact on          information from employee                 
                                      surveys to target training materials on 
             accuracy                   the types of calls assistors reported 
                                        receiving. However, they did not      
                                        successfully collect and analyze      
                                       assessments from internal and external 
                                           stakeholders to assess training in 
                                        terms of its impact on accuracy.      

Phase: application of evaluation results

Evaluate training program as a whole, in

 0L As discussed above, although L&E attempted to do a pilot test of a level 4

addition to individual course evaluations, and document the results

evaluation of the training program as a whole, the evaluation was
unsuccessful.

Incorporate evaluation results 0L  L&E and TEC reported using evaluation   
         into the training                 results in planning to make        
     and development program to       improvements to training courses or to  
              improve                    identify training needs for the      
                                      upcoming year. However, because there   
              accuracy                      was no successful level 4         
                                     evaluation or benefit cost comparison,   
                                     L&E and TEC's ability to make            
                                     informed decisions on improving training 
                                             to improve accuracy was limited. 

Source: GAO analysis.
Key:
Conforms to guidance to a great extent .0M
Conforms to guidance to some extent 0L
Conforms to guidance to little or no extent 0m

Appendix IX: Assessments of Evaluation: Tax Law Walk-in Service by W&I CARE

             Evaluation phase and practice Extent Evidence summary

Phase: evaluation plan

Establish an overall approach for evaluating 0L L&E had adopted a
four-level model for evaluating training, the Evaluation the impact of
training on accuracy (a level 4 Monitoring System-Integrated Training
Evaluation and Measurement evaluation) Services (EMS-ITEMS), based on the
widely-accepted Kirkpatrick model.

IRS's Human Capital Office L&E officials concluded that level 4
evaluations were appropriate. However, CARE and L&E officials had not
agreed on whether to conduct level 4 evaluations. In addition, L&E
officials had not documented an analysis of what level of evaluation was
appropriate.

Systematically analyze the feasibility and cost 0m Because L&E and CARE
officials had not agreed to do a level 4 evaluation, effectiveness of
alternative methodologies for L&E had not analyzed the feasibility and
cost effectiveness of alternative evaluating the impact of training and
level 4 methodologies. In addition, L&E and CARE officials did not look at
development on accuracy methodologies specific to questions on tax law
assistance. The approach

for training and evaluating staff who provided assistance by answering
walk-in customers' tax law questions and staff preparing returns were the
same.

Before implementing training, develop 0L Because L&E and CARE officials
had not agreed to do a level 4 evaluation,
analysis plans, including data to be collected, L&E did not have a level 4
data analysis plan. However, EMS-ITEMS did
analyses to be conducted, and how the provide general guidance for
different levels of evaluation.
analyses will be used

Phase: data collection

Ensure that data are collected systematically 0L EMS-ITEMS had controls to
help ensure systematic and timely collection of

and in a timely manner 	data for level 1 and 3 evaluations. EMS-ITEMS has
guidance on the collection of data including responsible parties and
timing. However, the Administrative Corporate Education System (ACES)
database used to collect level 2 data was no longer in place, and as a
result, there was no system to consistently collect and store level 2
data. In addition, because L&E and CARE officials had not agreed to do a
level 4 evaluation, L&E had not collected the data necessary to do a level
4 evaluation.

Ensure that data are accurate, valid, 0L EMS-ITEMS had level 1 and some
level 3 data and the system had controls complete, and reliable in place
to help ensure that data were accurate, valid, and reliable.

However, because L&E and CARE had not agreed to do a level 4 evaluation,
L&E did not collect level 4 data. The ACES database used to collect level
2 data no longer exists and there was no replacement system planned.

Phase: data analysis

Analyze data collected to assess the impact

 0L L&E conducted some level 1, 2, and 3 evaluations of participants' opinions,
                            of training on accuracy

learning, and subsequent job performance. For example, L&E analyzed course
evaluations, tests, and supervisory evaluations after employees completed
coursework to identify needed improvements to training. However, L&E had
done no level 4 evaluations. In addition, there was no distinction between
analysis of walk-in tax law and return preparation assistance.

Compare accuracy benefits to training costs 0m	Because L&E and CARE had
not agreed to do a level 4 evaluation, L&E did not do a comparison of
benefits to cost.

Appendix IX: Assessments of Evaluation: Tax Law Walk-in Service by W&I
CARE

             Evaluation phase and practice Extent Evidence summary

Benchmark training cost, training outcomes 0m None.
(accuracy), and analytical methods against
high-performing organizations

Analyze internal and external 0L         L&E and CARE analyzed stakeholder 
           stakeholders'                  assessments of training to identify 
    assessments of training to      needed changes. For example, focus groups 
         include impact on          were used to determine if                 
                                        changes were needed to improve course 
             accuracy                         material, training environment, 
                                          timing, and objectives of training. 
                                            However, because L&E and CARE had 
                                       not agreed to do a level 4 evaluation, 
                                         there were no efforts to collect and 
                                    analyze assessments from internal and     
                                    external stakeholders to assess           
                                       training in terms of its impact on     
                                                    accuracy.                 

                    Phase: application of evaluation results

Evaluate training program as a whole, in 0m Evaluations were for
individual courses, not for the program as a whole.
addition to individual course evaluations, and
document the results

Incorporate evaluation results 0L L&E and CARE reported using evaluation   
         into the training           results in planning to make              
     and development program to       improvements to training courses or to  
              improve                    identify training needs for the      
                                       upcoming year. L&E and CARE used the   
              accuracy                     available level 1 through 3        
                                     evaluations to make planned improvements 
                                              on individual courses.          
                                      However, because L&E and CARE had not   
                                              agreed to do a level 4          
                                     evaluation or cost-benefit comparisons,  
                                     L&E and CARE's ability to make           
                                     informed decisions on improving training 
                                             to improve accuracy was limited. 

Source: GAO analysis.
Key:
Conforms to guidance to a great extent .0M
Conforms to guidance to some extent 0L
Conforms to guidance to little or no extent 0m

Appendix X: Assessments of Evaluation: Return Preparation Walk-in Service by W&I
CARE

  Evaluation phase and practice Extent Evidence summary Phase: evaluation plan

Establish an overall approach for evaluating 0L L&E has adopted a
four-level model for evaluating training, the Evaluation the impact of
training on accuracy (a level 4 Monitoring System-Integrated Training
Evaluation and Measurement Services evaluation) (EMS-ITEMS), based on the
widely-accepted Kirkpatrick model. IRS's Human

Capital Office L&E officials had concluded that level 4 evaluations were
appropriate. However, CARE and L&E officials had not agreed on whether to
conduct a level 4 evaluation. In addition, L&E officials had not
documented an analysis of what level of evaluation was appropriate.

Systematically analyze the feasibility and 0m Because L&E and CARE
officials had not agreed to do a level 4 evaluation,
cost effectiveness of alternative L&E had not analyzed the feasibility and
cost effectiveness of alternative level
methodologies for evaluating the impact of 4 methodologies. In addition,
L&E and CARE officials did not look at
training and development on accuracy methodologies specific to questions
on return preparation. The approaches for

training and evaluating staff that provided assistance by preparing
returns and staff answering walk-in customers' tax law questions were the
same.

Before implementing training, develop 0L Because L&E and CARE officials
had not agreed to do a level 4 evaluation, analysis plans, including data
to be L&E did not have a level 4 data analysis plan. However, EMS-ITEMS
did collected, analyses to be conducted, and provide general guidance for
different levels of evaluation. how the analyses will be used

Phase: data collection

Ensure that data are collected systematically 0L EMS-ITEMS had controls to
help ensure systematic and timely collection of

and in a timely manner 	data for level 1 and 3 evaluations. EMS-ITEMS had
guidance on the collection of data including responsible parties and
timing. However, the Administrative Corporate Education System (ACES)
database used to collect level 2 data was no longer in place, and as a
result, there was no system to consistently collect and store level 2
data. In addition, because L&E and CARE officials had not agreed to do a
level 4 evaluation, L&E had not collected the data necessary to do a level
4 evaluation.

Ensure that data are accurate, valid, 0L EMS-ITEMS had level 1 and some
level 3 data and the system had controls in complete, and reliable place
to help ensure that data were accurate, valid, and reliable. However,
because L&E and CARE had not agreed to do a level 4 evaluation, L&E did
not collect level 4 data. The ACES database used to collect level 2 data
no longer existed and there was no replacement system planned.

                              Phase: data analysis

Analyze data collected to assess the impact of training on accuracy

0L	L&E conducted some level 1, 2, and 3 evaluations of participants'
opinions, learning, and subsequent job performance. For example, L&E
analyzed course evaluations, tests, and supervisory evaluations after
employees completed coursework to identify needed improvements to
training. However, L&E had done no level 4 evaluations. In addition, there
was no distinction between analysis of walk-in tax law and return
preparation assistance.

Compare accuracy benefits to training costs 0m	Because L&E and CARE had
not agreed to do a level 4 evaluation, L&E did not do a comparison of
benefits to cost.

Benchmark training cost, training outcomes 0m None.
(accuracy), and analytical methods against
high-performing organizations

Appendix X: Assessments of Evaluation: Return Preparation Walk-in Service
by W&I CARE

  Evaluation phase and practice Extent Evidence summary Phase: application of
                               evaluation results

Analyze internal and external 0L     L&E and CARE analyzed stakeholder     
           stakeholders'               assessments of training to identify    
    assessments of training to      needed changes. For example, focus groups 
         include impact on                  were used to determine if         
                                      changes were needed to improve course   
             accuracy                    material, training environment,      
                                          timing, and objectives of training. 
                                        However, because L&E and CARE had not 
                                     agreed to do a level 4 evaluation, there 
                                       were no efforts to collect and analyze 
                                    assessments from internal and external    
                                    stakeholders to assess training in        
                                        terms of its impact on accuracy.      

Evaluate training program as a whole, in 0m Evaluations were for
individual courses, not for the program as a whole.
addition to individual course evaluations,
and document the results

    Incorporate evaluation  0L L&E and CARE reported using evaluation results 
       results into the                     in planning to make               
training and development            improvements to training courses or to 
          program to                 identify training needs for the upcoming 
       improve accuracy        year. L&E and CARE used the available levels 1 
                                          through 3 evaluations to            
                               make planned improvements on individual        
                               courses. However, because L&E                  
                                  and CARE had not agreed to do a level 4     
                                         evaluation or cost-benefit           
                                comparisons, L&E and CARE's ability to make   
                                           informed decisions on              
                                 improving training to improve accuracy was   
                                                  limited.                    

Source: GAO analysis.
Key:
Conforms to guidance to a great extent .0M
Conforms to guidance to some extent 0L
Conforms to guidance to little or no extent 0m

Appendix XI: Information on the Kirkpatrick Model of Training and Development
Evaluation

In recent years, a growing number of organizations have adopted a
balanced, multilevel approach to evaluating their training and development
efforts. Such an approach can help provide varied data and perspectives on
the effect that training efforts have on the organization. One commonly
accepted model consists of four levels of assessment.1 The first level
measures the training participants' reaction to, and satisfaction with,
the training program or planned actions to use new or enhanced
competencies. The second level measures the extent to which learning has
occurred because of the training effort. The third level measures the
application of this learning to the work environment through changes in
behavior that trainees exhibit on the job because of the training or
development program. The fourth level measures the impact of the training
program on the agency's program or organizational results. The fourth
level is sometimes split into two levels with the fifth level representing
a comparison of costs and benefits quantified in dollars. This fifth
level-often referred to as return on investment (ROI)-compares the
benefits (quantified in dollars) to the costs of the training and
development program.2

Not all training and development programs require, or are suitable for,
higher levels of evaluation. Indeed, higher levels of evaluation can be
challenging to conduct because of the difficulty and costs associated with
data collection and the complexity in directly linking training and
development programs to improved individual and organizational
performance. Figure 2 depicts an example gradation of the extent to which
an agency could use the various levels of evaluation to assess its
training and development programs. For example, an agency may decide to
evaluate participants' reactions for all (100 percent) of its programs,
while conducting an ROI analysis for 5 percent of its programs.

1Donald L. Kirkpatrick (author of Evaluating Training Programs: The Four
Levels) conceived a commonly recognized four-level model for evaluating
training and development efforts.

2Jack J. Phillips conceived the fifth level in the model for evaluating
training and development efforts as discussed in the book, Measuring ROI
in the Public Sector.

Appendix XI: Information on the Kirkpatrick Model of Training and
Development Evaluation

Figure 2: Example Agency's Training and Development Programs Assessed
Using Each Level of Evaluation

                        Appendix XII: Comments from IRS

                        Appendix XII: Comments from IRS

                        Appendix XII: Comments from IRS

                        Appendix XII: Comments from IRS

Appendix XIII: GAO Contact and Staff Acknowledgments

GAO Contact James R. White (202) 512-9110

Acknowledgments 	In addition to the contact named above, Charlie Daniel,
David Dornisch, Chad Gorman, Jason Jackson, Ronald Jones, Veronica
Mayhand, and Katrina Taylor made key contributions to this report.

  GAO's Mission

Obtaining Copies of GAO Reports and Testimony

The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no cost
is through GAO's Web site (www.gao.gov). Each weekday, GAO posts newly
released reports, testimony, and correspondence on its Web site. To have
GAO e-mail you a list of newly posted products every afternoon, go to
www.gao.gov and select "Subscribe to Updates."

Order by Mail or Phone 	The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:

U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548

To order by Phone: 	Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061

  To Report Fraud, Contact:

Waste, and Abuse in Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: [email protected] Programs Automated answering system: (800)
424-5454 or (202) 512-7470

Gloria Jarmon, Managing Director, [email protected] (202)
512-4400Congressional U.S. Government Accountability Office, 441 G Street
NW, Room 7125 Relations Washington, D.C. 20548

Public Affairs 	Paul Anderson, Managing Director, [email protected] (202)
512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548

                           PRINTED ON RECYCLED PAPER
*** End of document. ***