IRS Telephone Assistance: Limited Progress and Missed
Opportunities to Analyze Performance in the 2001 Filing Season
(07-DEC-01, GAO-02-212).
Congress has been long concerned about the quality of service
taxpayers received when calling the Internal Revenue Service
(IRS) for help in understanding and meeting their tax
obligations. IRS has taken steps to improve its responsiveness to
the tens of millions of telephone calls it receives each year,
such as expanding the hours of service and increasing the use of
automation. In the 2000 tax filing season, the quality of
telephone assistance was mixed and below IRS' long-term goal of
providing world-class service. Overall, IRS made limited progress
toward its goal of providing world-class telephone service. When
compared with the 2000 tax filing season, access and accuracy
performance for 2001 improved in two of six comparable measures,
declined in one, and changed two percentage points or less in the
others. When compared with 2001 performance targets, IRS did not
meet any of its access goals, falling considerably below its
target to shorten the time taxpayers spend waiting to speak with
an assistor. Although assistors exceeded quality-of-service
targets when responding to taxpayer questions, they did not meet
higher targets for providing correct answers and account
adjustments. IRS officials missed opportunities to analyze data
to better understand the factors affecting telephone performance,
including the actions it took to improve performance. IRS
managers sometimes reached conclusions about key factors without
conducting analyses to test their conclusions. IRS officials also
missed opportunities to plan evaluations to determine the
effectiveness of the actions taken to improve access and
accuracy. Because of the limited progress made in the 2001 filing
season, IRS cannot afford to miss opportunities to analyze
performance data and evaluate improvement actions.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-02-212
ACCNO: A02443
TITLE: IRS Telephone Assistance: Limited Progress and Missed
Opportunities to Analyze Performance in the 2001 Filing Season
DATE: 12/07/2001
SUBJECT: Customer service
Taxpayers
Telephone
Agency missions
Performance measures
Internal Revenue Manual
IRS TeleTax System
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Testimony. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-02-212
Report to the Chairman, Subcommittee on Oversight, Committee on Ways and
Means, House of Representatives
United States General Accounting Office
GAO
December 2001 IRS TELEPHONE ASSISTANCE
Limited Progress and Missed Opportunities to Analyze Performance in the 2001
Filing Season
GAO- 02- 212
Page i GAO- 02- 212 IRS' 2001 Filing Season Telephone Assistance Letter 1
Results in Brief 2 Background 3 Scope and Methodology 5 IRS Made Limited
Progress Toward Providing World- Class
Telephone Service 6 IRS Missed Some Opportunities to Better Understand
Telephone
Performance 8 Conclusions 15 Recommendation for Executive Action 15 Agency
Comments and Our Evaluation 16
Appendix I Descriptions and Formulas for Measures of IRS Telephone
Assistance Accessibility and Accuracy 18
Appendix II Comments From the Internal Revenue Service 19
Tables
Table 1: IRS Tax Filing Season Telephone Assistance Performance 6 Table 2:
Descriptions and Formulas for Measures 18
Abbreviations
CQRS Centralized Quality Review System GPRA Government Performance and
Results Act IRS Internal Revenue Service Contents
Page 1 GAO- 02- 212 IRS' 2001 Filing Season Telephone Assistance
December 7, 2001 The Honorable Amo Houghton Chairman, Subcommittee on
Oversight Committee on Ways and Means House of Representatives
Dear Mr. Chairman: For years, Congress has been concerned about the quality
of service taxpayers received when calling the Internal Revenue Service
(IRS) for help in understanding and meeting their tax obligations. IRS? goal
is to make its telephone operation a ?world- class customer service
organization? that provides taxpayers with accessible and accurate
assistance comparable to the best practices in the private and public
sectors. IRS has taken steps intended to improve how it responds to the tens
of millions of telephone calls received each year, such as expanding the
hours of service and increasing the use of automation. However, as we
previously reported to you, IRS has continued to struggle to provide
accessible and accurate telephone assistance. In the 2000 tax filing season,
1 the quality of telephone assistance was mixed and below IRS? long- term
goal of providing world- class service. 2
Because of your continuing interest in the quality of IRS? telephone
assistance, you asked us to assess the performance and management of IRS?
telephone operations in the 2001 tax filing season. More specifically, our
objectives were to (1) compare IRS? performance in providing accessible and
accurate telephone assistance in the 2001 tax filing season with the 2000
tax filing season and 2001 performance targets and (2) assess IRS? efforts
to determine the factors that affected performance in the 2001 tax filing
season and its plans to evaluate the actions it took to improve performance.
To address these objectives, we collected and analyzed data on eight
telephone assistance performance measures: four on taxpayers? ability to
1 According to IRS officials, the tax filing season generally begins January
1 and ends around July 15 each year. 2 IRS Telephone Assistance: Quality of
Service Mixed in the 2000 Filing Season and Below IRS? Long- Term Goal (GAO-
01- 189, Apr. 6, 2001).
United States General Accounting Office Washington, DC 20548
Page 2 GAO- 02- 212 IRS' 2001 Filing Season Telephone Assistance
gain access to IRS and four on the accuracy of IRS?s response to taxpayers.
We also interviewed IRS officials involved in managing telephone operations
and obtained and analyzed supporting documentation. The interviews included
surveys of the 10 field directors responsible for managing IRS? 26 call
sites, which employ about 10, 000 telephone assistors. Our scope and
methodology are discussed in greater detail in a separate section of this
report.
Overall, IRS made limited progress in the 2001 tax filing season toward its
goal of providing world- class telephone service. When compared with the
2000 tax filing season, access and accuracy performance for 2001 improved
considerably in two of six comparable measures, declined in one, and changed
2 percentage points or less in the others. In addition, performance did not
meet six of the eight 2001 targets IRS set to help it move IRS toward its
goal of world- class service. More specifically, when access measures were
compared with the 2000 tax filing season, there was a 4- percentage- point
decline in callers that hung up while waiting to speak with an assistor, but
they waited 15- percent longer to speak with an assistor. The other two
comparable measures of access showed little change. For the accuracy
measures, the quality of service when responding to account inquiries
increased by 10 percentage points, but the quality of service when
responding to tax law questions showed little change. When compared with
2001 performance targets, IRS did not meet any of its access goals, falling
considerably below its target to shorten the time taxpayers spend waiting to
speak with an assistor. Although assistors exceeded quality- of- service
targets when responding to taxpayer questions about their accounts and tax
law, they did not meet higher targets for providing taxpayers with correct
answers and account adjustments.
IRS officials missed some opportunities to analyze data to better understand
the factors affecting telephone performance, including the actions it took
to improve performance. IRS collected and analyzed a variety of data about
the key factors affecting access and accuracy. However, IRS managers
sometimes reached conclusions about these key factors without conducting
analyses to test their conclusions. IRS officials also missed opportunities
to plan evaluations to determine the effectiveness of the actions IRS took
to improve access and accuracy. For example, field directors disagreed about
whether the new Accounts Resolution Guide, a computer- based guide to help
assistors resolve Results in Brief
Page 3 GAO- 02- 212 IRS' 2001 Filing Season Telephone Assistance
taxpayers? questions, improved accuracy in 2001. At the time the guide was
implemented, IRS did not have a plan to evaluate it, and no evaluation was
done. The Government Performance and Results Act of 1993 (GPRA) 3 and IRS
guidance stress the need for conducting analyses to help management
understand how factors affect performance so that corrective actions can be
targeted toward those factors. We recognize that some data analyses can be
costly; however, some of the missed opportunities involve key factors
affecting performance. Also, since IRS devotes significant resources- 10,
000 assistors- to telephone assistance, the benefit of more effectively
using the resources to improve service could be substantial.
Because of the limited progress made in the 2001 filing season, IRS cannot
afford to miss opportunities to analyze performance data and evaluate
improvement actions. We are recommending that the Commissioner ensure that
managers at the appropriate levels of IRS follow existing guidance to plan
and conduct analyses of the factors affecting performance and to evaluate
improvement actions.
In a letter dated December 3, 2001, the IRS Commissioner agreed with our
recommendations. (See p. 16 for a discussion of agency comments, which are
reprinted in app. II.)
In the 2001 tax filing season, IRS received more than 70.7 million calls on
its three toll- free assistance numbers and answered over 50.5 million
calls- assistors answered 22.7 million calls and automated systems answered
27.8 million calls. 4 As in previous years, IRS had three toll- free
telephone numbers that taxpayers could call with questions about tax law,
taxpayer accounts, and refunds. 5 Located at 26 call sites, IRS has about
10,000 assistors that help taxpayers with a variety of questions ranging
from the applicability of tax laws to the status of their accounts. IRS?
call
3 P. L. 103- 62. 4 Automated systems include the Telephone Routing and
Interactive System (TRIS) and TeleTax interactive applications that allow
taxpayers with certain questions to obtain the information, such as the
status of their refunds, or service they need without speaking to an
assistor. The three telephone numbers are part of TRIS, while TeleTax is a
separate number that also provides recorded information on about 150 tax
topics. Taxpayers who called a TRIS number about refund status were given
the option of being routed to TeleTax in the 2001 tax filing season.
5 IRS?s published telephone number for providing taxpayer assistance is 800-
829- 1040. IRS lists additional toll- free telephone numbers on the notices
it sends to taxpayers asking them to call about an account or refund issue.
Background
Page 4 GAO- 02- 212 IRS' 2001 Filing Season Telephone Assistance
sites are supervised by 10 field directors, each of whom oversees two to
three sites.
IRS has four measures to evaluate the extent to which taxpayers are provided
with accessible telephone assistance, and four to evaluate the extent to
which taxpayers are provided with accurate telephone assistance. (For more
information on IRS? telephone assistance access and accuracy measures see
app. I.) IRS? measures of access are based on actual counts of calls using
data collected by IRS? telephone system. IRS? measures of the accuracy of
assistance, the quality and correct response measures, are estimates based
on representative samples of nationwide calls that quality assurance staff
monitor and score for accuracy. IRS began collecting data on correct
responses in June 2000, so there are no data for the 2000 tax filing season
to compare with 2001.
Over the years, IRS has studied its telephone performance and made changes
designed to improve it. For example, in 1999, IRS extended its hours of
service to 24 hours a day, 7 days a week. By providing around- theclock
service, IRS expected to distribute demand more evenly and thus improve
taxpayers? access to service. With the increased use of callrouting
technology in 1999, IRS began to manage its telephone operations centrally
at the Joint Operations Center in Atlanta. Routing calls to the first
available assistor who had the necessary skills to answer the taxpayer?s
question was expected to improve taxpayers? access to service and lessen the
disparity in the level of service across sites. However, the level of
service declined in 1999, 6 and the quality of service was mixed in the 2000
tax filing season and below IRS? long- term goal of providing world- class
customer service.
According to IRS, some of the key factors that affected performance in the
2000 tax filing season were the demand for assistance, staffing levels,
assistor productivity, assistor skills, and IRS? guidance for assistors. As
we discussed in a previous report, 7 IRS? analyses did not cover all key
management decisions or other key factors that could have affected telephone
performance. Additionally, determining how each factor affected performance
was made even more difficult because many of the factors are interrelated;
changes in one can affect another. The IRS Commissioner has recognized the
complex interrelationships within the
6 See Tax Administration: IRS? 1999 Tax Filing Season (GAO/ GGD- 00- 37,
Dec. 15, 1999). 7 GAO- 01- 189, Apr. 6, 2001.
Page 5 GAO- 02- 212 IRS' 2001 Filing Season Telephone Assistance
telephone- operating environment and has stated that years of sustained
effort will be required for IRS to achieve its goal of providing world-
class telephone service.
To address our objectives, we interviewed IRS officials involved in managing
toll- free telephone operations and obtained and analyzed supporting
documentation as follows:
To assess IRS? performance in responding to calls on the three main
telephone assistance toll- free numbers, we compared the 2001 tax filing
season performance for accessibility and accuracy measures with IRS?
performance in the 2000 tax filing season and its 2001 performance targets.
8
To assess IRS? efforts to determine the factors that affected performance
in the 2001 tax filing season, including actions it took to improve
performance, we used as criteria GPRA and IRS? own guidance on analyzing
performance data. We interviewed IRS officials in the Wage and Investment
and Small Business and Self- Employed Divisions, 9 and the Joint Operations
Center. We also analyzed various documents, including reports on IRS?
efforts to determine the factors that affect telephone performance and the
results of actions to improve performance. In addition, we used a
questionnaire to obtain information from the 10 field directors about their
efforts to identify the factors that affected performance and assess the
effectiveness of actions taken to improve performance.
While we did not independently assess the accuracy of IRS? performance data,
we verified that IRS had procedures in place intended to ensure data
reliability. We did our work from February 2001 through October 2001 in
accordance with generally accepted government auditing standards.
8 Accessibility measures are based on comparable data for the weeks
beginning January 1 and ending July 14. Tax law and accounts accuracy rates
are based on data for the months of January through June.
9 IRS restructured its operation into four divisions: Wage and Investment
Income (W& I), Small Business and Self- Employed (SB/ SE), Large and Mid-
Size Business, and Tax- Exempt and Government Entities. The W& I and SB/ SE
Divisions provide IRS? toll- free assistance. Scope and
Methodology
Page 6 GAO- 02- 212 IRS' 2001 Filing Season Telephone Assistance
IRS made limited progress in the 2001 tax filing season toward its longterm
goal of providing world- class telephone service. When compared with the
2000 tax filing season, access and accuracy performance improved by 2
percentage points or less in three of the six comparable measures. The
quality of responses to account inquiries increased 10 percentage points,
and there was a 4 percentage point decline in callers who hung up while
waiting to speak with an assistor; however, taxpayers waited 15- percent
longer to speak with an assistor. When compared with 2001 performance
targets, IRS did not meet any of its accessibility goals. These targets were
intended to move IRS toward its goal of providing world- class service.
Although it met or exceeded the 2001 quality targets, IRS did not meet the
current year?s higher targets for providing taxpayers with correct
responses.
Table 1 compares IRS? actual 2000 performance levels with its 2001
performance levels and targets. (See app. I for more information on the
measures.)
Table 1: IRS Tax Filing Season Telephone Assistance Performance 2000 2001
Accessibility measures a Actual Actual Target
Assistor level of service 60% 61% 69% Assistor response level 39 41 49
Abandon rate 22 18 12 Average speed of answer 237 seconds 273 seconds 132
seconds
Accuracy measures a
Tax law quality rate 73% +/- 2% b 75% +/- 1% b 74% Accounts quality rate 59
+/- 2% b 69 +/- 1% b 63 Tax law correct response rate c 79 +/- 1% b 82
Accounts correct response rate c 88 +/- 1% b 91 a Accessibility measures are
based on actual counts. Accuracy measures are based on representative
samples. b Actual values are estimated at the 90- percent confidence level.
c Comparable data do not exist. Source: IRS data.
According to IRS officials, the access measures are similar to those
commonly used by world- class customer service organizations. They are
designed to focus efforts on enhancing taxpayers? experience in getting IRS
Made Limited
Progress Toward Providing World- Class Telephone Service
Page 7 GAO- 02- 212 IRS' 2001 Filing Season Telephone Assistance
access to assistance. For example, the ?assistor level of service? measure
10 is intended to show IRS? effectiveness in providing callers with access
to an assistor. 11 The ?assistor response level? is to measure the
percentage of taxpayers that waited 30 seconds or less to speak with an
assistor. The
?abandon rate? measure is to show the percentage of taxpayers who hang up
while waiting to speak with an assistor, while the ?average speed of answer?
measure is to show the average number of seconds taxpayers wait to speak to
an assistor.
IRS? accuracy measures are designed to gauge the taxpayers? experience in
getting accurate assistance. The ?quality? measures are to show, for a
representative sample of calls, the percentage for which assistors followed
all procedures, such as properly identifying themselves at the beginning of
calls, doing appropriate research on taxpayers? accounts, and providing
accurate information to taxpayers. The new ?correct response rate? measures
are intended to show the percentage of calls for which IRS assistors
provided correct responses to inquiries without taking into account
procedural errors that would not affect the accuracy of the information
given the taxpayer. IRS began collecting these data in June 2000, so there
were no data for the 2000 filing season to compare with 2001.
IRS also measures its performance in answering calls through the use of
automation. However, we did not consider this measure-? automated service
completion rate?- in assessing IRS? performance because it assumes that
callers who get through to TeleTax are served. The TeleTax system does not
have data on how many callers hung up before completing an automated
service. Although IRS officials recognize that the measure had limitations,
according to them, routing refund status calls to TeleTax allowed IRS to
answer about 11.3 million more calls made to its three main toll- free
assistance numbers as compared with the 2000 tax filing season. We are
continuing to assess many of IRS? new performance measures, including those
used to evaluate telephone assistance.
10 This measure includes callers who received automated service when the
assistor service queue was full. 11 IRS? previous access measure, level of
service, was calculated by dividing the total calls answered by total call
attempts on IRS? toll- free assistance lines. The Commissioner of IRS
discontinued use of this measure in February 2001, replacing it with the
assistor level of service measure.
Page 8 GAO- 02- 212 IRS' 2001 Filing Season Telephone Assistance
IRS missed some opportunities to better understand the factors that affected
performance and to plan evaluations of the actions it took to improve
performance. GPRA and IRS guidance outline the benefits of first gathering
and then analyzing data to help managers understand the reasons for
performance. To this end, IRS collected performance data on access and
conducted some analyses. Even so, IRS missed opportunities to do other
analyses of the factors affecting performance including actions to
improvement it. Contributing to the missed opportunities was a lack of
planning for the evaluation of those actions.
IRS has a variety of systems in place to make data on the access and
accuracy of telephone assistance available to managers. Two of these systems
are its Joint Operations Center in Atlanta and Centralized Quality Review
System (CQRS) in Philadelphia. Managers at call sites also collect data on
factors affecting access and accuracy performance.
IRS? Joint Operations Center in Atlanta manages the activities of the 26
call sites, including monitoring access data and routing calls to the next
available assistor anywhere in the country. The Center collects data on
various accessibility measures and makes those data available daily to IRS
managers through an internal Web site. According to IRS officials, IRS
improved the collection of performance data in the 2001 tax filing season.
For example, IRS implemented the Enterprise Telephone Database to provide a
central call information database. The database was designed to provide IRS
analysts and management with the most accurate information for analysis and
program decision- making by centralizing data collection and producing a
standard set of management reports.
IRS? CQRS staff in Philadelphia are responsible for collecting data on the
accuracy of telephone assistance. CQRS provides call- site officials with
daily access through its internal Web site on the results of the sample of
calls answered by their sites. It also provides weekly and monthly reports
on the quality of sites? responses to taxpayers? questions about tax law or
about their accounts- two of the accuracy performance measures. These data
show the call sites what errors assistors are making so site managers can
quickly take action to reduce these errors. IRS officials told us that they
made better use of the data in the 2001 filing season. They said that Wage
and Investment Division and site officials developed strategies to reduce
assistor errors based on CQRS reports.
IRS call sites collect data on factors affecting access and accuracy in
various ways. For example, supervisors use real- time data and historical
IRS Missed Some
Opportunities to Better Understand Telephone Performance
IRS Collected Performance Data on Access and Accuracy
Page 9 GAO- 02- 212 IRS' 2001 Filing Season Telephone Assistance
reports available at the call sites on how assistors spent their time,
including average handle time- the time an assistor spends talking with the
taxpayer, keeping the taxpayer on hold, and finishing the call and
indicating readiness to receive another call. Also, local staff monitor
calls to provide more detailed information on what errors assistors are
making and in what units the errors are being made.
IRS, both at the national and call- site levels, conducted some analyses of
performance data intended to determine the factors affecting performance.
GPRA and IRS guidance stress that analysis is a key part of understanding
performance and identifying improvement options. Analysis of performance
data is intended to help managers understand changes in performance,
determine root causes, and identify improvement options.
We identified several examples of analytical efforts to determine the
factors affecting performance at both the national and call- site levels. In
one example with regard to access, IRS officials analyzed data provided by
the Joint Operations Center to determine the reasons for the lower-
thanexpected level of service in the first 3 months of fiscal year 2001.
They concluded that declining assistors? productivity, as measured by
average handle time, was the major reason for the decline in access.
Officials from the operating divisions and the call sites conducted a series
of assessments to determine the underlying reasons for the increase in
average handle time. The assessments included focus groups with managers and
employees to solicit their views on productivity and monitoring of telephone
calls to determine how assistors use the time between calls. The assessments
identified three major categories of factors that had negatively affected
average handle time: management practices, work processes, and computer
systems. According to IRS officials, some management practices adversely
affected the level of service because managers did not take actions to
improve assistors? use of time between calls, the primary factor that
increased average handle time. IRS officials said that they took immediate
corrective actions, such as briefing assistors and supervisors and
eliminating unnecessary data entry and taxpayer notification requirements.
They also organized teams to further evaluate and resolve the more
complicated work process and computer systems issues.
According to IRS officials, IRS improved the analysis of Joint Operations
Center performance data in the 2001 tax filing season. For example, analysts
studied the factors that affected the demand for live assistance regarding
refunds, including the impact of increased electronic filing. IRS Did Some
Analyses to
Determine the Factors Affecting Performance
Page 10 GAO- 02- 212 IRS' 2001 Filing Season Telephone Assistance
Center analysts also began developing quantitative models of the time
taxpayers and IRS spend on telephone questions, with the intent to better
match IRS? resources with taxpayer needs.
Regarding accuracy, CQRS staff analyzed assistor errors and made nationwide
and individual site suggestions for addressing the causes of the errors. The
suggestions included changes to the assistors? training and guidance. Also,
IRS field directors conducted some analyses to determine the factors that
affected access and accuracy. For example, one director said analysis staff
at one of her sites was doing a study to determine if the site?s extensive
use of faxing negatively affected access. She said she believed the site?s
average handle time was longer than others owing to the site?s policy to
keep the taxpayer on the line until the accounts issue was resolved, even
while the taxpayer faxed documents. Another director said he monitored
assistors to determine whether the computer- based research tools assistors
used to answer taxpayers? questions met assistors? needs.
Although IRS conducted some analyses of performance data, it missed
opportunities to do other analyses at the field level that could have
provided a better understanding of the factors affecting telephone
assistance performance, including the actions it took to improve
performance. Identifying the key factors that most affect performance is
important, yet difficult because those factors that can affect telephone
access and accuracy are often numerous and interrelated. IRS guidance
recognizes that there are a variety of approaches to conducting analyses,
such as hypothesis testing, which involves forming a tentative conclusion
that is tested using the data. We recognize that some analyses can be
costly, but as already noted, GPRA and IRS guidance stress that analysis is
a key part of identifying improvement options.
Field directors sometimes reached conclusions about the factors affecting
access and accuracy without conducting analyses to test their conclusions.
Seven of 10 directors said that the relative inexperience of assistors,
caused primarily by higher- than- usual attrition, was a key factor
affecting performance in the 2001 tax filing season. They said many
experienced seasonal assistors had taken permanent positions in other parts
of IRS, and the new hires who replaced them tended to take longer on calls
and make more errors. The second most common factor they cited was problems
with the computer- based research tools that assistors used to answer
taxpayers? questions. Five of 10 directors cited such problems, including
difficulties in using the Servicewide Electronic Research Project to search
the Internal Revenue Manual, the assistors? primary guidance for IRS Missed
Some
Opportunities to Analyze Performance
Page 11 GAO- 02- 212 IRS' 2001 Filing Season Telephone Assistance
handling calls regarding taxpayers? accounts. Some directors said the
computer systems were cumbersome and difficult to navigate, causing
assistors to take longer on calls and make errors, and some said computer
systems often failed and thus hampered assistors? ability to research
questions.
Although directors cited high attrition and computer problems as key factors
affecting performance, only two directors identified a specific analysis to
support their conclusions. These directors said that focused monitoring was
done at their sites that confirmed the limitations of a computer system
assistors used to answer taxpayers? questions. When we asked other directors
whether they or their staff had conducted analyses to confirm or refute
their conclusions about the factors that affected performance, they
acknowledged that they had not.
We identified several opportunities to conduct analyses of performance. One
way directors could have analyzed the impact of high attrition on access and
accuracy would have been to monitor a sample of calls handled by experienced
and inexperienced assistors to compare error rates and average handle time.
One director acknowledged that her analysis staff could have done more to
learn about how accepting additional calls from businesses affected
performance, such as comparing the handle time for business taxpayer calls
with individual taxpayer calls. In another example, the program manager for
the new Accounts Resolution Guide, a computer- based, step- by- step guide
on how to resolve an account- related telephone call, agreed that more could
have been done to evaluate the guide?s effectiveness. For example, local
managers could have observed and compared assistors who used and did not use
the guide. According to IRS officials, additional analyses such as these
could have been done at relatively low cost.
Analyzing performance data can be important for several reasons. First,
there can be disagreement about which actions improve performance. For
example, some directors cited the Accounts Resolution Guide as a reason for
the significant improvement in their accounts quality rate. However, another
director said that the guide actually had a negative effect on accounts
quality, saying that because the guide was new to some assistors, the
?learning curve? to become proficient in using the guide caused assistors to
make errors.
Second, when multiple factors affect performance, knowing the extent to
which each factor has an impact can help managers decide where to focus
scarce managerial attention. For example, the solutions for addressing
Page 12 GAO- 02- 212 IRS' 2001 Filing Season Telephone Assistance
high attrition and computer problems are likely to be different.
Understanding the relative importance of high attrition and computer
problems could help prioritize improvement actions. In addition, in the case
of multiple factors, IRS? use of performance measures to determine the
effect of one factor without controlling for other factors can be
misleading. In one case, a field director noted the risks of using average
handle time as an indicator of the effectiveness of actions taken to improve
the productivity of telephone assistors. The director noted that other
factors, such as the complexity of calls handled, could also affect average
handle time.
Third, the interrelationship among factors makes it difficult to determine
which factors most affect performance. For example, as we reported last
year, the quality of guidance assistors use can affect not only the accuracy
but also the accessibility of telephone assistance. 12 Although step- by-
step guidance on how to respond to questions would likely improve accuracy,
it could also cause assistors to take more time answering calls, thereby
negatively affecting taxpayers? access to service.
As we previously reported, conducting systematic analyses of program
performance is important for determining the factors affecting performance
and identifying opportunities for improvement. 13 IRS guidance states that
analysis to understand the underlying factors influencing the performance
reflected in the balanced measures is necessary to determine how to improve
performance and warns that managers should not ?jump to conclusions? about
the causes of performance problems. 14 As we said in a report on management
reform,
?an organization cannot improve performance and customer satisfaction if it
does not know what it does that causes current levels of performance and
customer satisfaction.? 15 Because the factors affecting telephone
performance are numerous and are often interrelated, conducting analyses is
essential to determining the factors that have the most effect on
12 GAO- 01- 189, Apr. 6, 2001. 13 Managing for Results: Challenges Agencies
Face in Producing Credible Performance Information (GAO/ GGD- 00- 52, Feb.
4, 2000). 14 Internal Revenue Manual 105.4.1.
15 Management Reform: Using the Results Act in Quality Management to Improve
Federal Performance (GGD/ T- 99- 151, July 29, 1999).
Page 13 GAO- 02- 212 IRS' 2001 Filing Season Telephone Assistance
performance so that corrective actions can be targeted toward those factors.
We recognize that some analysis can be costly. Consequently, the costs need
to be balanced against the benefits. Considering that IRS devotes
significant resources (about 10,000 assistors) to telephone assistance, the
benefits of analysis- identifying ways to more effectively use resources and
improve service- could be substantial.
IRS missed opportunities to plan evaluations to determine the effectiveness
of actions it took to improve the access and accuracy of its telephone
assistance. IRS guidance presents a seven- step process designed to guide
data collection and analysis to identify ways to improve performance. The
last step states that managers should establish a plan that tracks the
effectiveness of actions taken to improve performance. Without such a plan,
IRS may not collect the data needed to judge the action?s effectiveness.
Additionally, planning to collect the data before the improvement action is
implemented may be less costly than developing the data and evaluating the
action later.
IRS field directors cited several different actions they took to address
factors that negatively affected access and accuracy in the prior filing
season:
Assistor skill gaps (the difference between the skills assistors had and
the skills needed by IRS). To address skill gaps, field directors most
frequently cited training as the action taken, with all 10 directors
referring to training as the primary, and most often only, action taken.
Although training was designed at the division level, field directors and
managers were responsible for implementing it in the field, such as
selecting the trainers and determining which assistors need to be trained.
Errors caused by flaws in the guidance assistors used to respond to
taxpayers? account questions. To address the flaws in assistors? guidance
for answering taxpayer calls, 5 of the 10 field directors cited the
implementation and use of computer- based tools to improve guidance,
including the Accounts Resolution Guide. IRS Missed Opportunities
to Plan Evaluations of Actions to Improve Access and Accuracy
Page 14 GAO- 02- 212 IRS' 2001 Filing Season Telephone Assistance
Declining assistor productivity. All 10 field directors said that the
primary actions taken to address assistor productivity declines were
nationwide managers? training and employee briefings. 16
Although field directors said all three of these actions were key to
improving telephone assistance performance this year, none of the field
directors cited specific evaluations or plans for assessing the
effectiveness of these actions. Instead, field directors based their
assessment of actions on performance trends, not taking into account the
multiple factors or the interrelatedness of factors that can affect a
performance measure.
One example of a missed opportunity to plan an evaluation of an improvement
action on the national level is the lack of a systematic plan to assess the
impact the Accounts Resolution Guide had on access and accuracy. The program
manager for the guide said that IRS did not develop such a plan because
assistors were not required to use the guide and IRS? remote monitoring
system was unable to determine when the guide was used. The program manager
agreed that local managers could have done more to evaluate its
effectiveness because they could have observed and compared the results of
assistors who did or did not use the guide. Having an evaluation plan when
the new Accounts Resolution Guide was distributed to the field would have
provided local managers with guidance on the type of data to collect.
Another example of a missed opportunity is the lack of evaluation plans in
filing season readiness plans. IRS field directors complete a standard plan
each year, adding any items unique to their sites, to ensure that the sites
have taken all the necessary steps to provide phone assistance in the tax
filing season. Such steps include providing appropriate training and having
the equipment and guidance assistors need to respond to taxpayer calls. The
readiness plans we reviewed, however, did not include steps to ensure that
sites collect and analyze data to evaluate the effectiveness of any
improvement actions.
IRS officials noted that since some of the improvement actions were national
in scope, field directors would not have been individually responsible for
evaluating the effectiveness of the actions. We recognize
16 Management training was designed to improve managers? skills in
addressing productivity issues, and the briefings were designed to increase
assistor awareness of the increase in average handle time and its impact on
access.
Page 15 GAO- 02- 212 IRS' 2001 Filing Season Telephone Assistance
that evaluations of national improvement actions, such as the Accounts
Resolution Guide and managers? training and employee briefings, to address
productivity may involve the higher levels in IRS that are responsible for
the action. Accordingly, as noted above, we discussed the Guide with its
national program manager and were told that no systematic evaluation was
done. We also discussed the actions to address productivity with division-
level officials. Similar to the field directors, division officials
evaluated the actions by monitoring trends in average handle time and
comparing average handle time with previous performance.
IRS made limited progress in the 2001 tax filing season toward its longterm
goal of providing world- class customer service. To speed progress toward
its long- term goal, IRS managers need to identify the causes for
performance, plan strategies to improve performance, and evaluate how well
those strategies worked. Unfortunately, IRS sometimes missed opportunities
to conduct analysis to help managers understand the reasons for performance
and to evaluate actions taken to improve performance. The decision on the
type of analysis to be done and who will do it should consider the costs and
benefits of the analysis and which organizational levels are most
responsible for the factor or improvement action being analyzed. We
recognize that some analyses can be costly; however, some of the missed
opportunities were low- cost and some involved key factors affecting actions
taken to improve performance. In addition, there are costs in not using the
10,000 assistors as effectively as possible. Considering IRS? limited
progress, it cannot afford to miss opportunities without determining the
most effective use of its resources to improve performance.
We recommend that the Commissioner ensure that managers follow IRS guidance
on analyzing the factors that affect performance and evaluating improvement
actions. Specifically, we recommend that (1) field directors be required to
develop and follow written plans to collect and analyze data to test their
conclusions about the key local factors affecting performance and, when
appropriate, evaluate local improvement actions, such as actions involving
training; (2) field directors include in filing season readiness plans a
step to ensure that site managers have plans to evaluate the effectiveness
of any local improvement actions; and (3) program managers and other
appropriate national officials be required to develop and follow written
plans to evaluate the effectiveness of key national improvement actions,
such as the Accounts Resolution Guide. Conclusions
Recommendation for Executive Action
Page 16 GAO- 02- 212 IRS' 2001 Filing Season Telephone Assistance
The Commissioner of Internal Revenue provided written comments on a draft of
this report in a December 3, 2001, letter, which is reprinted in appendix
II. The Commissioner stated that while the report categorized IRS? progress
toward providing world- class telephone assistance as limited, he is
confident that IRS is moving in the right direction. He noted that IRS had
initiated a number of strategies to improve telephone assistance and agreed
with our recommendation. Specifically, he agreed that IRS needs ?better
testing, documentation, and analytical activities to determine the factors
that affect performance and assess the results of our improvement actions.?
In his comments, the Commissioner noted that the report focuses on
accessibility to telephone assistors and stated that IRS also assists
taxpayers through automated telephone services and other means, such as IRS?
Internet Web site and walk- in Tax Assistance Centers. As noted in the
report, we did not assess IRS? automated telephone services because IRS?
method of measuring its performance in providing automated services had
limitations. The measure assumed that all callers that go through one of its
automated systems- TeleTax- were served because the TeleTax system does not
have data on how many taxpayers hung up before completing an automated
service. Other taxpayer services, such as walk- in assistance, are to be
addressed in our upcoming report on various aspects of the 2001 tax filing
season.
As agreed with your staff, unless you publicly release its contents earlier,
we will make no further distribution of this report until 30 days after its
issue date. At that time, we will send copies of this report to the Chairmen
and Ranking Minority Members of the Senate Committee on Finance and the
House Committee on Ways and Means and the Ranking Minority Member of the
Subcommittee. We will also send copies to the Secretary of the Treasury; the
Commissioner of Internal Revenue; the Director, Office of Management and
Budget; and other interested parties. We will make copies available to
others on request. Agency Comments
and Our Evaluation
Page 17 GAO- 02- 212 IRS' 2001 Filing Season Telephone Assistance
If you have any questions or would like additional information, please call
me at (202) 512- 9110 or Carl Harris at (404) 679- 1900. Key contributors to
this report are Ronald W. Jones and Ronald J. Heisterkamp.
James R. White Director, Tax Issues
Appendix I: Descriptions and Formulas for Measures of IRS Telephone
Assistance Accessibility and Accuracy
Page 18 GAO- 02- 212 IRS' 2001 Filing Season Telephone Assistance
Table 2: Descriptions and Formulas for Measures Measure Description Formula
Assistor level of service The percentage of callers who were routed to an
assistor or who call when the assistor queue is full and who reached an
assistor or completed an automated service.
The number of calls answered by assistors, and by automated services, when
the waiting queue for speaking with an assistor was full, divided by the
total of these calls plus the number of calls that hung up while waiting to
speak with an assistor or when the queue for speaking to assistor was full,
plus busy signals. Assistor response level The percentage of callers who
waited
30 seconds or less to speak with an assistor.
Total number of services provided that were delivered within the threshold
of 30 seconds divided by the total services handled. Abandon rate The
percentage of callers who hung
up while waiting to speak with an assistor.
Total calls that hang up after waiting in line for an assistor divided by
the total call attempts offered to assistors.
Average speed of answer The average number of seconds callers waited before
an assistor took their call.
Total number of seconds that customers waited in line for service by
assistor divided by the total services provided.
Tax law quality rate An estimate of the percentage of calls dealing with tax
law in which assistors followed all required procedures.
The estimate is calculated by taking the site quality rate from the
Centralized Quality Review System (CQRS) and weighting it using total site
volume and number reviewed. Accounts quality rate An estimate of the
percentage of calls
dealing with accounts issues in which assistors followed all required
procedures.
The estimate calculated by taking the site quality rate from CQRS and
weighting it using total site volume and number reviewed.
Tax law correct response rate An estimate of the percentage of callers who
received accurate responses to tax law inquiries.
The estimate calculated by taking the site rate from CQRS and weighting it
using total site volume and number reviewed. Accounts correct response rate
An estimate of the percentage of
callers who received accurate responses to account inquiries.
The estimate calculated by taking the site rate from the Centralized Quality
Review System and weighting it using total site volume and number reviewed.
Appendix I: Descriptions and Formulas for Measures of IRS Telephone
Assistance Accessibility and Accuracy
Appendix II: Comments From the Internal Revenue Service
Page 19 GAO- 02- 212 IRS' 2001 Filing Season Telephone Assistance
Appendix II: Comments From the Internal Revenue Service
Appendix II: Comments From the Internal Revenue Service
Page 20 GAO- 02- 212 IRS' 2001 Filing Season Telephone Assistance
Appendix II: Comments From the Internal Revenue Service
Page 21 GAO- 02- 212 IRS' 2001 Filing Season Telephone Assistance
Appendix II: Comments From the Internal Revenue Service
Page 22 GAO- 02- 212 IRS' 2001 Filing Season Telephone Assistance (440050)
The General Accounting Office, the investigative arm of Congress, exists to
support Congress in meeting its constitutional responsibilities and to help
improve the performance and accountability of the federal government for the
American people. GAO examines the use of public funds; evaluates federal
programs and policies; and provides analyses, recommendations, and other
assistance to help Congress make informed oversight, policy, and funding
decisions. GAO?s commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
The fastest and easiest way to obtain copies of GAO documents is through the
Internet. GAO?s Web site (www. gao. gov) contains abstracts and full- text
files of current reports and testimony and an expanding archive of older
products. The Web site features a search engine to help you locate documents
using key words and phrases. You can print these documents in their
entirety, including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as ?Today?s Reports,? on its Web
site daily. The list contains links to the full- text document files. To
have GAO E- mail this list to you every afternoon, go to our home page and
complete the easy- to- use electronic order form found under ?To Order GAO
Products.?
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:
U. S. General Accounting Office P. O. Box 37050 Washington, D. C. 20013
To order by phone: Voice: (202) 512- 6000 TDD: (301) 413- 0006 Fax: (202)
258- 4066
GAO Building Room 1100, 700 4th Street, NW (corner of 4th and G Streets, NW)
Washington, D. C. 20013
Contact: Web site: www. gao. gov/ fraudnet/ fraudnet. htm, E- mail:
fraudnet@ gao. gov, or 1- 800- 424- 5454 (automated answering system).
Jeff Nelligan, Managing Director, NelliganJ@ gao. gov (202) 512- 4800 U. S.
General Accounting Office, 441 G. Street NW, Room 7149, Washington, D. C.
20548 GAO?s Mission
Obtaining Copies of GAO Reports and Testimony
Order by Mail or Phone Visit GAO?s Document Distribution Center
To Report Fraud, Waste, and Abuse in Federal Programs
Public Affairs
*** End of document. ***