Tax Systems Modernization: Imaging System's Performance Improving but
Still Falls Short of Expectations (Letter Report, 01/16/97,
GAO/GGD-97-29).
Pursuant to a congressional request, GAO reviewed the performance of the
Internal Revenue Service's (IRS) Service Center Recognition/Image
Processing System (SCRIPS) in 1996, focusing on: (1) the primary causes
for performance problems that occurred in 1995; (2) whether SCRIPS
performance improved in 1996 as of September 30, 1996; and (3) the
status of IRS future plans for SCRIPS.
GAO found that: (1) SCRIPS experienced significant performance problems
in 1995; (2) two problems were system downtime and slow processing
rates; (3) because of performance problems, two of the five centers
stopped processing Forms 1040EZ on SCRIPS and reverted to using manual
data entry; (4) SCRIPS performance deficiencies in 1995 stemmed
primarily from both hardware and software problems; (5) although IRS had
expected that SCRIPS would be processing five document types, forms
1040EZ, 941, and 1040PC, federal tax deposit (FTD) coupons, and
information returns, IRS postponed plans to process forms 941 and 1040PC
on SCRIPS; (6) of the three remaining document types, the Cincinnati
test certified the software application only for FTD coupons, therefore,
the software applications for information returns and Form 1040EZ were
not thoroughly tested before they were put into production, (7) to
improve the performance of SCRIPS for 1996, IRS made hardware and
software modifications, some of which were made before the start of the
1996 filing season; (8) as a result of some of these enhancements and
more staff familiarity with SCRIPS, according to IRS officials in all
five SCRIPS service centers, SCRIPS performed significantly better
during the 1996 filing season than it did in 1995; (9) in addition to a
slower processing rate for information returns, SCRIPS is not processing
all the forms that it was expected to process in 1996, is expected to
cost more than originally estimated, and is expected to provide lower
labor cost savings than IRS originally anticipated; (10) the latest cost
estimate for SCRIPS is $288 million, considerably more than previous
cost estimates, which, according to IRS' post-implementation review
report on SCRIPS, ranged from +$133 million to $209 million; (11) July
1995, IRS decided to terminate all software programming for the 1040PC
because of SCRIPS' instability, lack of system capacity, and a number of
software application problems; (12) IRS has also decided not to use
SCRIPS to process Forms 941 beyond fiscal year (FY) 1997; and (13) in
addition to funding constraints, according to IRS officials, a decision
on using SCRIPS for Forms 941 will depend on the results of a September
1996 capacity test of SCRIPS, which were not available when GAO
completed its audit work, and decisions by IRS' Investment Review Board.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: GGD-97-29
TITLE: Tax Systems Modernization: Imaging System's Performance
Improving but Still Falls Short of Expectations
DATE: 01/16/97
SUBJECT: Systems conversions
Tax administration systems
Image data bases
Computer software verification and validation
Electronic forms
Tax returns
Application software
Information processing operations
Cost control
IDENTIFIER: IRS Service Center Recognition/Image Processing System
IRS Document Processing System
Cincinnati (OH)
IRS 1040PC Program
IRS Tax System Modernization Program
TSM
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to the Chairman, Subcommittee on Oversight, Committee on Ways
and Means, House of Representatives
January 1997
TAX SYSTEMS MODERNIZATION -
IMAGING SYSTEM'S PERFORMANCE
IMPROVING BUT STILL FALLS SHORT OF
EXPECTATIONS
GAO/GGD-97-29
Performance of SCRIPS
(268708)
Abbreviations
=============================================================== ABBREV
DPS - Document Processing System
FTD - Federal Tax Deposit
IRS - Internal Revenue Service
OCR - optical character recognition
TSM - Tax Systems Modernization
SCRIPS - The Service Center Recognition/Image Processing System
Letter
=============================================================== LETTER
B-270263
January 16, 1997
The Honorable Nancy L. Johnson
Chairman, Subcommittee on Oversight
Committee on Ways and Means
House of Representatives
Dear Chairman Johnson:
One of the major objectives of the Internal Revenue Service's (IRS)
modernization effort is to move away from a labor-intensive tax
return processing system that relies on thousands of employees
transcribing data from paper tax returns and move to an electronic
system that reduces processing costs and eliminates transcription
errors. One strategy for achieving that objective is to reduce the
number of paper returns by increasing the number of returns filed
electronically. For returns that will continue to be filed on paper,
IRS plans to achieve its objective through document imaging and
optical character recognition (OCR). The Service Center
Recognition/Image Processing System (SCRIPS) is the first of what was
to have been two document imaging and OCR systems.\1
SCRIPS became operational in the latter part of 1994, and we reported
that it experienced numerous performance problems during the 1995
filing season.\2 This report responds to your request that we follow
up on those problems and assess the performance of SCRIPS in 1996.
Specifically, this report (1) identifies the primary causes for
performance problems that occurred in 1995; (2) assesses whether
SCRIPS performance improved in 1996 as of September 30, 1996; and (3)
provides a status report on IRS' future plans for SCRIPS.
--------------------
\1 The second system, the Document Processing System (DPS), was
intended to replace SCRIPS and expand the imaging capability to more
complex tax forms. However, on October 8, 1996, IRS announced that
it was terminating that project.
\2 The 1995 Tax Filing Season: IRS Performance Indicators Provide
Incomplete Information About Some Problems (GAO/GGD-96-48, Dec. 29,
1995).
BACKGROUND
------------------------------------------------------------ Letter :1
In July 1995, we reported on IRS' progress in implementing some of
the business and technological components of its modernization
effort, known as Tax Systems Modernization (TSM).\3 Although we said
that IRS had made some progress, we also said that pervasive
management and technical weaknesses existed that placed the
modernization effort at risk. Among other things, (1) IRS did not
have a business strategy to maximize electronic filing, the result of
which could slow the planned decrease in the workload of paper
processing systems; and (2) IRS lacked the full range of managerial
and technical foundations to realize its modernization objectives.
Some of these key foundation components were a complete cost/benefit
analysis of the overall modernization effort and thorough testing of
individual systems before they were implemented. In September 1996,
we reported on IRS' progress in addressing the managerial and
technical weaknesses we identified in July 1995.\4 We concluded that
although IRS is working to resolve these weaknesses, it had not fully
satisfied any of our recommendations.
SCRIPS was one of the systems that was designed under the conditions
cited in our July 1995 report. It was intended to replace the aging
OCR equipment that IRS had been using to process all of the paper
Federal Tax Deposit (FTD) coupons; almost all of the paper
information returns (e.g, Forms 1099); some of the individual income
tax returns filed on Form 1040EZ; and some employment tax returns
(Form 941). In addition, SCRIPS was expected to process Form 1040PC,
a paper form that taxpayers can generate when they use computer
software to prepare a tax return.
Figure 1 shows the various SCRIPS components. Under the character
recognition and image capture component of SCRIPS, scanners (1) read
information from the document and convert the information to
machine-readable format for later computer processing and (2) create
an image of the document. In the event of recognition errors during
document scanning, IRS staff can access an image of the tax return in
lieu of having to locate the original paper tax return to make
corrections. IRS expected that having an image of the tax return
would improve the productivity of staff doing data validation. In
addition, the images of FTD coupons and certain information return
documents are stored on optical disk for later use. The older OCR
systems used microfilming as the storage medium--essentially a manual
process that required more physical storage space than optical disks.
IRS continues to retain a paper copy of the Form 1040EZ rather than
store the image because IRS has certain legal concerns.\5
Figure 1: SCRIPS Components
(See figure in printed
edition.)
Cont.
(See figure in printed
edition.)
Source: GAO graphic based on IRS data.
IRS expected that SCRIPS would provide faster, more accurate document
processing. Specifically, IRS expected that SCRIPS would result in a
20 percent productivity increase over manual data entry and a 10
percent productivity increase over older OCR equipment. Other
expected benefits included lower costs for system maintenance and
storage of tax return data.
Originally, IRS planned to implement SCRIPS in all 10 service centers
where it currently processes paper returns. However, after the
contract was awarded in February 1993, IRS decided to consolidate
paper tax return processing in five centers. Accordingly, SCRIPS was
tested in Cincinnati in the summer of 1994, and the other four SCRIPS
centers began using SCRIPS between September and November 1994.
--------------------
\3 Tax Systems Modernization: Management and Technical Weaknesses
Must Be Corrected If Modernization Is To Succeed (GAO/AIMD-95-156,
July 26, 1995).
\4 Internal Revenue Service: Business Operations Need Continued
Improvement (GAO/AIMD/GGD-96-152, Sept. 9, 1996).
\5 One concern is the validity of an imaged signature. Another
concern is that SCRIPS images only one side of the two-sided Form
1040EZ. The second page includes a worksheet that taxpayers use for
tax calculations, and IRS considers it part of the tax return.
RESULTS IN BRIEF
------------------------------------------------------------ Letter :2
SCRIPS experienced significant performance problems in 1995. Two
problems were system downtime and slow processing rates. For
example, although SCRIPS was expected to increase productivity by 20
percent over the manual data entry system, SCRIPS actually processed
Forms 1040EZ in 1995 at a rate that was about 7 percent slower than
manual data entry. Because of performance problems, two of the five
centers stopped processing Forms 1040EZ on SCRIPS and reverted to
using manual data entry.
SCRIPS performance deficiencies in 1995 stemmed primarily from both
hardware and software problems. Hardware problems occurred with the
scanner that captures the image of the document. Software problems
occurred with the operating system for the image controller that
reconciles the number of documents scanned to the number of documents
in the SCRIPS database. The hardware and software problems were not
detected before the rollout of SCRIPS because testing in Cincinnati
was incomplete. For example, SCRIPS was never tested at the peak
volumes that IRS experiences during a filing season.
Although IRS had expected that SCRIPS would be processing five
document types--forms 1040EZ, 941, and 1040PC; FTD coupons; and
information returns--IRS postponed plans to process forms 941 and
1040PC on SCRIPS. Of the three remaining document types, the
Cincinnati test certified the software application only for FTD
coupons. Therefore, the software applications for information
returns and Form 1040EZ were not thoroughly tested before they were
put into production.
To improve the performance of SCRIPS for 1996, IRS made hardware and
software modifications, some of which were made before the start of
the 1996 filing season. As a result of some of these enhancements
and more staff familiarity with SCRIPS, according to IRS officials in
all five SCRIPS service centers, SCRIPS performed significantly
better during the 1996 filing season than it did in 1995. For
example, between April and June, unscheduled system downtime
decreased from about 791 hours in 1995 to about 43 hours in 1996.
Also, performance on another indicator---the number of documents
processed per hour--improved for Forms 1040EZ and FTD coupons.\6
Specifically, the number of Forms 1040EZ processed per hour in 1996
increased by four documents per hour (about 6 percent) and the number
of FTD coupons increased by 3 documents per hour (less than 1
percent). However, the number of information returns processed per
hour decreased by five documents per hour (about 3 percent).
In addition to a slower processing rate for information returns,
SCRIPS (1) is not processing all the forms that it was expected to
process in 1996, (2) is expected to cost more than originally
estimated, and (3) is expected to provide lower labor cost savings
than IRS originally anticipated. IRS' October 1994 business case for
SCRIPS stated that in 1996 SCRIPS would be processing all FTD coupons
and information returns, all Forms 1040EZ, 50 percent of the Forms
1040PC, and 93 percent of the Forms 941. However, during the first 9
months of 1996, SCRIPS processed only FTD coupons, 60 percent of the
information returns, and about 50 percent of the Forms 1040EZ. This
reduced level of performance stems in large part from (1) IRS'
decision, after the contract was awarded, to consolidate all paper
processing in 5 service centers instead of 10; and (2) IRS' decision
to purchase only 5 systems when the contractor was planning to
provide 10 systems--1 in each of the 10 service centers.
Furthermore, IRS officials did not know the extent to which the
hardware and software modifications that had already been made for
1996 and those that were planned for later in the year would increase
SCRIPS' capability for processing more 1040EZs and information
returns. Therefore, IRS officials did not significantly increase the
expectations for SCRIPS in 1996 over those that they had in 1995.
The latest cost estimate for SCRIPS is $288 million--considerably
more than previous cost estimates, which, according to IRS'
post-implementation review report on SCRIPS, ranged from $133 million
to $209 million. At least $20 million of that increase is attributed
to maintenance costs that were not included in either the $133
million or $209 million estimates.\7 Also, IRS estimated in October
1994 that SCRIPS would provide about $17 million in labor savings
from fiscal years 1994 through 2000. In September 1995 IRS lowered
that estimate to about $5 million. Furthermore, a September 1996 IRS
investment evaluation of SCRIPS estimated that SCRIPS will yield a
negative return on investment (i.e., SCRIPS' costs will exceed its
benefits) from 1991 to 2001. However, the evaluation noted that the
estimate is based on performance in 1995 and does not reflect the
software and hardware modifications that were made in 1996.
In July 1995, IRS decided to terminate all software programming for
the 1040PC because of SCRIPS' instability, a lack of system capacity,
and a number of software application problems. Also, IRS has decided
not to use SCRIPS to process Forms 941 in fiscal year 1997.
Decisions on using SCRIPS to process Forms 941 beyond fiscal year
1997 have not been finalized. In addition to funding constraints,
according to IRS officials, a decision on using SCRIPS for Forms 941
will depend on (1) the results of a September 1996 capacity test of
SCRIPS, which were not available when we completed our audit work;
and (2) decisions by IRS' Investment Review Board\8 on
recommendations made by a paper processing task team.
--------------------
\6 The number of documents processed per hour, commonly referred to
as the composite rate, includes not only the time to scan the
documents but also the time spent by work station operators
validating and correcting tax return data. Validation and correction
are necessary in those instances when the scanner and secondary
recognition systems cannot read certain tax return data or the system
detects an error made by a taxpayer.
\7 The supporting documentation for IRS' $288 million cost estimate
showed a separate breakout for maintenance costs for only fiscal
years 1995 and 1996. We were unable to determine what, if any,
portion of the costs for fiscal years 1997 to 2000 was attributed to
maintenance. According to IRS officials, maintenance costs may have
been combined with other costs for those fiscal years.
\8 In October 1995, IRS created an executive-level Investment Review
Board, currently co-chaired by the Deputy Commissioner and the Chief
Financial Officer, for selecting, controlling, and evaluating all of
IRS' information technology investments.
OBJECTIVES, SCOPE, AND
METHODOLOGY
------------------------------------------------------------ Letter :3
Our objectives were to (1) determine the primary causes for SCRIPS
performance problems in 1995; (2) assess whether those problems were
corrected as of September 30, 1996; and (3) provide a status report
on IRS' future plans for SCRIPS.
To accomplish these objectives, we did the following:
-- We interviewed National Office officials in the SCRIPS project
office and the taxpayer service function, which has
responsibility for those tax forms processed on SCRIPS, to
obtain their views on the extent to which SCRIPS' performance
improved during fiscal year 1996.
-- We interviewed IRS contracting office officials to determine
what, if any, performance requirements the contractor was being
held to in 1995 and 1996.
-- We interviewed officials at all five SCRIPS service centers to
determine whether performance had improved in 1996 and to
identify the performance indicators officials were using to
evaluate SCRIPS performance.
-- We observed SCRIPS in operation at the service centers in
Memphis and Cincinnati and at the program development site in
Washington, D.C.
-- We interviewed contractor officials about the workload
requirements that IRS had specified for SCRIPS.
-- We reviewed IRS business cases\9 for SCRIPS that documented the
objectives of the system, its expected benefits, and its
estimated cost. IRS prepared business cases for SCRIPS in
February 1992, April 1992, December 1993, October 1994, and
December 1995. To assess SCRIPS, we used the performance
expectations that were included in the October 1994 business
case because it was the one that was in effect at the beginning
of the 1995 filing season.
-- We reviewed IRS evaluations and reports on the implementation of
SCRIPS, including an August 1995 performance evaluation report;
a December 1995 post- implementation review report that assessed
whether business goals were met; a May 1995 Internal Audit
report on SCRIPS testing in Cincinnati; a February 1996 Internal
Audit report on the rollout of SCRIPS; and a September 1996
report on the results of an investment evaluation of SCRIPS.
-- We compared available SCRIPS performance data for January
through September 1995 and 1996.
-- We computed composite rates--the number of documents processed
per hour--for Form 1040EZ, information returns, and FTD coupons.
To compute those rates, we used IRS data on the number of hours
spent for various aspects of tax return processing, including
scanning, data correction, and data validation for January
through September 1995 and 1996. Our composite rate
calculations, similar to those done by IRS shortly after SCRIPS
was implemented, did not include time for document preparation
or another function that is referred to as code and edit.\10
We did our audit work between October 1995 and September 1996 and in
accordance with generally accepted government auditing standards.
Other than identifying obvious reporting errors, we did not verify
the accuracy of IRS' data on the number of hours spent for various
aspects of tax return processing. We did attempt to detemine what
elements were included in IRS' cost estimates for SCRIPS, but we did
not attempt to verify the accuracy of the costs for those elements.
We requested comments on a draft of this report from the Commissioner
of Internal Revenue or her designated representative. Responsible
IRS officials, including the National Director for Submission
Processing, the Assistant Commissioner for Forms and Submission
Processing, and the SCRIPS project manager, provided IRS' comments in
a November 13, 1996, meeting. Their comments on our recommendation
were reiterated in a November 19, 1996, memorandum from the Acting
Chief of Taxpayer Service. IRS' comments are summarized and
evaluated on pages 22 and 23.
--------------------
\9 IRS prepares business cases to justify an information technology
project and to demonstrate that it is cost beneficial.
\10 For SCRIPS, the IRS data we received for computing composite
rates showed the combined hours for both the code and edit function
and the document preparation function. During code and edit, staff
prepare returns for computer entry by, among other things, ensuring
that all data are present and legible. For the manual entry system,
the code and edit function was not combined with the document
preparation function, and the hours associated with document
preparation were not shown. Therefore, we did not include these
aspects of processing in our composite rate comparison.
SCRIPS PERFORMED SUBSTANTIALLY
BELOW EXPECTATIONS IN 1995
------------------------------------------------------------ Letter :4
SCRIPS performed well below expectations in 1995. Extensive,
unscheduled system downtime and slower than expected processing rates
affected SCRIPS' ability to meet the expectations that IRS had
established before the start of the 1995 filing season. Hardware
problems with the scanner contributed to significant amounts of
system downtime; various software problems contributed to slow
processing rates.
SCRIPS PROCESSED LESS VOLUME
AND AT SLOWER RATES THAN
EXPECTED
---------------------------------------------------------- Letter :4.1
In 1995, SCRIPS processed 19 percent more FTD coupons than IRS
expected. However, as shown in table 1, SCRIPS did not meet IRS'
volume expectations for the three other document types scheduled for
processing in 1995--information returns, Form 1040EZ, and Form 941
(IRS did not expect to start processing Form 1040PC on SCRIPS until
1996).
Table 1
Number of Documents SCRIPS Expected to
Process and Number Actually Processed in
1995
Number IRS
expected to Number actually
Document type process in 1995\a processed in 1995
------------------------------ ------------------ ------------------
FTD coupons 76,422,540 90,630,944
Information returns
57,369,000 24,897,860
Form 1040EZ 8,594,500 4,873,419
Form 941 4,826,140 0\b
----------------------------------------------------------------------
\a IRS expectations as of October 1994.
\b Sometime between October and December 1994 IRS decided not to use
SCRIPS to process Forms 941.
Source: IRS data.
SCRIPS processed 19 percent more FTD coupons than IRS had expected
because more FTD coupons were filed than expected, and service center
officials placed the highest priority on processing those forms.
That priority stemmed from (1) IRS procedures that require that 90
percent of all FTD coupons be processed within 24 hours of receipt;
and (2) the absence of a backup processing system, because IRS had
cancelled its maintenance contract for the older OCR equipment that
had been processing FTD coupons. IRS had backup systems for the
other documents that SCRIPS was expected to process in 1995. The
manual data entry system could be used for forms 1040EZ and 941, and
IRS extended the maintenance contract for the older OCR equipment
that was processing information returns.
Not only did SCRIPS not process the number of forms expected, but the
speed with which it did its processing was slower than expected. As
shown in table 2, SCRIPS did not meet the processing rate
expectations for any of the three document types it processed in
1995. Moreover, the actual processing rate for Forms 1040EZ in 1995
was about 7 percent less than the rate achieved in 1995 by manual
data entry.
Table 2
Expected and Actual Processing Rates for
SCRIPS in 1995 Compared to Rates for
Systems SCRIPS Was to Replace
Processing rate (documents per hour) in
1995
-------------------------------------------
Expected for Actual for Actual for
Document type SCRIPS SCRIPS older systems
------------------------- ------------- ------------- -------------
1040EZ 186 63 68
Information returns 178 147 131
FTD coupons 1,170 758 Not
applicable\a
----------------------------------------------------------------------
\a IRS cancelled the maintenance contract for the OCR system that had
been processing FTD coupons. SCRIPS was the only system used to
process FTD coupons in 1995.
Source: GAO computations using IRS data for January through December
1995.
Because of performance problems, two of the five service centers
stopped using SCRIPS to process Forms 1040EZ in 1995.\11 Instead,
they reverted to manual data entry, which required using more staff
resources than planned, thus increasing processing costs. IRS had
planned to use 25.6 staff years to process other-than-full-paid\12
Forms 1040EZ during the 1995 filing season in the five SCRIPS centers
but used 66.5 staff years instead.\13
Hardware problems contributed to SCRIPS' performance deficiencies in
1995. According to IRS' August 1995 performance evaluation report,
the SCRIPS scanner experienced the most hardware failures, which
contributed to a "substantial amount of downtime" at the centers.\14
According to IRS data, between April and June 1995, the five service
centers experienced about 791 hours of unscheduled downtime. In
addition, the scanner jammed when paper was extremely thin, which was
sometimes the case with information returns.
Software problems also occurred. According to IRS officials and
evaluations of SCRIPS, the image controller did not operate fast
enough to keep up with the scanner because the operating system
software was inefficient. The image controller is to track each
scanned document image file to ensure that it moves from the scanner
and ultimately to the server where it is stored. This component
helps reconcile the number of documents scanned to the number of
documents in the database. IRS officials and evaluations of SCRIPS
also indicated that SCRIPS did not provide accurate reports on this
reconciliation--referred to as "run-to-run" balancing. The FTD
coupon run-to-run balancing report, for example, did not provide
total counts to help IRS staff confirm that all of the FTD coupons in
a block that was scanned had in fact been processed. This balancing
is particularly important for FTD coupons because IRS needs to
accurately classify tax deposits to the appropriate Treasury account.
At one center we visited, officials told us that they had to revert
to manual counts for doing run-to-run balancing.
--------------------
\11 According to IRS officials, service center directors had
discretion in managing the processing workload. Some chose to revert
to the manual system; others did not.
\12 Other-than-full-paid tax returns are those returns filed by
taxpayers who either were due a refund or did not pay the full amount
of tax owed at the time of filing. These returns were about 92
percent of the Forms 1040EZ filed at the five SCRIPS service centers
during the 1995 filing season.
\13 The number of staff years for Form 1040EZ includes those used on
SCRIPS and manual data entry.
\14 IRS' evaluation of SCRIPS performance during the 1995 filing
season noted that wide variations in scanner operator skills also
affected the scanner's performance. IRS officials also noted that
even though the scanner may have been "down," other SCRIPS components
could have been operating.
PERFORMANCE PROBLEMS NOT
THOROUGHLY IDENTIFIED BEFORE
ROLLOUT DUE TO INCOMPLETE
TESTING
---------------------------------------------------------- Letter :4.2
Many of the problems experienced with SCRIPS in 1995 might have been
anticipated if IRS had thoroughly tested SCRIPS before installing the
system in the other four service centers. The pilot test of SCRIPS
was incomplete because it (1) did not certify all software
applications that were to be used during 1995; and (2) did not test
SCRIPS' ability to handle peak processing volumes, such as those
experienced in the tax return filing season.
An organization tests a new or modified system to detect system
design and development errors and to correct them before putting a
system into operation. In our July 1995 report on TSM, we said that
although IRS recognized the importance of testing, it had not yet
developed a complete and comprehensive testing plan for TSM.\15 We
said that individual TSM systems were developing their own test
plans, which IRS described as rudimentary and inadequate. This
testing environment was in effect when IRS did a pilot test of SCRIPS
at the Cincinnati Service Center in the summer of 1994.
The purpose of a pilot test is to evaluate the performance of a
system in one location before deciding whether to implement the
system at other locations. IRS uses the pilot test to certify that
the system is meeting its program or business objectives. During the
pilot test, IRS is to collect data on the performance of the system
and compare the data against established performance goals to certify
that the system is performing as expected.
Due to delays in receiving and testing software, only the FTD coupon
application was certified as a result of the pilot test. The
certifications for Forms 1040EZ and information returns were not done
due to management concerns that conducting these certifications would
delay the rollout of SCRIPS to five centers, which was scheduled for
January 1995. As a result of incomplete testing, for example, IRS
did not identify problems with the information returns software until
SCRIPS was fully operational. Due to these software problems, a
higher percentage of returns were sent to data validation than
originally anticipated, thus decreasing system productivity.
According to IRS officials, those problems were corrected before the
1996 filing season.
The pilot test was also incomplete because it did not (1) provide IRS
a clear indication of SCRIPS' ability to perform under peak workload
conditions and (2) show the impact of running multiple software
applications. If the system had been tested at larger volumes,
running multiple software applications, problems may have been
identified earlier and resources could have been diverted to address
performance problems before the system's rollout. According to an
Internal Audit report on the pilot test, the contract required that
SCRIPS process 200,000 documents a day during peak periods. During
the 1-day volume test in the pilot, only 45,000 documents were
scanned into the database. The remaining components of the system
processed an additional 142,000 documents that had been scanned
before the test. Thus, the scanner, one of the primary sources of
downtime during 1995, was not tested under levels IRS experiences in
a production environment. Also, only 10,000 documents were processed
completely--only 5 percent of the daily production workload specified
in the contract.
According to IRS officials, a full production test plan had been
developed in December 1993 that would simulate peak filing season
volumes. However, the plan was never implemented. One reason IRS
officials provided for not implementing the test plan was that IRS
never had full access to a SCRIPS system to do the test. The system
that IRS was to use for the test was the same system that the
contractor was using for systems development work. Another factor
that IRS officials cited was a sense of urgency to roll out SCRIPS
for FTD coupons because IRS had no backup processing system.\16
IRS' December 1995 report on the post-implementation review of SCRIPS
cited several problems with the testing of SCRIPS in addition to
those that occurred in the pilot test. For example, although an
equipment acceptance test was done, the test was not adequate to
validate the system's readiness for production. The report also
stated that IRS waived an "operational capabilities demonstration"
that would have included (1) tests for the readability of forms and
(2) measures of the number of documents scanned per hour. The report
said that such a test, done as part of the contract award process,
may have shown that the system could not meet the minimum performance
requirements.
The post-implementation review team proposed that tests be done to
verify that the system could handle the production workload. It also
stated that if such a test could not be done, a simulated test should
be done focusing on the performance of key areas of the system,
particularly those that could lead to processing bottlenecks. It
also proposed a 1-year pilot test for forms that may be added to
SCRIPS in the future and proposed that the pilot test include the
peak volumes of a filing season.
--------------------
\15 GAO/AIMD-95-156.
\16 As previously stated, IRS had cancelled the maintenance contract
for the OCR system that IRS was using to process FTD coupons. In
addition, IRS had redistributed the workload of paper FTD coupons in
1995--consolidating the work of 10 service centers into 4 service
centers.
PERFORMANCE IMPROVED FOR 1996,
BUT SCRIPS PERFORMED BELOW
ORIGINAL EXPECTATIONS AND COSTS
MAY BE HIGHER THAN EXPECTED
------------------------------------------------------------ Letter :5
Officials in the five service centers that used SCRIPS in 1996 said
that it is performing significantly better than it did in 1995. IRS
officials told us that the primary performance expectation for SCRIPS
in 1996 was system stabilization. One of the primary indicators
these officials used in evaluating stabilization was the amount of
downtime. According to IRS service center officials and available
IRS data, downtime decreased substantially in 1996. This reduction
enabled SCRIPS to process more Forms 1040EZ and information returns
in 1996 while continuing to process all FTD coupons. In addition,
the processing rates for SCRIPS (i.e., the number of documents
processed per hour) improved for two document types--Form 1040EZ and
FTD coupons.
Despite these improvements, the system, as of September 30, 1996, (1)
was not processing all the forms that the October 1994 business case
said it would process, (2) was expected to cost more than original
estimates, and (3) was not expected to provide the estimated labor
savings that were cited in the October 1994 business case.
SCRIPS PERFORMANCE IMPROVED
IN 1996
---------------------------------------------------------- Letter :5.1
According to service center officials, SCRIPS performed
"significantly better" in 1996 than it did in 1995 because the system
experienced significantly less downtime than in 1995. IRS did not
begin tracking downtime in 1995 until April. Comparable data for
April through June 1995 and 1996 show that unscheduled downtime did
decrease significantly--from about 791 hours to 43 hours.
As shown in table 3, mostly because of less unscheduled system
downtime, SCRIPS processed many more documents during the first 9
months of 1996 than it did during the first 9 months of 1995.
Table 3
Number of Documents SCRIPS Processed in
1995 and 1996 as of September 30
Number processed Number processed
as of September as of September
Document type 30, 1995 30, 1996
------------------------------ ------------------ ------------------
Form 1040EZ 4,870,493 8,309,384
Information returns 24,631,809 22,618,452
FTD coupons 67,247,198 70,857,389
======================================================================
Total 96,749,500 101,785,225
----------------------------------------------------------------------
Source: IRS data.
Also, according to IRS officials, new software was installed in
November 1995 to correct the run-to-run balancing problems that we
discussed earlier. According to these officials, since that software
was installed, SCRIPS centers have not reported problems with
tracking the number of documents scanned and processed.
Also in 1996, as shown in table 4, processing rates increased for
Forms 1040EZ and FTD coupons.
Table 4
Actual Processing Rates by Document Type
as of September 30, 1995, and 1996
Processing rate (documents per hour)
-------------------------------------------
SCRIPS actual SCRIPS actual Actual for
rate as of rate as of older systems
Sept. 30, Sept. 30, as of Sept.
Form type 1995 1996 30, 1996
------------------------- ------------- ------------- -------------
Form 1040EZ 63 67 60
Information returns 150 145 133
FTD coupons 719 722 Not
applicable\a
----------------------------------------------------------------------
\a IRS cancelled the maintenance contract for the OCR system that had
been processing FTD coupons. SCRIPS was the only system used to
process paper FTD coupons in 1996.
Source: GAO computations based on IRS data.
SCRIPS PROCESSED MANY FEWER
DOCUMENTS THAN ORIGINALLY
EXPECTED IN 1996 WHILE ITS
ESTIMATED COSTS HAVE
INCREASED
---------------------------------------------------------- Letter :5.2
Despite the improved performance in 1996, SCRIPS is still doing much
less than expected. In addition, estimated costs have increased and
estimated labor cost savings have decreased.
SCRIPS PROCESSED FEWER
DOCUMENTS THAN EXPECTED
IN 1996
-------------------------------------------------------- Letter :5.2.1
The October 1994 business case stated that in 1996 SCRIPS would be
processing (1) all Forms 1040EZ, (2) all FTD coupons, (3) all
information returns, (4) 93 percent of the Forms 941, and (5) 50
percent of the Forms 1040PC. During the first 9 months of 1996, as
in 1995, SCRIPS processed all the FTD coupons IRS had received but no
forms 1040PC or 941. Also, although SCRIPS processed more Forms
1040EZ and information returns during the first 9 months of 1996
compared with the first 9 months of 1995, it still processed only
about 50 percent of the Forms 1040EZ and about 60 percent of paper
information returns. As discussed in the next few paragraphs, this
reduced level of performance, compared to the October 1994 business
case, stems in large part from a major change in IRS' plans after the
SCRIPS contract was awarded. Furthermore, IRS officials did not know
the extent to which hardware and software modifications that had been
made for 1996 and those that were planned for later in the year would
affect SCRIPS' ability to process more Forms 1040EZ and information
returns. Therefore, IRS officials did not significantly increase the
expectations for SCRIPS in 1996 over those that they had in 1995.
As specified in the contract, SCRIPS was designed on the assumption
that it would be installed in each of the 10 service centers that
were then processing paper returns. However, in December 1993--10
months after the contract was awarded--IRS announced plans to
consolidate the processing of paper tax returns in five centers. As
a result, although the total workload for SCRIPS remained the same,
the volume to be processed by any one of the five SCRIPS service
centers on the average doubled. Thus, the system that the contractor
had designed would not meet the workload requirements without further
systems development work.
In May 1994, IRS attempted to revise the original contract to meet
the volume requirements at five service centers. According to IRS'
post-implementation review report, a statement of work was written to
revise the contract requirements to accommodate the five service
center scenario. In December 1994, at IRS' request, the contractor
proposed changes totaling about $21 million. According to contractor
officials, that proposal represented an interim attempt to meet the
new workload requirements under a five service center scenario.
According to IRS officials, IRS could not afford all of the proposed
modifications. Also, IRS officials did not believe that all the
proposed changes were needed. As a result, negotiations on these
proposals were never completed. Thus, IRS decided to purchase five
systems, one for each of the five SCRIPS service centers. To
compensate for having fewer systems than intended, IRS decided to (1)
continue paper processing of Forms 1040EZ in the other five service
centers and (2) extend the maintenance contract for the OCR equipment
that was being used to process information returns.
After the 1995 filing season, IRS issued a statement of work for
system enhancements that would help stabilize SCRIPS performance in
1996. According to IRS officials, they purchased those enhancements
that they believed offered the greatest potential for improving
SCRIPS' performance in fiscal year 1996. For example, IRS purchased
a third scanner for each service center that could (1) be used as a
backup if one of the two primary scanners failed and (2) provide the
ability to scan documents ahead and have them wait in queue for
further processing. Many of these enhancements were implemented late
in the 1996 filing season, after most of the Forms 1040EZ and
information returns had been processed.
According to IRS' Office of Assistant Chief Counsel, IRS cannot hold
the contractor to any specific performance requirements for the
number of documents that SCRIPS must process within a specific time
period, (e.g., per week, per hour). When IRS modified the contract
to reflect that SCRIPS would be put in 5 centers instead of 10,
according to the Assistant Chief Counsel's Office, it did not clearly
establish throughput requirements--the number of documents to be
scanned per hour. Thus, despite paying for enhancements for 1996,
IRS has determined that it cannot currently hold the contractor to
any specific performance requirements. In February 1996, IRS
Internal Audit recommended that IRS finalize throughput requirements
for SCRIPS and do a test to determine whether the contractor is
meeting the requirements. IRS officials told us that they are
examining options for incorporating throughput requirements into the
contract.
IRS officials said that they would have a better foundation for
establishing throughput requirements once IRS evaluates the impact of
the enhancements on SCRIPS' capacity. According to contractor
officials, given the enhancements made in 1996, they believe SCRIPS
is capable of processing more documents than it has. They pointed
out that SCRIPS' capability to process documents depends not only on
the system's design but also on IRS' human resource decisions, such
as the number of work stations that are staffed and employee
incentives (or lack thereof).
IRS officials tested SCRIPS in late September 1996. According to the
test plan, the purpose of the test was to determine (1) the maximum
number of FTD coupons, Forms 1040EZ, and information returns that
SCRIPS can process; (2) the amount of free time, if any, that will be
available to process Forms 941; (3) any bottlenecks in the SCRIPS
system that can be eliminated to increase system throughput; and (4)
the performance thresholds that could be put into the SCRIPS
contract. However, because the software application for Form 941 was
not complete, it would have been difficult to fully assess SCRIPS'
processing capability. The results of the test were not available
when we completed our audit work.
EXPECTED COSTS HAVE
INCREASED AND ANTICIPATED
LABOR COST SAVINGS HAVE
DECREASED
-------------------------------------------------------- Letter :5.2.2
Although the workload being processed on SCRIPS continues to be less
than expected, SCRIPS' estimated costs have risen. Also, anticipated
labor cost savings have decreased.
According to IRS' post-implementation review report, previous cost
estimates for SCRIPS have ranged from $133 million to $209 million.
The estimate of $133 million was made in February 1992 and again in
April 1992. That cost estimate, however, assumed that SCRIPS would
be implemented in 1994 and did not include the cost of maintaining
SCRIPS. The current life-cycle cost estimate for SCRIPS is $288
million, which includes at least $20 million for maintenance.\17 That
estimate was included in the Department of the Treasury's May 6,
1996, report to Congress on IRS' progress in responding to our
recommendations on the managerial and technical weaknesses of TSM.\18
We could not determine how much IRS has already spent on SCRIPS since
its inception because IRS does not have an accurate cost accounting
system. Using the latest life-cycle cost estimate, SCRIPS is
estimated to have cost about $145 million from fiscal year 1989
through fiscal year 1996.
In October 1994, IRS estimated that SCRIPS would provide about $17
million in labor cost savings from fiscal years 1994 through fiscal
year 2000. In September 1995, IRS lowered that estimate to about $5
million. Also, IRS' September 1996 investment evaluation report on
SCRIPS concluded that the system will yield a negative return on
investment (i.e., costs will exceed benefits) from 1991 to 2001.
However, the evaluation report stated that its return on investment
estimate does not fully capture the operational benefits of the
hardware and software enhancements that were made since the end of
fiscal year 1995. Therefore, the report concludes that the final
judgment on SCRIPS' performance cannot be made until after the 1997
filing season.
--------------------
\17 See footnote 7.
\18 Report to House and Senate Appropriations Committees: Progress
Report on IRS' Management and Implementation of Tax Systems
Modernization, May 6, 1996.
USE OF SCRIPS FOR FORM 941 IS
UNCERTAIN
------------------------------------------------------------ Letter :6
As discussed previously, SCRIPS was not used to process forms 1040PC
and 941 in 1995 or 1996. In July 1995, IRS decided to terminate all
software programming for the 1040PC because of SCRIPS' instability, a
lack of system capacity, and a number of software application
problems. It is uncertain when, if at all, SCRIPS will be used to
process Forms 941.
The President's fiscal year 1997 budget request included $850 million
for TSM, about $38 million of which was for SCRIPS. In a June 6,
1996, letter, Treasury submitted a revised TSM funding request of
$664 million to the House Appropriations Committee, of which about
$30 million was for SCRIPS. According to the letter, at this funding
level, SCRIPS would not be used to process Forms 941 for fiscal year
1997. Congress subsequently appropriated $336 million for TSM for
fiscal year 1997. About 60 percent of that appropriation is
earmarked for operational TSM projects, such as SCRIPS, but it was
unclear when we prepared this report how much IRS would allocate to
SCRIPS. However, we would expect that IRS will continue funding
SCRIPS because the project is operational and IRS has no backup
system for FTD coupons.
In addition to funding constraints, decisions on the use of SCRIPS
beyond fiscal year 1997 for Forms 941, according to IRS officials,
will hinge on: (1) the results of the September 1996 test of SCRIPS,
which we discussed earlier; and (2) Investment Review Board actions
on recommendations made by a paper processing task team.
In March 1996, IRS convened a task team to develop a paper processing
strategy due to delays in implementing various aspects of IRS' tax
return processing vision. Specifically, IRS had originally expected
that (1) a significant number of returns would be received
electronically, thereby reducing the need for some manual data entry;
and (2) DPS would be positioned to begin processing the remaining
paper tax returns. As we reported in October 1995, IRS' electronic
filing program is falling short of expectations.\19 Current estimates
indicate that IRS may receive only 33 million electronic returns in
2001 rather than the 80 million that IRS had set as its goal. The
shortfall in meeting IRS' 80-million goal stems from the lack of a
comprehensive business strategy to attract taxpayers to electronic
filing. IRS is currently trying to develop such a strategy. Also,
IRS announced on October 8, 1996, that it was terminating the DPS
project because of "revised priorities and budget realities for the
next several years."
Among other things, the task team developed a paper processing
strategy based on the assumption that the maximum number of
electronic returns that can be expected in 2001 is about 33 million.
In addition, the team evaluated the need for a replacement system for
IRS' current manual data entry system. According to IRS officials,
one part of the paper processing strategy may be to contract out the
processing of some documents. These documents could include
information returns because the processing of those documents is not
as time-sensitive as the processing of income tax returns. In
addition, in the future, more FTD coupons are to be submitted
electronically as called for in the North American Free Trade
Agreement. For example, IRS estimates that about 44 percent of the
FTD coupons will be received electronically in fiscal year 1998,
compared with only 1 percent in fiscal year 1996. Thus, given that
some of SCRIPS' existing workload could decrease or be contracted
out, IRS could possibly increase the number of Forms 1040EZ that
SCRIPS could process or add additional forms, such as Forms 941,
without requiring any additional system capacity beyond what is
currently available.
--------------------
\19 Tax Administration: Electronic Filing Falling Short of
Expectations (GAO/GGD-96-12, Oct. 31, 1995).
CONCLUSIONS
------------------------------------------------------------ Letter :7
Because SCRIPS was developed before IRS started taking actions to
address the managerial and technical weaknesses of TSM that we
identified in July 1995, SCRIPS suffered from some of those
weaknesses that we said would contribute to such things as cost
overruns and failure to meet mission goals. SCRIPS is expected to
cost more than originally expected and, according to IRS' September
1995 estimates, could provide less than one-third of the originally
expected labor cost savings. We recognize that IRS is now completely
reliant on SCRIPS for processing paper FTD coupons. However, the
decrease in expected labor cost savings and the increase in estimated
costs raise questions about the cost-effectiveness of SCRIPS.
In addition, one of the most critical weaknesses for SCRIPS was a
lack of thorough and complete system testing before the system was
rolled out to five service centers. Although IRS tested SCRIPS in
September 1996 to determine the maximum number of Forms 1040EZ,
information returns, and FTD coupons SCRIPS can process, that test
does not substitute for a test of SCRIPS' ability to process any new
document types along with the existing ones under a production
environment that replicates peak volume conditions.
RECOMMENDATION TO THE
COMMISSIONER OF INTERNAL
REVENUE
------------------------------------------------------------ Letter :8
We recommend that before deciding to increase the percentage of Forms
1040EZ or information returns that SCRIPS processes or using SCRIPS
to process other tax forms, such as Forms 941, the Commissioner (1)
do a cost-benefit analysis that includes examining the costs and
benefits of alternative ways for processing those forms, such as
those developed by the paper processing task team; and (2) if the
analysis shows that it is cost-effective to have SCRIPS process Forms
941, ensure that IRS tests SCRIPS' ability to process the existing
software applications along with any new software applications that
may be added using peak volumes or volumes simulating peak workload
conditions to ensure that SCRIPS can meet performance expectations.
AGENCY COMMENTS AND OUR
EVALUATION
------------------------------------------------------------ Letter :9
We requested comments on a draft of this report from the Commissioner
of Internal Revenue or her designated representative. Responsible
IRS officials, including the National Director for Submission
Processing, the Assistant Commissioner for Forms and Submission
Processing, and the SCRIPS project manager, provided IRS' comments in
a November 14, 1996, meeting. Their comments on our recommendation
were reiterated in a November 19, 1996, memorandum from the Acting
Chief of Taxpayer Service. Besides commenting on our recommendation,
the Deputy Chief Information Officer, Systems Development, provided
several factual clarifications that we incorporated in the report
where appropriate. The officials also asked that we update our
report to reflect processing rate information through September 30,
1996, and they gave us the relevant data. We made that update.
IRS officials generally agreed with our recommendation. With respect
to the first part of our recommendation, the officials generally
agreed that IRS should do a cost-benefit analysis before deciding to
increase the percentage of Forms 1040EZ or information returns that
SCRIPS processes or to add additional forms, such as Forms 941. They
said that this analysis would be done by the end of fiscal year 1997
and would take into account approved results from the paper
processing task team. IRS officials said that in the event of
workload imbalances during the 1997 tax return filing season, if
necessary, they may decide to increase the workload for SCRIPS before
the cost/benefit analysis is completed. We recognize that IRS'
priority must be to process tax returns during the filing season in a
timely manner, and it needs to reserve the right to do so.
With respect to the second part of our recommendation, IRS officials
said that they plan to work with the contractor to identify and
implement any necessary contract modifications that may be needed to
ensure complete testing. They said the testing will ensure that
SCRIPS can meet performance expectations in a peak production
environment.
IRS officials said that a few changes are planned for the 1997 filing
season that could help improve SCRIPS' future performance.
Specifically, they mentioned that IRS would be testing different
incentive systems for SCRIPS operators to determine the extent to
which incentives affect operator performance, which could also affect
overall SCRIPS' performance. They also mentioned that IRS is
negotiating with the contractor to provide a new scanner feeder that
should resolve some of the problems experienced when information
returns are filed on extremely thin paper.
IRS officials expressed concern about using $133 million as the
baseline cost estimate for SCRIPS. That estimate was included in a
February 1992 business case and again in a revision dated April 23,
1992. IRS officials said that we should have used an earlier cost
estimate of $209 million that was included in IRS' Information
Systems Initiative Summary Database in August 1991. IRS'
post-implementation review report on SCRIPS also stated that the
SCRIPS project office considered the $209 million as the baseline
life-cycle cost estimate. However, the report also noted that
between fiscal years 1993 and 1996, IRS cited four other different
life-cycle cost estimates for budget purposes, including an estimate
of $132 million. We revised the report to acknowledge that $209
million was one of the cost estimates for SCRIPS, but we believe that
the business case estimate is an appropriate baseline because the
business case was a major basis for decisions to go forward with
SCRIPS. IRS officials said that IRS developed the SCRIPS business
case before it began taking steps to manage information technology
projects as investments. Since that time, for example, IRS has
developed an investment justification handbook to help ensure that
project cost and benefit analyses that are included in business cases
are standardized and complete.
---------------------------------------------------------- Letter :9.1
We are sending copies of this report to the Subcommittee's Ranking
Minority Member, the Chairman and Ranking Minority Member of the
House Committee on Ways and Means, the Chairman and Ranking Minority
Member of the Senate Committee on Finance, various other
congressional committees, the Secretary of the Treasury, the
Commissioner of Internal Revenue, the Director of the Office of
Management and Budget, and other interested parties.
Major contributors to this report are listed in the appendix. Please
contact me on (202) 512-9110 if you have any questions.
Sincerely yours,
Lynda D. Willis
Director, Tax Policy and
Administration Issues
MAJOR CONTRIBUTORS TO THIS REPORT
=========================================================== Appendix I
GENERAL GOVERNMENT DIVISION,
WASHINGTON, D.C.
--------------------------------------------------------- Appendix I:1
David Attianese, Assistant Director, Tax Policy and Administration
Issues Sherrie Russ, Evaluator-in-Charge
Christopher Hess, Evaluator
Monika Gomez, Evaluator
KANSAS CITY FIELD OFFICE
--------------------------------------------------------- Appendix I:2
Cecelia Ball, Senior Evaluator
Marvin McGill, Evaluator
*** End of document. ***