    Tax Administration: Planning for IRS's Enforcement Process
    Changes Included Many Key Steps but Can Be Improved (20-JAN-04,
    GAO-04-287).

    In recent years, the Internal Revenue Service (IRS) has
    experienced declines in most of its enforcement programs,
    including declines in audits and in efforts to collect delinquent
    taxes. Increasing enforcement productivity is one strategy that
    can help reverse these declines. To this end, IRS is currently
    planning and has begun implementing enforcement process
    improvement projects. GAO was asked to assess the extent to which
    the planning for the projects followed steps consistent with both
    published GAO guidance and the experiences of private sector and
    government organizations. Specifically, GAO assessed the extent
    to which four judgmentally selected projects followed the 20
    planning steps.
    -------------------------Indexing Terms-------------------------
    REPORTNUM:   GAO-04-287
        ACCNO:   A09139
      TITLE:     Tax Administration: Planning for IRS's Enforcement
    Process Changes Included Many Key Steps but Can Be Improved
         DATE:   01/20/2004
      SUBJECT:   Audits
    	     Delinquent taxes
    	     Taxpayers
    	     Internal controls
    	     Tax administration
    	     Strategic planning
    	     Performance measures

    ******************************************************************
    ** This file contains an ASCII representation of the text of a  **
    ** GAO Product.                                                 **
    **                                                              **
    ** No attempt has been made to display graphic images, although **
    ** figure captions are reproduced.  Tables are included, but    **
    ** may not resemble those in the printed version.               **
    **                                                              **
    ** Please see the PDF (Portable Document Format) file, when     **
    ** available, for a complete electronic file of the printed     **
    ** document's contents.                                         **
    **                                                              **
    ******************************************************************
    GAO-04-287

    United States General Accounting Office

    GAO

    Report to the Chairman and Ranking

                  Minority Member, Committee on Finance, U.S. Senate

    January 2004

    TAX ADMINISTRATION

    Planning for IRS's Enforcement Process Changes Included Many Key Steps but Can Be Improved

                                           a

    GAO-04-287

    January 2004

    TAX ADMINISTRATION

    Planning for IRS's Enforcement Process Changes Included Many Key Steps but Can
    Be Improved

    Planning for the four enforcement process improvement projects GAO
    reviewed included most of the 20-step framework developed to assess the
    projects. This increases the likelihood that projects target the right
    processes for improvement, choose the best target process from among
    alternatives, effectively implement the project, accurately assess project
    outcomes, and properly manage the change to the new process. However, none
    of the projects completed all of the steps. For example, some projects did
    not fully identify the causes of productivity shortfalls, leaving a risk
    that the project did not fix the right problem. In the course of this
    work, GAO found that IRS managers do not have guidance about the steps to
    follow in planning process improvement projects, increasing the
    possibility of omitting steps.

    A recurring issue in the four projects was that IRS's enforcement data
    only partially adjust for the complexity and quality of cases worked. This
    issue is also a problem for IRS enforcement productivity data generally.
    Failing to adjust for both complexity and quality increases the risk that
    trends in productivity will be misunderstood. For example, a decline in
    the number of cases closed per employee at the same time that case
    complexity is increasing may not be a real decline in productivity. GAO
    recognizes that some options for improving productivity data could be
    costly. However, costs could be mitigated by using existing statistical
    methods and IRS complexity and quality data.

    Highlights of GAO-04-287, a report to the Committee on Finance, U.S.
    Senate

    In recent years, the Internal Revenue Service (IRS) has experienced
    declines in most of its enforcement programs, including declines in audits
    and in efforts to collect delinquent taxes. Increasing enforcement
    productivity is one strategy that can help reverse these declines. To this
    end, IRS is currently planning and has begun implementing enforcement
    process improvement projects.

    GAO was asked to assess the extent to which the planning for the projects
    followed steps consistent with both published GAO guidance and the
    experiences of private sector and government organizations. Specifically,
    GAO assessed the extent to which four judgmentally selected projects
    followed the 20 planning steps summarized in the figure.

    GAO recommends that the Commissioner of Internal Revenue take actions to
    (1) put in place a framework to guide the planning for process improvement
    projects and (2) invest in better enforcement program productivity data,
    recognizing the costs and benefits of doing so.

    www.gao.gov/cgi-bin/getrpt?GAO-04-287.

    To view the full product, including the scope and methodology, click on
    the link above. For more information, contact James White at (202)
    512-9110 or whitej@gao.gov.

                         20-Step Process Improvement Framework

    Contents

      Letter                                                                   1
                                     Results in Brief                          5
                                        Background                             6
              SB/SE Process Improvement Projects Included Most Key Steps but
                        Productivity Measurement Could Be Improved             9
                                       Conclusions                            18
                           Recommendations for Executive Action               18
                            Agency Comments and Our Evaluation                19

    Appendixes
                     Appendix I:    Process Improvement Project Framework     21
                                              Decision to Change              22
                                         Target Process Development           24
                                               Implementation                 26
                                             Outcome Assessment               27
                                              Change Management               27
                    Appendix II:            Scope and Methodology             30
                                    GAO Assessment of Four Selected SB/SE
                   Appendix III:                   Process
                                             Improvement Projects             32
                   Appendix IV:   Descriptions of SB/SE Process Improvement   37
                                                   Projects
                     Appendix V:  Comments from the Internal Revenue Service  41

    Table Table 1:
       	Major Process Improvement Projects in IRS's Small Business/Self Employed Operating Division

    Figures 	Figure 1: Figure 2:

    Figure 3: Figure 4:

    Figure 5:

    Figure 6:

    Figure 7:

    20Step Process Improvement Framework 3
    Key Steps Included in Selected SB/SE Process
    Improvement Projects 10
    20Step Process Improvement Framework 21
    Key Steps Included in the Field Examination
    Reengineering Project 33
    Key Steps Included in the Compliance Support Case
    Processing Redesign Project 34
    Key Steps Included in the Collection Taxpayer Delinquent
    Account Support Project 35
    Key Steps Included in the Collection Field Function
    Consultative Initiative Project 36

    Contents

    Abbreviations

    CQMS Collection Quality Measurement System
    EQMS Examination Quality Measurement System
    GAO General Accounting Office
    IRS Internal Revenue Service
    SB/SE Small Business/Self Employed Division

    This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.

    A

    United States General Accounting Office

    Washington, D.C. 20548

    January 20, 2004

    The Honorable Charles E. Grassley
    Chairman
    The Honorable Max Baucus
    Ranking Minority Member
    Committee on Finance
    United States Senate

    In recent years, as we have reported, the Internal Revenue Service (IRS)
    has experienced declines in most of its enforcement programs, including
    steep declines in audits and broad declines in its efforts to collect
    delinquent taxes.1 Factors we have cited as contributing to these declines
    include the growth in tax returns filed each year, reallocation of
    enforcement resources to improve telephone and other services to
    taxpayers, the addition of new taxpayer rights and protections by the IRS
    Restructuring and Reform Act of 1998,2 decreased enforcement staffing,
    and decreased enforcement productivity as measured by cases closed per
    staff time.3 The declines have triggered concerns that taxpayers'
    motivation to voluntarily comply with their tax obligations could be
    undermined.

    Increasing enforcement productivity is one strategy that could help reverse
    these declines. To this end, IRS's Small Business/Self Employed (SB/SE)
    operating division is currently planning and has begun implementing 15
    enforcement process improvement projects. These projects aim to make
    incremental improvements to enforcement processes and are distinct from
    the major changes expected to result from IRS's longterm business
    systems modernization effort. Because of your concern about the declines
    in IRS's enforcement programs, including the reported declines in their
    productivity, you asked us to examine IRS's planning of its enforcement
    process improvement projects. Specifically, as agreed with your offices,

    1U.S. General Accounting Office, Compliance and Collection: Challenges for
    IRS in Reversing Trends and Implementing New
    Initiatives, GAO03732T (Washington, D.C.: May 7, 2003) and U.S. General Accounting Office, IRS
    Modernization: Continued Progress Necessary for Improving Service to
    Taxpayers and Ensuring
    Compliance, GAO03796T (Washington, D.C.: May 20, 2003).

    2P.L. 105206.

    3U.S. General Accounting Office, Tax Administration: Impact of Compliance
    and Collection Program Declines on
    Taxpayers, GAO02674 (Washington, D.C.: May 22, 2002).

    our objective was to assess the extent to which SB/SE's planning followed steps consistent with both GAO guidance and the experiences of private sector and government organizations. Our focus was on SB/SE's planning and development of their projects; we did not evaluate the results of any projects because only 1 of the 15 projects had started implementation in August 2003, when we were choosing projects to review, and that project was too new to be evaluated.

    To assess SB/SE's planning, we needed to identify criteriathe steps to follow in planning process improvement projects. We developed a 20step framework based on both GAO's Business
    Process Reengineering Assessment
    Guide and discussions with private sector and other government managers with experience planning process improvement projects.4 GAO's Guide recognizes that the steps for planning process improvement need to be adapted for the magnitude of the projects and the particular circumstances of an organization. To do this, we first held a roundtable meeting with the Private Sector Council and two of its member companies, Northrop Grumman and CNF.5 We also discussed process improvement planning with managers from the tax departments of California, Florida, and Minnesota; and the Minerals Management Service in the U.S. Department of the Interior. We then used GAO's guidance and the experiences of the above organizations to develop 20 key planning steps appropriate for SB/SE's incremental improvement projects. The 20 key steps, organized by project stages, are summarized in figure 1.

    4U.S. General Accounting Office, Business Process Reengineering Assessment
    Guide, GAO/AIMD10.1.15 (Washington, D.C.: April 1997).

    5The Private Sector Council is a nonprofit, nonpartisan public service organization created to help the federal government improve its efficiency, management, and productivity through cooperative sharing of knowledge between the public and private sectors. It is comprised of member companies-businesses from across North America in industries such as telecommunications, defense, finance, and energy. Corporate executives from member companies provide their time and expertise at no cost to the government.

    Figure 1: 20-Step Process Improvement Framework

    The first six steps are related to an organization's decision to change a process. These steps help an organization to understand the extent and causes of any weaknesses that need to be addressed. For example, in step 1, an organization should begin investigating change by using productivity data to define a baseline. Productivity measures generally take the form of a ratio of outputs to inputs. The remaining steps in this stage refine the organization's understanding of current performance and where the organization wants to be.

    The next six steps help an organization develop the new, or target, process. These steps emphasize understanding and analyzing alternatives and planning for implementation.

    The next four steps concern an organization's implementation of the new process. These steps help an organization through the most difficult phase of process improvement, where ideas about the new process are turned into actions. Two of these steps are related to pilot testing. Pilot testing

    allows corrective actions to be taken if needed to correct problems prior to full implementation. The other two steps address employee responsibilities.

    In the next stage, an organization makes plans to assess the outcome of the new process. The single step in this stage ensures that an organization makes plans early for evaluating the success the success of the new process. It is important to develop assessment plans prior to full project implementation in order to ensure that the data necessary for evaluation are collected.

    The last three steps are related to how an organization manages the change to the new process. Successfully managing change reduces the risk that improvement efforts will fail due to the natural resistance to change within an organization.

    Our 20step framework emphasizes productivity measurement and analysis both because the SB/SE projects focus on productivity gains and because the managers we consulted, particularly our roundtable participants, said that meaningful productivity data are an important foundation for design and implementation of process improvement efforts.

    We recognized in our review that there is some judgment involved in defining these steps and that some steps may not be appropriate on every project. GAO's Business
    Process Reengineering Assessment
    Guide notes that the guide only provides a general framework for assessing key issues. It says that it should not be regarded as a specific, rigid set of steps for conducting a project because reengineering is far too situational for such a rigid approach.6 The same caveat applies to the process improvement 20step framework we developed for this review. Appendix I provides an expanded discussion of each planning step we identified as appropriate for SB/SE's projects.

    We judgmentally selected four projects to study in detail, including at least one project in each of the three main enforcement areas that SB/SE was revamping-audits (or examinations), collection, and compliance support. We also looked for projects that were sufficiently far along in their planning for us to expect to see either completed steps or plans for the remaining steps.

    6GAO/AIMD10.1.15.

    For the four projects we selected, we assessed the degree to which IRS followed our 20step framework by first interviewing officials and examining the extensive documentation they provided. We then returned to officials responsible for each project and asked for additional information, particularly in areas where our initial assessment was that key steps were not taken. Where IRS officials showed us that certain steps had been addressed, we then revised our initial assessment. We also recognized the need for flexibility in the application of our criteria, in that some of the steps we identified may not necessarily be appropriate for every project. For instance, one project determined that completing a pilot was not necessary; accordingly they could not adjust the process based on the pilot, so that step was listed "not applicable" in our assessment. We describe the development of our framework and our assessment of selected SB/SE projects in detail in appendix II.

    Results in Brief
       	Planning for the four projects we reviewed included most of the 20step framework we developed to assess SB/SE's planning of enforcement process improvement projects. By including so many key planning steps in its projects, SB/SE increased the likelihood that project teams target the right processes for improvement, choose the best target process from among alternatives, effectively implement the project, accurately assess project outcomes, and properly manage the change to the new process. However, none of the projects completed all of the steps. For example, some projects did not fully identify the causes of productivity shortfalls, leaving a risk that the projects would not fix the right problem. In the course of this work, we found that SB/SE managers do not have guidance about the steps to follow in planning process improvement projects, increasing the possibility of omitting steps. GAO's Business
    Process Reengineering Assessment
    Guide notes that a framework, such as the one in this report, could help ensure that key steps are followed.7

    A recurring issue in the four projects we examined in detail was that SB/SE's enforcement data only partially adjust for the complexity and quality of cases worked. This issue is also a problem for SB/SE enforcement productivity data generally. Failing to adjust for both complexity and quality increases the risk that trends in productivity will be misunderstood. For example, a decline in the number of cases closed per

    7GAO/AIMD10.1.15.

    employee at the same time that case complexity is increasing may not be a real decline in productivity-more complex cases may require more time per case. We recognize that some options for improving productivity data, such as collecting more data on complexity and quality, could be costly. However, costs could be mitigated by using existing statistical methods to analyze SB/SE's current complexity and quality.

    To improve the planning of future enforcement process changes, we are recommending that the Commissioner of Internal Revenue ensure that SB/SE (1) puts in place a framework to guide planning of future process improvement projects and (2) invests in enforcement productivity data that better adjust for complexity and quality, taking into consideration the costs and benefits of doing so.

    In commenting on a draft of this report, the Commissioner of Internal Revenue agreed with our first recommendation and agreed in principle with our second, but also raised concerns about cost and feasibility. The Commissioner's comments are discussed later in this report. The Commissioner's letter is reprinted in appendix V.

    Background
       	SB/SE is one of IRS's four business operating divisions. SB/SE is responsible for enforcement, taxpayer education, and account services for about 45 million taxpayers, including 33 million selfemployed taxpayers and 7 million small businesses with assets of less than $10 million. SB/SE also performs some collection functions for other IRS operating divisions.

    SB/SE managers told us that the reorganization of IRS in 2000-including the creation of SB/SE-presented an opportunity for them to examine enforcementrelated processes from a new perspective. Prior to this, the agency was organized around functional and geographic lines, with separate groups responsible for activities such as processing returns, audits, and collection in particular areas. The reorganization eliminated or substantially modified this national, regional, and district structure and established organizational units serving particular groups of taxpayers with similar needs. Officials told us that with the reorganization, they were now responsible for functions that they had not controlled so directly before. They said that there was general agreement among the managers of the newly created division that there were opportunities to make processes more efficient and effective, and that this led them to start several enforcement process improvement projects. They also distinguished between enforcement process improvement projects, which are generally incremental in their approach, and more farreaching efforts to modernize IRS and transform processes through business systems modernization and other significant changes. We noted in our recent Performance
    and Accountability
    Series that IRS has made important progress in these larger efforts but its transformation continues to be a work in progress.8

    Though many of the SB/SE projects include the word "reengineering" in their titles, SB/SE managers agreed that process improvement projects was a better description, given the scope of the changes these projects were making. As described in GAO's Business
    Process Reengineering Assessment
    Guide, reengineering entails fundamentally rethinking how an organization's work should be done while process improvement efforts focus on functional or incremental improvements.9 SB/SE managers explained that they purposefully avoided technologydriven changes of the sort under development in the IRSwide business systems modernization effort. They said that their goal was to make shorter term, more SB/SEfocused changes in the meantime, while the more sweeping changes, and their longer planning and implementation horizons, were still years away from completion. In this report, we refer to the 15 SB/SE efforts under way as of November 2003 as "process improvement projects."

    8U.S. General Accounting Office, Major Management Challenges and Program
    Risks: A Governmentwide Perspective, GAO0395 (Washington, D.C.: January
    2003).

    9GAO/AIMD10.1.15.

    We have reported on declining enforcement trends, finding in 2002 that there were large and pervasive declines in six of eight major compliance and collection programs we reviewed, with the only exceptions in returns processing and in the automated underreporter program. In addition to these declines, we reported on the large and growing gap between collection workload and collection work completed and the resultant increase in the number of cases where IRS has had to defer collection action on delinquent accounts.10 In 2003, we reported on the declining percentage of individual income tax returns that IRS was able to examine each year, with this rate falling from .92 percent to .57 percent between 1993 and 2002.11 We also reported on enforcement productivity measured by cases closed per fulltime equivalent employees, finding that IRS's telephone and field collection productivity declined by about 25 percent from 19962001 and productivity in IRS's three audit programs-individual, corporate, and other audit-declined by 31 to 48 percent.12

    Improving productivity by changing processes is a strategy SB/SE is using to address these declining trends. As of November 2003, SB/SE had 15 ongoing process improvement projects under way, most of them in three broad enforcement areas-audit, collection, and compliance support. Audit projects entail changes to field and office examination processes.13 Collection projects include changes to automated collection programs, field collections, and other programs.14 Compliance support is the term SB/SE uses to describe processing functions related to audit and collection such as updating IRS information systems for the results of enforcement work and preparing examination closing letters and liens on taxpayer property. Compliance support projects include changes to technical services and case processing.

    10GAO02674.

    11U.S. General Accounting Office, Tax Administration: IRS Should Continue
    to Expand Reporting on Its Enforcement
    Efforts, GAO03378 (Washington, D.C.: Jan. 31, 2003).

    12GAO02674.

    13Field examinations are the most complex audits and are done at the taxpayer's location. In an office examination, the taxpayer comes to an IRS office with his or her records and meets with an auditor.

    14IRS employs a number of means to collect overdue taxes from taxpayers who owe them. These include letters, phone calls, office appointments, and visits to business locations.

    We selected four SB/SE process improvement projects to review in detail for this report. Field Examination Reengineering includes changes to preaudit processes to better identify specific issues on tax returns for auditors to focus on, among other changes intended to improve examination efficiency and reduce taxpayer burden. The Compliance Support Case Processing Redesign project seeks to centralize data entry into online information systems that monitor the status of active audit and collection cases and their results from many different locations with widely variable workload to just a few with more consistent, predictable workload. The Collection Taxpayer Delinquent Account Support Project involves the development of two computer models to improve setting priorities for collections cases to assign to collections staff. The Collection Field Function Consultative Initiative seeks to improve timeliness on collections cases through regular managerial involvement as cases are being worked. Brief descriptions of all of SB/SE's projects can be found in appendix IV.

      SB/SE Process Improvement Projects Included Most Key Steps but Productivity Measurement Could Be Improved

    SB/SE process improvement project teams completed most of the steps we identified as key to SB/SE's process improvement project planning, but none of the projects we reviewed completed all of the key steps. Guidance on project planning steps, such as our 20step framework, could help ensure that key steps are followed more consistently. Also, SB/SE enforcement productivity data presented problems in that the data available to SB/SE managers to assess the productivity of their enforcement activities, identify processes that need improvement, and assess the success of their process improvement efforts are only partially adjusted for complexity and quality of cases worked.

        Enforcement Process Improvement Projects Included Many Key Steps but SB/SE Lacked a Planning Framework

    The planning for each of the four projects we reviewed included most of the key steps in our process improvement framework, but none of the projects included all of the steps. Figure 2 presents our findings, organized by project stages, for each of the four projects we studied. A full circle means a step was fully completed in project planning and a partial circle means that only part of a step was completed. Our basis for each "no" or "partial" finding is explained in appendix III. Following figure 2, we discuss our findings in more detail with selected examples from the four projects we reviewed.

    Figure 2: Key Steps Included in Selected SB/SE Process Improvement
    Projects

    aSB/SE has not completed the implementation plan for this step in the project. bSB/SE has not completed the monitoring and evaluation plan for this project. N/A means a step was not applicable.

    The four SB/SE projects we reviewed largely included the productivity baseline definition and process mapping steps under the "Decision to Change" stage, where SB/SE had to determine whether any of its processes should be improved. The Field Examination Reengineering project team and both collection project teams had baseline data showing that the time needed to complete casework was rising and all four project teams had extensive flowcharts mapping the details of current processes. By helping managers understand the strengths and weaknesses of current processes, such information contributes to more informed decisions about which processes to change.

    However, SB/SE did not as consistently include the complexity and quality of work being done in productivity baselines, compare productivity data to external benchmarks, identify root causes of productivity declines, or measure the gap between current and desired productivity. Weaknesses in these steps leave SB/SE managers without information that could be useful when making decisions about which processes to change. For example, on three of the four projects, productivity data were not adjusted for case complexity and only partially adjusted for quality. This could cause productivity trends to be misinterpreted, leaving SB/SE at risk of trying to redesign processes that are already working well or missing opportunities to fix processes with potential for improvement. Because GAO's Business
    Process Reengineering Assessment
    Guide and our roundtable participants stressed the importance of complete productivity data and because this was a recurring issue we identified in our assessment of the four SB/SE projects, we discuss the importance of adjusting for case complexity and quality when measuring productivity in more detail in the next section of this report.15

    Another example of not consistently following our key steps in the "Decision to Change" stage is found in the Field Examination Reengineering project. The project team sought the advice of many large, noted organizations to benchmark its productivity. However, the work did not lead to measuring how SB/SE's productivity compared to others' because the team did not believe that operations in other organizations were comparable. Without this benchmarking, the team did not know whether and by how much it could improve productivity by updating operations based on the experiences of other large organizations. Both GAO's Business
    Process Reengineering Assessment Guide and our

    15GAO/AIMD10.1.15.

    roundtable participants stressed that although processes may seem unique to government, they likely have counterparts at a process level in the private sector. Moreover, GAO's Guide says that looking at dissimilar organizations can actually lead to the most fruitful improvements because it stimulates thinking about new approaches.

    During the "Target Process Development" stage, the projects we reviewed consistently included the steps that prepare for implementation. Planning on all four of the projects we studied included obtaining executive support, assessing barriers to implementing changed processes, and assessing resource needs and availability. The Compliance Support Case Processing Redesign team, for example, originally identified the need for a computer programming change to implement part of their process redesign. When the programming change could not be made immediately, they continued with a manual process in order to keep the project moving forward.

    However, SB/SE less consistently included key steps in this stage related to designing the new process. For example, in the Collection Taxpayer Delinquent Account Support project, SB/SE did not consider alternatives to achieving the project's goal of identifying the best cases to assign to collections staff. Because options were not considered, the team ran the risk of missing a more effective approach than the one they took. Another team did not design the new process based on analysis of a gap between current and desired productivity. It is important at this stage for projects to include factbased performance analysis to assess how to change processes that are in greatest need of improvement in terms of cost, quality, and timeliness. By analyzing the gap between an existing process's performance and where that performance should be, projects can target those processes that are most in need of improvement, analyze alternatives, and develop and justify implementation plans. Using these steps can increase the likelihood of determining the best new process.

    During the "Implementation" stage, three of the four projects we reviewed had completed implementation plans and all three included key implementation steps. These steps focus on the challenge of turning project concepts into a workable program. For example, in the Collection Taxpayer Delinquent Account Support project, the team clearly defined who was responsible for updating the existing computer programs to select cases for priority collection action and who was responsible for evaluating the implemented process. We also found that three of the four teams conducted pilot tests and used their results to modify the new processes

    prior to implementation-steps important for ensuring that process problems are worked out prior to project implementation.

    SB/SE was less consistent, however, in establishing employee performance expectations for the new processes. In the Field Examination Reengineering project, SB/SE plans to implement changes to audit planning steps in order to streamline audits and reduce demands on taxpayers for additional information. SB/SE's plan includes monitoring the deployment of the new process using measures such as the percent of personnel trained. However, SB/SE's plan does not specify performance expectations for employees or how it will measure whether its auditors are using the new techniques properly.

    Two projects had completed plans for outcome assessments at the time of our review. One of these, the Collection Taxpayer Delinquent Account Support project, included an evaluation plan using available data to develop measures of how accurately the new models were working. The other two projects were in the process of developing evaluation plans-an important step to ensure that the correct data are available and collected once the change is implemented.

    Three of four initiatives incorporated change management principles throughout their initiatives. In the fourth, we agreed with SB/SE managers that change management key steps were not a factor because the changes to the method of prioritizing collection cases did not affect collections staff. These are key steps because successful process improvement depends on overcoming a natural resistance to change and giving staff the training to implement the changes. The three project teams where change management was a factor consistently completed all of the key steps in the "Change Management" stage.

    In the course of our discussions with SB/SE managers about the steps that their projects did and did not include, we learned that SB/SE does not have its own guidance or framework that describes the steps to be followed in planning process improvement projects. SB/SE managers said that projects had been planned and carried out without such a framework. Contractors provided substantial assistance in designing SB/SE's process improvement projects, and managers told us that they relied in large part on the contractor staffs' expertise and experience in planning the projects.

    A framework laying out the steps to be followed is an important internal control for projects such as these because it provides top managers assurance that the steps that the organization has determined to be important are either taken on each project or that project managers have explained why they should be omitted. GAO's Business
    Process Reengineering Assessment Guide
    notes that an established framework is important for projects in that it defines in detail the activities the project team needs to complete and alerts the team to key issues that it must address.16 Without a process improvement framework and a consistent set of steps to follow, IRS runs the risk of future projects also missing key steps. This, in turn, exacerbates the risk of projects not addressing appropriate process problems, developing a less than optimal target process, ineffectively implementing the project, inaccurately assessing project outcomes, or mismanaging the change to the new process. A framework such as the one we developed for this report is an important internal control tool for SB/SE managers to guard against these risks. The internal control is needed whether process improvement is planned by SB/SE staff or contractors. Such a framework may also prove useful in other IRS units besides SB/SE. As with the 20step framework we used to assess SB/SE's approach, however, any such guidelines should allow for appropriate managerial discretion in cases where certain steps are not relevant.

        IRS Enforcement Productivity Data Only Partially Adjusted for Complexity and Quality

    The data available to SB/SE managers to assess the productivity of their enforcement activities, identify processes that need improvement, and assess the success of their process improvement efforts are only partially adjusted for complexity and quality of cases worked. Productivity measures the efficiency with which resources are used to produce outputs. Specific productivity measures take the form of ratios of outputs to inputs such as cases closed or dollars collected per staff year. The accurate measurement of enforcement productivity requires data about the quantity of outputs produced and inputs used that are accurate and consistent over time and that link the outputs directly to the inputs used to produce them.17

    16GAO/AIMD10.1.15.

    17We did not evaluate the accuracy or consistency of the data that IRS used to measure these quantities, or whether the specific outputs and inputs that IRS chose for its productivity measures were the most appropriate. These evaluations were beyond the scope of our report. However, for a discussion of the issues involved in choosing output and input indicators for productivity measures, see app. I.

    The accurate measurement of productivity also requires good data on the relative complexity or difficulty of the cases and the quality of the work done by IRS staff. Case complexity can vary with the type of tax (employment vs. income), the type of taxpayer (individual vs. business) and the type and sources of income and expenses. A measure of productivity like cases closed per staff year that shows an increase may not indicate a real gain in efficiency if the mix of cases worked has shifted to less difficult cases or the quality of the work has declined. This problem of adjusting for quality and complexity is not unique to SB/SE process improvement projects-the data available to process improvement project managers are the same data used throughout SB/SE to measure productivity and otherwise manage enforcement operations.

    SB/SE managers used data on the number of cases completed and the time it takes to complete them to measure output. Such data were usually only partially adjusted for quality and only once were they adjusted for complexity. Opportunities to make more such adjustments were missed.

    An example of a complete adjustment for complexity is the Compliance Support Case Processing Redesign team's use of a proxy for complexity. The project illustrates both the shortcomings of SB/SE's productivity data and the feasibility of some adjustments using other currently available information. The team wanted to measure the work needed to enter examination and collection case data into the information system, holding complexity constant, but direct measures of complexity were not available. While developing their new process, the team knew that more complex cases were to be assigned to highergrade clerks.18 The team used the grade of the clerk to adjust output for complexity. Although not a direct measure of relative complexity, the grade level of the clerks provided managers a means to adjust for complexity and better identify performance increases that were due to changes in productivity by holding complexity constant. Such an adjustment increases the credibility of the team's estimate that IRS would save up to 385 positions from the proposed redesign.

    18SB/SE staff have general schedule grades, with higher grades equating to more education and/or experience and to more complex job responsibilities.

    SB/SE has systems in place that measure quality against current standards but do not account adequately for changes in standards of quality. The Exam Quality Measurement System (EQMS) and the Collection Quality Measurement System (CQMS) use samples of audit and collection cases, respectively, to determine if IRS standards were followed and compute scores that summarize the quality of the case.19 Generally, the scoring is done on a numerical scale. For example, EQMS uses quality scores that range on a scale from 0 to 100. To SB/SE's credit, most of the projects that we reviewed used EQMS and CQMS scores in an attempt to control for quality changes. Unfortunately, these scores may not adequately reflect changes in standards of quality. For example, the IRS Restructuring and Reform Act of 1998 placed additional documentation requirements for certain collection actions on SB/SE collections staff, such as certifications that they had verified that taxes were past due and that sanctions were appropriate given the taxpayers' circumstances. SB/SE has changed the standards used in EQMS and CQMS to reflect the new requirements but has not changed its quality scale to account for the new, higher level of quality implied by the new standards. As a result, two exams with the same quality scores, one done before passage of the act and one after, may not have the same level of quality. If the way that SB/SE computes its quality scores does not adequately reflect such changes in quality standards, an increase in staff time needed to complete the additional requirement may be misinterpreted as a decline in productivity.

    19The Internal Revenue
    Manual describes standards for both the collection and examination system. Standards for collection requirements include determining if the casework isolated the right issues at the right time, the right actions were taken timely and efficiently, the right legal procedures were followed, and the case was closed correctly. The exam system uses eight standards to define quality, each defined by elements representing components that are present in a quality examination. Each exam is scored on each of the eight standards and the total score is the sum of the scores for each standard.

    Opportunities exist to improve SB/SE's enforcement productivity data. Statistical methods that are widely used in both the public and private sectors can be used to adjust SB/SE productivity measures for quality and complexity. In particular, by using these methods, managers can distinguish productivity changes that represent real efficiency gains or losses from those that are due to changes in quality standards. These methods could be implemented using data currently available at SB/SE. The cost of implementation would be chiefly the staff time required to adapt the statistical models to SB/SE. Although the computations are complex, the methods can be implemented using existing software.20 We currently have under way a separate study that illustrates how these methods can be used to create better productivity measures at IRS. We plan to report the results of that study later in 2004.

    We recognize that better incorporating the complexity and quality of enforcement cases in enforcement productivity data could entail costs to SB/SE. Collecting additional data on complexity and quality may require longterm planning and investment of additional resources. However, as discussed in the previous paragraph, there are options available now to mitigate such costs. Existing statistical methods could be used in the short term, with currently available data on case complexity and quality to improve productivity measurement. In addition, IRS's ongoing business systems modernization effort may provide additional opportunities for collecting data.

    20The statistical methods use data on outputs, inputs, quality, and complexity to derive a composite productivity index. This index can be further analyzed (or "decomposed") to adjust for the effect of factors like the IRS Restructuring and Reform Act of 1998. Examples of these methods include stochastic frontier and data envelopment analysis. These methods have been applied extensively in both the public and private sectors. For a survey of studies, see L. Sieford, "Data Envelopment Analysis: The Evolution of the State of the Art, 19781995," Journal
    of Productivity
    Analysis, 1996, 7, pp. 99137. For an example of an application of these methods to the banking industry, see D. Wheelock and P. Wilson, "Technological Progress, Inefficiency, and Productivity Change in U.S. Banking, 19841993, Journal
    of Money, Credit and
    Banking, 31(2), May 1999, pp.212234. For an application to public school productivity measurement, see J. Ruggiero and D. Vitaliano, "Assessing the Efficiency of Public Schools Using Data Envelopment Analysis and Frontier Regression," Contemporary
    Economic Policy, July 1999, 17(3), pp. 32131.

    Our roundtable participants stressed the benefits of productivity analysis. They said that an inadequate understanding of productivity makes it harder to distinguish processes with a potential for improvement from those without such potential. GAO's Business
    Process Reengineering Assessment
    Guide also highlighted the importance of being able to identify processes that are in greatest need of improvement.21

    Conclusions"	SB/SE deserves recognition for embracing process improvement and for including so many key steps in planning the projects. To the extent that IRS succeeds in improving enforcement productivity through these projects, resources will be made available for providing additional services to taxpayers and addressing the declines in tax enforcement programs.

    While the SB/SE projects we reviewed included most of the key steps in our framework, putting guidance in place for future projects to follow would help ensure that all key steps are included and improve project planning. The 20step framework that we developed for this report is an example of such guidance.

    More complete productivity data-input and output measures adjusted for the complexity and quality of cases worked-would give SB/SE managers a more informed basis for decisions on how to improve processes. We recognize that better productivity will mean additional costs for SB/SE and that, therefore, SB/SE will have to weigh these costs against the benefits of better data. GAO currently has under way a separate study, illustrating how data on complexity and quality could be combined with output and input data to create better productivity measures. This may prove useful to SB/SE managers as they evaluate the current state of their productivity measures. We will report the results of that review later in 2004.

    Recommendations for We recommend that the Commissioner of Internal Revenue ensure that Executive Action SB/SE take the following two actions:

    o
    "	Put in place a framework to guide planning of future SB/SE process improvement projects. The framework that GAO developed for this report is an example of such a framework.

    21GAO/AIMD10.1.15.

    o
    "	Invest in enforcement productivity data that better adjust for complexity and quality, taking into consideration the costs and benefits of doing so.

      Agency Comments and Our Evaluation

    The Commissioner of Internal Revenue provided written comments on a draft of this report in a January 14, 2004, letter, which is reprinted in appendix V. The Commissioner agreed with our recommendation that IRS develop a framework to guide future improvement projects. He notes that SB/SE used outside experts to help direct the projects we discuss in our report, and how the expertise gained from SB/SE's projects puts the organization in a position to create a framework for future projects.

    In regard to our second recommendation, the Commissioner agreed in principle with the value of adding to current enforcement productivity data, but also expressed concerns about cost and feasibility. His letter also discusses initiatives in progress to improve program management and monitoring in the short term, as well as his intent to explore the use of statistical methods to improve enforcement program productivity measurement and to ensure that they are included in modernization projects. The careful consideration of costs and benefits and steps to improve measures in the long term are at the heart of our recommendation and we encourage his ongoing commitment to these efforts.

    The Commissioner's letter also notes that employee performance goals- one of the steps in our framework-must not violate legal restrictions on the use of certain enforcement data to evaluate employee performance. We agree and clarified language in our report to make it clear that our framework step concerns employee performance expectations, not using enforcement data to evaluate employees or otherwise imposing production goals or quotas.

    In addition to commenting on our recommendations, IRS provided supplemental data on the results of some reengineering projects. Reviewing project results was not part of the scope of our review and time did not permit us to verify the supplemental data provided by IRS on project results.

    We conducted our work from September 2002 through November 2003 in accordance with generally accepted government auditing standards.

    As agreed with your offices, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days after its date. At that time, we will send copies of this report to the Secretary of the Treasury, The Commissioner of Internal Revenue, and other interested parties. This report is available at no charge on GAO's Web site at http://www.gao.gov.

    If you or your staffs have any questions, please contact me at (202) 5129110 or David Lewis, Assistant Director, at (202) 5127176. We can also be reached by email at whitej@gao.gov or lewisd@gao.gov, respectively. Key contributors to this assignment were Tara Carter, Kevin Daly, Leon Green, Landis Lindsey, and Amy Rosewarne.

    James R. White Director, Tax Issues Strategic Issues Team

    Appendix I#

                         Process Improvement Project Framework#

    The 20step process improvement we identified is broken out into four broad stages, from deciding to make changes to assessing the results of the changes. A fifth stage deals with managing the changes being made and takes place throughout the latter part of the project. Figure 3 places the stages in their chronological order, with the change management stage shown taking place simultaneously with other stages.

    Figure 3: 20-Step Process Improvement Framework

    Within each of the stages of this framework are specific key steps that we developed based on GAO guidance and what we learned from managers in other organizations about the steps they took to ensure that they were embarking on the right projects, designing and implementing the projects appropriately, and accurately assessing their projects' results. The sections below describe the nature and purpose of the key steps that fall under the different stages. We recognize that some steps may not be appropriate for some projects and that managers need to apply judgment in using this or any other process improvement framework. Development of this framework is described in appendix II.

                                       Appendix I
                         Process Improvement Project Framework

    Decision to Change"	Organizations that base the decision to redesign processes on accurate productivity data and a clear understanding of current processes increase the likelihood that a project will avoid misdiagnosing a problem or designing a less than optimal outcome target. Six key steps are important to accomplishing this.

      Identify Productivity Baseline

    Baseline data are information on the current process that provide the metrics against which to compare improvements and use in benchmarking. Productivity measures the efficiency with which an organization uses resources, or inputs, to produce outputs. Specific productivity measures generally take the form of a ratio of outputs to inputs. By establishing a baseline using such measures, a process improvement can be measured in terms of real efficiency gains. For example, the baseline could be used to measure the effect of a successful process improvement as additional output produced by the organization with the same or fewer inputs.

      Include Complexity and Quality in Productivity Measures

    Productivity measures may give misleading information if they do not account for the relative complexity and quality of outputs and inputs. A measure of productivity like cases closed per staff year that shows an increase may not indicate a real gain in efficiency if the mix of cases has shifted to less difficult cases or the quality of the work has declined. Besides accounting for complexity and quality, the organization must also choose the appropriate indicators of its outputs and inputs and measure them accurately. Organizations like IRS that are service providers have outputs that often consist of complex, interrelated activities that, in many cases, may require multiple indicators of outputs to accurately measure productivity. The specific data needed depend on the characteristics of particular IRS processes. For example, the number and type of output indicators appropriate for services that have direct contact with taxpayers, such as audits, may be larger and more varied (to reflect the full impact of these services on taxpayers) than those appropriate for other services with less or no direct contact, such as forms processing. However, factors like complexity and quality are necessary for accurate productivity measurement for any process in IRS, regardless of how the specific quantitative measures are defined.

                                       Appendix I
                         Process Improvement Project Framework

      Compare Current Productivity to Internal and External Benchmarks

    A benchmark is a measurement or standard that serves as a point of reference by which process performance is measured. During the "Decision to Change" stage, benchmarking is the solutionbuilding component of process improvement through which an organization compares data used to measure existing internal processes with external data on similar processes in other organizations, or in other components of the same organization, to identify process improvement needs and outcome targets. Through benchmarking an organization is able to identify gaps between an organization's process performance and that of other organizations or other components of the same organization. Benchmarking is a key tool for performance improvement because it provides "real world" models and reference points for setting ambitious improvement goals.

    Map Current Process "	Process mapping is a graphical representation depicting the inputs, outputs, constraints, responsibilities, and interdependencies of the core processes of an organization. Acceptable modeling tools and other analysis techniques include flowcharting, tree diagrams, fishbone diagrams, and business activity maps. It is important that a process map defines what the components of each process are, as well as the process's boundaries, dependencies, and interconnections with other processes. If initial process mapping is done at a high level, more detailed modeling is necessary to identify all of the current process's activities and tasks, staff roles and responsibilities, and links and dependencies with other processes, customers, and suppliers. Performance data (e.g., costs, time, throughput) for all activities and tasks should be included on the map, or made available elsewhere. The people who actually do the work as well as the process owner should validate the mapping. Regulations, policies, laws, and assumptions underlying the processes should be identified.

    Identify Causes of Weak Causal factors are the conditions that initiate the occurrence of an

    Performance"	undesired activity or state. Causal factors that are within the span of control of an organization should be addressed during the Target Process Development stage. Causal factors that are beyond the span of control of an organization should be isolated when identifying a problem. Examples of causal factors are legal requirements, mix of inputs, quality of inputs, and staff constraints.

                                       Appendix I
                         Process Improvement Project Framework

    Measure Gap Between An empirical basis for the decision to make a process change is an Current and Desired important step leading towards an improvement that is optimal and Productivity attainable. An empirical basis can be established by using productivity

    data to define the gap between where the organization currently is and

    where it wants to be.

    Target Process "	After deciding to undergo a process improvement, an organization can increase the likelihood of determining the best new process by using

    Development "	productivity data, assessing implementation barriers, and developing feasible alternatives.

      Understand the Best Practices of Others

    Solutions can be adapted from best practices found in other organizations. Best practices are processes and procedures that highperforming organizations use to achieve results. An organization should evaluate the pros and cons of each best practice and if possible, apply its own productivity standards. Ideally, this form of benchmarking should be done with an external organization. Many processes that seem unique to the government actually have counterparts in the private sector, especially in generic areas such as claims processing, loan management, inventory management, etc. Also, it is important to note that the other organizations do not have to be particularly similar, or even do similar work. For example, Xerox used L.L. Bean to improve order fulfillment. Looking at processes in dissimilar organizations can actually lead to the most fruitful improvements because it stimulates new thinking about traditional approaches to doing work.

    Analyze Alternatives "	Alternatives are different process designs that would likely result in the same or a similar outcome. An organization's analysis of alternative processes should consider benefits, costs, and risks. Performance results that each alternative could be expected to achieve should be determined. This can be done using methods such as prototyping, limited pilot testing, and modeling and/or computer simulation. In addition to performance, alternatives can be scored by any number of factors including, feasibility, budget, political appeal, implementation time, payback time, and risk. The team should explore each alternative thoroughly enough to convincingly demonstrate its potential to achieve the desired performance goals and

                                       Appendix I
                         Process Improvement Project Framework

    fully describe the types of technical and organization changes necessary to support each goal, and if possible, test key assumptions.

    Design New Process to The selection of a target process from among alternatives needs an

    Close Productivity Gap "	empirical basis in the form of some sort of quantitative analysis. The decision to improve and forming the target process should be linked by an analysis of productivity data that shows how the new process can close the gap between the productivity baseline and the desired outcome.

    Obtain Executive Support "	Executive support should come in the form of an executive steering committee-a group headed by the organization's leader to support and oversee the process improvement effort from start to finish. Executive involvement is important because they are in a position to build credible support among customers and stakeholders, mobilize the talent and resources for a reengineering project, and authorize the actions necessary to change agencywide operations. An executive steering committee's roles include defining the scope of the improvement project, allotting resources, ensuring that project goals align with the agency's strategic goals and objectives, integrating the project with other improvement efforts, monitoring the project's progress, and approving the reengineering team's recommendations. While carrying out these responsibilities the steering committee must also keep stakeholders apprised of the reengineering team's efforts.

      Assess Barriers to Implementing Changed Process

    Implementation barriers are obstacles that the organization will need to overcome to implement a new process. Examples of implementation barriers include political issues, entrenched workplace attitudes or values, an insufficient number of employees with the skills required for the redesigned roles, collective bargaining agreements, incompatible organization or physical infrastructure, current laws and regulations, and funding constraints. The impact of these barriers and the costs of addressing them (such as staff training, hiring, and relocation) need to be factored into the process selection decision. If the reengineering team determines that the risks and costs of implementing a preferred new process appear too great, they may need to pursue one of the less ideal, but more feasible alternatives.

                                       Appendix I
                         Process Improvement Project Framework

    Assess Resource Needs and Prior to taking on a process improvement project, GAO guidance and the

    Availability"	other organizations we consulted stress the importance of ensuring the availability of staff and other resources necessary to complete design and implementation of the changed process. Without adequate resources, an organization undertaking a change runs the risk of an incompletely implemented project.

    Implementation "	A carefully designed process improvement project needs a similarly well thoughtout implementation in order to be successful.

    Conduct Pilot Tests "	Pilot tests are trial runs of the redesigned process. Pilot testing is a tool used to move the organization successfully to full implementation. Pilot testing allows the organization to (1) evaluate the soundness of the proposed process in actual practice, (2) identify and correct problems with the new design, and (3) refine performance measures. Also, successful pilot testing will help strengthen support for fullscale implementation from employees and stakeholders.

    Adjust Target Process Based Postpilot adjustments are corrective actions taken to correct trouble spots

    on Pilot Results"	prior to full implementation. Trouble spots can be pinpointed through the formal evaluation of pilot projects designed to determine the efficiency and effectiveness of the new process.

    Define Roles and Process owners are the individuals with the responsibility for the process

    Responsibilities "	being improved. Designating process owners is necessary to ensure accountability.

                                       Appendix I
                         Process Improvement Project Framework

      Establish Employee Expectations for New Process

    New employee and/or team performance expectations should be established to account for changes in roles and career expectations caused by the new process. Measurable indicators that are currently being used to track and assess employee or team progress should be analyzed to determine if adjustments will be required after the new process is implemented. In the case of IRS enforcement activities, the agency must ensure that the expectations do not violate the legal prohibition on using tax enforcement results to evaluate employee performance or imposing or suggesting production quotas or goals.1 In 2002, we reported on IRS's progress towards improving its performance management system; these changes were brought on, in part, by this requirement.2

    Outcome Assessment "	Careful assessment of the results of a process improvement project is important in that it may lead to further changes in the process being addressed and may suggest lessons for other projects.

                   Develop Plans to Monitor and Evaluate New Process

    An evaluation plan is a way to collect and analyze data in order to determine how well a process is meeting its performance goals and whether further improvements are needed. Good performance measures generally include a mix of outcome, output, and efficiency measures. Outcome measures assess whether the process has actually achieved the intended results, such as an increase in the number of claims processed. Efficiency measures evaluate such things as the cost of the process and the time it takes to deliver the output of the process (a product or service) to the customer. The data needed to conduct outcome assessments later on need to be identified during project planning to ensure that they are available and collected once implementation begins.

    Change Management "	Change management focuses on the adjustments that occur in the culture of an organization as a result of a redesigned process. Research suggests that the failure to adequately address-and often even consider-a wide

    1IRS Restructuring and Reform Act of 1998, section 1204, 26 U.S.C. S:7804 note (2000).

    2U.S. General Accounting Office, Performance Management Systems: IRS's
    Systems for Frontline Employees and Managers Align with Strategic Goals
    but Improvements Can Be Made, GAO02804 (Washington, D.C.: July 12, 2002).

                                       Appendix I
                         Process Improvement Project Framework

    variety of people and cultural issues is at the heart of unsuccessful organizational transformations. Similarly for process improvement efforts, redesigning a process is not only the technical or operational aspect of change, but also overcoming a natural resistance to change. Successfully managing change reduces the risk that improvement efforts will fail due to a natural resistance to change within an organization.

      Establish a Change Management Strategy

    An organization needs to establish a change management strategy that addresses cultural changes, builds consensus among customers and stakeholders, and communicates the planning, testing, and implementation of all aspects of the transition to the new process. Change management activities focus on (1) defining and instilling new values, attitudes, norms, and behaviors within an organization that support new ways of doing work and overcome resistance to change, (2) building consensus among customers and stakeholders on specific changes designed to better meet their needs, and (3) planning, testing, and implementing all aspects of the transition from one organization structure or process to another. Executive involvement is important for successful change management. Executive support helps strengthen upper management's support for the project and serves to reinforce the organization's commitment to the proposed changes. In a roundtable meeting held by GAO to obtain the perspectives of the private sector, one organization mentioned that providing continuous feedback to its employees is a critical element of a change management program. They also described the importance of consistently updating those employees who would be directly affected by a change initiative. Keeping employees informed of decisions and recognizing their contributions are important elements of developing positive employee attitudes toward implementing process improvement initiatives. Ongoing communication about the goals and progress of the reengineering effort is crucial, since negative perceptions could be formed and harden at an early stage, making the implementation of the new process more difficult to achieve. If change management is delayed it will be difficult to build support and momentum among the staff for implementing the new process, however good it might be.

    Establish a Transition Team "	A transition team is a group of people tasked with managing the implementation phase of process improvement projects. A transition team should include the project sponsor, the process owner, members of the process improvement project team, and key executives, managers, and staff from the areas directly affected by changeover from the old process to

                                       Appendix I
                         Process Improvement Project Framework

    the new. Agency executives and the transition team should develop a detailed implementation plan that lays out the road to the new process, including a fully executable communication plan. The process owners responsible for managing the project will not effectively convey the goals and implementation strategy of the project if a viable mechanism is not set up by the transition team to keep employees and external stakeholders informed.

    Develop Workforce Training Training and redeploying the workforce is often a major challenge and

    Plans "	generally requires substantial preparation time. When a process is redesigned and new information systems are introduced, many of the tasks workers perform are radically changed or redistributed. Some positions may be eliminated or cut back, while others are created or modified. Workers may need to handle a broader range of responsibilities, rely less on direct supervision, and develop new skills.

    Appendix II#

    Scope and Methodology#

    We began development of a process improvement framework by reviewing previously developed GAO guidance related to business process reengineering.1 We also reviewed guidance that GAO has recently issued on assessment frameworks for other major management areas.2

    GAO's Business Process Reengineering Assessment
    Guide recognizes that the steps for planning process improvement need to be adapted for the magnitude of the projects and the particular circumstances of an organization. To supplement the GAO business process reengineering guidance, we held a halfday roundtable meeting with the Private Sector Council and two of its member companies, Northrop Grumman (a $25 billion defense enterprise) and CNF (a $5 billion transportation services company).3 We also discussed process improvement planning with public sector managers with experience in revamping complex processes. Reviewing publicly available information and in discussions with SB/SE staff, we found that the tax agencies in the states of California, Minnesota, and Florida had gone through substantial process improvement efforts in recent years. Similarly, the Department of the Interior's Minerals Management Service had carried out substantial process improvement projects. We interviewed officials from these organizations and reviewed documents that they provided. We then used all of this information to adapt GAO's guidance to a 20step framework appropriate to the SB/SE projects.

    We judgmentally selected 4 projects to study in detail from the 15 projects SB/SE had under way. Our goal in selecting projects for detailed review was to cover at least one project in each of the three main enforcement

    1GAO/AIMD10.1.15.

    2U.S. General Accounting Office, Human Capital: A Guide for Assessing
    Strategic Training and Development Efforts in the Federal
    Government, GAO03893G (Washington, D.C.: July 2003); U.S. General Accounting Office, Homeland
    Security: Management Challenges Facing Federal
    Leadership, GAO03260 (Washington, D.C.: Dec. 20, 2002), p. 60; U.S. General Accounting Office, Information
    Technology: A Framework for Assessing and Improving Enterprise
    Architecture Management (Version
    1.1), GAO03584G (Washington, D.C.; April 2003).

    3The Private Sector Council is a nonprofit, nonpartisan public service organization created to help the federal government improve its efficiency, management, and productivity through cooperative sharing of knowledge between the public and private sectors. It is comprised of member companies-businesses from across North America in industries such as telecommunications, defense, finance, and energy. Corporate executives from member companies provide their time and expertise at no cost to the government.

    Appendix II
    Scope and Methodology

    areas that IRS was revamping- audit, collection, and compliance support. We also looked for projects that were sufficiently far along that we considered it reasonable to expect to see either completed steps or plans for remaining steps for most of the project. We selected one project each in the audit and compliance support areas. We found that there were 2 projects underway in the collections area that were significantly far along, so we selected both of them for our detailed review.

    For the four projects we selected, we used the documentation previously provided to us to identify evidence that SB/SE managers had taken or were in the process of taking the key process improvement project steps we identified. We then discussed our initial findings with IRS officials responsible for the four projects and they provided additional evidence, both orally and in writing, concerning the elements we had identified as present or not in our initial document review. We then revised our initial assessments based on the additional evidence that the officials provided. Our assessments also included review by a GAO project design specialist, in addition to our usual quality control procedures.

    We also recognized the need for flexibility in the application of our criteria, in that not all of the steps we identified necessarily make sense for every project. Where a particular step did not logically apply to a particular project, we listed it as "not applicable" in our assessment. For instance, the Collection Taxpayer Delinquent Account Support project we reviewed in detail did not change processes that staff were asked to carry out, so we rated the step about developing a training plan as "not applicable." Where a step was not fully completed but the project team did a number of elements of the step, we assessed that step as "partial" in our matrix. We did not evaluate the success so far or the likelihood of success for any of the projects we reviewed. We also did not evaluate the effectiveness with which project steps were completed. For example, we did not evaluate the quality of the pilot tests.

    To determine the usefulness of IRS productivity data as a basis for determining the direction and eventual success of SB/SE process improvement efforts, we reviewed the literature on productivity measurement in tax agencies and in the public sector generally. We also reviewed studies on productivity measurement in service industries with functions similar to IRS.

    Appendix III#

    GAO Assessment of Four Selected SB/SE Process Improvement Projects

    The following four figures provide summaries of the evidence we used to make specific assessments of four selected SB/SE process improvement projects.

    Appendix III
    GAO Assessment of Four Selected SB/SE
    Process Improvement Projects

      Figure 4: Key Steps Included in the Field Examination Reengineering Project

    Appendix III
    GAO Assessment of Four Selected SB/SE
    Process Improvement Projects

    Figure 5: Key Steps Included in the Compliance Support Case Processing Redesign
                                        Project

    aSB/SE has not completed the implementation plan for this step in the project.

    Appendix III
    GAO Assessment of Four Selected SB/SE
    Process Improvement Projects

    Figure 6: Key Steps Included in the Collection Taxpayer Delinquent Account
                                    Support Project

    Appendix III
    GAO Assessment of Four Selected SB/SE
    Process Improvement Projects

    Figure 7: Key Steps Included in the Collection Field Function Consultative
                                   Initiative Project

    aSB/SE has not completed the monitoring and evaluation plan for the new process.

    Appendix IV

    Descriptions of SB/SE Process Improvement Projects

    SB/SE management capitalized on the opportunity presented by the IRS reorganization that created their operating division and saw declining productivity trends as an impetus to change. SB/SE had 15 distinct process improvement efforts under way as of November 2003, many with multiple subprojects. Table 1 provides descriptive information of the 15 projects.

    Table 1: Major Process Improvement Projects in IRS's Small Business/Self
    Employed Operating Division

    Problem identified IRS's process Project by IRS improvement goals Status
    Target completion date Examination

    Office Examination 27 percent increase in

    & Field Examination 	calendar time associated with a field audit

    75 percent increase in the hours per Office Examination Improve examination efficiency, effectiveness, and consistent treatment of taxpayers

    Improve examiner's ability to be flexible

    Reduce taxpayer burden through more focused audits

    Increase collaboration between examiner, manager, taxpayer, and representative

    Full implementation in Office Exam: Phase 1 - progress February 2004 Phase 2 - May 2004

    Field Exam:
    Phase 1 - in progress
    Field Exam: Phase 2 -
    February 2004

    Exam Life Cycle
       	SB/SE examination life cycle from date tax return is filed to examination closing averages 780 days

    Identify shortterm and Recommendations
    longterm pending
    recommendations to
    dramatically reduce
    examination life cycle

    Shortterm recommendations and preliminary longterm recommendations: March 2004

    Implementation of shortterm recommendations and final longterm recommendations: April 2004

                                   Compliance Support

    Case Processing       Nonprofessional           Centralize case       Full implementation     Implementation -
                       operations disbursed     processing to four SB/SE        pending        Between January 2005
                       across 86 posts of duty          campuses                                and January 2006, with
                                                                                               90 percent completed by
                      Lack of flexibility in      Implement electronic                                June 2005
                     changing staff priorities  case closure processing
                            and duties

    Inconsistent processes

    Appendix IV
    Descriptions of SB/SE Process Improvement
    Projects

                             (Continued From Previous Page)

    Problem identified IRS's process Project by IRS improvement goals Status
    Target completion date

    Technical Services
       	Lowvolume, highcomplexity programs are handled across all areas, limiting development of staff expertise Realign workload with staff expertise

    Reassign work from professionals to paraprofessionals

    Centralize national and multiarea programs

    Automate responses to frequently asked questions

    Develop program knowledge and subject expertise with specialization Full implementation in Implementation complete progress - April 2004

    Collection Operations  Resource constraints limit  Identify opportunities for  Recommendations  Recommendations due -
                           ability to accomplish work      specialization and          pending           April 2004
                           within established time       consolidation of work
                                     frames                    processes
                                 Paperintensive          Develop and implement
                                   processes             automation initiatives
                                                         that will reduce paper
                                                               processing

    Collection

    Taxpayer Delinquent Accounts Support Project SB/SE can only work on 40 percent of its top priority delinquent tax accounts

    Total dollars of tax accounts continues to rise Determine the optimal balance between workload and inventory for telephone and field collection

    Identify characteristics of cases that provide for productive disposition

    Develop and implement models that predict payment characteristics Full implementation Implementation

    complete 	completed - January 2003

    Collection Field Average calendar time to Decrease calendar time Training and Final phase of training

    Function 	complete a collection and improve quality implementation partially and implementation - case is 354 days completed March 2004

    Tax Delinquent Appropriate selection Identify characteristics of Modeling implementation Implementation due -
    Investigations/ criteria must be cases that provide for complete July 04
    Nonfiler Initiative developed to identify productive nonfiler cases

    nonfiler cases

    Appendix IV
    Descriptions of SB/SE Process Improvement
    Projects

                             (Continued From Previous Page)

    Problem identified IRS's process Project by IRS improvement goals Status
    Target completion date

    Automated Collection  The volume of phone call        Determine optimal      Site implementation  Site implementation
      System Operating         from taxpayers to         balance in telephone         complete        complete - December
           Model             telephone collection       collection between IRS                               2003
                          continues to rise resulting  initiated phone calls to
                              in capacity issues       taxpayers and taxpayer
                                                             calls to IRS

    Organizational Structure
       	Additional compliance Determine if Taxpayer Implementation pending Transition plan to be resources are needed for Education and developed by January highrisk accounts Communication 2004

    resources can be shifted
    to compliance with Implementation complete
    minimal impact on by December 2004
    achieving mission

    Installment Agreement
       	Currently, 36 percent of installment agreements and 49 percent of installment agreement dollars default within 5 years Segment taxpayers with installment agreements based on probability of default

    Develop alternative treatment process to leverage current enforcement tools based on taxpayer risk characteristics to reduce default rates and increase dollars collected

    Develop protocols for applying the appropriate agreement structures to taxpayer segments Full implementation Final phase of

    pending 	implementation to be completed in September 2004

    Collection Cycle Time Calendar days (cycle Develop Recommendations Final shortterm time) to process collection recommendations to pending recommendations - work from beginning to decrease cycle time March 2004 end needs to be reduced Final longterm recommendations and implementation of shortterm recommendations - April 2004

    Appendix IV
    Descriptions of SB/SE Process Improvement
    Projects

                             (Continued From Previous Page)

    Problem identified IRS's process Project by IRS improvement goals Status
    Target completion date

    Business Master File     Tolerance levels for           Develop a revenue        Implementation in   Accounts management
     Tolerance Modeling      business taxpayers in     sensitive model for setting       progress         tolerances in place in
                          submissions processing            business taxpayer                                  August 2003
                                 and accounts                  tolerances           Adjustments pending
                               management needs to be                                                    Submission processing
                             set consistently and                                                        phase I - January 2004
                          effectively in coordination
                          with compliance levels                                                         Additional changes based

    on a new model - January 2005

    Workload Driver Model Analysis is needed to Identify unknown causes Recommendations Initial recommendations - identify causes of for increased volume or pending July 2004 increased volume of work work received in SB/SE in SB/SE Customer Account Services using datamining techniques and regression analysis and develop improvements to mitigate impact

    Source: IRS.

                                      Appendix V

                      Comments from the Internal Revenue Service

    Appendix V
    Comments from the Internal Revenue Service

    Appendix V
    Comments from the Internal Revenue Service

    GAO's Mission
       	The General Accounting Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability.

      Obtaining Copies of GAO Reports and Testimony

    The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO's Web site (www.gao.gov) contains abstracts and fulltext files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics.

    Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as "Today's Reports," on its Web site daily. The list contains links to the fulltext document files. To have GAO email this list to you every afternoon, go to www.gao.gov and select "Subscribe to email alerts" under the "Order GAO Products" heading.

    Order by Mail or Phone
       	The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to:

    U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C. 20548

    To order by Phone: 	Voice: (202) 5126000 TDD: (202) 5122537 Fax: (202) 5126061

    To Report Fraud, 	Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm

      Waste, and Abuse in Email: fraudnet@gao.gov

    Federal Programs Automated answering system: (800) 4245454 or (202) 5127470

    Public Affairs
       	Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 5124800 U.S. General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C. 20548

                                   Presorted Standard
                                  Postage & Fees Paid
                                          GAO
                                    Permit No. GI00

    United States
    General Accounting Office
    Washington, D.C. 20548-0001

    Official Business
    Penalty for Private Use $300

    Address Service Requested
    *** End of document. ***