    Business Systems Modernization: IRS Needs to Complete Recent
    Efforts to Develop Policies and Procedures to Guide Requirements
    Development and Management (20-MAR-06, GAO-06-310).

    The Internal Revenue Service's (IRS) effort to modernize its tax
    administrative and financial systems--Business Systems
    Modernization (BSM)--has suffered delays and cost overruns due to
    a number of factors, including inadequate development and
    management of requirements. Recognizing these deficiencies, IRS
    created a Requirements Management Office (RMO) to establish
    policies and procedures for managing requirements. GAO's
    objectives were to assess (1) whether the office has established
    adequate requirements development and management policies and
    procedures and (2) whether BSM has effectively used requirements
    development and management practices for key systems development
    efforts.
    -------------------------Indexing Terms-------------------------
    REPORTNUM:   GAO-06-310
        ACCNO:   A49457
      TITLE:     Business Systems Modernization: IRS Needs to Complete
    Recent Efforts to Develop Policies and Procedures to Guide
    Requirements Development and Management
         DATE:   03/20/2006
      SUBJECT:   Cost overruns
    	     Financial management systems
    	     Program evaluation
    	     Program management
    	     Schedule slippages
    	     Strategic planning
    	     Systems management
    	     Technology modernization programs
    	     Policies and procedures
    	     IRS Business Systems Modernization
    	     Program

    ******************************************************************
    ** This file contains an ASCII representation of the text of a  **
    ** GAO Product.                                                 **
    **                                                              **
    ** No attempt has been made to display graphic images, although **
    ** figure captions are reproduced.  Tables are included, but    **
    ** may not resemble those in the printed version.               **
    **                                                              **
    ** Please see the PDF (Portable Document Format) file, when     **
    ** available, for a complete electronic file of the printed     **
    ** document's contents.                                         **
    **                                                              **
    ******************************************************************
    GAO-06-310

         * Contents
              * Results in Brief
              * Background
                   * History of IRS Modernization
                   * Requirements Development and Management
                   * Requirements Development and Management Processes
                        * Elicitation
                        * Documentation
                        * Verification and Validation
                        * Management
              * BSM Lacks Policies and Procedures for Requirements Management
                   * BSM Lacks Comprehensive Policies and Procedures for
                     Requirements Development and Management, but Has Initiated
                     Their Development
              * BSM Projects Have Not Consistently Followed Disciplined
                Requirements Development and Management Practices
                   * Project Teams Have Inconsistent Requirements Elicitation
                     Practices
                   * Projects Do Not Fully Document Requirements
                   * Projects Do Not Fully Verify Requirements; Validation
                     Activities Completed, but Problems Remain
                        * Projects Do Not Fully Verify Requirements through Peer
                          Reviews
                        * Project Teams Performing Validation, but Problems
                          Remain
                   * Project Teams Not Adequately Managing Requirements
                        * Project Teams Not Updating Cost and Schedule to Reflect
                          Requirements Changes
                        * Project Teams Not Ensuring Full Traceability of
                          Requirements
              * Conclusions
              * Recommendations for Executive Action
              * Agency Comments
         * Scope and Methodology
         * Project Descriptions
              * Modernized e-File
              * Filing and Payment Compliance
              * Customer Account Data Engine
         * Comments from the Department of the Treasury
         * GAO Contact and Staff Acknowledgments
         * cov1&2corrected.pdf
              * Report to the Chairman, Subcommittee on Transportation, Treasury,
                the Judiciary, HUD, and Related Agencies, Committee on
                Appropriations, U.S. Senate
                   * March 2006
              * BUSINESS SYSTEMS MODERNIZATION
                   * IRS Needs to Complete Recent Efforts to Develop Policies and
                     Procedures to Guide Requirements Development and Management

    Contents

    Tables

    Figures

    March 20, 2006Letter

    The Honorable Christopher S. Bond Chairman Subcommittee on Transportation,
    Treasury, the Judiciary, HUD, and Related Agencies Committee on
    Appropriations United States Senate

    Dear Mr. Chairman:

    The Internal Revenue Service (IRS) has long relied on obsolete automated
    systems for key operational and management functions; its attempts to
    modernize these systems span several decades. IRS's current
    effort-Business Systems Modernization (BSM)-is a highly complex,
    multibillion dollar effort to modernize its technology and related
    business processes. Over the past 7 years, IRS has been appropriated
    approximately $1.9 billion for BSM. These systems modernization projects
    have experienced cost overruns and schedule delays due to, among other
    things, inadequate development and management of requirements.1 Lack of
    attention to these crucial processes has led to projects not meeting cost,
    schedule, and performance goals.

    IRS has recognized these deficiencies and established a Requirements
    Management Office (RMO) to, among other things, develop the policies and
    procedures that are to cover all aspects of requirements development and
    management. This report responds to your request that we assess (1)
    whether the RMO has established adequate requirements development and
    management policies and procedures and (2) whether BSM has used effective
    requirements development and management practices for key systems
    development efforts.

    To accomplish our objectives, we reviewed BSM requirements development and
    management policies, guidance, procedures, and tools. We also analyzed two
    completed and one ongoing BSM systems development projects2 and
    interviewed appropriate IRS and contractor officials. Further details on
    our objectives, scope, and methodology are provided in appendix I. Our
    work was performed from June 2005 through February 2006 in accordance with
    generally accepted government auditing standards.

    Results in Brief

    BSM does not yet have adequate policies and procedures in place to guide
    its systems modernization projects in developing and managing
    requirements. In January 2006, the RMO developed a set of draft policies
    that address key areas of requirements development and management; these
    policies are to serve as interim guidance while the final policies and
    processes are being developed. At the conclusion of our review, BSM
    provided us the draft policies and a high-level plan that includes
    milestones for completing these policies. Since critical BSM projects
    continue to be pursued and completion of the policies and procedures is
    not expected until March 2007, it is critical that BSM immediately
    implement the draft policies and continue to develop the final policies.

    As a result of the lack of policies and procedures, BSM projects did not
    consistently follow disciplined requirements development and management
    practices. For example, all three projects had a change management process
    in place that requires approvals and impact assessments to be completed
    when there were changes to requirements. However, none of the projects met
    all of the practices needed for effective requirements management. For
    example, two did not implement all needed practices in eliciting
    (gathering) requirements; two did not have fully documented requirements;
    and two could not produce fully traceable requirements (i.e., the
    requirements could not be tracked through development and testing). Unless
    BSM takes the steps needed to develop and institutionalize disciplined
    requirements development and management processes and to improve interim
    guidance, it will continue to face risks, including cost overruns,
    schedule delays, and performance shortfalls.

    We are recommending that the Commissioner of Internal Revenue direct the
    Associate Chief Information Officer for BSM to ensure that BSM completes
    the delivery of policies and procedures for requirements development and
    management, as planned. In addition, we also recommend that the interim
    policies for BSM be immediately implemented while the final policies and
    procedures are being developed.

    In providing written comments on a draft to this report, the Commissioner
    of Internal Revenue agreed with our findings and stated that the report
    provided a sound and balanced representation of the progress IRS has made
    to date as well as work that remains to be completed. The Commissioner
    also described the actions that IRS is taking to implement our
    recommendations, including establishing a schedule to complete the
    development of policies that address the areas of requirements
    elicitation, documentation, verification and validation, and management.
    The Commissioner's written comments are reprinted in appendix III.

    Background

    IRS is currently replacing its antiquated tax administration and financial
    systems. This effort, as we have reported numerous times,3 has suffered
    delays and cost overruns due to a number of reasons, including inadequate
    development and management of requirements.

    History of IRS Modernization

    The IRS tax administration system, which collects approximately $2
    trillion in annual revenues, is critically dependent on a collection of
    obsolete computer systems. Congress and IRS designed the BSM program to
    bring IRS tax administration systems to a level equivalent to private and
    public sector best practices, while managing the risks inherent in one of
    the largest, most visible, and sensitive modernization programs under way.
    Over the past 7 years, IRS has been appropriated approximately $1.9
    billion for BSM (see fig. 1).

    Figure 1: IRS BSM Funding Fiscal Year 1999 to Fiscal Year 2005

    BSM is critical to supporting IRS's taxpayer service and enforcement
    goals. For example, BSM includes projects to allow taxpayers to file and
    retrieve information electronically and to help reduce the backlog of
    collection cases. It also provides IRS with the reliable and timely
    financial management information it routinely needs to account for the
    nation's largest revenue stream.

    BSM has had some recent successes with its modernization efforts. During
    2004, IRS implemented initial versions of (1) Modernized e-File (MeF),
    which provides electronic filing for large corporations and tax-exempt
    organizations; (2) e-Services, which created a Web portal and other
    electronic services to promote the goal of conducting most IRS
    transactions with taxpayers and tax practitioners electronically; (3)
    Customer Account Data Engine (CADE), which will replace the current system
    that contains the agency's repository of taxpayer information; and (4) the
    Integrated Financial System, which replaced aspects of IRS's core
    financial systems and is ultimately intended to operate as its new
    accounting system of record.

    However, despite these successes, IRS has had difficulty developing and
    managing requirements for its modernization efforts over the years. We
    reported in 19954 that IRS did not have the requisite software development
    capability to successfully complete a major modernization effort and that
    the success of modernization would depend on whether IRS would promptly
    address the weaknesses in several software development areas, including
    requirements management. In 1998, we assessed IRS's systems life cycle
    document and reported5 a lack of sufficient information to document how
    business requirements were to be specified. More recently, in February and
    November of 2004, we reported in testimony and a report6 that cost
    overruns in various BSM projects, including CADE, MeF, and e-Services,
    were due in part to inadequate definition of requirements for their new
    systems, leading to incorporation of additional requirements late in the
    system's life cycle and at a higher cost than if they had been included in
    the initial requirements baseline. We continue to highlight management
    control weaknesses such as these in our annual expenditure plan reviews.7

    Other organizations that have assessed BSM projects have also found that
    IRS has not developed and managed requirements sufficiently on various
    projects. In 2001, the Treasury Inspector General for Tax Administration
    (TIGTA) reviewed8 key systems development practices of four BSM projects9
    and reported that weaknesses in several process areas, including
    requirements management, were responsible for cost increases and schedule
    delays. TIGTA noted that these weaknesses raised the risk that systems
    would be developed that would not meet the needs of the businesses they
    were intended to support and recommended that BSM strengthen and/or
    implement aspects of these key systems development practices. In 2003, an
    independent technical assessment10 of CADE noted significant breakdowns in
    developing and managing requirements, which resulted in the inability of
    CADE to meet its original schedule. The assessment further stated that IRS
    focused primarily on the high-level business requirements and paid less
    attention to the development of specific, testable requirements developed
    later in the development life cycle and that responsibility for developing
    and managing the requirements was distributed among their various
    organizational component, instead of being concentrated in a centralized
    authority.

    BSM has acknowledged that it has weaknesses in developing and managing
    requirements; since 2004, requirements management has been one of its
    high-priority initiatives.11 To demonstrate their commitment to improving
    the development and management of requirements, it created an RMO in
    October 2004. This office is to address issues related to (1) lack of
    quality and completeness of modernization requirements, (2) lack of
    alignment of modernization requirements with business strategy and needs,
    (3) risks incurred by projects transitioning to development without a
    sufficient requirements baseline, and (4) lack of visibility into a fully
    traceable set of modernization requirements. During 2005, the RMO created
    a Concept of Operations that showed, at a high level, the RMO's plans to
    address requirements practices, and, in November 2005, it obtained
    contractor support to develop new policies and procedures.

    Requirements Development and Management

    According to the Software Engineering Institute's (SEI) Capability
    Maturity Model Integration12 (CMMIsm), the requirements for a system
    describe the functionality needed to meet user needs and perform as
    intended in the operational environment. A disciplined process for
    developing13 and managing14 requirements can help reduce the risks of
    developing or acquiring a system. A well-defined and managed requirements
    baseline can, in addition, improve understanding among stakeholders and
    increase stakeholder buy-in and acceptance of the resulting system. The
    practices underlying requirements development and management include
    eliciting, documenting, verifying and validating, and managing the
    requirements through the systems life cycle (see fig. 2). This set of
    activities translates customer needs from statements of high-level
    business requirements into validated, testable systems requirements.

    Figure 2: Requirements Development and Management Process and Typical Work
    Products/Activities

    Requirements Development and Management Processes

    Elicitation

    The requirements development process starts with project teams eliciting,
    or gathering, requirements from stakeholders or participants involved in
    the project (e.g., customers and users). Since the usefulness of the
    system to its users and stakeholders is critically dependent on the
    accuracy and completeness of the requirements, all user groups and
    stakeholders should be identified and involved in defining requirements.
    In addition to gathering requirements from users and other stakeholders,
    analysis and/or research can be used to identify additional requirements
    that balance stakeholder needs against constraints and ensure that the
    requirements can be met in the proposed operational environment.

    Documentation

    After requirements have been elicited, they are analyzed in detail;
    documented as the business, or high-level, requirements; and agreed to by
    all stakeholders. Stakeholder agreement is an important part of this
    activity and is needed to demonstrate that the requirements accurately
    define intended uses. The business requirements should then be decomposed
    into detailed system requirements, which are analyzed to ensure that they
    can be implemented in the expected operational environment and that they
    can satisfy the objectives of higher-level requirements. The final
    requirements are approved by all stakeholders and documented as the
    requirements baseline. Once the baseline is established, it is placed
    under configuration management (CM)15 control.

    Verification and Validation

    Once the requirements baseline has been developed, the requirements are
    analyzed and broken down into more specific system-level requirements and
    eventually into the code that makes up the system. The verification
    process ensures that the system-level requirements and the resulting code
    are an accurate representation of stakeholder needs. This process includes
    checking selected work products, such as software code, against the
    initial baseline requirements to ensure that the lower-level items fully
    satisfy the higher-level requirements. It is an inherently incremental
    process, occurring throughout the development of the product. This
    agreement between work products, such as code and baseline requirements,
    is verified by conducting peer reviews. Peer reviews can also be used to
    identify action items that need to be addressed. Without such reviews, an
    organization is taking a risk that substantial defects will not be
    detected until late in the development and/or testing phases, or even
    after the system is implemented.

    While the system is being developed, each component must be tested to
    ensure that its outputs are accurate. Testing (e.g., unit, system
    integration, and user acceptance) is the process of executing a program
    with the intent of finding errors. Clear, complete, and well-documented
    requirements are needed in order to design and implement an effective
    testing program. Linking the testing activities back to the requirements
    assures the organization that, once testing activities are successfully
    completed, all requirements have been addressed and will be met by the
    system. Without such assurance, it is possible for a requirement to be
    missed in development and the resulting lack of functionality not noticed
    until late in testing, or even after deployment.

    Management

    Requirements, once developed and approved, also need to be managed
    throughout the system life cycle. Two key areas of requirements management
    are addressing changes to requirements and establishing and maintaining
    bidirectional traceability from high-level requirements through detailed
    work products to test cases and scenarios. As mentioned earlier, once a
    set of high-level requirements is documented, verified, and approved, it
    is placed under configuration control. From this point, changes to the
    requirements are evaluated and validated as part of the

    change control process.16 Change control includes reviewing and assessing
    proposed changes to the requirements to determine the reasons for the
    changes, determining if these changes are occurring due to flaws in the
    requirements development process, and ensuring that any effects of the
    change on other requirements as well as on the cost, schedule, and
    performance goals of the project are determined and assessed.

    Establishing and maintaining traceability from initial requirements to
    work products and the resulting system is also important. A requirements
    traceability matrix demonstrates forward and backward (bidirectional)
    traceability from business requirements to detailed system requirements
    all the way through to test cases.

    BSM Lacks Policies and Procedures for Requirements Management

    BSM does not yet have adequate policies and procedures in place to guide
    its systems modernization projects in developing and managing
    requirements. In January 2006, the RMO developed a set of draft policies
    that address key areas of requirements development and management; these
    policies are to serve as interim guidance while the final policies and
    processes are being developed. At the conclusion of our review, the RMO
    provided us the draft policies and a high-level plan that includes
    milestones for completing these policies. Since critical BSM projects
    continue to be pursued and completion of the policies and procedures is
    not expected until March 2007, it is critical that BSM immediately
    implement the draft policies and continue to develop the final policies.

    BSM Lacks Comprehensive Policies and Procedures for Requirements
    Development and Management, but Has Initiated Their Development

    BSM does not have comprehensive, detailed policies and procedures for
    requirements management and development activities that include
    requirements elicitation, documentation, verification and validation, and
    management. During our review, BSM officials were unable to provide us
    with detailed policies and procedures and agreed that they do not have
    such documents. Project teams were not consistent in their understanding
    of which guidance they should use for the development and management of
    requirements; some project team members mentioned BSM's Enterprise

    Life Cycle17 (ELC); others said they were waiting for guidance from the
    RMO. Our review of the ELC showed that it did not provide the procedures
    project managers would need to properly perform the steps in the
    requirements development and management process. BSM program officials
    agreed that the ELC did not provide the needed guidance.

    In December 2005, the RMO completed an analysis of requirements
    development and management areas that need improvement. The RMO found, as
    we did, that BSM lacks detailed guidance; their recommendations included
    developing process handbooks for aspects of requirements elicitation,
    documentation, verification and validation, and management. Subsequently,
    in January 2006, BSM officials developed draft guidance that covers
    aspects of requirements development and management. However, this guidance
    does not fully address requirements elicitation, documentation,
    verification and validation, and management. At that time, BSM also
    provided us with a high-level plan that contains interim milestones and
    establishes a March 2007 completion date for the final set of policies and
    procedures. BSM officials told us that these draft policies are to serve
    as interim guidance while the remaining policies and procedures are being
    developed. In addition, IRS also uses its governance processes,
    particularly the milestone exit reviews, to find and mitigate issues and
    problems with requirements development and management on existing
    projects. Finally, the RMO is allocating resources to key projects-such as
    F&PC version 1.2-to assist them in developing requirements.

    Without a formal set of documents that detail organizational policies and
    associated procedures, employees and contractors will rely on their
    individual knowledge and expertise to complete requirements development
    and management activities. This raises the risk of cost overruns, schedule
    delays, and reduction of functionality. Since critical BSM projects are
    already under way, and completion of the policies and procedures is not
    set until March 2007, it is urgent that BSM immediately implement the
    draft policies. Until BSM develops and implements policies and procedures
    that fully address the key areas of requirements development and
    management, including elicitation, documentation, tracking of cost and
    schedule impacts associated with requirements changes, and establishing
    and maintaining full bidirectional traceability, ongoing projects will
    continue to run greater risk of cost and schedule overruns and poor system
    performance.

    BSM Projects Have Not Consistently Followed Disciplined Requirements
    Development and Management Practices

    As a result of the lack of policies and procedures, BSM projects varied in
    the extent to which they followed disciplined requirements practices. All
    three projects we reviewed-MeF release 3.2 (to be deployed March 2006),
    and F&PC release 1.1 (deployed January 2006), and CADE release 1.1
    (deployed July 2004)-performed some of the practices associated with sound
    requirements development and management. For example, all three projects
    had a change management process in place that requires approvals and
    impact assessments to be completed when changes are made to requirements.
    However, none of the three BSM project releases we reviewed consistently
    performed all of the practices needed for effective requirements
    management. Specifically:

    o Project teams did not have a clear, documented, and consistent method of
    eliciting requirements.

    o Project teams did not adequately document all requirements.

    o Project teams did not effectively verify requirements.

    o Project teams did not demonstrate adequate management of requirements.

    Table 1: Requirements Activities Completed on BSM Projects

                       Project                     MeF 3.2    F&PC 1.1   CADE 1.1
    Eliciting requirements
    Documenting requirements
    Verifying and validating requirements
    Managing requirements

    practice fully implemented

    practice partially implemented

    practice not implemented

    Source: GAO analysis of BSM data.

    Project Teams Have Inconsistent Requirements Elicitation Practices

    Based on stakeholder information such as customer expectations,
    constraints, and interfaces for a system, the requirements elicitation
    team discovers, defines, refines, and documents business-level
    requirements. Due to the importance of this activity, plans or strategies
    should be in place to guide project officials in defining
    elicitation-related activities and in outlining how the requirements will
    be gathered (e.g., interviewing the users or analyzing the current or
    expected business processes).

    BSM project teams did not have a clear, consistent, and documented method
    of eliciting requirements for the projects. For example, although the
    teams identified stakeholders in their project plans, only MeF 3.2
    provided evidence that working group meetings were conducted with
    stakeholders to understand their needs and to identify their problems and
    expectations and that strategies or plans were developed for eliciting
    requirements. CADE 1.1 project team members could not describe how they
    elicited requirements or provide a requirements plan that documented
    elicitation procedures or strategies. An F&PC project team member stated
    that, for release 1.1, the project did not have a fully documented process
    for elicitation; however, the team member stated that the team had held
    workshops and obtained resources and assistance from the RMO to help
    mitigate the lack of a process. The RMO used lessons learned from this
    effort to develop a new requirements elicitation process, which they
    expect will assist F&PC in elicitation for its next release.

    BSM project and program officials agreed that requirements elicitation
    processes could be improved and stated that they were planning to address
    some of the problems we found. For example, when we asked project
    officials about the policies and procedures underlying their current
    requirements elicitation activities, some stated that they were waiting
    for new policies to be issued by the RMO, and others cited the ELC as
    guidance. As mentioned earlier, the ELC does not provide the information
    needed for the requirements elicitation process. Furthermore, F&PC
    officials could not state which sections in the ELC described the
    requirements elicitation process. BSM program management and RMO officials
    acknowledged the lack of policies and procedures and stated that the RMO
    has since developed new guidance for eliciting requirements that will be
    piloted on F&PC version 1.2, which is currently entering the requirements
    development phase.

    BSM project teams performed elicitation processes in a nonstandard manner
    due to the lack of policies, procedures, and guidance. Without
    standardized policies and procedures to guide this key part of
    requirements development, BSM program officials cannot ensure that its
    systems development projects have collected and documented all the
    necessary requirements, which could result in systems being developed that
    do not meet user needs.

    Projects Do Not Fully Document Requirements

    After collecting and documenting high-level requirements from customers,
    users, and other stakeholders, the requirements team should analyze these
    high-level requirements against the conceptual (or expected) operational
    environment to balance user needs and constraints and to ensure that the
    system developed will perform as intended. The resulting lower-level
    requirements should also be analyzed to make sure they can be performed in
    the expected environment and that they satisfy the objectives of the
    higher-level requirements. The final requirements are documented in the
    requirements baseline.

    The BSM projects we reviewed did not complete all of the activities needed
    to adequately document requirements. Although project teams provided
    evidence that they created a set of high-level requirements and obtained
    approvals from stakeholders on this set of requirements, two of the three
    projects did not provide evidence that requirements were thoroughly
    analyzed and decomposed to lower-level system requirements. For example,
    part of this analysis would link all lower-level systems requirements to
    the original higher-level business requirements. Only one of the project
    teams-F&PC-provided documentation showing the necessary linking or mapping
    of lower-level system requirements to the business requirements. MeF and
    CADE provided documentation; however, their documentation did not fully
    demonstrate the linking of system-level requirements to the business
    requirements.

    A MeF project official agreed that full linkage of system-level
    requirements to business requirements should be implemented. The MeF
    official stated that they planned to implement this in their next
    version-version 4.0. In addition, a BSM program official indicated that
    additional project guidance on requirements documentation would be part of
    the RMO's deliverables and would help to address this issue.

    Without feasible and clearly defined requirements, projects run the risk
    of cost overruns, schedule delays, and deployment of systems with limited
    functionality. For example, incomplete definition of requirements has been
    cited as one reason for schedule delays and cost overruns for both CADE
    and MeF.

    Projects Do Not Fully Verify Requirements; Validation Activities
    Completed, but Problems Remain

    Once requirements are fully documented, software code and other work
    products that will guide development and testing activities need to be
    verified using peer review techniques against the original requirements.
    In addition, these products should be validated through testing to ensure
    that they will operate effectively in the intended environment.

    Projects Do Not Fully Verify Requirements through Peer Reviews

    Requirements verification ensures that the lower-level requirements,
    software code, and other work products that will guide development and
    testing activities are an accurate representation of stakeholder needs.
    Peer reviews are an important part of the verification process and are a
    proven mechanism for effective defect removal. During peer reviews, teams
    of peers18 examine code and other work products to identify defects,
    determine the causes of the defects, and make recommendations that address
    changes needed to help ensure that the system will meet stakeholder and
    developer needs. Peer reviews should follow a structured, formalized
    process; peer review events should be planned in advance, with items, such
    as code and other work products, selected in advance; the results of the
    sessions should be incorporated into peer review reports that project
    teams are expected to address before moving further into development
    activities.

    The BSM project teams did not provide evidence that work products had been
    verified against requirements through the use of a formalized peer review
    process and project officials did not follow recommended practices for
    conducting peer reviews. BSM project team members stated that they
    conducted customer technical reviews and milestone exit reviews that they
    considered to be peer reviews; however, these kinds of reviews do not meet
    the criteria for peer reviews. They were not structured, did not select
    code and other items in advance to be evaluated, and did not produce
    formal peer reports with action items that projects were required to
    address.

    Project Teams Performing Validation, but Problems Remain

    Requirements validation is the process of demonstrating that a product
    fulfills its intended use in its environment. It differs from the
    verification activities previously described, in that validation
    determines that the product will fulfill its intended use, while
    verification ensures that work products properly reflect the baseline
    requirements. Validation includes tests conducted on the product during
    development to prove that the product performs its intended functions
    correctly. In a disciplined software development process, planning for
    validation activities begins as requirements are developed; testing
    activities are critical to determining that a system not only operates
    effectively but addresses all baseline requirements. To complete
    validation activities, testing is conducted at several levels, each of
    which validates that the system will operate effectively at a different
    level. For example, unit testing validates individual sections of code,
    and system integration testing ensures that the system as a whole can
    operate effectively in its environment. User acceptance testing allows the
    user community to determine whether the system, as developed, can be used
    to effectively support their work. It also validates that the system meets
    user expectations. An effective testing process confirms the functionality
    and performance of the product prior to delivery. It is a crucial process
    and needs to be well planned, well structured, well documented, and
    carried out in a controlled and managed way.

    The BSM projects provided evidence of validation activities, such as test
    plans and test results. CADE 1.1 officials provided both test plans and
    test reports. MeF release 3.2 and F&PC release 1.1 are still in the
    testing phase; they provided available test plans but do not yet have test
    reports.

    Despite the existence of test plans and reports, requirements are not
    fully documented or fully traced. In addition, while the ELC provides
    guidance on testing that discusses test planning, activities, and test
    responsibilities, program officials say that this guidance is limited.
    BSM's Enterprise Services organization has initiated an effort to review,
    revise, and enhance test procedures across BSM. Therefore, until BSM
    ensures full documentation and traceability of requirements, questions
    about the completeness of its testing will remain.

    Project Teams Not Adequately Managing Requirements

    Finally, requirements must be managed through the system development life
    cycle. We found that the three projects did not fully demonstrate adequate
    management of their requirements. Although the projects had a formal
    change control process in place to analyze and manage changes to
    requirements, associated costs and schedule changes resulting from
    requirements changes were not always tracked or updated. In addition,
    projects' documentation did not demonstrate adequate traceability of
    requirements from the business requirements (high-level requirements) to
    system requirements (low-level requirements) to test cases.

    Project Teams Not Updating Cost and Schedule to Reflect Requirements
    Changes

    Managing changes to the original requirements is a formal process to
    identify, evaluate, track, report, and approve these changes. As work
    products are developed and more is learned about the system that is being
    developed, information is occasionally found that requires a change to the
    original requirements. Modifications to project scope or design can also
    result in requirements changes. Therefore, projects need to manage these
    changes to requirements in a structured way.

    The BSM project teams used a change management process to manage changes
    to requirements that included documenting the rationale for changes,
    developing assessments of the impact of the change, and obtaining
    approvals by the Configuration Change Board. However, only the MeF and
    F&PC teams provided evidence that their cost and schedule baselines were
    updated when changes to requirements impacted cost and/or schedule. For
    example, CADE officials did not provide any evidence to show how they
    updated and tracked cost changes resulting from changes to requirements,
    nor did they provide evidence that the work breakdown structure19 was
    updated to reflect schedule changes. F&PC officials provided evidence for
    tracking changes to the cost and schedule. MeF officials provided a
    document that tracked the cost implications of changes to requirements and
    the work breakdown structure to reflect schedule changes.

    A BSM project official indicated that the BSM project was implementing
    cost and schedule tracking on its current releases. However, it was not
    clear whether BSM was doing this consistently or whether appropriate
    guidance for tracking cost and schedule would be provided by BSM.

    Project teams that do not effectively track cost and schedule changes as a
    result of changes to requirements will not be able to effectively mitigate
    the potential impact of these changes to overall cost, schedule, and
    performance goals, thus raising the risk of cost overruns, schedule
    delays, and deferral of functionality.

    Project Teams Not Ensuring Full Traceability of Requirements

    Another key element of requirements management is establishing and
    maintaining the traceability of requirements. Traceability of requirements
    is tracking the requirements from the inception of the project and
    agreement on a specific set of business requirements to development of the
    lower-level system requirements, detailed design, code implementation, and
    test cases necessary for validating the requirements. Tracing a
    requirement throughout the development cycle provides evidence that the
    requirements are met in the developed system and ensures that the product
    or system will work as intended. Requirements must be traceable forward
    and backward through the life cycle. Each business requirement must be
    traceable to associated system requirements and test cases. Without
    adequate traceability, errors in functionality could occur and not be
    found until the testing phase, when problems are more costly to fix and
    time frames for fixing problems without causing a schedule slip for
    deployment are limited.

    Of the three projects, only the F&PC team showed evidence of full
    traceability of the requirements from high-level requirements to low-level
    requirements. MeF and CADE documentation did not demonstrate clear
    traceability from the business requirements to lower-level system
    requirements, coding, and test cases. MeF program officials acknowledged
    weaknesses in this area and stated that they planned to develop full
    bidirectional traceability to the business level requirements as part of
    MeF release 4.0.

    According to project officials, one reason they do not have full
    bidirectional traceability is due to the lack of detailed procedures and
    guidance for traceability of requirements. Until recently, BSM projects
    were not required to develop and use a traceability matrix. While interim
    guidance issued by BSM does require the use of traceability matrices and
    use of its configuration management repository to manage requirements, the
    guidance lacks the detail needed to ensure that projects meet criteria.
    BSM program officials agreed that this was an area that needed additional
    guidance. The RMO is currently reviewing new guidance on how to improve
    requirements traceability.

    Without adequate traceability of requirements, system requirements can be
    missed during development and the agency cannot be assured that validation
    activities fully demonstrate that all the agreed-upon requirements have
    been developed, fully tested, and will work as intended.

    Conclusions

    BSM lacks policies and procedures to develop and manage requirements for
    their systems modernization projects. BSM has acknowledged this deficiency
    since late 2004 when it listed requirements management as one of its
    high-priority initiatives and created an RMO. The office has now developed
    draft policies that cover aspects of eliciting, documenting, verifying and
    validating, and managing requirements. These draft policies are to serve
    as guidance to projects teams as BSM projects are pursued. It is critical
    that BSM implement these draft policies immediately and continue to
    develop the remaining policies.

    The three BSM development projects that we reviewed showed significant
    differences in how they implemented practices for developing and managing
    requirements. Until BSM and the RMO complete the development of policies
    and procedures to ensure disciplined requirements development and
    management practices, projects will not have sufficient guidance to ensure
    implementation of these practices, which will impair their ability to
    effectively manage the development and acquisition of critical systems and
    increase the risk of cost overruns, schedule delays, and deferral of
    functionality.

    Recommendations for Executive Action

    To improve the requirements development and management policies and
    practices of the IRS's BSM, we recommend that the Commissioner of Internal
    Revenue direct the Associate Chief Information Officer for BSM to take the
    following two actions:

    1.Ensure that BSM completes the delivery of policies and procedures for
    requirements development and management as planned. The policies and
    procedures should fully describe the processes, include a minimum set of
    activities required for each project, and provide detailed procedures for
    each of the key areas of requirements elicitation, documentation,
    verification and validation, and management. As part of this effort, the
    policies and procedures should specifically include the following:

    o A standardized process for the elicitation of requirements that ensures
    that projects fully investigate the requirements needed for a specific
    system, including gathering requirements from all relevant users and
    stakeholders.

    o A standardized process for the documentation of requirements that
    ensures full documentation of the baseline requirements.

    o A process for ensuring formal peer reviews are planned and completed for
    key products.

    o Guidance on tracking cost and schedule impacts of changes to
    requirements for all projects.

    o Guidance on establishing and maintaining full bidirectional requirements
    traceability.

    2.In addition, since BSM has ongoing projects that are developing and
    managing requirements and the development of new policies and procedures
    is not scheduled to be complete until March 2007, the Commissioner should
    direct the Associate CIO for BSM to immediately implement its draft
    policies while the final policies and procedures are being developed.

    Agency Comments

    In providing written comments on a draft to this report, the Commissioner
    of Internal Revenue agreed with our findings and stated that the report
    provided a sound and balanced representation of the progress IRS has made
    to date as well as work that remains to be completed. The Commissioner
    also described the actions that IRS is taking to implement our
    recommendations, including establishing a schedule to complete the
    development of policies that address the areas of requirements
    elicitation, documentation, verification and validation, and management.
    The Commissioner's written comments are reprinted in appendix III.

    As agreed with your office, unless you publicly announce the contents of
    this report earlier, we plan no further distribution until 30 days from
    the date of this letter. At that time, we will send copies of this report
    to the Chairmen and Ranking Members of other Senate and House committees
    and subcommittees that have appropriation, authorization, and oversight
    responsibilities for IRS. We are also sending copies to the Commissioner
    of Internal Revenue, the Secretary of the Treasury, the Chairman of the
    BSM Oversight Board, and the Director of the Office of Management and
    Budget. Copies are also available at no charge on the GAO Web site at
    http://www.gao.gov.

    report, please contact David A. Powner at (202) 512-9286 or p
    ownerd@gao.gov or Keith A. Rhodes at (202) 512-6412 or r  hodesk@gao.gov.
    Contact points for our Offices of Congressional Relations and Public
    Affairs may be found on the last page of this report. GAO staff who made
    major contributions to this report are listed in appendix III.

    Sincerely yours,

    David A. Powner Director, Information Technology Management Issues

    Keith A. Rhodes, Chief Technologist, Center for Technology and Engineering

    Appendix I

    Scope and Methodology

    The objectives of our review were to assess (1) whether the Requirements
    Management Office (RMO) has established adequate requirements development
    and management policies and procedures and (2) whether the Business
    Systems Modernization (BSM) has effectively used requirements development
    and management practices for key systems development efforts.

    To assess the adequacy of BSM's requirements development and management
    policies and procedures, including IRS's Enterprise Life Cycle (ELC),1 we
    compared it against criteria based on industry standards and best
    practices, including the Software Engineering Institute's (SEI) Capability
    Maturity Model Integration (CMMIsm) version 1.1. We also reviewed draft
    policies and procedures provided by the RMO in February 2006 and compared
    them against this criteria. In addition, we interviewed appropriate BSM
    officials to discuss the creation and goals of the RMO and whether there
    were BSM requirements development and management policies and procedures
    in place.

    To assess whether BSM project teams effectively used requirements
    development and management practices on its systems acquisitions, we
    selected three BSM projects to review: (1) Modernized e-File (MeF) release
    3.2, which is to be deployed in March 2006; (2) Filing and Payment
    Compliance (F&PC) release 1.1, which was deployed in January 2006; and
    Customer Account Data Engine (CADE) Individual Master File release 1.1,
    which was deployed July 2004. We selected these investments because they
    were (1) important to the goals and mission of the agency, (2) large-scale
    development efforts with significant costs, and (3) at different points in
    their development life cycles.

    To evaluate whether each of the three projects had effectively used
    requirements development and management practices for key systems
    development efforts, we compared the project's documentation and processes
    against criteria based on industry standards and best practices, including
    SEI's CMMIsm version 1.1. The documentation reviewed for each of the
    projects included requirements management plans, traceability matrices,
    testing plans, baseline requirements, and other items. We also interviewed
    the program officials from each of these three projects to further clarify
    issues on their requirements development and management activities.

    Our work was performed from June 2005 to February 2006 in Washington D.C.,
    in accordance with generally accepted government auditing standards.

    Appendix II

    Project Descriptions

    The following are descriptions of the three projects we selected to
    review: Modernized e-File (MeF) release 3.2, Filing and Payment Compliance
    (F&PC) release 1.1, and Customer Account Data Engine (CADE) release 1.1.

    Modernized e-File

    In fiscal year 2004, BSM introduced the Modernized e-File (MeF) system,
    which allows e-filing for tax-exempt organizations and large corporations
    and reduces the time to process their tax forms. The goal for MeF is to
    replace the current e-filing technology with a modernized, Internet-based
    electronic filing system. MeF is also expected to result in an increase in
    the use of electronic filing because it is efficient and easy to access,
    use, and maintain.

    Projected benefits of the MeF program are as follows:

    o Reduction in the BSM's effort associated with receiving, processing,
    manually entering data, and resolving data entry errors from paper
    returns;

    o Reduction in system maintenance costs;

    o Savings in time and money for taxpayers and tax practitioners due to not
    copying, assembling, and mailing a return; and

    o Sharing of tax and information return data electronically throughout
    state agencies.

    The MeF project is projected to provide the capability for Internet-based
    filing of 330 different BSM forms. The following is table 2, describing
    MeF releases deployed and their functionalities, followed by table 3,
    which describes MeF financial data.

    Table 2: MeF Releases and Functionality

         Release                           Functionality
    Release 1   Deployed in February 2004

                   Provides 53 forms and schedules for 1120/1120S and 5 forms for
                   990 e-filing, along with the functionality to support those
                   forms, including applicable interfaces, validations, retrieval
                   and display options, the capability for large taxpayers to
                   file using the Internet, and the capability to attach Adobe
                   files.
    Release 2   Deployed in August 2004

                   Provides the remaining 43 forms and schedules for 1120/1120S
                   and required public access (access to redacted information for
                   nonprofit organizations) to the filed 990s.
    Release 3.1 Deployed in January 2005

                   Provides 990PF series forms, Form 7004 (Application for
                   Automatic Extension of Time to File Corporation Return), and
                   M-TRDB processing codes.
    Release 3.2 Projected deployment in January 2006

                   Expected to provide the Federal/State Single Point Filing
                   System platform and retrieval system for all state, corporate,
                   and tax-exempt tax returns and acknowledgments.

    Source: IRS.

    Table 3: Development and Steady State Costs for MeF

          Dollars in millions
              Fiscal year              Development       Steady state       Total
    FY 04                                 $57.1               $2.3       $59.4
    FY 05                                 $67.1               $5.2       $72.3
    FY 06                                 $60.6               $5.6       $66.2

    Source: IRS.

    Filing and Payment Compliance

    The Filing and Payment Compliance (F&PC) project is intended to improve
    technologies and processes that support BSM's compliance activities.
    According to BSM, their collection operations rely on 30-year-old
    technology and processes that are no longer compatible with the realities
    of today's taxpayer environment. F&PC plans to provide support for
    detecting, scoring, and working nonfiler and payment delinquency cases. It
    is to use advanced software to analyze tax collection cases and divide
    them into cases that require BSM involvement and those that can be handled
    by private collection agencies. Case attributes are to be identified,
    segmented, and prioritized to select the individual taxpayer cases that
    have a greater probability of paying the tax liabilities in full or
    through installment agreements.

    The F&PC project is also to serve as an inventory management system to
    assign, exchange, monitor, control, and update delinquent taxpayer
    accounts between the BSM Authoritative Data Source and the private
    collection agencies with whom BSM will contract.

    The F&PC project is expected to increase the following:

    o collection case closures by 10 million annually by 2014,

    o voluntary taxpayer compliance, and

    o BSM's capacity to resolve the buildup of delinquent taxpayer cases.

    The BSM intends to deliver an initial limited private debt collection
    capability in January 2006. Full implementation of this aspect of the F&PC
    is projected to be completed by January 2008 with additional functionality
    to follow in later years. Following is table 4, describing F&PC releases
    deployed and their functionality, followed by table 5, which describes
    F&PC financial data.

    Table 4: F&PC Releases and Functionality

         Release                           Functionality
    Release 1.1 Projected deployment in January 2006

                   Expected to provide limited functionality of commercial-off
                   the-shelf (COTS) software with many manual processes employed
                   by BSM, small case volumes and minimal number of private
                   collection agencies supported.
    Release 1.2 Projected deployment in January 2007

                   Expected to provide full implementation, testing, and
                   deployment of inventory management capabilities using the BSM
                   infrastructure.
    Release 1.3 Projected deployment in January 2008

                   Expected to provide full COTS software functionality, private
                   collection agencies fully installed, electronic data transfer
                   between them fully established and operational.

    Source: IRS.

    Table 5: Development Costs for F&PC

          Dollars in millions
              Fiscal year              Development       Steady state       Total
    FY 04                                  $0.4               $0.0        $0.4
    FY 05                                  $0.0               $0.0        $0.0
    FY 06                                  $5.9               $0.0        $5.9

    Source: IRS.

    Customer Account Data Engine

    The Customer Account Data Engine (CADE), intended to replace BSM's
    antiquated tax administration system, is BSM's highest priority project
    and is intended to house tax information for more than 200 million
    individual and business taxpayers. The CADE databases and related
    applications are also to enable the implementation of other systems that
    will improve customer service and compliance and allow the online posting
    and updating of taxpayer account and return data.

    The CADE project is intended to

    o generate refund notices, detect potential fraudulent transactions, and
    calculate taxes;

    o replace the group of BSM tax master files with a single database-the Tax
    Account Data Store;

    o accept, validate, and store taxpayer return and account data, along with
    financial account activity data, such as tax payments, liabilities, and
    installment agreements; and

    o enable future business application systems.

    In July 2004 and January 2005, BSM implemented the initial releases of
    CADE, which have been used to process Form 1040EZ returns. CADE posted
    more than 1.4 million returns for filing season 2005 and generated more
    than $427 million in refunds. In 2006, CADE is expected to expand the
    number and type of returns beyond the Form 1040EZ. BSM is also projecting
    that CADE will process 33 million returns during 2007. Following is table
    6, describing CADE releases deployed and their functionality, followed by
    table 7 describing CADE financial data.

    Table 6: CADE Releases and Functionality

          Release                           Functionality
    Release 1.0   Deployed in July 2004

                     Provided the filing capabilities for 1040 EZ forms.
    Release 1.1   Deployed in July 2004 (concurrent with Release 1.0)

                     Provided filing season 2003 and 2004 tax law changes.
    Release 1.2   Deployed in January 2005

                     Provided filing season 2005 tax law changes.
    Release 1.3.1 Deployed in September 2005

                     Provided new functionality to improve performance and allow
                     address changes on tax returns as well as some filing season
                     2006 tax law changes.
    Release 1.3.2 Projected deployment in January 2006

                     Expected to provide the remaining filing season 2006 tax law
                     changes and some additional functionality.

    Source: IRS.

    Table 7: Development Costs for CADE

          Dollars in millions
              Fiscal year              Development       Steady state       Total
    FY 04                                $100.6               $0.0      $100.6
    FY 05                                $109.9               $0.0      $109.9
    FY 06                                $109.9               $0.0      $109.9

    Source: IRS.

    Appendix III

    Comments from the Department of the Treasury

    Appendix IV

    GAO Contact and Staff Acknowledgments

    GAO Contact

    David A. Powner, 202-512-9286, pownerd@gao.gov
    Keith A. Rhodes, 202-512-6412, rhodesk@gao.gov

    Staff Acknowledgments

    In addition to those named above, Neil Doherty, Nancy Glover, George
    Kovachick, Tonia Johnson, Tammi Nguyen, Madhav Panwar, and Rona Stillman
    made key contributions to this report.

    (310490)

    www.gao.gov/cgi-bin/getrpt? GAO-06-310 .

    To view the full product, including the scope

    and methodology, click on the link above.

    For more information, contact David A. Powner at (202) 512-9286 or
    pownerd@gao.gov or Keith A. Rhodes at (202) 512-6412 or rhodesk@gao.gov.

    Highlights of GAO-06-310 , a report to the Subcommittee on Transportation,
    Treasury, the Judiciary, HUD, and Related Agencies, Committee on
    Appropriations, U.S. Senate

    March 2006

    BUSINESS SYSTEMS MODERNIZATION

    IRS Needs to Complete Recent Efforts to Develop Policies and Procedures to
    Guide Requirements Development and Management

    The Internal Revenue Service's (IRS) effort to modernize its tax
    administrative and financial systems-Business Systems Modernization
    (BSM)-has suffered delays and cost overruns due to a number of factors,
    including inadequate development and management of requirements.
    Recognizing these deficiencies, IRS created a Requirements Management
    Office (RMO) to establish policies and procedures for managing
    requirements. GAO's objectives were to assess (1) whether the office has
    established adequate requirements development and management policies and
    procedures and (2) whether BSM has effectively used requirements
    development and management practices for key systems development efforts.

    What GAO Recommends

    To improve the requirements development and management practices at IRS,
    GAO recommends that the Commissioner of Internal Revenue direct the
    Associate Chief Information Officer for BSM to (1) ensure that BSM
    completes delivery of policies and procedures for requirements development
    and management as planned and (2) immediately implement interim policies
    for BSM projects while final policies and procedures are being developed.
    The Commissioner agreed with our recommendations and described steps taken
    to address them.

    BSM does not yet have adequate policies and procedures in place to guide
    its systems modernization projects in developing and managing
    requirements. In January 2006, the RMO developed a set of draft policies
    that address some key areas of requirements development and management;
    these policies are to serve as interim guidance while the final policies
    and processes are being developed. At the conclusion of GAO's review, the
    RMO also provided a high-level plan that includes milestones for
    completing these policies. Since critical BSM projects continue to be
    pursued and completion of the policies and procedures is not expected
    until March 2007, it is critical that BSM immediately implement the draft
    policies and continue to develop the final policies.

    As a result of the lack of policies and procedures, the one ongoing
    project-Modernized e-File (MeF)-and the two completed projects-Filing and
    Payment Compliance (F&PC) and Customer Account Data Engine (CADE)-GAO
    reviewed did not consistently follow disciplined practices for systems
    development and management (see table).

    Requirements Activities Completed on BSM Projects

    Note: MeF release 3.2 will be deployed in March 2006, F&PC release 1.1 was
    deployed January 2006, and CADE release 1.1 was deployed July 2004.

    For example, all three projects had a key element of managing
    requirements-a change management process that requires approvals and
    impact assessments to be completed when there are changes to
    requirements-but none met all of the practices needed for effective
    requirements management. In addition, two projects did not have a clear,
    consistent way to elicit (gather) requirements, two did not have fully
    documented requirements, and two could not produce fully traceable
    requirements (i.e., the requirements could not be tracked through
    development and testing), which is another key element of managing
    requirements. Unless IRS takes the steps needed to develop and
    institutionalize disciplined requirements development and management
    processes and implements draft policies in the interim to cover key areas
    of requirements development and management, it will continue to face
    risks, including cost overruns, schedule delays, and performance
    shortfalls.

    Report to the Chairman, Subcommittee on Transportation, Treasury, the
    Judiciary, HUD, and Related Agencies, Committee on Appropriations, U.S.
    Senate

    March 2006

    BUSINESS SYSTEMS MODERNIZATION

    IRS Needs to Complete Recent Efforts to Develop Policies and Procedures to
    Guide Requirements Development and Management
    *** End of document. ***