Business Systems Modernization: IRS Needs to Complete Recent
Efforts to Develop Policies and Procedures to Guide Requirements
Development and Management (20-MAR-06, GAO-06-310).
The Internal Revenue Service's (IRS) effort to modernize its tax
administrative and financial systems--Business Systems
Modernization (BSM)--has suffered delays and cost overruns due to
a number of factors, including inadequate development and
management of requirements. Recognizing these deficiencies, IRS
created a Requirements Management Office (RMO) to establish
policies and procedures for managing requirements. GAO's
objectives were to assess (1) whether the office has established
adequate requirements development and management policies and
procedures and (2) whether BSM has effectively used requirements
development and management practices for key systems development
efforts.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-06-310
ACCNO: A49457
TITLE: Business Systems Modernization: IRS Needs to Complete
Recent Efforts to Develop Policies and Procedures to Guide
Requirements Development and Management
DATE: 03/20/2006
SUBJECT: Cost overruns
Financial management systems
Program evaluation
Program management
Schedule slippages
Strategic planning
Systems management
Technology modernization programs
Policies and procedures
IRS Business Systems Modernization
Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-06-310
* Contents
* Results in Brief
* Background
* History of IRS Modernization
* Requirements Development and Management
* Requirements Development and Management Processes
* Elicitation
* Documentation
* Verification and Validation
* Management
* BSM Lacks Policies and Procedures for Requirements Management
* BSM Lacks Comprehensive Policies and Procedures for
Requirements Development and Management, but Has Initiated
Their Development
* BSM Projects Have Not Consistently Followed Disciplined
Requirements Development and Management Practices
* Project Teams Have Inconsistent Requirements Elicitation
Practices
* Projects Do Not Fully Document Requirements
* Projects Do Not Fully Verify Requirements; Validation
Activities Completed, but Problems Remain
* Projects Do Not Fully Verify Requirements through Peer
Reviews
* Project Teams Performing Validation, but Problems
Remain
* Project Teams Not Adequately Managing Requirements
* Project Teams Not Updating Cost and Schedule to Reflect
Requirements Changes
* Project Teams Not Ensuring Full Traceability of
Requirements
* Conclusions
* Recommendations for Executive Action
* Agency Comments
* Scope and Methodology
* Project Descriptions
* Modernized e-File
* Filing and Payment Compliance
* Customer Account Data Engine
* Comments from the Department of the Treasury
* GAO Contact and Staff Acknowledgments
* cov1&2corrected.pdf
* Report to the Chairman, Subcommittee on Transportation, Treasury,
the Judiciary, HUD, and Related Agencies, Committee on
Appropriations, U.S. Senate
* March 2006
* BUSINESS SYSTEMS MODERNIZATION
* IRS Needs to Complete Recent Efforts to Develop Policies and
Procedures to Guide Requirements Development and Management
Contents
Tables
Figures
March 20, 2006Letter
The Honorable Christopher S. Bond Chairman Subcommittee on Transportation,
Treasury, the Judiciary, HUD, and Related Agencies Committee on
Appropriations United States Senate
Dear Mr. Chairman:
The Internal Revenue Service (IRS) has long relied on obsolete automated
systems for key operational and management functions; its attempts to
modernize these systems span several decades. IRS's current
effort-Business Systems Modernization (BSM)-is a highly complex,
multibillion dollar effort to modernize its technology and related
business processes. Over the past 7 years, IRS has been appropriated
approximately $1.9 billion for BSM. These systems modernization projects
have experienced cost overruns and schedule delays due to, among other
things, inadequate development and management of requirements.1 Lack of
attention to these crucial processes has led to projects not meeting cost,
schedule, and performance goals.
IRS has recognized these deficiencies and established a Requirements
Management Office (RMO) to, among other things, develop the policies and
procedures that are to cover all aspects of requirements development and
management. This report responds to your request that we assess (1)
whether the RMO has established adequate requirements development and
management policies and procedures and (2) whether BSM has used effective
requirements development and management practices for key systems
development efforts.
To accomplish our objectives, we reviewed BSM requirements development and
management policies, guidance, procedures, and tools. We also analyzed two
completed and one ongoing BSM systems development projects2 and
interviewed appropriate IRS and contractor officials. Further details on
our objectives, scope, and methodology are provided in appendix I. Our
work was performed from June 2005 through February 2006 in accordance with
generally accepted government auditing standards.
Results in Brief
BSM does not yet have adequate policies and procedures in place to guide
its systems modernization projects in developing and managing
requirements. In January 2006, the RMO developed a set of draft policies
that address key areas of requirements development and management; these
policies are to serve as interim guidance while the final policies and
processes are being developed. At the conclusion of our review, BSM
provided us the draft policies and a high-level plan that includes
milestones for completing these policies. Since critical BSM projects
continue to be pursued and completion of the policies and procedures is
not expected until March 2007, it is critical that BSM immediately
implement the draft policies and continue to develop the final policies.
As a result of the lack of policies and procedures, BSM projects did not
consistently follow disciplined requirements development and management
practices. For example, all three projects had a change management process
in place that requires approvals and impact assessments to be completed
when there were changes to requirements. However, none of the projects met
all of the practices needed for effective requirements management. For
example, two did not implement all needed practices in eliciting
(gathering) requirements; two did not have fully documented requirements;
and two could not produce fully traceable requirements (i.e., the
requirements could not be tracked through development and testing). Unless
BSM takes the steps needed to develop and institutionalize disciplined
requirements development and management processes and to improve interim
guidance, it will continue to face risks, including cost overruns,
schedule delays, and performance shortfalls.
We are recommending that the Commissioner of Internal Revenue direct the
Associate Chief Information Officer for BSM to ensure that BSM completes
the delivery of policies and procedures for requirements development and
management, as planned. In addition, we also recommend that the interim
policies for BSM be immediately implemented while the final policies and
procedures are being developed.
In providing written comments on a draft to this report, the Commissioner
of Internal Revenue agreed with our findings and stated that the report
provided a sound and balanced representation of the progress IRS has made
to date as well as work that remains to be completed. The Commissioner
also described the actions that IRS is taking to implement our
recommendations, including establishing a schedule to complete the
development of policies that address the areas of requirements
elicitation, documentation, verification and validation, and management.
The Commissioner's written comments are reprinted in appendix III.
Background
IRS is currently replacing its antiquated tax administration and financial
systems. This effort, as we have reported numerous times,3 has suffered
delays and cost overruns due to a number of reasons, including inadequate
development and management of requirements.
History of IRS Modernization
The IRS tax administration system, which collects approximately $2
trillion in annual revenues, is critically dependent on a collection of
obsolete computer systems. Congress and IRS designed the BSM program to
bring IRS tax administration systems to a level equivalent to private and
public sector best practices, while managing the risks inherent in one of
the largest, most visible, and sensitive modernization programs under way.
Over the past 7 years, IRS has been appropriated approximately $1.9
billion for BSM (see fig. 1).
Figure 1: IRS BSM Funding Fiscal Year 1999 to Fiscal Year 2005
BSM is critical to supporting IRS's taxpayer service and enforcement
goals. For example, BSM includes projects to allow taxpayers to file and
retrieve information electronically and to help reduce the backlog of
collection cases. It also provides IRS with the reliable and timely
financial management information it routinely needs to account for the
nation's largest revenue stream.
BSM has had some recent successes with its modernization efforts. During
2004, IRS implemented initial versions of (1) Modernized e-File (MeF),
which provides electronic filing for large corporations and tax-exempt
organizations; (2) e-Services, which created a Web portal and other
electronic services to promote the goal of conducting most IRS
transactions with taxpayers and tax practitioners electronically; (3)
Customer Account Data Engine (CADE), which will replace the current system
that contains the agency's repository of taxpayer information; and (4) the
Integrated Financial System, which replaced aspects of IRS's core
financial systems and is ultimately intended to operate as its new
accounting system of record.
However, despite these successes, IRS has had difficulty developing and
managing requirements for its modernization efforts over the years. We
reported in 19954 that IRS did not have the requisite software development
capability to successfully complete a major modernization effort and that
the success of modernization would depend on whether IRS would promptly
address the weaknesses in several software development areas, including
requirements management. In 1998, we assessed IRS's systems life cycle
document and reported5 a lack of sufficient information to document how
business requirements were to be specified. More recently, in February and
November of 2004, we reported in testimony and a report6 that cost
overruns in various BSM projects, including CADE, MeF, and e-Services,
were due in part to inadequate definition of requirements for their new
systems, leading to incorporation of additional requirements late in the
system's life cycle and at a higher cost than if they had been included in
the initial requirements baseline. We continue to highlight management
control weaknesses such as these in our annual expenditure plan reviews.7
Other organizations that have assessed BSM projects have also found that
IRS has not developed and managed requirements sufficiently on various
projects. In 2001, the Treasury Inspector General for Tax Administration
(TIGTA) reviewed8 key systems development practices of four BSM projects9
and reported that weaknesses in several process areas, including
requirements management, were responsible for cost increases and schedule
delays. TIGTA noted that these weaknesses raised the risk that systems
would be developed that would not meet the needs of the businesses they
were intended to support and recommended that BSM strengthen and/or
implement aspects of these key systems development practices. In 2003, an
independent technical assessment10 of CADE noted significant breakdowns in
developing and managing requirements, which resulted in the inability of
CADE to meet its original schedule. The assessment further stated that IRS
focused primarily on the high-level business requirements and paid less
attention to the development of specific, testable requirements developed
later in the development life cycle and that responsibility for developing
and managing the requirements was distributed among their various
organizational component, instead of being concentrated in a centralized
authority.
BSM has acknowledged that it has weaknesses in developing and managing
requirements; since 2004, requirements management has been one of its
high-priority initiatives.11 To demonstrate their commitment to improving
the development and management of requirements, it created an RMO in
October 2004. This office is to address issues related to (1) lack of
quality and completeness of modernization requirements, (2) lack of
alignment of modernization requirements with business strategy and needs,
(3) risks incurred by projects transitioning to development without a
sufficient requirements baseline, and (4) lack of visibility into a fully
traceable set of modernization requirements. During 2005, the RMO created
a Concept of Operations that showed, at a high level, the RMO's plans to
address requirements practices, and, in November 2005, it obtained
contractor support to develop new policies and procedures.
Requirements Development and Management
According to the Software Engineering Institute's (SEI) Capability
Maturity Model Integration12 (CMMIsm), the requirements for a system
describe the functionality needed to meet user needs and perform as
intended in the operational environment. A disciplined process for
developing13 and managing14 requirements can help reduce the risks of
developing or acquiring a system. A well-defined and managed requirements
baseline can, in addition, improve understanding among stakeholders and
increase stakeholder buy-in and acceptance of the resulting system. The
practices underlying requirements development and management include
eliciting, documenting, verifying and validating, and managing the
requirements through the systems life cycle (see fig. 2). This set of
activities translates customer needs from statements of high-level
business requirements into validated, testable systems requirements.
Figure 2: Requirements Development and Management Process and Typical Work
Products/Activities
Requirements Development and Management Processes
Elicitation
The requirements development process starts with project teams eliciting,
or gathering, requirements from stakeholders or participants involved in
the project (e.g., customers and users). Since the usefulness of the
system to its users and stakeholders is critically dependent on the
accuracy and completeness of the requirements, all user groups and
stakeholders should be identified and involved in defining requirements.
In addition to gathering requirements from users and other stakeholders,
analysis and/or research can be used to identify additional requirements
that balance stakeholder needs against constraints and ensure that the
requirements can be met in the proposed operational environment.
Documentation
After requirements have been elicited, they are analyzed in detail;
documented as the business, or high-level, requirements; and agreed to by
all stakeholders. Stakeholder agreement is an important part of this
activity and is needed to demonstrate that the requirements accurately
define intended uses. The business requirements should then be decomposed
into detailed system requirements, which are analyzed to ensure that they
can be implemented in the expected operational environment and that they
can satisfy the objectives of higher-level requirements. The final
requirements are approved by all stakeholders and documented as the
requirements baseline. Once the baseline is established, it is placed
under configuration management (CM)15 control.
Verification and Validation
Once the requirements baseline has been developed, the requirements are
analyzed and broken down into more specific system-level requirements and
eventually into the code that makes up the system. The verification
process ensures that the system-level requirements and the resulting code
are an accurate representation of stakeholder needs. This process includes
checking selected work products, such as software code, against the
initial baseline requirements to ensure that the lower-level items fully
satisfy the higher-level requirements. It is an inherently incremental
process, occurring throughout the development of the product. This
agreement between work products, such as code and baseline requirements,
is verified by conducting peer reviews. Peer reviews can also be used to
identify action items that need to be addressed. Without such reviews, an
organization is taking a risk that substantial defects will not be
detected until late in the development and/or testing phases, or even
after the system is implemented.
While the system is being developed, each component must be tested to
ensure that its outputs are accurate. Testing (e.g., unit, system
integration, and user acceptance) is the process of executing a program
with the intent of finding errors. Clear, complete, and well-documented
requirements are needed in order to design and implement an effective
testing program. Linking the testing activities back to the requirements
assures the organization that, once testing activities are successfully
completed, all requirements have been addressed and will be met by the
system. Without such assurance, it is possible for a requirement to be
missed in development and the resulting lack of functionality not noticed
until late in testing, or even after deployment.
Management
Requirements, once developed and approved, also need to be managed
throughout the system life cycle. Two key areas of requirements management
are addressing changes to requirements and establishing and maintaining
bidirectional traceability from high-level requirements through detailed
work products to test cases and scenarios. As mentioned earlier, once a
set of high-level requirements is documented, verified, and approved, it
is placed under configuration control. From this point, changes to the
requirements are evaluated and validated as part of the
change control process.16 Change control includes reviewing and assessing
proposed changes to the requirements to determine the reasons for the
changes, determining if these changes are occurring due to flaws in the
requirements development process, and ensuring that any effects of the
change on other requirements as well as on the cost, schedule, and
performance goals of the project are determined and assessed.
Establishing and maintaining traceability from initial requirements to
work products and the resulting system is also important. A requirements
traceability matrix demonstrates forward and backward (bidirectional)
traceability from business requirements to detailed system requirements
all the way through to test cases.
BSM Lacks Policies and Procedures for Requirements Management
BSM does not yet have adequate policies and procedures in place to guide
its systems modernization projects in developing and managing
requirements. In January 2006, the RMO developed a set of draft policies
that address key areas of requirements development and management; these
policies are to serve as interim guidance while the final policies and
processes are being developed. At the conclusion of our review, the RMO
provided us the draft policies and a high-level plan that includes
milestones for completing these policies. Since critical BSM projects
continue to be pursued and completion of the policies and procedures is
not expected until March 2007, it is critical that BSM immediately
implement the draft policies and continue to develop the final policies.
BSM Lacks Comprehensive Policies and Procedures for Requirements
Development and Management, but Has Initiated Their Development
BSM does not have comprehensive, detailed policies and procedures for
requirements management and development activities that include
requirements elicitation, documentation, verification and validation, and
management. During our review, BSM officials were unable to provide us
with detailed policies and procedures and agreed that they do not have
such documents. Project teams were not consistent in their understanding
of which guidance they should use for the development and management of
requirements; some project team members mentioned BSM's Enterprise
Life Cycle17 (ELC); others said they were waiting for guidance from the
RMO. Our review of the ELC showed that it did not provide the procedures
project managers would need to properly perform the steps in the
requirements development and management process. BSM program officials
agreed that the ELC did not provide the needed guidance.
In December 2005, the RMO completed an analysis of requirements
development and management areas that need improvement. The RMO found, as
we did, that BSM lacks detailed guidance; their recommendations included
developing process handbooks for aspects of requirements elicitation,
documentation, verification and validation, and management. Subsequently,
in January 2006, BSM officials developed draft guidance that covers
aspects of requirements development and management. However, this guidance
does not fully address requirements elicitation, documentation,
verification and validation, and management. At that time, BSM also
provided us with a high-level plan that contains interim milestones and
establishes a March 2007 completion date for the final set of policies and
procedures. BSM officials told us that these draft policies are to serve
as interim guidance while the remaining policies and procedures are being
developed. In addition, IRS also uses its governance processes,
particularly the milestone exit reviews, to find and mitigate issues and
problems with requirements development and management on existing
projects. Finally, the RMO is allocating resources to key projects-such as
F&PC version 1.2-to assist them in developing requirements.
Without a formal set of documents that detail organizational policies and
associated procedures, employees and contractors will rely on their
individual knowledge and expertise to complete requirements development
and management activities. This raises the risk of cost overruns, schedule
delays, and reduction of functionality. Since critical BSM projects are
already under way, and completion of the policies and procedures is not
set until March 2007, it is urgent that BSM immediately implement the
draft policies. Until BSM develops and implements policies and procedures
that fully address the key areas of requirements development and
management, including elicitation, documentation, tracking of cost and
schedule impacts associated with requirements changes, and establishing
and maintaining full bidirectional traceability, ongoing projects will
continue to run greater risk of cost and schedule overruns and poor system
performance.
BSM Projects Have Not Consistently Followed Disciplined Requirements
Development and Management Practices
As a result of the lack of policies and procedures, BSM projects varied in
the extent to which they followed disciplined requirements practices. All
three projects we reviewed-MeF release 3.2 (to be deployed March 2006),
and F&PC release 1.1 (deployed January 2006), and CADE release 1.1
(deployed July 2004)-performed some of the practices associated with sound
requirements development and management. For example, all three projects
had a change management process in place that requires approvals and
impact assessments to be completed when changes are made to requirements.
However, none of the three BSM project releases we reviewed consistently
performed all of the practices needed for effective requirements
management. Specifically:
o Project teams did not have a clear, documented, and consistent method of
eliciting requirements.
o Project teams did not adequately document all requirements.
o Project teams did not effectively verify requirements.
o Project teams did not demonstrate adequate management of requirements.
Table 1: Requirements Activities Completed on BSM Projects
Project MeF 3.2 F&PC 1.1 CADE 1.1
Eliciting requirements
Documenting requirements
Verifying and validating requirements
Managing requirements
practice fully implemented
practice partially implemented
practice not implemented
Source: GAO analysis of BSM data.
Project Teams Have Inconsistent Requirements Elicitation Practices
Based on stakeholder information such as customer expectations,
constraints, and interfaces for a system, the requirements elicitation
team discovers, defines, refines, and documents business-level
requirements. Due to the importance of this activity, plans or strategies
should be in place to guide project officials in defining
elicitation-related activities and in outlining how the requirements will
be gathered (e.g., interviewing the users or analyzing the current or
expected business processes).
BSM project teams did not have a clear, consistent, and documented method
of eliciting requirements for the projects. For example, although the
teams identified stakeholders in their project plans, only MeF 3.2
provided evidence that working group meetings were conducted with
stakeholders to understand their needs and to identify their problems and
expectations and that strategies or plans were developed for eliciting
requirements. CADE 1.1 project team members could not describe how they
elicited requirements or provide a requirements plan that documented
elicitation procedures or strategies. An F&PC project team member stated
that, for release 1.1, the project did not have a fully documented process
for elicitation; however, the team member stated that the team had held
workshops and obtained resources and assistance from the RMO to help
mitigate the lack of a process. The RMO used lessons learned from this
effort to develop a new requirements elicitation process, which they
expect will assist F&PC in elicitation for its next release.
BSM project and program officials agreed that requirements elicitation
processes could be improved and stated that they were planning to address
some of the problems we found. For example, when we asked project
officials about the policies and procedures underlying their current
requirements elicitation activities, some stated that they were waiting
for new policies to be issued by the RMO, and others cited the ELC as
guidance. As mentioned earlier, the ELC does not provide the information
needed for the requirements elicitation process. Furthermore, F&PC
officials could not state which sections in the ELC described the
requirements elicitation process. BSM program management and RMO officials
acknowledged the lack of policies and procedures and stated that the RMO
has since developed new guidance for eliciting requirements that will be
piloted on F&PC version 1.2, which is currently entering the requirements
development phase.
BSM project teams performed elicitation processes in a nonstandard manner
due to the lack of policies, procedures, and guidance. Without
standardized policies and procedures to guide this key part of
requirements development, BSM program officials cannot ensure that its
systems development projects have collected and documented all the
necessary requirements, which could result in systems being developed that
do not meet user needs.
Projects Do Not Fully Document Requirements
After collecting and documenting high-level requirements from customers,
users, and other stakeholders, the requirements team should analyze these
high-level requirements against the conceptual (or expected) operational
environment to balance user needs and constraints and to ensure that the
system developed will perform as intended. The resulting lower-level
requirements should also be analyzed to make sure they can be performed in
the expected environment and that they satisfy the objectives of the
higher-level requirements. The final requirements are documented in the
requirements baseline.
The BSM projects we reviewed did not complete all of the activities needed
to adequately document requirements. Although project teams provided
evidence that they created a set of high-level requirements and obtained
approvals from stakeholders on this set of requirements, two of the three
projects did not provide evidence that requirements were thoroughly
analyzed and decomposed to lower-level system requirements. For example,
part of this analysis would link all lower-level systems requirements to
the original higher-level business requirements. Only one of the project
teams-F&PC-provided documentation showing the necessary linking or mapping
of lower-level system requirements to the business requirements. MeF and
CADE provided documentation; however, their documentation did not fully
demonstrate the linking of system-level requirements to the business
requirements.
A MeF project official agreed that full linkage of system-level
requirements to business requirements should be implemented. The MeF
official stated that they planned to implement this in their next
version-version 4.0. In addition, a BSM program official indicated that
additional project guidance on requirements documentation would be part of
the RMO's deliverables and would help to address this issue.
Without feasible and clearly defined requirements, projects run the risk
of cost overruns, schedule delays, and deployment of systems with limited
functionality. For example, incomplete definition of requirements has been
cited as one reason for schedule delays and cost overruns for both CADE
and MeF.
Projects Do Not Fully Verify Requirements; Validation Activities
Completed, but Problems Remain
Once requirements are fully documented, software code and other work
products that will guide development and testing activities need to be
verified using peer review techniques against the original requirements.
In addition, these products should be validated through testing to ensure
that they will operate effectively in the intended environment.
Projects Do Not Fully Verify Requirements through Peer Reviews
Requirements verification ensures that the lower-level requirements,
software code, and other work products that will guide development and
testing activities are an accurate representation of stakeholder needs.
Peer reviews are an important part of the verification process and are a
proven mechanism for effective defect removal. During peer reviews, teams
of peers18 examine code and other work products to identify defects,
determine the causes of the defects, and make recommendations that address
changes needed to help ensure that the system will meet stakeholder and
developer needs. Peer reviews should follow a structured, formalized
process; peer review events should be planned in advance, with items, such
as code and other work products, selected in advance; the results of the
sessions should be incorporated into peer review reports that project
teams are expected to address before moving further into development
activities.
The BSM project teams did not provide evidence that work products had been
verified against requirements through the use of a formalized peer review
process and project officials did not follow recommended practices for
conducting peer reviews. BSM project team members stated that they
conducted customer technical reviews and milestone exit reviews that they
considered to be peer reviews; however, these kinds of reviews do not meet
the criteria for peer reviews. They were not structured, did not select
code and other items in advance to be evaluated, and did not produce
formal peer reports with action items that projects were required to
address.
Project Teams Performing Validation, but Problems Remain
Requirements validation is the process of demonstrating that a product
fulfills its intended use in its environment. It differs from the
verification activities previously described, in that validation
determines that the product will fulfill its intended use, while
verification ensures that work products properly reflect the baseline
requirements. Validation includes tests conducted on the product during
development to prove that the product performs its intended functions
correctly. In a disciplined software development process, planning for
validation activities begins as requirements are developed; testing
activities are critical to determining that a system not only operates
effectively but addresses all baseline requirements. To complete
validation activities, testing is conducted at several levels, each of
which validates that the system will operate effectively at a different
level. For example, unit testing validates individual sections of code,
and system integration testing ensures that the system as a whole can
operate effectively in its environment. User acceptance testing allows the
user community to determine whether the system, as developed, can be used
to effectively support their work. It also validates that the system meets
user expectations. An effective testing process confirms the functionality
and performance of the product prior to delivery. It is a crucial process
and needs to be well planned, well structured, well documented, and
carried out in a controlled and managed way.
The BSM projects provided evidence of validation activities, such as test
plans and test results. CADE 1.1 officials provided both test plans and
test reports. MeF release 3.2 and F&PC release 1.1 are still in the
testing phase; they provided available test plans but do not yet have test
reports.
Despite the existence of test plans and reports, requirements are not
fully documented or fully traced. In addition, while the ELC provides
guidance on testing that discusses test planning, activities, and test
responsibilities, program officials say that this guidance is limited.
BSM's Enterprise Services organization has initiated an effort to review,
revise, and enhance test procedures across BSM. Therefore, until BSM
ensures full documentation and traceability of requirements, questions
about the completeness of its testing will remain.
Project Teams Not Adequately Managing Requirements
Finally, requirements must be managed through the system development life
cycle. We found that the three projects did not fully demonstrate adequate
management of their requirements. Although the projects had a formal
change control process in place to analyze and manage changes to
requirements, associated costs and schedule changes resulting from
requirements changes were not always tracked or updated. In addition,
projects' documentation did not demonstrate adequate traceability of
requirements from the business requirements (high-level requirements) to
system requirements (low-level requirements) to test cases.
Project Teams Not Updating Cost and Schedule to Reflect Requirements
Changes
Managing changes to the original requirements is a formal process to
identify, evaluate, track, report, and approve these changes. As work
products are developed and more is learned about the system that is being
developed, information is occasionally found that requires a change to the
original requirements. Modifications to project scope or design can also
result in requirements changes. Therefore, projects need to manage these
changes to requirements in a structured way.
The BSM project teams used a change management process to manage changes
to requirements that included documenting the rationale for changes,
developing assessments of the impact of the change, and obtaining
approvals by the Configuration Change Board. However, only the MeF and
F&PC teams provided evidence that their cost and schedule baselines were
updated when changes to requirements impacted cost and/or schedule. For
example, CADE officials did not provide any evidence to show how they
updated and tracked cost changes resulting from changes to requirements,
nor did they provide evidence that the work breakdown structure19 was
updated to reflect schedule changes. F&PC officials provided evidence for
tracking changes to the cost and schedule. MeF officials provided a
document that tracked the cost implications of changes to requirements and
the work breakdown structure to reflect schedule changes.
A BSM project official indicated that the BSM project was implementing
cost and schedule tracking on its current releases. However, it was not
clear whether BSM was doing this consistently or whether appropriate
guidance for tracking cost and schedule would be provided by BSM.
Project teams that do not effectively track cost and schedule changes as a
result of changes to requirements will not be able to effectively mitigate
the potential impact of these changes to overall cost, schedule, and
performance goals, thus raising the risk of cost overruns, schedule
delays, and deferral of functionality.
Project Teams Not Ensuring Full Traceability of Requirements
Another key element of requirements management is establishing and
maintaining the traceability of requirements. Traceability of requirements
is tracking the requirements from the inception of the project and
agreement on a specific set of business requirements to development of the
lower-level system requirements, detailed design, code implementation, and
test cases necessary for validating the requirements. Tracing a
requirement throughout the development cycle provides evidence that the
requirements are met in the developed system and ensures that the product
or system will work as intended. Requirements must be traceable forward
and backward through the life cycle. Each business requirement must be
traceable to associated system requirements and test cases. Without
adequate traceability, errors in functionality could occur and not be
found until the testing phase, when problems are more costly to fix and
time frames for fixing problems without causing a schedule slip for
deployment are limited.
Of the three projects, only the F&PC team showed evidence of full
traceability of the requirements from high-level requirements to low-level
requirements. MeF and CADE documentation did not demonstrate clear
traceability from the business requirements to lower-level system
requirements, coding, and test cases. MeF program officials acknowledged
weaknesses in this area and stated that they planned to develop full
bidirectional traceability to the business level requirements as part of
MeF release 4.0.
According to project officials, one reason they do not have full
bidirectional traceability is due to the lack of detailed procedures and
guidance for traceability of requirements. Until recently, BSM projects
were not required to develop and use a traceability matrix. While interim
guidance issued by BSM does require the use of traceability matrices and
use of its configuration management repository to manage requirements, the
guidance lacks the detail needed to ensure that projects meet criteria.
BSM program officials agreed that this was an area that needed additional
guidance. The RMO is currently reviewing new guidance on how to improve
requirements traceability.
Without adequate traceability of requirements, system requirements can be
missed during development and the agency cannot be assured that validation
activities fully demonstrate that all the agreed-upon requirements have
been developed, fully tested, and will work as intended.
Conclusions
BSM lacks policies and procedures to develop and manage requirements for
their systems modernization projects. BSM has acknowledged this deficiency
since late 2004 when it listed requirements management as one of its
high-priority initiatives and created an RMO. The office has now developed
draft policies that cover aspects of eliciting, documenting, verifying and
validating, and managing requirements. These draft policies are to serve
as guidance to projects teams as BSM projects are pursued. It is critical
that BSM implement these draft policies immediately and continue to
develop the remaining policies.
The three BSM development projects that we reviewed showed significant
differences in how they implemented practices for developing and managing
requirements. Until BSM and the RMO complete the development of policies
and procedures to ensure disciplined requirements development and
management practices, projects will not have sufficient guidance to ensure
implementation of these practices, which will impair their ability to
effectively manage the development and acquisition of critical systems and
increase the risk of cost overruns, schedule delays, and deferral of
functionality.
Recommendations for Executive Action
To improve the requirements development and management policies and
practices of the IRS's BSM, we recommend that the Commissioner of Internal
Revenue direct the Associate Chief Information Officer for BSM to take the
following two actions:
1.Ensure that BSM completes the delivery of policies and procedures for
requirements development and management as planned. The policies and
procedures should fully describe the processes, include a minimum set of
activities required for each project, and provide detailed procedures for
each of the key areas of requirements elicitation, documentation,
verification and validation, and management. As part of this effort, the
policies and procedures should specifically include the following:
o A standardized process for the elicitation of requirements that ensures
that projects fully investigate the requirements needed for a specific
system, including gathering requirements from all relevant users and
stakeholders.
o A standardized process for the documentation of requirements that
ensures full documentation of the baseline requirements.
o A process for ensuring formal peer reviews are planned and completed for
key products.
o Guidance on tracking cost and schedule impacts of changes to
requirements for all projects.
o Guidance on establishing and maintaining full bidirectional requirements
traceability.
2.In addition, since BSM has ongoing projects that are developing and
managing requirements and the development of new policies and procedures
is not scheduled to be complete until March 2007, the Commissioner should
direct the Associate CIO for BSM to immediately implement its draft
policies while the final policies and procedures are being developed.
Agency Comments
In providing written comments on a draft to this report, the Commissioner
of Internal Revenue agreed with our findings and stated that the report
provided a sound and balanced representation of the progress IRS has made
to date as well as work that remains to be completed. The Commissioner
also described the actions that IRS is taking to implement our
recommendations, including establishing a schedule to complete the
development of policies that address the areas of requirements
elicitation, documentation, verification and validation, and management.
The Commissioner's written comments are reprinted in appendix III.
As agreed with your office, unless you publicly announce the contents of
this report earlier, we plan no further distribution until 30 days from
the date of this letter. At that time, we will send copies of this report
to the Chairmen and Ranking Members of other Senate and House committees
and subcommittees that have appropriation, authorization, and oversight
responsibilities for IRS. We are also sending copies to the Commissioner
of Internal Revenue, the Secretary of the Treasury, the Chairman of the
BSM Oversight Board, and the Director of the Office of Management and
Budget. Copies are also available at no charge on the GAO Web site at
http://www.gao.gov.
report, please contact David A. Powner at (202) 512-9286 or p
[email protected] or Keith A. Rhodes at (202) 512-6412 or r [email protected].
Contact points for our Offices of Congressional Relations and Public
Affairs may be found on the last page of this report. GAO staff who made
major contributions to this report are listed in appendix III.
Sincerely yours,
David A. Powner Director, Information Technology Management Issues
Keith A. Rhodes, Chief Technologist, Center for Technology and Engineering
Appendix I
Scope and Methodology
The objectives of our review were to assess (1) whether the Requirements
Management Office (RMO) has established adequate requirements development
and management policies and procedures and (2) whether the Business
Systems Modernization (BSM) has effectively used requirements development
and management practices for key systems development efforts.
To assess the adequacy of BSM's requirements development and management
policies and procedures, including IRS's Enterprise Life Cycle (ELC),1 we
compared it against criteria based on industry standards and best
practices, including the Software Engineering Institute's (SEI) Capability
Maturity Model Integration (CMMIsm) version 1.1. We also reviewed draft
policies and procedures provided by the RMO in February 2006 and compared
them against this criteria. In addition, we interviewed appropriate BSM
officials to discuss the creation and goals of the RMO and whether there
were BSM requirements development and management policies and procedures
in place.
To assess whether BSM project teams effectively used requirements
development and management practices on its systems acquisitions, we
selected three BSM projects to review: (1) Modernized e-File (MeF) release
3.2, which is to be deployed in March 2006; (2) Filing and Payment
Compliance (F&PC) release 1.1, which was deployed in January 2006; and
Customer Account Data Engine (CADE) Individual Master File release 1.1,
which was deployed July 2004. We selected these investments because they
were (1) important to the goals and mission of the agency, (2) large-scale
development efforts with significant costs, and (3) at different points in
their development life cycles.
To evaluate whether each of the three projects had effectively used
requirements development and management practices for key systems
development efforts, we compared the project's documentation and processes
against criteria based on industry standards and best practices, including
SEI's CMMIsm version 1.1. The documentation reviewed for each of the
projects included requirements management plans, traceability matrices,
testing plans, baseline requirements, and other items. We also interviewed
the program officials from each of these three projects to further clarify
issues on their requirements development and management activities.
Our work was performed from June 2005 to February 2006 in Washington D.C.,
in accordance with generally accepted government auditing standards.
Appendix II
Project Descriptions
The following are descriptions of the three projects we selected to
review: Modernized e-File (MeF) release 3.2, Filing and Payment Compliance
(F&PC) release 1.1, and Customer Account Data Engine (CADE) release 1.1.
Modernized e-File
In fiscal year 2004, BSM introduced the Modernized e-File (MeF) system,
which allows e-filing for tax-exempt organizations and large corporations
and reduces the time to process their tax forms. The goal for MeF is to
replace the current e-filing technology with a modernized, Internet-based
electronic filing system. MeF is also expected to result in an increase in
the use of electronic filing because it is efficient and easy to access,
use, and maintain.
Projected benefits of the MeF program are as follows:
o Reduction in the BSM's effort associated with receiving, processing,
manually entering data, and resolving data entry errors from paper
returns;
o Reduction in system maintenance costs;
o Savings in time and money for taxpayers and tax practitioners due to not
copying, assembling, and mailing a return; and
o Sharing of tax and information return data electronically throughout
state agencies.
The MeF project is projected to provide the capability for Internet-based
filing of 330 different BSM forms. The following is table 2, describing
MeF releases deployed and their functionalities, followed by table 3,
which describes MeF financial data.
Table 2: MeF Releases and Functionality
Release Functionality
Release 1 Deployed in February 2004
Provides 53 forms and schedules for 1120/1120S and 5 forms for
990 e-filing, along with the functionality to support those
forms, including applicable interfaces, validations, retrieval
and display options, the capability for large taxpayers to
file using the Internet, and the capability to attach Adobe
files.
Release 2 Deployed in August 2004
Provides the remaining 43 forms and schedules for 1120/1120S
and required public access (access to redacted information for
nonprofit organizations) to the filed 990s.
Release 3.1 Deployed in January 2005
Provides 990PF series forms, Form 7004 (Application for
Automatic Extension of Time to File Corporation Return), and
M-TRDB processing codes.
Release 3.2 Projected deployment in January 2006
Expected to provide the Federal/State Single Point Filing
System platform and retrieval system for all state, corporate,
and tax-exempt tax returns and acknowledgments.
Source: IRS.
Table 3: Development and Steady State Costs for MeF
Dollars in millions
Fiscal year Development Steady state Total
FY 04 $57.1 $2.3 $59.4
FY 05 $67.1 $5.2 $72.3
FY 06 $60.6 $5.6 $66.2
Source: IRS.
Filing and Payment Compliance
The Filing and Payment Compliance (F&PC) project is intended to improve
technologies and processes that support BSM's compliance activities.
According to BSM, their collection operations rely on 30-year-old
technology and processes that are no longer compatible with the realities
of today's taxpayer environment. F&PC plans to provide support for
detecting, scoring, and working nonfiler and payment delinquency cases. It
is to use advanced software to analyze tax collection cases and divide
them into cases that require BSM involvement and those that can be handled
by private collection agencies. Case attributes are to be identified,
segmented, and prioritized to select the individual taxpayer cases that
have a greater probability of paying the tax liabilities in full or
through installment agreements.
The F&PC project is also to serve as an inventory management system to
assign, exchange, monitor, control, and update delinquent taxpayer
accounts between the BSM Authoritative Data Source and the private
collection agencies with whom BSM will contract.
The F&PC project is expected to increase the following:
o collection case closures by 10 million annually by 2014,
o voluntary taxpayer compliance, and
o BSM's capacity to resolve the buildup of delinquent taxpayer cases.
The BSM intends to deliver an initial limited private debt collection
capability in January 2006. Full implementation of this aspect of the F&PC
is projected to be completed by January 2008 with additional functionality
to follow in later years. Following is table 4, describing F&PC releases
deployed and their functionality, followed by table 5, which describes
F&PC financial data.
Table 4: F&PC Releases and Functionality
Release Functionality
Release 1.1 Projected deployment in January 2006
Expected to provide limited functionality of commercial-off
the-shelf (COTS) software with many manual processes employed
by BSM, small case volumes and minimal number of private
collection agencies supported.
Release 1.2 Projected deployment in January 2007
Expected to provide full implementation, testing, and
deployment of inventory management capabilities using the BSM
infrastructure.
Release 1.3 Projected deployment in January 2008
Expected to provide full COTS software functionality, private
collection agencies fully installed, electronic data transfer
between them fully established and operational.
Source: IRS.
Table 5: Development Costs for F&PC
Dollars in millions
Fiscal year Development Steady state Total
FY 04 $0.4 $0.0 $0.4
FY 05 $0.0 $0.0 $0.0
FY 06 $5.9 $0.0 $5.9
Source: IRS.
Customer Account Data Engine
The Customer Account Data Engine (CADE), intended to replace BSM's
antiquated tax administration system, is BSM's highest priority project
and is intended to house tax information for more than 200 million
individual and business taxpayers. The CADE databases and related
applications are also to enable the implementation of other systems that
will improve customer service and compliance and allow the online posting
and updating of taxpayer account and return data.
The CADE project is intended to
o generate refund notices, detect potential fraudulent transactions, and
calculate taxes;
o replace the group of BSM tax master files with a single database-the Tax
Account Data Store;
o accept, validate, and store taxpayer return and account data, along with
financial account activity data, such as tax payments, liabilities, and
installment agreements; and
o enable future business application systems.
In July 2004 and January 2005, BSM implemented the initial releases of
CADE, which have been used to process Form 1040EZ returns. CADE posted
more than 1.4 million returns for filing season 2005 and generated more
than $427 million in refunds. In 2006, CADE is expected to expand the
number and type of returns beyond the Form 1040EZ. BSM is also projecting
that CADE will process 33 million returns during 2007. Following is table
6, describing CADE releases deployed and their functionality, followed by
table 7 describing CADE financial data.
Table 6: CADE Releases and Functionality
Release Functionality
Release 1.0 Deployed in July 2004
Provided the filing capabilities for 1040 EZ forms.
Release 1.1 Deployed in July 2004 (concurrent with Release 1.0)
Provided filing season 2003 and 2004 tax law changes.
Release 1.2 Deployed in January 2005
Provided filing season 2005 tax law changes.
Release 1.3.1 Deployed in September 2005
Provided new functionality to improve performance and allow
address changes on tax returns as well as some filing season
2006 tax law changes.
Release 1.3.2 Projected deployment in January 2006
Expected to provide the remaining filing season 2006 tax law
changes and some additional functionality.
Source: IRS.
Table 7: Development Costs for CADE
Dollars in millions
Fiscal year Development Steady state Total
FY 04 $100.6 $0.0 $100.6
FY 05 $109.9 $0.0 $109.9
FY 06 $109.9 $0.0 $109.9
Source: IRS.
Appendix III
Comments from the Department of the Treasury
Appendix IV
GAO Contact and Staff Acknowledgments
GAO Contact
David A. Powner, 202-512-9286, [email protected]
Keith A. Rhodes, 202-512-6412, [email protected]
Staff Acknowledgments
In addition to those named above, Neil Doherty, Nancy Glover, George
Kovachick, Tonia Johnson, Tammi Nguyen, Madhav Panwar, and Rona Stillman
made key contributions to this report.
(310490)
www.gao.gov/cgi-bin/getrpt? GAO-06-310 .
To view the full product, including the scope
and methodology, click on the link above.
For more information, contact David A. Powner at (202) 512-9286 or
[email protected] or Keith A. Rhodes at (202) 512-6412 or [email protected].
Highlights of GAO-06-310 , a report to the Subcommittee on Transportation,
Treasury, the Judiciary, HUD, and Related Agencies, Committee on
Appropriations, U.S. Senate
March 2006
BUSINESS SYSTEMS MODERNIZATION
IRS Needs to Complete Recent Efforts to Develop Policies and Procedures to
Guide Requirements Development and Management
The Internal Revenue Service's (IRS) effort to modernize its tax
administrative and financial systems-Business Systems Modernization
(BSM)-has suffered delays and cost overruns due to a number of factors,
including inadequate development and management of requirements.
Recognizing these deficiencies, IRS created a Requirements Management
Office (RMO) to establish policies and procedures for managing
requirements. GAO's objectives were to assess (1) whether the office has
established adequate requirements development and management policies and
procedures and (2) whether BSM has effectively used requirements
development and management practices for key systems development efforts.
What GAO Recommends
To improve the requirements development and management practices at IRS,
GAO recommends that the Commissioner of Internal Revenue direct the
Associate Chief Information Officer for BSM to (1) ensure that BSM
completes delivery of policies and procedures for requirements development
and management as planned and (2) immediately implement interim policies
for BSM projects while final policies and procedures are being developed.
The Commissioner agreed with our recommendations and described steps taken
to address them.
BSM does not yet have adequate policies and procedures in place to guide
its systems modernization projects in developing and managing
requirements. In January 2006, the RMO developed a set of draft policies
that address some key areas of requirements development and management;
these policies are to serve as interim guidance while the final policies
and processes are being developed. At the conclusion of GAO's review, the
RMO also provided a high-level plan that includes milestones for
completing these policies. Since critical BSM projects continue to be
pursued and completion of the policies and procedures is not expected
until March 2007, it is critical that BSM immediately implement the draft
policies and continue to develop the final policies.
As a result of the lack of policies and procedures, the one ongoing
project-Modernized e-File (MeF)-and the two completed projects-Filing and
Payment Compliance (F&PC) and Customer Account Data Engine (CADE)-GAO
reviewed did not consistently follow disciplined practices for systems
development and management (see table).
Requirements Activities Completed on BSM Projects
Note: MeF release 3.2 will be deployed in March 2006, F&PC release 1.1 was
deployed January 2006, and CADE release 1.1 was deployed July 2004.
For example, all three projects had a key element of managing
requirements-a change management process that requires approvals and
impact assessments to be completed when there are changes to
requirements-but none met all of the practices needed for effective
requirements management. In addition, two projects did not have a clear,
consistent way to elicit (gather) requirements, two did not have fully
documented requirements, and two could not produce fully traceable
requirements (i.e., the requirements could not be tracked through
development and testing), which is another key element of managing
requirements. Unless IRS takes the steps needed to develop and
institutionalize disciplined requirements development and management
processes and implements draft policies in the interim to cover key areas
of requirements development and management, it will continue to face
risks, including cost overruns, schedule delays, and performance
shortfalls.
Report to the Chairman, Subcommittee on Transportation, Treasury, the
Judiciary, HUD, and Related Agencies, Committee on Appropriations, U.S.
Senate
March 2006
BUSINESS SYSTEMS MODERNIZATION
IRS Needs to Complete Recent Efforts to Develop Policies and Procedures to
Guide Requirements Development and Management
*** End of document. ***