Veterans Benefits Modernization: VBA Has Begun to Address Software
Development Weaknesses But Work Remains (Letter Report, 09/15/97,
GAO/AIMD-97-154).
Pursuant to a congressional request, GAO conducted a follow-up review to
determine the actions taken by the Veterans Benefits Administration
(VBA) to address management and technical weaknesses identified in the
June 19, 1996, hearing on the agency's modernization effort, focusing on
the agency's actions to improve its software development capability.
GAO noted that: (1) VBA has taken action to improve its software
development capability; (2) among other things, it has launched a
software process improvement initiative, chartered a software
engineering process group, and obtained the services of an experienced
contractor to assist in developing and implementing a software process
improvement effort; (3) although it has made progress, VBA has not yet
fully addressed needed software development improvements; (4) these
include a need for: (a) a defined strategy to reach the repeatable level
and a baseline to measure improvements; (b) a process improvement
training program for its software developers; and (c) a process to
ensure that VBA's software development contractors are at the repeatable
level; (5) VBA generally agrees that these issues need to be addressed
and has efforts underway to do so; and (6) until these deficiencies are
sufficiently addressed, VBA's software development capability remains ad
hoc and chaotic, subjecting the agency to continuing risk of cost
overruns, poor quality software, and schedule delays in software
development.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: AIMD-97-154
TITLE: Veterans Benefits Modernization: VBA Has Begun to Address
Software Development Weaknesses But Work Remains
DATE: 09/15/97
SUBJECT: Management information systems
Information resources management
Computer software
Veterans benefits
Customer service
Strategic information systems planning
Computer software verification and validation
Human resources training
Contract monitoring
IDENTIFIER: VBA Best Practices Round Up Project
VBA Standards, Policies, and Procedures Project
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to the Chairman, Subcommittee on Oversight and Investigations,
Committee on Veterans' Affairs, House of Representatives
September 1997
VETERANS BENEFITS MODERNIZATION -
VBA HAS BEGUN TO ADDRESS SOFTWARE
DEVELOPMENT WEAKNESSES BUT WORK
REMAINS
GAO/AIMD-97-154
Veterans Benefits Modernization
(511221)
Abbreviations
=============================================================== ABBREV
SEI - Software Engineering Institute
VA - Department of Veterans Affairs
VBA - Veterans Benefits Administration
Letter
=============================================================== LETTER
B-277370
September 15, 1997
The Honorable Terry Everett
Chairman, Subcommittee on Oversight
and Investigations
Committee on Veterans' Affairs
House of Representatives
Dear Mr. Chairman:
The Veterans Benefits Administration (VBA), a component of the
Department of Veterans Affairs (VA), is in the process of modernizing
its information systems to improve its administrative operations and
thereby upgrade the quality and timeliness of its services. Software
development has been identified by many experts as one of the
riskiest and most costly aspects of systems development. In June
1996, we reported and testified that VBA was operating at a level 1
software development capability, defined as ad hoc and chaotic.\1 In
its response to that report, VBA stated it would take actions to
address the weaknesses we identified.
At your request, we conducted a follow-up review to determine the
actions taken by VBA to address management and technical weaknesses
identified in the June 19, 1996, hearing on the agency's
modernization effort. This report, the second in a series on actions
taken by VBA to address these weaknesses, summarizes our assessment
of the agency's actions to improve its software development
capability.\2
--------------------
\1 Software Capability Evaluation: VA's Software Development Process
Is Immature (GAO/AIMD-96-90, June 19, 1996) and Veterans Benefits
Modernization: Management and Technical Weaknesses Must Be Overcome
If Modernization Is to Succeed (GAO/T-AIMD-96-103, June 19, 1996).
\2 The first report, Veterans Benefits Computer Systems: Risks of
VBA's Year-2000 Efforts (GAO/AIMD-97-79, May 30, 1997), discussed our
assessment of VBA's Year 2000 efforts.
RESULTS IN BRIEF
------------------------------------------------------------ Letter :1
VBA has taken action to improve its software development capability.
Among other things, it has launched a software process improvement
initiative, chartered a software engineering process group, and
obtained the services of an experienced contractor to assist in
developing and implementing a software process improvement effort.
Although it has made progress, VBA has not yet fully addressed needed
software development improvements. These include a need for (1) a
defined strategy to reach the repeatable level and a baseline to
measure improvements, (2) a process improvement training program for
its software developers, and (3) a process to ensure that VBA's
software development contractors are at the repeatable level. VBA
generally agrees that these issues need to be addressed and has
efforts underway to do so. Until these deficiencies are sufficiently
addressed, VBA's software development capability remains ad hoc and
chaotic, subjecting the agency to continuing risk of cost overruns,
poor quality software, and schedule delays in software development.
BACKGROUND
------------------------------------------------------------ Letter :2
Since 1986, VBA has been trying to modernize its old, inefficient
information systems. It reportedly spent an estimated $294 million
on these activities between October 1, 1986, and February 29, 1996.
The modernization program can have a major impact on the efficiency
and accuracy with which about $20 billion in benefits and other
services is paid annually to our nation's veterans and their
dependents. Software development is a critical component of this
modernization effort. Also, a mature software development capability
will provide added assurance that software developers will be able to
effectively make changes to the software needed to address the Year
2000 computing problem.
To evaluate VBA's software development processes, in 1996, we applied
the Software Engineering Institute's (SEI)\3 software capability
evaluation methodology to those projects identified by VBA as using
the best development processes. This evaluation compares agencies'
and contractors' software development processes against SEI's
five-level software capability maturity model, with 5 being the
highest level of maturity and 1 being the lowest.\4
In June 1996, we reported that VBA was operating at a level 1
capability. At this level, VBA cannot reliably develop and maintain
high-quality software on any major project within existing cost and
schedule constraints, which places VBA software development projects
at significant risk. Accordingly, VBA must rely on the various
capabilities of individuals rather than on an institutional process
that will yield repeatable, or level 2, results. VBA did not satisfy
any of the criteria for a repeatable or level 2 capability, the
minimum level necessary to significantly improve productivity and
return on investment. For example, VBA is extremely weak in the
requirements management, software project planning, and software
subcontract management areas, with no identifiable strengths or
improvement activities.
Because of VBA's software development weaknesses, we recommended that
the Secretary of Veterans Affairs
-- obtain expert advice to improve VBA's ability to develop
high-quality software;
-- develop and expeditiously implement an action plan that
describes a strategy for reaching the repeatable (level 2) level
of process maturity;
-- ensure that any future contracts for software development
require the contractor to have a software development capability
of at least level 2; and
-- delay any major investment in new software development--beyond
what is needed to sustain critical day-to-day operations--until
the repeatable level of process maturity is attained.
In commenting on a draft of the June 1996 report, VBA agreed with
three of our recommendations but disagreed with delaying major
investments in software development. VBA stated that while it agreed
that a repeatable level of process maturity is a goal that must be
attained, it disagreed that "all software development beyond that
which is day-to-day critical must be curtailed." VBA stated that the
payment system replacement projects, the migration of legacy systems,
and other activities to address the change of century must continue.
In our response to VBA's comments, we agreed that the change of
century and changes to legislation must be continued, and we
characterized these changes as sustaining day-to-day operations.
However, for those projects that do not meet this criterion, we
continue to believe that VBA should delay software development
investments until a maturity of level 2 is reached.
--------------------
\3 This is a nationally recognized, federally funded research and
development center established at Carnegie Mellon University in
Pittsburgh, Pennsylvania, to address software development issues.
\4 The following are the five levels of the software capability
maturity model: level 5 (optimizing) is continuous process
improvement; level 4 (managed) is detailed measure collection and
process/product control; level 3 (defined) is software process
documentation, standardization, and integration; level 2 (repeatable)
is software project cost, schedule and functionality tracking and
repeatability of successes; and level 1 (initial) is ad hoc and
chaotic software processes. The six key process areas for a
repeatable capability are (1) requirements management, (2) project
planning, (3) project tracking and oversight, (4) subcontract
management, (5) quality assurance, and (6) configuration management.
SCOPE AND METHODOLOGY
------------------------------------------------------------ Letter :3
To assess actions taken by VBA to improve its software capability, we
reviewed VBA documents, such as its "Software Process Improvement
Initiative Strategic Plan," dated March 1997; the Best Practices
Round Up Method; and the Interagency Agreement with the Air Force,
dated September 1996, to obtain expert software process improvement
assistance. We also reviewed SEI's IDEAL\SM: A User's Guide for
Software Process Improvement, dated February 1996, and technical
report on Best Training Practices Within the Software Engineering
Industry, dated November 1996. In addition, we reviewed VBA
contracts, correspondence to contractors, and supporting documents to
determine what VBA has done to ensure that VBA's software development
contractors are at the repeatable level.
We interviewed VBA officials and contractor personnel involved with
the software process improvement effort to determine what actions VBA
has taken to improve its software capability. We also interviewed
selected VBA project managers involved in new systems development on
their knowledge of the software process improvement initiative.
We performed our work from October 1996 through August 1997 in
accordance with generally accepted government auditing standards. We
requested comments on a draft of this report from the
Secretary-Designate of Veterans Affairs. The Secretary-Designate
provided us with written comments, which are discussed in the "Agency
Comments and Our Evaluation" section and reprinted in appendix I.
VBA HAS INITIATED ACTIONS TO
IMPROVE ITS SOFTWARE
DEVELOPMENT CAPABILITY
------------------------------------------------------------ Letter :4
VBA has initiated several actions to improve its software development
capability. For example, in response to our recommendation that it
obtain expert assistance, VBA has hired a contractor--SEI--to (1)
assist it in developing an integrated set of software practices that
will position VBA for successful, lasting improvements, (2) help
formulate a software improvement program, and (3) provide expertise
in executing software improvement activities. SEI is also expected
to provide expertise in strategic and tactical planning, training,
policy preparation, and action planning.
In addition, in response to our recommendation that it develop and
implement an action plan describing a strategy for reaching the
repeatable level, VBA launched a software process improvement
initiative in June 1996 to lay the foundation and build the context
for sustainable, measurable improvements to its software development
capability. It has developed a strategic plan that describes the
purpose and goals of this improvement initiative. One of the plan's
goals is to establish organization policies and guidelines for the
management, planning, and tracking of software projects that will
enable VBA to repeat earlier successes on projects with similar
applications.
VBA has also recently initiated two software process improvement
projects. The first project, called "Best Practices Round Up," will
identify software development practices that VBA software development
teams are doing correctly. VBA believes that there are "pockets of
excellence" within its organization not identified in our June 1996
report that it can build upon. The Best Practices Round Up project
team started its review in April 1997 and briefed VBA's chief
information officer on its results in August 1997.
The second project, called "Standards, Policies, and Procedures,"
will assess whether VBA's software development teams are following
current software development policies, procedures, and standards.
This project was initiated in June 1997 and is expected to be
completed by the end of September 1997. The project team is expected
to make recommendations to ensure compliance with standards.
SOFTWARE PROCESS IMPROVEMENT
INITIATIVE LACKS SPECIFICITY
AND A BASELINE
------------------------------------------------------------ Letter :5
Although VBA has launched a software process initiative and an
accompanying strategic plan, VBA has not yet clearly presented how it
intends to move from an ad hoc and chaotic level of software
development capability to a repeatable level. SEI's IDEAL\SM: A
User's Guide for Software Process Improvement requires that a plan be
developed that includes a schedule for initial activities, basic
resource requirements, and benefits to the organization. VBA's
current plan contains no milestones--beginning, interim, or
completion dates--by which to measure the agency's progress and to
identify problems. The plan also contains no analysis or information
on costs, benefits, or risks.
VBA officials stated that they recognize that the agency's strategic
plan for software process improvement lacks this specificity. At the
conclusion of our review, these officials said that VBA intends to
address this area in an upcoming action plan for the software process
improvement initiative.
VBA has also not yet established a baseline from which to measure its
software process improvements. According to the SEI IDEAL\SM: A
User's Guide for Software Process Improvement, an organization needs
to understand its current software process baseline so that it can
develop a plan to achieve the business changes necessary to reach its
software process improvement goals. At the conclusion of our review,
VBA officials told us that they plan to use as their baseline the
results of our June 1996 report, along with the results from their
Best Practices Round Up and Standards, Policies, and Procedures
projects. They stated that the baseline should be established by
September 1997.
VBA SOFTWARE PROCESS
IMPROVEMENT TRAINING PLAN NOT
EFFECTIVELY IMPLEMENTED
------------------------------------------------------------ Letter :6
Training of key staff is critical to achieving level 2 repeatability.
According to SEI's technical report entitled, Best Practices Within
the Software Engineering Industry, best training practices include
defining a process for software engineering education.
Although VBA has provided process improvement training to many of the
managers in its software engineering process group and management
steering group,\5 key software personnel--software developers,
project managers, and line managers--have not been trained in the
process improvement methodology, the principles behind it, and the
key process areas. VBA's software process improvement project
manager explained that these key people had not yet been trained
because VBA did not want to train them too long before implementing
the process improvement projects. The project manager said that VBA
plans to train these staff during fiscal years 1998 and 1999.
However, VBA does not have a documented training plan to help ensure
that these personnel receive training. Unless these individuals are
trained in the process improvement methodology, its principles, and
the key process areas, it will be difficult for them to implement the
new policies and procedures required to reach the repeatable level.
At the conclusion of our review, VBA officials stated that a training
plan is now under development and will be made part of the software
process improvement initiative action plan.
--------------------
\5 The software engineering process group is comprised of 8 people, 4
of whom are involved in systems development and 4 are not; the
management steering group is comprised of 14 people, primarily senior
managers.
PROCESS FOR ENSURING THAT
CONTRACTORS ARE AT THE
REPEATABLE LEVEL IS NOT CLEARLY
ESTABLISHED
------------------------------------------------------------ Letter :7
In responding to our recommendation that it ensure that contractors
have a repeatable software development capability, VBA intends to use
a new provision in future software development contracts. This
provision, however, does not require potential contractors to submit
supporting documentation to VBA certifying their level of maturity.
Validation of potential contractors' software development capability
maturity level should be a key factor in VBA's software contracting
decisions. The Internal Revenue Service, for example, recently
started requiring that all current, in-process, and future contract
solicitations for software development services require that
contractors submit documentation to verify how their software
development practices and processes satisfy the repeatable key
process areas specified by SEI's capability maturity model. The
Internal Revenue Service plans to use this information when selecting
software development services. Also, the Department of the Air
Force's acquisition policy states that software capability
evaluations should be used for selecting software contractors.
Although VBA has asserted that two of its current contractors are at
the repeatable level, VBA could not provide documentation to support
this. VBA subsequently requested the documentation from the
contractors, but the information the contractors provided did not
clearly show that they were at the repeatable level. In one case,
the contractor presented information on how it assisted federal
agencies in achieving the repeatable and/or higher levels of software
development capability but did not provide documentation that the
contractor was certified. In the second case, a component of the
contractor's organization asserted that it was at the repeatable
level but did not provide documentation supporting this assertion.
CONCLUSIONS
------------------------------------------------------------ Letter :8
Recognizing the importance of a mature software development
capability, VBA has initiated actions to address the weaknesses
identified in our June 1996 report. These actions will help it move
toward a repeatable software capability maturity level, but
additional efforts are needed. Specifically, VBA has not (1)
developed a detailed strategy for how VBA plans to achieve a
repeatable level of software development capability, (2) established
a baseline to measure performance improvements, (3) trained its
software development teams in the process improvement methodology,
and (4) established a process for ensuring that its software
development contractors are at the repeatable level. Recognizing
that these deficiencies need to be addressed, VBA has efforts
underway to do so. If these deficiencies are not sufficiently
addressed, VBA's software development capability will remain ad hoc
and chaotic, subjecting the agency to continuing risk of cost
overruns, poor quality software, and schedule delays.
RECOMMENDATIONS
------------------------------------------------------------ Letter :9
We recommend that the Secretary of Veterans Affairs direct the Under
Secretary for Benefits, in conjunction with VBA's chief information
officer, to
-- define the milestones, costs, tasks, and risks of the software
process improvement initiative in order to provide a clear
strategy for how VBA plans to improve its software development
capability to a repeatable level;
-- develop and use a baseline showing VBA's current software
development capability from which to measure VBA's software
improvement effort;
-- ensure that a training plan is developed and implemented that
will provide key software development staff training in the
software process improvement methodology, its principles, and
key process areas; and
-- establish a source selection process to ensure that VBA's
software development contractors have the mature processes
necessary for timely, high-quality software development,
including evaluating and validating documentation provided by
potential contractors establishing that they are at the
repeatable level or higher.
AGENCY COMMENTS AND OUR
EVALUATION
----------------------------------------------------------- Letter :10
In comments on a draft of this report, VA concurred with our
recommendations. VA also agreed that a repeatable level of process
maturity is a goal that VBA must attain and described a number of
activities underway to improve its software development capability.
For example, VBA has developed a draft action plan to define a
strategy to reach the repeatable level and specify the
activities/tasks, milestones, costs, and timeliness associated with
the process improvement effort. VBA also was reviewing and revising
the draft plan to fully address the issues raised in our report. VA
added that a significant amount of work still remains before this
plan is finalized.
We are encouraged by VBA's response and will continue to monitor the
agency's progress in implementing its software improvement effort.
--------------------------------------------------------- Letter :10.1
We are sending copies of this report to the Ranking Minority Member
of the Subcommittee on Oversight and Investigations and the Chairman
and Ranking Minority Member of the Subcommittee on Benefits, House
Committee on Veterans' Affairs. We will also provide copies to the
Chairmen and Ranking Minority Members of the House and Senate
Committees on Veterans' Affairs and the House and Senate Committees
on Appropriations; the Secretary-Designate of Veterans Affairs; and
the Director of the Office of Management and Budget. Copies will
also be
made available to other parties upon request. Please contact me at
(202) 512-6253 or by e-mail at [email protected] if you have
any questions concerning this report. Major contributors to this
report are listed in appendix II.
Sincerely yours,
Joel C. Willemssen
Director, Information Resources Management
(See figure in printed edition.)Appendix I
COMMENTS FROM THE DEPARTMENT OF
VETERANS AFFAIRS
============================================================== Letter
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
The following is GAO's comment on the Department of Veterans Affairs
letter dated September 4, 1997.
GAO'S COMMENT
1. Enclosure (2) has not been included.
MAJOR CONTRIBUTORS TO THIS REPORT
========================================================== Appendix II
ACCOUNTING AND INFORMATION
MANAGEMENT DIVISION, WASHINGTON,
D.C.
Helen Lew, Assistant Director
Leonard J. Latham, Technical Assistant Director
K. Alan Merrill, Technical Assistant Director
David Chao, Senior Technical Advisor
Tonia L. Johnson, Senior Information Systems Analyst
*** End of document. ***