Veterans Benefits Administration: Progress Made in Long-Term
Effort to Replace Benefits Payment System, but Challenges Persist
(27-APR-07, GAO-07-614).
Since 1996, the Veterans Benefit Administration (VBA) has been
working on an initiative to replace its aging system for paying
compensation and pension benefits. In 2005, concerned about the
slow pace of development, VBA contracted with the Software
Engineering Institute (SEI) for an independent evaluation of the
project, known as the Veterans Service Network (VETSNET). SEI
advised VBA to continue working on the project at a reduced pace
while addressing management and organization weaknesses that it
determined had hampered the project's progress. GAO was requested
to determine to what extent the VETSNET project has followed the
course of action recommended by SEI and describe the project's
current status. To perform its review, GAO analyzed project
documentation, conducted site visits, and interviewed key program
officials.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-07-614
ACCNO: A68851
TITLE: Veterans Benefits Administration: Progress Made in
Long-Term Effort to Replace Benefits Payment System, but
Challenges Persist
DATE: 04/27/2007
SUBJECT: Claims processing
Contractors
Performance measures
Program management
Records
Software
Systems conversions
Veterans benefits
Veterans disability compensation
Veterans pensions
Program evaluation
Policies and procedures
Veterans Service Network
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-07-614
* [1]Results in Brief
* [2]Background
* [3]The Processes Supporting the Variety of Compensation and Pen
* [4]Our Prior Products Identified Weaknesses in the Development
* [5]SEI Assessed System Replacement Initiative and Recommended a
* [6]VBA Is Following Course Recommended by SEI, but Weaknesses R
* [7]VBA Is Taking Action on Certain Overall Management Concerns
* [8]Top Management Included in Revised Governance Structure
* [9]VBA Took Action to Address Project Planning and
Management
* [10]Conversion Efforts Were Suspended as Advised, but Issues
Rem
* [11]VBA Is Improving Software Development Processes, but Improve
* [12]Risk Management Plan Has Been Revised
* [13]Requirements Management Has Improved
* [14]Management Attention Is Being Focused on Major Software
Defe
* [15]The Replacement Initiative Is Tracking Certain
Performance M
* [16]VETSNET Is Currently Processing a Portion of Compensation Cl
* [17]Conclusions
* [18]Recommendations for Executive Action
* [19]Agency Comments and Our Evaluation
* [20]GAO Contact
* [21]Staff Acknowledgments
* [22]GAO's Mission
* [23]Obtaining Copies of GAO Reports and Testimony
* [24]Order by Mail or Phone
* [25]To Report Fraud, Waste, and Abuse in Federal Programs
* [26]Congressional Relations
* [27]Public Affairs
Report to the Chairman, Committee on Veterans' Affairs, House of
Representatives
United States Government Accountability Office
GAO
April 2007
VETERANS BENEFITS ADMINISTRATION
Progress Made in Long-Term Effort to Replace Benefits Payment System, but
Challenges Persist
GAO-07-614
Contents
Letter 1
Results in Brief 3
Background 4
VBA Is Following Course Recommended by SEI, but Weaknesses Remain to Be
Addressed 10
VETSNET Is Currently Processing a Portion of Compensation Claims, but Much
Work Remains 22
Conclusions 26
Recommendations for Executive Action 27
Agency Comments and Our Evaluation 28
Appendix I Objectives, Scope, and Methodology 30
Appendix II Comments from the Department of Veterans Affairs 32
Appendix III GAO Contact and Staff Acknowledgments 38
Tables
Table 1: Roles and Responsibilities of VETSNET Organizational Elements 12
Table 2: VETSNET Component Applications and Status of Development 24
Table 3: Software Releases Planned to Complete the VETSNET System 25
Table 4: Conversion Phases 26
Figure
Figure 1: VETSNET System Life Cycle and Review Concept 15
Abbreviations
BDN Benefits Delivery Network
CIO Chief Information Officer
FAS Finance and Accounting System
IT Information Technology
MAP-D Modern Award Processing-Development
RBA 2000 Rating Board Automation 2000
SEI Carnegie Mellon Software Engineering Institute
VA Department of Veterans Affairs
VBA Veterans Benefits Administration
VETSNET Veterans Service Network
VRE Vocational Rehabilitation and Employment
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
United States Government Accountability Office
Washington, DC 20548
April 27, 2007
The Honorable Bob Filner Chairman,
Committee on Veterans' Affairs
House of Representatives
Dear Mr. Chairman:
Since 1996, the Veterans Benefits Administration (VBA), a major component
of the Department of Veterans Affairs (VA), has been undertaking a project
to replace its compensation and pension benefits payment system. VBA's
existing system, the Benefits Delivery Network (BDN), is essential to
ensuring the accurate processing of compensation and pension benefits
payments to over 3.5 million veterans and their dependents each month.
However, this system, which has been in operation for more than 40 years,
is based on antiquated software programs, which have become increasingly
difficult and costly to maintain. VBA is in the process of replacing the
aging BDN with a faster, more flexible, and higher capacity system. When
it began this project in 1996, which it generally refers to as the
Veterans Service Network (VETSNET),1 VBA had planned to complete the
replacement system in May 1998 at an estimated cost of $8 million.
Over the years, we have reported on numerous problems that VBA has
encountered in completing the replacement system.2 Our prior work found,
for example, that the project was begun before VBA had fully developed its
business requirements, resulting in confusion over the requirements to be
addressed; in addition, VBA's software development capability was too
immature to ensure that the agency could reliably develop and maintain
high-quality software on any major project within cost and schedule
constraints. In 2002, we offered a number of recommendations to improve
managerial and program weaknesses, including that VBA appoint a project
manager; thoroughly analyze its initiative; and develop a number of plans,
including a revised compensation and pension replacement strategy and an
integrated project plan. VA concurred with our recommendations, and as we
last reported in June 2006,3 it took several actions to address them. For
example, it appointed a full-time project manager, and the project team
reported that it had completed certifications of users' requirements for
the system's applications. Nonetheless, these actions did not implement
all our recommendations and were not sufficient to establish the program
on a solid footing: certain basic requirements of sound project
management, such as an integrated project plan for the replacement system,
continued to be lacking.
1It also refers to the initiative as the compensation and pension or C&P
replacement system.
2GAO, Software Capability Evaluation: VA's Software Development Process Is
Immature, [28]GAO/AIMD-96-90 (Washington, D.C.: June 19, 1996); Veterans
Benefits Modernization: VBA Has Begun to Address Software Development
Weaknesses but Work Remains, [29]GAO/AIMD-97-154 (Washington, D.C.: Sept.
15, 1997); VA Information Technology: Progress Continues Although
Vulnerabilities Remain, [30]GAO/T-AIMD-00-321 (Washington, D.C.: Sept. 21,
2000); VA Information Technology: Important Initiatives Begun, Yet Serious
Vulnerabilities Persist, [31]GAO-01-550T (Washington, D.C.: Apr. 4, 2001);
VA Information Technology: Management Making Important Progress in
Addressing Key Challenges, [32]GAO-02-1054T (Washington, D.C.: Sept. 26,
2002); and Information Technology: VA and DOD Face Challenges in
Completing Key Efforts, [33]GAO-06-905T (Washington, D.C.: June 22, 2006).
In 2005, the VA Chief Information Officer (CIO) and Under Secretary for
Benefits became concerned about continuing problems with the replacement
project and contracted for an independent assessment of the department's
options for the project, including whether the project should be
terminated. This assessment, conducted by the Carnegie Mellon Software
Engineering Institute (SEI), concluded that the replacement project faced
many risks arising from management and organizational issues, but no
technical barriers that could not be overcome.4 According to SEI, a new
system was still needed, and VBA would not be able to successfully deliver
a full, workable solution unless it addressed its management and
organizational weaknesses. SEI recommended that VBA continue to work on
the project at a reduced pace, while taking an aggressive approach to
addressing the identified weaknesses.
Given the importance of ensuring the effective and efficient delivery of
veterans' benefits, we were requested to review the department's
continuing efforts to develop and implement the compensation and pension
replacement system. Specifically, our objectives were to determine (1) to
what extent VA has followed the course of action recommended by SEI and
addressed the concerns that it raised and (2) the current status of the
replacement project.
3 [34]GAO-06-905T .
4Kathryn Ambrose, William Novak, Steve Palmquist, Ray Williams, and Carol
Woody, Report of the Independent Technical Assessment on the Department of
Veterans Affairs VETSNET Program (Carnegie Mellon Software Engineering
Institute, September 2005).
In conducting this study, we visited VBA's Nashville Regional Office to
observe processing of new benefits claims in the compensation and pension
replacement system, and we visited the St. Petersburg Regional Office to
observe processes and procedures used to test and validate key
functionalities of the replacement system. We obtained and analyzed
documents related to the replacement initiative and to SEI's review of the
initiative. We supplemented our analysis with interviews of VBA and
contractor personnel at headquarters and the two regional offices.
Appendix I contains a more detailed discussion of our objectives, scope,
and methodology. We conducted our study between April 2006 and April 2007
in accordance with generally accepted government auditing standards.
Results in Brief
VBA is generally following the course of action recommended by SEI by
continuing to work on the replacement initiative at a reduced pace, while
taking action to address identified weaknesses in overall management and
software development processes. For example, VBA established a new
governance structure for the initiative that included senior management
and involved all stakeholders, and it incorporated all critical areas of
system development in an integrated master schedule. However, VBA did not
address all of SEI's management concerns. Specifically, SEI advised VBA to
ensure that stakeholders take ownership responsibility for the project,
including the total system and process operating costs; however, although
VBA is tracking costs incurred by contractors, it is not yet tracking and
reporting in-house costs incurred by the project. Further, VBA has not yet
developed sufficient plans for performing the substantial task of moving
records from the BDN to the replacement system (approximately 3.5 million
beneficiaries are currently being served by the older system). VBA did
take steps to improve its software development processes, such as
establishing risk and requirements management processes, but some have not
been addressed. For example, still to be addressed are processes for
capacity planning and management, which will be important for ensuring
that further development does not lead to slowdowns in processing of
benefits. Finally, VBA has not yet documented policies and procedures to
institutionalize all the improvements that it has incorporated in the
replacement initiative. According to the replacement project's management
team, it made a conscious decision first to establish the governance and
build the organization, among other things, and it is still prioritizing
remaining tasks. However, if VBA does not institutionalize the
improvements made, it increases the risk that these process improvements
may not be maintained through the life of the project or be available for
application to other development initiatives.
After more than 10 years of effort, including the recent management,
organizational, and process improvements, VBA has developed critical
functionalities needed to process and pay certain original compensation
claims using the replacement system, but it remains far from completing
the project. According to VBA officials, all five of the major software
applications that make up the new system are now being used in VA's
regional offices to establish and process new compensation claims for
veterans. In total, the replacement system is currently providing monthly
compensation payments to almost 50,000 veterans (out of about 3 million
veterans who receive such payments); the system was used to process about
83 percent of all new compensation claims completed in March 2007.
Nonetheless, the system requires further development before it can be used
to process claims for the full range of compensation and pension benefits
available to veterans and their dependents. In addition, VBA still faces
the substantial task of converting records for the approximately 3.5
million beneficiaries currently being served by the BDN to the replacement
system.
To sustain the improved management and software development processes
currently being used by VETSNET project management, we are making
recommendations to the Secretary of Veterans Affairs in several areas,
including cost tracking, capacity planning and management, and performance
measures.
In providing written comments on a draft of this report, the Secretary of
Veterans Affairs agreed with our conclusions and concurred with the
report's recommendations. (The department's comments are reproduced in
app. II.) The comments described actions planned that respond to our
recommendations, as well as providing further information on relevant
actions already taken. If the planned actions are properly implemented,
they could help strengthen the department's management of the replacement
system project and improve the chances that the system will be
successfully completed.
Background
VBA provides benefits for veterans and their families through five
programs: (1) [35]compensation and pension , (2) [36]education , (3)
vocational rehabilitation and employment (VRE) services, (4) loan
guaranty, and (5) [37]life insurance . It relies on the BDN to administer
benefit programs for three of VBA's five programs: compensation and
pension, education, and VRE services.
Replacing the aging BDN has been a focus of systems development efforts at
VBA since 1986.5 Originally, the administration planned to modernize the
entire system, but after experiencing numerous false starts and spending
approximately $300 million on the overall modernization of the BDN, VBA
revised its strategy in 1996. It narrowed its focus to replacing only
those functionalities that support the compensation and pension program,
and began developing a replacement system, which it called VETSNET.
As reported by the department in its fiscal year 2008 budget submission,
the compensation and pension program is the largest of the three programs
that the BDN supports:
o The compensation and pension program paid about $35 billion in
benefits in fiscal year 2006 to about 3.6 million veterans or
veterans' family members.
o Of this amount, compensation programs paid benefits
of about $31 billion to about 3.1 million recipients.
o Pension programs paid benefits of about $3.5
billion to about 535,000 recipients.
o The education program paid about $2.8 billion to about 498,000
veterans or their dependents in fiscal year 2006.6
o The VRE services program paid about $574 million for VRE
services in 2006 and provides rehabilitation services to
approximately 65,700 disabled veteran participants per year.7
The Processes Supporting the Variety of Compensation and Pensions
Benefits Are Complex
One of the challenges of developing the replacement system is that
it must include processes to support the administration of a
complex set of benefits. Different categories of veterans and
their families are eligible for a number of different types of
benefits and payments, some of which are based on financial need.
Compensation programs, which are based on service-connected
disability or death, provide direct payments to veterans and/or
veterans' dependents and survivors. These programs are not based
on income. Pension benefits programs, on the other hand, are
income based; these are designed to provide income support to
eligible veterans and their families who experience financial
hardship. Eligible veterans are those who served in wartime and
are permanently and totally disabled for reasons that are not
service-connected (or who are age 65 or older). Veterans are also
eligible for burial benefits.
Survivor benefits may be paid to eligible survivors of veterans,
depending on the circumstances. Some of these benefits are based
on financial need, such as death pensions for some surviving
spouses and children of deceased wartime veterans, and Dependency
and Indemnity Compensation to some surviving parents.
Finally, certain benefits may be paid to third parties, such as
individuals to whom a veteran has given power of attorney or
medical service providers designated to receive payments on the
veteran's behalf.
Generally, VBA administers benefit programs through 57 veterans
benefits regional offices8 in a process that requires a number of
steps, depending on the type of claim. When a veteran submits, for
example, a compensation claim to any of the regional offices, a
veterans service representative must obtain the relevant evidence
to evaluate the claim (such as the veteran's military service
records, medical examinations, and treatment records from VA
medical facilities or private medical service providers). In the
case of pension claims, income information would also be
collected.
Once all the necessary evidence has been compiled, a rating
specialist evaluates the claim and determines whether the claimant
is eligible for benefits. If the veteran is determined to be
eligible for disability compensation, the Rating Veterans Service
Representative assigns a percentage rating based on the veteran's
degree of disability. This percentage is used in calculating the
amount of payment.
Benefits received by veterans are subject to change depending on
changing circumstances. More than half of VBA's workload consists
of dealing with such changes. If a veteran believes that a
service-connected condition has worsened, for example, the veteran
may ask for additional benefits by submitting another claim.9 The
first claim submitted by a veteran is referred to as the original
claim, and a subsequent change is referred to as a reopened claim.
Our Prior Products Identified Weaknesses in the Development and
Implementation of the Replacement System
Since its inception, VETSNET has been plagued by problems. Over
the years, we have reported on the project, highlighting concerns
about VBA's software development capabilities.
In 1996, our assessment of the department's software development
capability determined that it was immature.10 In our assessment,
we specifically examined VETSNET and concluded that VBA could not
reliably develop and maintain high-quality software on any major
project within existing cost and schedule constraints. The
department showed significant weaknesses in requirements
management, software project planning, and software subcontract
management, with no identifiable strengths. We also testified that
VBA did not follow sound systems development practices on VETSNET,
and we concluded that its modernization efforts had inherent
risks.
Between 1996 and 2002, we continued to identify the department's
weak software development capability as a significant factor
contributing to persistent problems in developing and implementing
the system. We also reported that VBA continued to work on VETSNET
without an integrated project plan. As a result, the development
of the system continued to suffer from problems in several areas,
including project management, requirements development, and
testing.
Over the years, we made several recommendations aimed at improving
VA's software development capabilities. Among our recommendations
was that the department take actions to achieve greater maturity
in its software development processes11 and that it delay any
major investment in software development (beyond that needed to
sustain critical day-to-day operations) until it had done so. In
addition, we made specific recommendations aimed at improving
VETSNET development. For example, we recommended that VA appoint a
project manager, thoroughly analyze its current initiative, and
develop a number of plans, including a revised compensation and
pension replacement strategy and an integrated project plan.
VA concurred with our recommendations and took several actions to
address them. For example, it appointed a full-time project
manager and ensured that business needs were met by certification
of user requirements for the system applications. The actions
taken addressed some of our specific concerns; however, they were
not sufficient to fully implement our recommendations or to
establish the program on a sound footing.
SEI Assessed System Replacement Initiative and Recommended a
Changed Approach
As a result of continuing concerns about the replacement project,
in 2005 VA's CIO and its Under Secretary for Benefits contracted
for an independent assessment of the department's options for the
initiative.12 The chosen contractor, SEI, is a federally funded
research and development center operated by Carnegie Mellon
University. Its mission is to advance software engineering and
related disciplines to ensure the development and operation of
systems with predictable and improved cost, schedule, and quality.
SEI recommended that the department reduce the pace of development
while at the same time taking an aggressive approach to dealing
with management and organizational weaknesses hampering VBA's
ability to complete the replacement system. According to SEI,
these management and organizational concerns needed to be
addressed before the replacement initiative or any similar project
could deliver a full, workable solution.
For example, the contractor stressed the importance of setting
realistic deadlines and commented that there was no credible
evidence that VETSNET would be complete by the target date, which
at the time of the review had slipped to December 2006. According
to the assessment, because this deadline was unrealistic, VBA
needed to plan and budget for supporting the BDN so that its
ability to pay veterans' benefits would not be disrupted. SEI also
noted that different organizational components had independent
schedules and priorities, which caused confusion and deprived the
department of a program perspective. Further, the contractor
concluded that VBA needed to give priority to establishing sound
program management to ensure that the project could meet targeted
dates. These and other observations were consistent with our
long-standing concerns regarding fundamental deficiencies in VBA's
management of the project.
To help VBA implement the overall recommendation, the contractor's
assessment included numerous discussions of activities needed to
address these areas of concern, which can be generally categorized
as falling into two major types:
o Overall management concerns with regard to the initiative
included
o governance structure, including assigning ownership
for the project and its costs;
o project planning, including the development
schedule and capacity planning; and
o conversion of records currently on the BDN to the
replacement system.
o Software development process improvements were needed in the
following areas:
VBA Is Following Course Recommended by SEI, but Weaknesses Remain
to Be Addressed
As recommended by SEI, VBA is continuing to work on the
replacement initiative at a reduced pace and taking action to
address identified weaknesses in the project's overall management
and software development processes. For example, VBA has
established a new governance structure and has developed an
integrated master schedule that provides additional time and
includes the full range of project activities. However, additional
effort is needed to complete a number of the corrective actions,
such as improving project accountability through monitoring and
reporting all project costs. Further, VBA has not yet
institutionalized many of the improvements that it has undertaken
for the initiative. In particular, process improvements remain in
draft and have not been established through documented policies
and procedures. According to the VETSNET management team, it gave
priority to other activities, such as establishing appropriate
governance and organizational structures, and it is still
gathering information to assist in prioritizing the activities
that remain. Nonetheless, if VBA does not institutionalize these
improvements, it increases the risk that these process
improvements may not be maintained through the life of the project
or be available for application to other development initiatives.
VBA Is Taking Action on Certain Overall Management Concerns
SEI concluded that VBA's management issues would need to be
addressed as part of the implementation of its overall
recommendation. SEI's overall management concerns focused on the
project governance, project planning, and conversion of records
currently on the BDN to the replacement system.
Top Management Included in Revised Governance Structure
SEI guidance for software development stresses the need for
organizational commitment and the involvement of senior management
in overall project governance.13 In its assessment, SEI noted that
because management of the VETSNET project had been assigned to
VBA's information technology (IT) group, certain activities
critical to the veterans' benefits program, but not traditionally
managed by the IT group, had not been visible to the project's
management. The contractor pointed out that the IT group, business
lines, and regional offices needed to share ownership and
management of the replacement project through an established
governance process and that the project management office should
include business representatives. According to SEI, the project
needed to establish ownership responsibility, including addressing
total system and process operating costs.
In response to the assessment, VBA developed a new governance
structure for the initiative, which the Under Secretary for
Benefits approved in March 2006. In the new structure, the VETSNET
Executive Board that had been in place was expanded and
reorganized to serve as a focal point and major governance
mechanism for the replacement initiative. A Special Assistant
(reporting directly to the Under Secretary) was appointed to
coordinate and oversee the initiative as the head of the VETSNET
Executive Team, which was established to provide day-to-day
operational control and oversight of the replacement initiative.
Implementation Teams were also established to conduct the
day-to-day activities associated with implementing the initiative.
This governance structure established a process for IT, business
lines, and regional offices to share ownership and management, as
SEI advised.
The roles and membership of each of the organizational elements in
the new governance structure are described in table 1.
o risk management,
o requirements management,
o defect management, and
o program measures.
5The BDN currently runs on aging software: COBOL programs and a
nonrelational database. Analysts have indicated that moving from a
nonrelational database of the BDN type to a more modern relational
database is a challenging task.
6This program provides veterans, service members, reservists, and certain
veterans' dependents with educational resources.
7To help veterans with service-connected disabilities become employable
and obtain and maintain suitable employment, the program provides a range
of direct and supportive services. These services include comprehensive
evaluation of rehabilitation needs (vocational or independent living);
training and employment services to obtain or maintain suitable
employment; and independent living services, such as training and
specialized equipment to enable independence in the activities of daily
living.
8Not all regional offices process all the different types of benefits. For
example, adjustments to pension claims are processed at three pension
maintenance centers in St. Paul, Milwaukee, and Philadelphia, and
education claims are processed only at the regional offices in Atlanta,
Buffalo, Muskogee, and St. Louis. The Philadelphia Regional Office and
Insurance Center has sole responsibility for insurance benefit processing,
and nine Regional Loan Centers administer the loan guaranty program.
9Claims may also be reopened if a veteran provides additional information
on a claim that was denied.
10 [38]GAO/AIMD-96-90 .
11Specifically, at the repeatable level of process maturity, basic project
management processes are established to track cost, schedule, and
functionality, and the necessary process discipline is in place to repeat
earlier successes on projects with similar applications.
12Kathryn Ambrose, William Novak, Steve Palmquist, Ray Williams, and Carol
Woody, Report of the Independent Technical Assessment on the Department of
Veterans Affairs VETSNET Program (Carnegie Mellon Software Engineering
Institute, September 2005).
13CMMI Product Team, CMMI for Development, Version 1.2 (Carnegie Mellon
Software Engineering Institute, August 2006).
Table 1: Roles and Responsibilities of VETSNET Organizational Elements
Organization Role Membership
VETSNET Executive Provides executive direction Deputy Under Secretary
Board to the VETSNET Executive for Benefits
Team.
Special Assistant to
the Under Secretary for
Benefits
VETSNET Program Manager
VA Office of
Information and
Technology Principal
Associate Deputy
Assistant Secretary
Chief Financial Officer
Director, Compensation
and Pension Service
Director, Office of
Performance Analysis
and Integrity
Associate Deputy Under
Secretary for Policy
and Programs
Associate Deputy Under
Secretary for Field
Operations
VBA Chief Information
Officer
VETSNET Executive Responsible for the delivery VETSNET Program
Team of the project and acts as Executive (Special
the review and approval Assistant to Under
authority for the Secretary)
Implementation Teams.
VETSNET Program Manager
Contract support
Business Architect
Technical Architect
MITRE support
VETSNET Responsible for the Compensation and
Implementation Teams day-to-day activities Pension
necessary to develop and
implement the replacement Office of Resource
system. Management
Office of Performance
Analysis and Integrity
Office of Field
Operation
Office of Information
Management, VA Central
Office
St. Petersburg Systems
Development Center
Hines Information
Technology Center
Austin Automation
Center
Source: GAO analysis of VBA information.
When the new governance structure was approved in March 2006, the Under
Secretary ensured that those involved in the project gave it high
priority, directing certain key personnel (such as members of the
executive and implementation teams) to make the initiative their primary
responsibility, and other personnel (technical staff that provide support
to other systems) with collateral (non-VETSNET) duties to make the project
their first priority. He also placed limitations on the transfer of
personnel away from the project, recognizing the importance of staff
continuity in successfully completing the initiative. Staff members
assigned project responsibilities could be reassigned (i.e., given
nonpromotion, lateral reassignments) only with approval from the Under
Secretary or his deputy.
By implementing the new governance and organizational structure and
ensuring that the project has priority, VBA partially responded to SEI's
concerns in this area; however, VBA has not yet taken action with regard
to ownership responsibility for total system and process operating costs,
as SEI advised. According to administration officials, the replacement
initiative is an in-house, contractor-assisted development effort, in
which three different contractors provide support for program management,
system development, and testing and validation of requirements. VA
reported VETSNET system costs to the Congress totaling about $89 million
for fiscal years 1996 through 2006, with additional estimated costs for
completion of the initiative in 2009 of about $62.4 million. However,
according to project management officials, these costs do not include
expenditures for in-house development work. This in-house work involves
many VA personnel, as well as travel to various locations for testing and
other project related activities. Thus, considerable costs other than
contract cost have been incurred, which have not been tracked and reported
as costs for the replacement initiative. Without comprehensive tracking
and reporting of costs incurred by the replacement project, the ability of
VBA and the Congress to effectively monitor progress could be impaired.
VBA Took Action to Address Project Planning and Management
A second major area of overall management concern was project planning. In
particular, the lack of an integrated master schedule for the VETSNET
project was a major concern articulated by SEI, as well as in our prior
work. An integrated project plan and schedule should incorporate all the
critical areas of system development and be used as a means of determining
what needs to be done and when, as well as measuring progress. Such an
integrated schedule should consider all dependencies and include subtasks
so that deadlines are realistic, and it should incorporate review
activities to allow oversight and approval by high-level managers. Among
other things, the program plan should also include capacity requirements
for resources and technical facilities to support development, testing,
user validation, and production.
SEI was specifically concerned that releases with overlapping
functionality were being developed at the same time, with insufficient
time to document or test requirements; this approach constrained resources
and added complexity because of the need to integrate completed
applications and newly developed functionality. In addition, SEI observed
that the VETSNET program suffered from lack of sufficient test facilities
because it did not have enough information to plan for adequate capacity.
In response to these project planning concerns, VETSNET management, with
contractor support, developed an integrated master schedule to guide
development and implementation of the remaining functionalities for the
replacement system. The VETSNET Integrated Master Schedule, finalized in
September 2006, includes an end-to-end plan and a master schedule.
According to VBA, the end-to-end plan documents the end state of the
project from a business perspective, which had not previously been done.
The master schedule identifies the necessary activities to manage and
control the replacement project through completion. The schedule also
describes a new software release process that provides more time to work
on requirements definition and testing, and allows for more
cross-organizational communications to lessen the possibility of not
meeting requirements.
In addition, the new release process includes a series of management
reviews to help control the software development process and ensure that
top management has continuous visibility of project related activities.
These reviews occur at major steps in the system development life cycle
(as described in fig. 1: initiation, preliminary design, and so on). Such
reviews are intended to ensure that the VETSNET Executive Team and the
VETSNET Executive Board agree and accept that the major tasks of each step
have been properly performed.
Figure 1: VETSNET System Life Cycle and Review Concept
Nonetheless, while the Integrated Master Schedule is an important
accomplishment, it may not ensure that the project sufficiently addresses
capacity planning, one of SEI's areas of concern. According to its
assessment, capacity requirements for the fully functional production
system were unclear. Capacity planning is important because program
progress depends on the availability of necessary system capacity to
perform development and testing; adjustments to such capacity take time
and must be planned. If systems do not have adequate capacity to
accommodate workload, interruptions or slowdowns could occur. According to
SEI, capacity adjustments cannot be made instantly, and program progress
will suffer without sufficient attention to resource requirements.
However, the VETSNET Integrated Master Schedule does not identify
activities or resources devoted to capacity planning. According to
officials, the capacity of the corporate environment (that is, corporate
information systems, applications, and networks) is being monitored by
operational teams with responsibility for maintaining this environment.
According to project officials, VETSNET representatives participate in
daily conference calls in which the performance of corporate applications
is discussed, and changes in application performance are reported to the
VETSNET developers for investigation and corrective action. Project
officials reported that when a performance degradation occurred in some
transactions during performance testing, it was determined that additional
computing capacity was needed and would be acquired. One reason why the
occurrence of degradation had not been anticipated by the VETSNET project
was that capacity planning had not taken place. Unless it ensures that
capacity planning and activities are included in the Integrated Master
Schedule, the replacement project may face other unanticipated
degradations that it must react to after the fact, thus jeopardizing the
project's cost, schedule, and performance.
Conversion Efforts Were Suspended as Advised, but Issues Remain
In its assessment, SEI questioned VBA's approach to developing
functionality while concurrently converting records from the BDN to the
replacement system. It noted that VBA had chosen to complete software
development according to location rather than according to the type of
functionality. Specifically, in 2004, VBA began an effort to remove all
claims activity (both new and existing claims) at one regional office
(Lincoln, Nebraska) from the BDN to the replacement system, developing the
software as necessary to accommodate processing the types of claims
encountered at that site. The intention was to address each regional
office in turn until all sites were converted. According to SEI, this
approach had resulted in the development being stalled by obstacles
arising from the variety of existing claims.14 The contractor advised VBA
to focus first on developing functionality to process original claims and
discontinue efforts to convert existing claims until all the necessary
functionality had been developed, and the replacement system's ability to
handle new cases of any complexity had been proven by actual experience.
14As previously noted, there are variations in types of compensation and
pension benefits, including death pensions for survivors of deceased
wartime veterans, burial benefits, Dependency and Indemnity Compensation
to some surviving parents, and benefits paid to third parties, such as
people to whom a veteran has given power of attorney, medical service
providers, and so on. In addition, different forms of payment must be
accommodated (check or electronic funds transfer).
In accordance with this advice, VBA stopped converting existing records
from the BDN and changed its focus to developing the necessary
functionality to process all new compensation claims. According to the
integrated master schedule, conversion activities are now timed to follow
the release of the needed functionality. That is, according to the
schedule, VBA plans to begin converting each type of record from the BDN
only after the necessary functionality for the replacement system has been
developed and deployed to process that type of record.
In addition, the project is mitigating risk by resuming conversions
beginning with a test phase. Its strategy is first to convert records for
terminated claims--claims that are no longer being paid. Conversion of the
terminated records will be followed by additional conversions of records
for claims receiving payment at Lincoln and Nashville (these two sites are
being used to test system functionality during development). The VETSNET
leadership will consider testing complete with the successful conversion
at these two sites.
However, SEI raised three additional issues with regard to the conversion
of records that VBA has not fully addressed:
o First, SEI expressed concerns that conversion failures could
lead to substantial numbers of records being returned to the BDN.
Because of differences in the database technologies used for the
old system and the replacement system, certain types of errors in
BDN records cause conversion to fail (according to SEI,
approximately 15 percent of all these records are estimated to
have such errors). If records fail to convert correctly, they may
need to be returned to the BDN so that benefits can continue to be
paid. However, this process is not simple and may involve manually
reentering the records.15
o Second, SEI observed that VBA was also depending on manual
processes for determining that records were converted
successfully, including the use of statistically random samples,
and that it was aiming to ensure correctness to a confidence level
of 95 percent. However, in the absence of a straightforward method
for automatically returning records to the BDN, SEI considered the
5 percent risk of error unacceptable for conversions of large
numbers of records.
o Finally, SEI observed that the lack of automated methods and the
complexity of the processes meant that conversions required
careful planning and assurance that adequate staff would be
available to validate records when the conversions took place.
However, the VETSNET leadership has not developed any strategy to
address the possibility that a large number of cases might need to
be returned to the BDN during the testing phase. For example, it
has not included this possibility as a risk in its risk management
plan. The absence of a strategy to address this possibility could
lead to delays in program execution.
Further, VBA has not yet decided whether a possible 5 percent
error rate is acceptable or developed a plan for addressing the
resulting erroneous records. If VBA does not address these issues
in its planning, it increases the risk that veterans may not
receive accurate or timely payments.
Finally, the VETSNET leadership has not yet developed detailed
plans that include the scheduled conversions for each regional
office and identified staff to perform the necessary validation.
Having such plans would reduce the risk that the conversion
process could be delayed or fail.
VBA Is Improving Software Development Processes, but Improvements
Are Not Yet Institutionalized
In addition to actions addressing the overall management concerns
identified by SEI, VBA has steps action to improve its software
development processes in risk management, requirements management,
defect/change management, and performance measures. SEI described
weaknesses in all of these areas. The steps taken have generally
been effective in addressing the identified weaknesses, but VBA
has not yet institutionalized many of these improvements.
According to the VETSNET management team, it made a conscious
decision first to establish the governance, build the
organization, implement processes to gain control, and gather
additional information about the project to assist in prioritizing
the remaining activities. The team also stated that some of the
processes are no longer VBA's responsibility but are now that of
the newly realigned Office of Information and Technology.16
Nonetheless, if VA does not develop and establish documented
policies and procedures to institutionalize these improvements,
they may not be maintained through the life of the project or
available to be applied to other development initiatives.
Risk Management Plan Has Been Revised
Risk management is a process for identifying and assessing risks,
their impact and status, the probability of their occurrence, and
mitigation strategies. Effective risk management includes the
development of a risk management plan and tracking and reporting
progress against the plan. According to SEI, to the extent that
risk management existed at all in the replacement program, it was
conducted on a pro forma basis without real effect on program
decisions. SEI said that risks and risk mitigation activities
needed to be incorporated into all aspects of program planning,
budgeting, scheduling, execution, and review.
In response to these concerns, VBA has instituted risk management
activities that, if properly implemented, should mitigate the
risks associated with the project. Specifically, the VETSNET team,
with contractor support, developed a risk management plan that was
adopted in January 2007. The plan includes procedures for
identifying, validating, analyzing, assessing, developing
mitigation strategies for, controlling and tracking, reporting,
and closing risks. It also establishes criteria for assessing the
severity of the risks and their impact.
The VETSNET leadership also developed a Risk Registry database,
and its contractor reviewed and prioritized the open risks. Each
open risk was evaluated, and a proposed disposition of the risk
was submitted to VETSNET management. Of the 39 open risks, all but
3 had been addressed as of January 2007.
The development documentation for each planned software release
also includes sections on risk. In accordance with these plans,
the VETSNET leadership is currently capturing potential risks and
tracking action items and issues. At weekly status meetings,
VETSNET leadership reviews Risk Registry reports of open risks.
According to the contractor, the reports identify each risk and
provide information on its age, ownership, and severity.
However, these risk management activities have not yet been
institutionalized through the definition and establishment of
associated policies and procedures. If it does not
institutionalize these improvements, VBA increases the possibility
that the VETSNET project's improvements in risk management may not
be maintained through the life of the project.
Requirements Management Has Improved
Requirements management is a process for establishing and
maintaining a common understanding between the business owners and
the developers of the requirements to be addressed, as well as
verifying that the system meets the agreed requirements. SEI's
report commented that the VETSNET project requirements were not
stable, and that the business owners (including subject-matter
experts) and developers were separated by many organizational
layers, resulting in confusion and delays in development of the
system. SEI suggested that VA restructure project activities to
focus on defining an effective requirements process. According to
SEI, the project needed to ensure that subject-matter experts were
included in developing requirements and that evaluation criteria
were established for prioritizing requests for changes to
requirements. Finally, business owners should confirm that the
system is meeting organizational needs.
VBA has instituted requirements management activities that, if
properly implemented, should help avoid the instability and other
requirements problems identified by SEI. Specifically, VBA took
steps to establish a requirements management process and to
stabilize the requirements. For example, the development release
process in the Integrated Master Schedule includes a phase for
requirements identification. In addition, the project has
established and begun applying evaluation criteria to prioritize
change requests for its development releases. Further, until all
claims are completely migrated from the BDN to the replacement
system, in July 2006, the Under Secretary directed that any
additional requirements would have to have his approval.
Responding to SEI's advice regarding the involvement of
subject-matter experts and business owners, VBA designed the new
release process to directly involve subject-matter experts in
requirements workshops. Further, the business teams participate in
user-acceptance testing.
However, these requirements management activities have not yet
been institutionalized through the definition and establishment of
policies and procedures. Until they are established, VBA runs the
risk that the improved processes will not be maintained through
the life of the VETSNET project or used in other software
development projects.
Management Attention Is Being Focused on Major Software Defects
SEI raised numerous concerns regarding the defect process for the
replacement system. These concerns for defect management included
(1) identification of defects, (2) determination of cause, and (3)
disposition of defects--either by correction or workaround.
According to SEI guidance, defect management prevents known
defects from hampering the progress of the program. The management
process should include clearly identifying and tracking defects,
analyzing defects to establish their cause, tracking their
disposition, clearly identifying the rationale for not addressing
any defects (as well as proposing workarounds), and making
information on defects and their resolution broadly available.
SEI's report stated that VBA needed to distinguish defects from
changes to requirements and develop a process for defect
management.
To respond to these concerns and focus program management
attention on major defects, the VETSNET Executive Team, with
contractor support, conducted an audit of existing defects and
revised the defect management process. The audit of the defect
database determined that the VETSNET database used to capture
software defects also included change requests; as a result, work
required to address processes that did not work properly was not
distinguished from requests for added or changed functionality,
which would require review and approval before being addressed. To
address this issue, the team separated defects from change
requests, and a new severity rating scale was developed. All open
defects were recategorized to ensure the major defects would
receive appropriate program management attention. Also, all defect
categorizations must meet the approval of the VETSNET Business
Architect and are scheduled for action as dictated by the severity
level.
Although these steps address many of SEI's concerns regarding
VBA's defect management process, more remains to be done before
the process is institutionalized. The Program Management Office
has reported that actions to revise the defect management process
are complete, but the process description is still in draft, and
policies and procedures have not been fully established. Without
institutionalized policies and procedures for the defect
management process, it may not be maintained consistently through
the life of the project.
The Replacement Initiative Is Tracking Certain Performance
Measures
According to SEI, performance measures are the only effective
mechanism that can provide credible evidence of a program's
progress. The chosen measures must link directly to the expected
accomplishments and goals of the system, and they must be applied
across all activities of the program. In its report, SEI stated
that although VBA was reporting certain types of performance
measures, it was not relating these to progress in system
development. For example, VBA reported the total number of
veterans paid, but did not provide estimates of how many
additional veterans would be paid when the system incorporated
specific functionalities that were under development. SEI
suggested several measures that would provide more evidence of
progress, such as increases in the percentage of original claims
being paid by the replacement system, as well as user satisfaction
and productivity gains resulting from use of the replacement
system applications at regional offices.
In response to these concerns, the replacement project has begun
tracking a number of the measures suggested by SEI, including
o increases in the percentage of original claims being paid by the
replacement system,
o increases in the percentage of veterans' service representatives
using the new system,
o decreases in the percentage of original claims being entered in
the BDN rather than the replacement system.
Although these measures provide indications of VA's progress,
other measures that could demonstrate the effectiveness of the
replacement system have not been developed. For example, VBA has
not developed results-oriented measures to capture user
satisfaction or productivity gains from the system. Without
measuring user satisfaction, VBA has reduced assurance that the
replacement system will be accepted by the users. In addition,
measures of productivity would provide VBA with another indication
of progress toward meeting business needs.
VETSNET Is Currently Processing a Portion of Compensation Claims,
but Much Work Remains
After more than 10 years of effort, including the recent
management, organizational, and process improvements, VBA has
achieved critical functionalities needed to process and pay
certain original compensation claims using the replacement system,
but it remains far from completing the project. For example, the
replacement system is currently being used to process a portion of
the original claims that veterans file for compensation.
Nonetheless, the system requires further development before it can
be used to process claims for the full range of compensation and
pension benefits available to veterans and their dependents. In
addition, VBA still faces the substantial task of moving
approximately 3.5 million beneficiaries who are currently being
served by the BDN to the replacement system.
As designed, VETSNET consists of five major system applications
that are used in processing benefits:
o Share--used to establish claims;17 it records and updates basic
information about veterans and dependents both in the BDN and the
replacement system.
o Modern Award Processing-Development (MAP-D)--used to manage the
claims development process, including the collection of data to
support the claims and the tracking of claims.
o Rating Board Automation 2000 (RBA 2000)--provides laws and
regulations pertaining to disabilities, which are used by rating
specialists in evaluating and rating disability claims.
o Award Processing (Awards)--used to prepare and calculate the
benefit award based on the rating specialist's determination of
the claimant's percentage of disability. It is also used to
authorize the claim for payment.
o Finance and Accounting System (FAS)--used to develop the actual
payment record. FAS generates various accounting reports and
supports generation and audit of benefit payments.
According to VBA officials, all five of the software applications
that make up the new system are now being used in VA's 57 regional
offices to establish and process new compensation claims for
veterans. As of March 2007, VBA leadership reported that the
replacement system was providing monthly compensation payments to
almost 50,000 veterans (out of about 3 million veterans who
receive such payments). In addition, the replacement system has
been processing a steadily increasing percentage of all new
compensation claims completed: this measure was 47 percent in
January 2007, increasing to 60 percent in February and 83 percent
in March.
Nonetheless, considerable work must be accomplished before VBA
will be able to rely on the replacement system to make payments to
all compensation and pension beneficiaries. Specifically, while
all five software applications can now be used to process original
compensation claims for veterans, two of the applications--Awards
and FAS--require further development before the system will be
able to process claims for the full range of benefits available to
veterans and their dependents. Table 2 shows the status of
development of all five applications.
15SEI quoted program staff as calling the process "extraordinarily time
consuming," saying that "It takes 5 minutes to get a veteran into VETSNET
and then 5 days to get him back out."
16This realignment was approved on February 27, 2007.
17The functionalities for the Search and Participant Profile, formerly a
separate application, are now in Share.
Table 2: VETSNET Component Applications and Status of Development
Application Development status Deployment status
Share Completed Fully deployed at
all regional
offices.
MAP-D Completed Fully deployed at
all regional
offices.
RBA 2000 Completed Fully deployed at
all regional
offices.
Awards Functionality to process In partial use in
third-party/nonveteran payee claims is in all regional
application testing. offices.
Requirements for processing survivor
benefits are being developed.
Functionality to process pensions is not yet
under development.
FAS Functionality to process In partial use in
third-party/nonveteran payee claims is in all regional
application testing. offices.
Functionality to generate management reports
is partially in requirements development and
partially in application testing.
Requirements for processing survivor
benefits are being developed.
Functionality to process pensions is not yet
under development.
Source: GAO analysis of VBA data.
According to VBA officials, Awards and FAS do not yet have the
capability to process original claims for payment to recipients
other than veterans: that is, the applications do not have the
functionality to process claims for survivor benefits18 and
third-party/nonveteran payee claims.19 In addition, further
development of these applications is needed to process pension
benefits for qualified veterans and their survivors. Until
enhancements are made to Awards and FAS, these claims must
continue to be processed and paid through the BDN. Also, according
to VBA, FAS does not yet have the capability to generate all the
necessary accounting reports that support the development of
benefits payments to claimants.
18Survivor benefits may be paid to survivors of veterans who were eligible
for either compensation or pension benefits, depending on the
circumstances.
19Examples of third-party/nonveteran payees include people to whom a
veteran has given power of attorney, medical service providers designated
to receive payments, and so on.
As described earlier, VBA now has an Integrated Master Schedule
that incorporates the activities that VBA needs to manage in order
to complete the replacement project. According to the schedule,
the remaining capabilities necessary to process compensation and
pension claims are to be developed and deployed in three software
releases, as shown in table 3.
Table 3: Software Releases Planned to Complete the VETSNET System
Estimated
Release number and title Functionality to be attained completion date
1. Complete Process all compensation claims August 2007
Compensation and add significant FAS
functionality (generate
management reports).
2. Nonincome-Based Process burial, accrued, and February 2008
Survivor Benefits survivor claims.
3. Income-Based Process nonservice-connected August 2008
Pension pension (veteran and survivor)
and parents' Dependent and
Indemnity Compensation Benefit.
Estimated start Estimated
Phases Conversions date completion date
Phase 1 Conversion testing on terminated March 2007 August 2007
records and selected live records
Phase 2 Conversion of live compensation January 2008 October 2008
records
Phase 3 Conversion of pension records November 2008 June 2009
Source: VBA.
As table 3 shows, VBA does not expect to complete the development
of the functionalities needed to process all new compensation and
pension claims until August 2008. However, according to VBA, the
estimated completion date is the planned date for completing all
development and testing, but it is not necessarily the date when
users will be able to begin using the new system. Before such use
can begin, other activities need to occur. For example, users must
receive training, and VETSNET program management must authorize
the use of the system at each regional office.
In addition to its remaining software development activities, VBA
also faces the challenge of converting records for claims
currently paid by the BDN to the replacement system. Existing
compensation and pension cases on the BDN number about 3 million
and about 535,000, respectively. Table 4 shows the phases in which
VA is planning to perform conversions, according to its Integrated
Master Schedule.
Table 4: Conversion Phases
Source: GAO analysis of VBA data.
As the table shows, VBA conversion efforts began in March 2007.
VBA first performed conversion testing on 310,000 terminated (that
is, inactive) compensation cases so that it could develop and
apply lessons learned to the conversion of live records. According
to VETSNET officials, VBA planned to continue testing by
converting live cases at two regional offices (Lincoln and
Nashville) that were used as testing sites during development. It
then plans to perform the conversion of all live compensation
cases. After the compensation conversion is complete, VBA plans to
begin efforts to convert pension benefits cases.
Based on VETSNET project documentation, activities supporting the
releases have so far been performed on time, consistent with the
milestones in the recently finalized Integrated Master Schedule.
For example, VA completed the Project Initiation and Review
Authorization for Release 1 on September 7, 2006, as scheduled
(see fig. 1, shown earlier in the report, for the phases of system
development and the required milestone reviews). It also completed
the Preliminary Design Review and the Critical Design Review as
scheduled (on November 20 and December 22, respectively). Planning
for Release 2 is also on schedule: a kickoff meeting was held on
January 24, 2007, which established the scope of the release, and
the Project Initiation and Review Authorization was conducted on
February 8.
Conclusions
VA has responded to SEI's assessment by making significant changes
in its approach to the project and its overall management,
including slowing the pace of development, establishing a new
governance structure, and ensuring staff resources. However, VBA
has not yet addressed all the issues raised by the SEI assessment.
That is, it has not ensured ownership responsibility for total
system and process operating costs, because it is not currently
monitoring and reporting in-house expenditures for the project. It
has not defined processes and resources for capacity planning for
the project. In addition, VBA has not yet addressed issues related
to the conversion of records now on the BDN to the replacement
system. Specifically, it has not addressed the risk that large
numbers of records may need to be returned to the BDN, decided on
the degree of confidence it will require that records are
converted accurately, or developed complete plans for converting
and validating records. In addition, although VBA has improved key
processes for managing the software development, these processes
have not yet been institutionalized in defined policies and
procedures, and performance measures of productivity and user
satisfaction have not been developed. VETSNET management has
stated that it gave priority to other activities, such as
establishing appropriate governance and organizational structures,
and that it is still gathering information to assist in
prioritizing the activities that remain.
Much work remains to be done to complete the VETSNET initiative.
Although VBA has substantially increased the number of claims
being paid by the replacement system, it must not only finish the
development and deployment of the software, it must also convert
the over 3.5 million records now on the BDN to the replacement
system. Addressing the remaining issues identified by SEI would
improve VBA's chances of successfully completing the replacement
system and ending reliance on the aged BDN to pay compensation and
pension benefits.
Recommendations for Executive Action
To enhance the likelihood that the replacement system will be
successfully completed and implemented, we are recommending that
the Secretary of Veterans Affairs take the following five actions:
o Direct the CIO to institute measures to track in-house
expenditures for the project.
o Direct the VETSNET project to include activities for capacity
planning in the VETSNET Integrated Master Schedule and ensure that
resources are available for these activities.
o Direct VBA to (1) develop a strategy to address the risk that
large numbers of records may need to be returned to the BDN; (2)
determine whether a greater confidence level for accuracy should
be required in the conversion process; and (3) develop a detailed
validation plan that includes the scheduled conversions for each
regional office and the validation team members needed for that
specific conversion.
o Direct the CIO to document and incorporate the improved
processes for managing risks, requirements, and defects into
specific policy and guidance for the replacement initiative and
for future use throughout VBA.
o Direct the replacement project to develop effective
results-oriented performance measures that show changes in
efficiency, economy, or improvements in mission performance, as
well as measures of user satisfaction, and to monitor and report
on the progress of the initiative according to these measures.
Agency Comments and Our Evaluation
In providing written comments on a draft of this report, the
Secretary of Veterans Affairs agreed with our conclusions and
concurred with the report's recommendations. (The department's
comments are reproduced in app. II.) The comments described
actions planned that respond to our recommendations, such as
incorporating processes developed for the VETSNET project in
standard project management policies, processes, and procedures
that would be used for all IT projects in the department. In
addition, the comments provided further information on actions
already taken, such as details of the records conversion process.
If the planned actions are properly implemented, they could help
strengthen the department's management of the replacement system
project and improve the chances that the system will be
successfully completed.
We are sending copies of this report to the Chairman and Ranking
Minority Member of Committee on Veterans' Affairs. We are also
sending copies to the Secretary of Veterans Affairs and
appropriate congressional committees. We will make copies
available to other interested parties upon request. Copies of this
report will also be made available at no charge on GAO's Web site
at http://www.gao.gov .
Should you or your staff have any questions about this report,
please contact me at (202) 512-6304 or by e-mail at
[email protected]. Contact points for our Offices of Congressional
Relations and Public Affairs may be found on the last page of this
report. Key contributors to this report are listed in appendix
III.
Sincerely yours,
Valerie C. Melvin
Acting Director
Human Capital and Management Information Systems
Appendix I: Objectives, Scope, and Methodology
Our objectives were to determine (1) to what extent the Department
of Veterans Affairs (VA) has followed the course of action
recommended by the Carnegie Mellon Software Engineering Institute
(SEI) and addressed the concerns that it raised and (2) the
current status of the replacement project, the Veterans Service
Network (VETSNET).
To determine the actions taken to implement SEI's recommended
approach and address the concerns it raised, we
o determined the recommended actions by analyzing the report;
o compared the concerns identified in the assessment to actions
planned, actions undertaken but not completed, and actions
implemented by VA officials or contractors;
o interviewed contractor, VA, and VETSNET program office officials
to gain an understanding about processes developed and procedures
implemented; and
o obtained and reviewed relevant VA and contractor documents that
disclosed or validated VA responses to SEI's concerns.
To determine the status of system development efforts and the
extent that tasks planned for the initiative were completed, we
analyzed VA and contractor documentation regarding system
operations and development, time frames, and activities planned.
We analyzed VA documents that disclosed costs to date and costs
planned for completion of the initiative. We did not assess the
accuracy of the cost data provided to us. We supplemented our
analyses with interviews of VA and contractor personnel involved
in the replacement initiative.
We visited the Nashville and St. Petersburg regional offices to
observe the replacement system in operation and the processes and
procedures used to test and validate the replacement system as it
was being developed and implemented. We analyzed VA documentation
and relevant evidence from contractors involved in the replacement
effort to establish the work remaining to complete the project.
Finally, we interviewed cognizant VA and contractor officials,
responsible for developing, testing, and implementing the
replacement system.
We performed our work at VA offices in Washington, D.C., and at VA
regional offices in Nashville, Tennessee, and St. Petersburg,
Florida, from April 2006 to April 2007 in accordance with
generally accepted government auditing standards.
Appendix II: Comments from the Department of Veterans Affairs
Appendix III: GAO Contact and Staff Acknowledgments
GAO Contact
Valerie Melvin, (202) 512-6304 or [email protected]
Staff Acknowledgments
In addition to the individual named above, key contributions were
made to this report by Barbara Oliver, Assistant Director;
Nabajyoti Barkakati; Barbara Collier; Neil Doherty; Matt Grote;
Robert Williams; and Charles Youman.
GAO�s Mission
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in
meeting its constitutional responsibilities and to help improve
the performance and accountability of the federal government for
the American people. GAO examines the use of public funds;
evaluates federal programs and policies; and provides analyses,
recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony
The fastest and easiest way to obtain copies of GAO documents at
no cost is through GAO's Web site ( www.gao.gov ). Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of
newly posted products every afternoon, go to www.gao.gov and
select "Subscribe to Updates."
Order by Mail or Phone
The first copy of each printed report is free. Additional copies
are $2 each. A check or money order should be made out to the
Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are
discounted 25 percent. Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM
Washington, D.C. 20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax:
(202) 512-6061
To Report Fraud, Waste, and Abuse in Federal Programs
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail:
[email protected] Automated answering system: (800) 424-5454 or
(202) 512-7470
Congressional Relations
Gloria Jarmon, Managing Director, [email protected] (202)
512-4400 U.S. Government Accountability Office, 441 G Street NW,
Room 7125 Washington, D.C. 20548
Public Affairs
Paul Anderson, Managing Director, [email protected] (202)
512-4800 U.S. Government Accountability Office, 441 G Street NW,
Room 7149 Washington, D.C. 20548
(310758)
www.gao.gov/cgi-bin/getrpt?GAO-07-614 .
To view the full product, including the scope
and methodology, click on the link above.
For more information, contact Valerie Melvin at (202) 512-6304 or
[email protected].
Highlights of [47]GAO-07-614 , a report to the Chairman, Committee on
Veterans' Affairs, House of Representatives
April 2007
VETERANS BENEFITS ADMINISTRATION
Progress Made in Long-Term Effort to Replace Benefits Payment System, but
Challenges Persist
Since 1996, the Veterans Benefit Administration (VBA) has been working on
an initiative to replace its aging system for paying compensation and
pension benefits. In 2005, concerned about the slow pace of development,
VBA contracted with the Software Engineering Institute (SEI) for an
independent evaluation of the project, known as the Veterans Service
Network (VETSNET). SEI advised VBA to continue working on the project at a
reduced pace while addressing management and organization weaknesses that
it determined had hampered the project's progress.
GAO was requested to determine to what extent the VETSNET project has
followed the course of action recommended by SEI and describe the
project's current status.
To perform its review, GAO analyzed project documentation, conducted site
visits, and interviewed key program officials.
[48]What GAO Recommends
To sustain the improved management and software development processes
currently being used by VETSNET project management, GAO is making
recommendations to the Secretary of Veterans Affairs in several areas,
including cost tracking, capacity planning and management, and performance
measures. The Secretary agreed with GAO's recommendations and described
actions planned in response.
VBA is generally following the course of action recommended by SEI by
continuing to work on the replacement initiative at a reduced pace, while
taking action to address identified weaknesses in overall management and
software development processes. For example, VBA established a new
governance structure for the initiative that included senior management
and involved all stakeholders, and it incorporated all critical areas of
system development in an integrated master schedule. However, not all of
SEI's management concerns have been addressed. For example, SEI advised
VBA to ensure that stakeholders take ownership responsibility for the
project, including the total system and process operating costs; however,
although VBA is tracking costs incurred by contractors, it is not yet
tracking and reporting in-house costs incurred by the project. Further,
although the project has improved its management processes, such as
establishing a process to manage and stabilize system requirements, it has
not yet developed processes for capacity planning and management. This
will be important for ensuring that further VETSNET development does not
lead to delays and slowdowns in processing of benefits. In addition,
although the project has established certain performance measures, it has
not yet established results-oriented measures for productivity and user
satisfaction, both of which will be important for measuring progress.
Finally, the process improvements that VBA has incorporated in the
replacement initiative remain in draft and have not been established
through documented policies and procedures. If VBA does not
institutionalize these improvements, it increases the risk that they may
not be maintained through the life of the project or be available for
application to other development initiatives.
After more than 10 years of effort, including the recent management,
organizational, and process improvements, VBA has developed critical
functionalities needed to process and pay certain original compensation
claims using the replacement system, but it remains far from completing
the project. According to VBA officials, all five of the major software
applications that make up the new system are now being used to establish
and process new compensation claims for veterans. In total, the
replacement system is currently providing monthly compensation payments to
almost 50,000 veterans (out of about 3 million veterans who receive such
payments); the system was used to process about 83 percent of all new
compensation claims completed in March 2007. Nonetheless, the system
requires further development before it can be used to process claims for
the full range of compensation and pension benefits available to veterans
and their dependents.
References
Visible links
28. http://www.gao.gov/cgi-bin/getrpt?GAO/AIMD-96-90
29. http://www.gao.gov/cgi-bin/getrpt?GAO/AIMD-97-154
30. http://www.gao.gov/cgi-bin/getrpt?GAO/T-AIMD-00-321
31. http://www.gao.gov/cgi-bin/getrpt?GAO-01-550T
32. http://www.gao.gov/cgi-bin/getrpt?GAO-02-1054T
33. http://www.gao.gov/cgi-bin/getrpt?GAO-06-905T
34. http://www.gao.gov/cgi-bin/getrpt?GAO-06-905T
35. http://www.vba.va.gov/bln/21/index.htm
36. http://www.gibill.va.gov/
37. http://www.insurance.va.gov/
38. http://www.gao.gov/cgi-bin/getrpt?GAO/AIMD-96-90
47. http://www.gao.gov/cgi-bin/getrpt?GAO-07-614
*** End of document. ***