Veterans Benefits Administration: Progress Made in Long-Term	 
Effort to Replace Benefits Payment System, but Challenges Persist
(27-APR-07, GAO-07-614).					 
                                                                 
Since 1996, the Veterans Benefit Administration (VBA) has been	 
working on an initiative to replace its aging system for paying  
compensation and pension benefits. In 2005, concerned about the  
slow pace of development, VBA contracted with the Software	 
Engineering Institute (SEI) for an independent evaluation of the 
project, known as the Veterans Service Network (VETSNET). SEI	 
advised VBA to continue working on the project at a reduced pace 
while addressing management and organization weaknesses that it  
determined had hampered the project's progress. GAO was requested
to determine to what extent the VETSNET project has followed the 
course of action recommended by SEI and describe the project's	 
current status. To perform its review, GAO analyzed project	 
documentation, conducted site visits, and interviewed key program
officials.							 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-07-614 					        
    ACCNO:   A68851						        
  TITLE:     Veterans Benefits Administration: Progress Made in       
Long-Term Effort to Replace Benefits Payment System, but	 
Challenges Persist						 
     DATE:   04/27/2007 
  SUBJECT:   Claims processing					 
	     Contractors					 
	     Performance measures				 
	     Program management 				 
	     Records						 
	     Software						 
	     Systems conversions				 
	     Veterans benefits					 
	     Veterans disability compensation			 
	     Veterans pensions					 
	     Program evaluation 				 
	     Policies and procedures				 
	     Veterans Service Network				 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-07-614

   

     * [1]Results in Brief
     * [2]Background

          * [3]The Processes Supporting the Variety of Compensation and Pen
          * [4]Our Prior Products Identified Weaknesses in the Development
          * [5]SEI Assessed System Replacement Initiative and Recommended a

     * [6]VBA Is Following Course Recommended by SEI, but Weaknesses R

          * [7]VBA Is Taking Action on Certain Overall Management Concerns

               * [8]Top Management Included in Revised Governance Structure
               * [9]VBA Took Action to Address Project Planning and
                 Management
               * [10]Conversion Efforts Were Suspended as Advised, but Issues
                 Rem

          * [11]VBA Is Improving Software Development Processes, but Improve

               * [12]Risk Management Plan Has Been Revised
               * [13]Requirements Management Has Improved
               * [14]Management Attention Is Being Focused on Major Software
                 Defe
               * [15]The Replacement Initiative Is Tracking Certain
                 Performance M

     * [16]VETSNET Is Currently Processing a Portion of Compensation Cl
     * [17]Conclusions
     * [18]Recommendations for Executive Action
     * [19]Agency Comments and Our Evaluation
     * [20]GAO Contact
     * [21]Staff Acknowledgments
     * [22]GAO's Mission
     * [23]Obtaining Copies of GAO Reports and Testimony

          * [24]Order by Mail or Phone

     * [25]To Report Fraud, Waste, and Abuse in Federal Programs
     * [26]Congressional Relations
     * [27]Public Affairs

Report to the Chairman, Committee on Veterans' Affairs, House of
Representatives

United States Government Accountability Office

GAO

April 2007

VETERANS BENEFITS ADMINISTRATION

Progress Made in Long-Term Effort to Replace Benefits Payment System, but
Challenges Persist

GAO-07-614

Contents

Letter 1

Results in Brief 3
Background 4
VBA Is Following Course Recommended by SEI, but Weaknesses Remain to Be
Addressed 10
VETSNET Is Currently Processing a Portion of Compensation Claims, but Much
Work Remains 22
Conclusions 26
Recommendations for Executive Action 27
Agency Comments and Our Evaluation 28
Appendix I Objectives, Scope, and Methodology 30
Appendix II Comments from the Department of Veterans Affairs 32
Appendix III GAO Contact and Staff Acknowledgments 38

Tables

Table 1: Roles and Responsibilities of VETSNET Organizational Elements 12
Table 2: VETSNET Component Applications and Status of Development 24
Table 3: Software Releases Planned to Complete the VETSNET System 25
Table 4: Conversion Phases 26

Figure

Figure 1: VETSNET System Life Cycle and Review Concept 15

Abbreviations

BDN Benefits Delivery Network

CIO Chief Information Officer
FAS Finance and Accounting System
IT Information Technology
MAP-D Modern Award Processing-Development
RBA 2000 Rating Board Automation 2000
SEI Carnegie Mellon Software Engineering Institute
VA Department of Veterans Affairs
VBA Veterans Benefits Administration
VETSNET Veterans Service Network
VRE Vocational Rehabilitation and Employment

This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.

United States Government Accountability Office
Washington, DC 20548

April 27, 2007

The Honorable Bob Filner Chairman,
Committee on Veterans' Affairs
House of Representatives

Dear Mr. Chairman:

Since 1996, the Veterans Benefits Administration (VBA), a major component
of the Department of Veterans Affairs (VA), has been undertaking a project
to replace its compensation and pension benefits payment system. VBA's
existing system, the Benefits Delivery Network (BDN), is essential to
ensuring the accurate processing of compensation and pension benefits
payments to over 3.5 million veterans and their dependents each month.
However, this system, which has been in operation for more than 40 years,
is based on antiquated software programs, which have become increasingly
difficult and costly to maintain. VBA is in the process of replacing the
aging BDN with a faster, more flexible, and higher capacity system. When
it began this project in 1996, which it generally refers to as the
Veterans Service Network (VETSNET),1 VBA had planned to complete the
replacement system in May 1998 at an estimated cost of $8 million.

Over the years, we have reported on numerous problems that VBA has
encountered in completing the replacement system.2 Our prior work found,
for example, that the project was begun before VBA had fully developed its
business requirements, resulting in confusion over the requirements to be
addressed; in addition, VBA's software development capability was too
immature to ensure that the agency could reliably develop and maintain
high-quality software on any major project within cost and schedule
constraints. In 2002, we offered a number of recommendations to improve
managerial and program weaknesses, including that VBA appoint a project
manager; thoroughly analyze its initiative; and develop a number of plans,
including a revised compensation and pension replacement strategy and an
integrated project plan. VA concurred with our recommendations, and as we
last reported in June 2006,3 it took several actions to address them. For
example, it appointed a full-time project manager, and the project team
reported that it had completed certifications of users' requirements for
the system's applications. Nonetheless, these actions did not implement
all our recommendations and were not sufficient to establish the program
on a solid footing: certain basic requirements of sound project
management, such as an integrated project plan for the replacement system,
continued to be lacking.

1It also refers to the initiative as the compensation and pension or C&P
replacement system.

2GAO, Software Capability Evaluation: VA's Software Development Process Is
Immature, [28]GAO/AIMD-96-90 (Washington, D.C.: June 19, 1996); Veterans
Benefits Modernization: VBA Has Begun to Address Software Development
Weaknesses but Work Remains, [29]GAO/AIMD-97-154 (Washington, D.C.: Sept.
15, 1997); VA Information Technology: Progress Continues Although
Vulnerabilities Remain, [30]GAO/T-AIMD-00-321 (Washington, D.C.: Sept. 21,
2000); VA Information Technology: Important Initiatives Begun, Yet Serious
Vulnerabilities Persist, [31]GAO-01-550T (Washington, D.C.: Apr. 4, 2001);
VA Information Technology: Management Making Important Progress in
Addressing Key Challenges, [32]GAO-02-1054T (Washington, D.C.: Sept. 26,
2002); and Information Technology: VA and DOD Face Challenges in
Completing Key Efforts, [33]GAO-06-905T (Washington, D.C.: June 22, 2006).

In 2005, the VA Chief Information Officer (CIO) and Under Secretary for
Benefits became concerned about continuing problems with the replacement
project and contracted for an independent assessment of the department's
options for the project, including whether the project should be
terminated. This assessment, conducted by the Carnegie Mellon Software
Engineering Institute (SEI), concluded that the replacement project faced
many risks arising from management and organizational issues, but no
technical barriers that could not be overcome.4 According to SEI, a new
system was still needed, and VBA would not be able to successfully deliver
a full, workable solution unless it addressed its management and
organizational weaknesses. SEI recommended that VBA continue to work on
the project at a reduced pace, while taking an aggressive approach to
addressing the identified weaknesses.

Given the importance of ensuring the effective and efficient delivery of
veterans' benefits, we were requested to review the department's
continuing efforts to develop and implement the compensation and pension
replacement system. Specifically, our objectives were to determine (1) to
what extent VA has followed the course of action recommended by SEI and
addressed the concerns that it raised and (2) the current status of the
replacement project.

3 [34]GAO-06-905T .

4Kathryn Ambrose, William Novak, Steve Palmquist, Ray Williams, and Carol
Woody, Report of the Independent Technical Assessment on the Department of
Veterans Affairs VETSNET Program (Carnegie Mellon Software Engineering
Institute, September 2005).

In conducting this study, we visited VBA's Nashville Regional Office to
observe processing of new benefits claims in the compensation and pension
replacement system, and we visited the St. Petersburg Regional Office to
observe processes and procedures used to test and validate key
functionalities of the replacement system. We obtained and analyzed
documents related to the replacement initiative and to SEI's review of the
initiative. We supplemented our analysis with interviews of VBA and
contractor personnel at headquarters and the two regional offices.
Appendix I contains a more detailed discussion of our objectives, scope,
and methodology. We conducted our study between April 2006 and April 2007
in accordance with generally accepted government auditing standards.

Results in Brief

VBA is generally following the course of action recommended by SEI by
continuing to work on the replacement initiative at a reduced pace, while
taking action to address identified weaknesses in overall management and
software development processes. For example, VBA established a new
governance structure for the initiative that included senior management
and involved all stakeholders, and it incorporated all critical areas of
system development in an integrated master schedule. However, VBA did not
address all of SEI's management concerns. Specifically, SEI advised VBA to
ensure that stakeholders take ownership responsibility for the project,
including the total system and process operating costs; however, although
VBA is tracking costs incurred by contractors, it is not yet tracking and
reporting in-house costs incurred by the project. Further, VBA has not yet
developed sufficient plans for performing the substantial task of moving
records from the BDN to the replacement system (approximately 3.5 million
beneficiaries are currently being served by the older system). VBA did
take steps to improve its software development processes, such as
establishing risk and requirements management processes, but some have not
been addressed. For example, still to be addressed are processes for
capacity planning and management, which will be important for ensuring
that further development does not lead to slowdowns in processing of
benefits. Finally, VBA has not yet documented policies and procedures to
institutionalize all the improvements that it has incorporated in the
replacement initiative. According to the replacement project's management
team, it made a conscious decision first to establish the governance and
build the organization, among other things, and it is still prioritizing
remaining tasks. However, if VBA does not institutionalize the
improvements made, it increases the risk that these process improvements
may not be maintained through the life of the project or be available for
application to other development initiatives.

After more than 10 years of effort, including the recent management,
organizational, and process improvements, VBA has developed critical
functionalities needed to process and pay certain original compensation
claims using the replacement system, but it remains far from completing
the project. According to VBA officials, all five of the major software
applications that make up the new system are now being used in VA's
regional offices to establish and process new compensation claims for
veterans. In total, the replacement system is currently providing monthly
compensation payments to almost 50,000 veterans (out of about 3 million
veterans who receive such payments); the system was used to process about
83 percent of all new compensation claims completed in March 2007.
Nonetheless, the system requires further development before it can be used
to process claims for the full range of compensation and pension benefits
available to veterans and their dependents. In addition, VBA still faces
the substantial task of converting records for the approximately 3.5
million beneficiaries currently being served by the BDN to the replacement
system.

To sustain the improved management and software development processes
currently being used by VETSNET project management, we are making
recommendations to the Secretary of Veterans Affairs in several areas,
including cost tracking, capacity planning and management, and performance
measures.

In providing written comments on a draft of this report, the Secretary of
Veterans Affairs agreed with our conclusions and concurred with the
report's recommendations. (The department's comments are reproduced in
app. II.) The comments described actions planned that respond to our
recommendations, as well as providing further information on relevant
actions already taken. If the planned actions are properly implemented,
they could help strengthen the department's management of the replacement
system project and improve the chances that the system will be
successfully completed.

Background

VBA provides benefits for veterans and their families through five
programs: (1) [35]compensation and pension , (2) [36]education , (3)
vocational rehabilitation and employment (VRE) services, (4) loan
guaranty, and (5) [37]life insurance . It relies on the BDN to administer
benefit programs for three of VBA's five programs: compensation and
pension, education, and VRE services.

Replacing the aging BDN has been a focus of systems development efforts at
VBA since 1986.5 Originally, the administration planned to modernize the
entire system, but after experiencing numerous false starts and spending
approximately $300 million on the overall modernization of the BDN, VBA
revised its strategy in 1996. It narrowed its focus to replacing only
those functionalities that support the compensation and pension program,
and began developing a replacement system, which it called VETSNET.

As reported by the department in its fiscal year 2008 budget submission,
the compensation and pension program is the largest of the three programs
that the BDN supports:

           o The compensation and pension program paid about $35 billion in
           benefits in fiscal year 2006 to about 3.6 million veterans or
           veterans' family members.

                        o Of this amount, compensation programs paid benefits
                        of about $31 billion to about 3.1 million recipients.

                        o Pension programs paid benefits of about $3.5
                        billion to about 535,000 recipients.

           o The education program paid about $2.8 billion to about 498,000
           veterans or their dependents in fiscal year 2006.6 
           o The VRE services program paid about $574 million for VRE
           services in 2006 and provides rehabilitation services to
           approximately 65,700 disabled veteran participants per year.7
		   
		   The Processes Supporting the Variety of Compensation and Pensions
		   Benefits Are Complex

           One of the challenges of developing the replacement system is that
           it must include processes to support the administration of a
           complex set of benefits. Different categories of veterans and
           their families are eligible for a number of different types of
           benefits and payments, some of which are based on financial need.
           Compensation programs, which are based on service-connected
           disability or death, provide direct payments to veterans and/or
           veterans' dependents and survivors. These programs are not based
           on income. Pension benefits programs, on the other hand, are
           income based; these are designed to provide income support to
           eligible veterans and their families who experience financial
           hardship. Eligible veterans are those who served in wartime and
           are permanently and totally disabled for reasons that are not
           service-connected (or who are age 65 or older). Veterans are also
           eligible for burial benefits.

           Survivor benefits may be paid to eligible survivors of veterans,
           depending on the circumstances. Some of these benefits are based
           on financial need, such as death pensions for some surviving
           spouses and children of deceased wartime veterans, and Dependency
           and Indemnity Compensation to some surviving parents.

           Finally, certain benefits may be paid to third parties, such as
           individuals to whom a veteran has given power of attorney or
           medical service providers designated to receive payments on the
           veteran's behalf.

           Generally, VBA administers benefit programs through 57 veterans
           benefits regional offices8 in a process that requires a number of
           steps, depending on the type of claim. When a veteran submits, for
           example, a compensation claim to any of the regional offices, a
           veterans service representative must obtain the relevant evidence
           to evaluate the claim (such as the veteran's military service
           records, medical examinations, and treatment records from VA
           medical facilities or private medical service providers). In the
           case of pension claims, income information would also be
           collected.

           Once all the necessary evidence has been compiled, a rating
           specialist evaluates the claim and determines whether the claimant
           is eligible for benefits. If the veteran is determined to be
           eligible for disability compensation, the Rating Veterans Service
           Representative assigns a percentage rating based on the veteran's
           degree of disability. This percentage is used in calculating the
           amount of payment.

           Benefits received by veterans are subject to change depending on
           changing circumstances. More than half of VBA's workload consists
           of dealing with such changes. If a veteran believes that a
           service-connected condition has worsened, for example, the veteran
           may ask for additional benefits by submitting another claim.9 The
           first claim submitted by a veteran is referred to as the original
           claim, and a subsequent change is referred to as a reopened claim.
		   
		   Our Prior Products Identified Weaknesses in the Development and
		   Implementation of the Replacement System

           Since its inception, VETSNET has been plagued by problems. Over
           the years, we have reported on the project, highlighting concerns
           about VBA's software development capabilities.

           In 1996, our assessment of the department's software development
           capability determined that it was immature.10 In our assessment,
           we specifically examined VETSNET and concluded that VBA could not
           reliably develop and maintain high-quality software on any major
           project within existing cost and schedule constraints. The
           department showed significant weaknesses in requirements
           management, software project planning, and software subcontract
           management, with no identifiable strengths. We also testified that
           VBA did not follow sound systems development practices on VETSNET,
           and we concluded that its modernization efforts had inherent
           risks.

           Between 1996 and 2002, we continued to identify the department's
           weak software development capability as a significant factor
           contributing to persistent problems in developing and implementing
           the system. We also reported that VBA continued to work on VETSNET
           without an integrated project plan. As a result, the development
           of the system continued to suffer from problems in several areas,
           including project management, requirements development, and
           testing.

           Over the years, we made several recommendations aimed at improving
           VA's software development capabilities. Among our recommendations
           was that the department take actions to achieve greater maturity
           in its software development processes11 and that it delay any
           major investment in software development (beyond that needed to
           sustain critical day-to-day operations) until it had done so. In
           addition, we made specific recommendations aimed at improving
           VETSNET development. For example, we recommended that VA appoint a
           project manager, thoroughly analyze its current initiative, and
           develop a number of plans, including a revised compensation and
           pension replacement strategy and an integrated project plan.

           VA concurred with our recommendations and took several actions to
           address them. For example, it appointed a full-time project
           manager and ensured that business needs were met by certification
           of user requirements for the system applications. The actions
           taken addressed some of our specific concerns; however, they were
           not sufficient to fully implement our recommendations or to
           establish the program on a sound footing.
		   
		   SEI Assessed System Replacement Initiative and Recommended a
		   Changed Approach

           As a result of continuing concerns about the replacement project,
           in 2005 VA's CIO and its Under Secretary for Benefits contracted
           for an independent assessment of the department's options for the
           initiative.12 The chosen contractor, SEI, is a federally funded
           research and development center operated by Carnegie Mellon
           University. Its mission is to advance software engineering and
           related disciplines to ensure the development and operation of
           systems with predictable and improved cost, schedule, and quality.

           SEI recommended that the department reduce the pace of development
           while at the same time taking an aggressive approach to dealing
           with management and organizational weaknesses hampering VBA's
           ability to complete the replacement system. According to SEI,
           these management and organizational concerns needed to be
           addressed before the replacement initiative or any similar project
           could deliver a full, workable solution.

           For example, the contractor stressed the importance of setting
           realistic deadlines and commented that there was no credible
           evidence that VETSNET would be complete by the target date, which
           at the time of the review had slipped to December 2006. According
           to the assessment, because this deadline was unrealistic, VBA
           needed to plan and budget for supporting the BDN so that its
           ability to pay veterans' benefits would not be disrupted. SEI also
           noted that different organizational components had independent
           schedules and priorities, which caused confusion and deprived the
           department of a program perspective. Further, the contractor
           concluded that VBA needed to give priority to establishing sound
           program management to ensure that the project could meet targeted
           dates. These and other observations were consistent with our
           long-standing concerns regarding fundamental deficiencies in VBA's
           management of the project.

           To help VBA implement the overall recommendation, the contractor's
           assessment included numerous discussions of activities needed to
           address these areas of concern, which can be generally categorized
           as falling into two major types:

           o Overall management concerns with regard to the initiative
           included

                        o governance structure, including assigning ownership
                        for the project and its costs;

                        o project planning, including the development
                        schedule and capacity planning; and

                        o conversion of records currently on the BDN to the
                        replacement system.

           o Software development process improvements were needed in the
           following areas:
		   
		   VBA Is Following Course Recommended by SEI, but Weaknesses Remain
		   to Be Addressed

           As recommended by SEI, VBA is continuing to work on the
           replacement initiative at a reduced pace and taking action to
           address identified weaknesses in the project's overall management
           and software development processes. For example, VBA has
           established a new governance structure and has developed an
           integrated master schedule that provides additional time and
           includes the full range of project activities. However, additional
           effort is needed to complete a number of the corrective actions,
           such as improving project accountability through monitoring and
           reporting all project costs. Further, VBA has not yet
           institutionalized many of the improvements that it has undertaken
           for the initiative. In particular, process improvements remain in
           draft and have not been established through documented policies
           and procedures. According to the VETSNET management team, it gave
           priority to other activities, such as establishing appropriate
           governance and organizational structures, and it is still
           gathering information to assist in prioritizing the activities
           that remain. Nonetheless, if VBA does not institutionalize these
           improvements, it increases the risk that these process
           improvements may not be maintained through the life of the project
           or be available for application to other development initiatives.
		   
		   VBA Is Taking Action on Certain Overall Management Concerns

           SEI concluded that VBA's management issues would need to be
           addressed as part of the implementation of its overall
           recommendation. SEI's overall management concerns focused on the
           project governance, project planning, and conversion of records
           currently on the BDN to the replacement system.
		   
		     Top Management Included in Revised Governance Structure

           SEI guidance for software development stresses the need for
           organizational commitment and the involvement of senior management
           in overall project governance.13 In its assessment, SEI noted that
           because management of the VETSNET project had been assigned to
           VBA's information technology (IT) group, certain activities
           critical to the veterans' benefits program, but not traditionally
           managed by the IT group, had not been visible to the project's
           management. The contractor pointed out that the IT group, business
           lines, and regional offices needed to share ownership and
           management of the replacement project through an established
           governance process and that the project management office should
           include business representatives. According to SEI, the project
           needed to establish ownership responsibility, including addressing
           total system and process operating costs.

           In response to the assessment, VBA developed a new governance
           structure for the initiative, which the Under Secretary for
           Benefits approved in March 2006. In the new structure, the VETSNET
           Executive Board that had been in place was expanded and
           reorganized to serve as a focal point and major governance
           mechanism for the replacement initiative. A Special Assistant
           (reporting directly to the Under Secretary) was appointed to
           coordinate and oversee the initiative as the head of the VETSNET
           Executive Team, which was established to provide day-to-day
           operational control and oversight of the replacement initiative.
           Implementation Teams were also established to conduct the
           day-to-day activities associated with implementing the initiative.
           This governance structure established a process for IT, business
           lines, and regional offices to share ownership and management, as
           SEI advised.

           The roles and membership of each of the organizational elements in
           the new governance structure are described in table 1.

                        o risk management,

                        o requirements management,

                        o defect management, and

                        o program measures.

5The BDN currently runs on aging software: COBOL programs and a
nonrelational database. Analysts have indicated that moving from a
nonrelational database of the BDN type to a more modern relational
database is a challenging task.

6This program provides veterans, service members, reservists, and certain
veterans' dependents with educational resources.

7To help veterans with service-connected disabilities become employable
and obtain and maintain suitable employment, the program provides a range
of direct and supportive services. These services include comprehensive
evaluation of rehabilitation needs (vocational or independent living);
training and employment services to obtain or maintain suitable
employment; and independent living services, such as training and
specialized equipment to enable independence in the activities of daily
living.

8Not all regional offices process all the different types of benefits. For
example, adjustments to pension claims are processed at three pension
maintenance centers in St. Paul, Milwaukee, and Philadelphia, and
education claims are processed only at the regional offices in Atlanta,
Buffalo, Muskogee, and St. Louis. The Philadelphia Regional Office and
Insurance Center has sole responsibility for insurance benefit processing,
and nine Regional Loan Centers administer the loan guaranty program.

9Claims may also be reopened if a veteran provides additional information
on a claim that was denied.

10 [38]GAO/AIMD-96-90 .

11Specifically, at the repeatable level of process maturity, basic project
management processes are established to track cost, schedule, and
functionality, and the necessary process discipline is in place to repeat
earlier successes on projects with similar applications.

12Kathryn Ambrose, William Novak, Steve Palmquist, Ray Williams, and Carol
Woody, Report of the Independent Technical Assessment on the Department of
Veterans Affairs VETSNET Program (Carnegie Mellon Software Engineering
Institute, September 2005).

13CMMI Product Team, CMMI for Development, Version 1.2 (Carnegie Mellon
Software Engineering Institute, August 2006).

Table 1: Roles and Responsibilities of VETSNET Organizational Elements

Organization         Role                          Membership              
VETSNET Executive    Provides executive direction  Deputy Under Secretary  
Board                to the VETSNET Executive      for Benefits            
                        Team.                                                 
                                                      Special Assistant to    
                                                      the Under Secretary for 
                                                      Benefits                
                                                                              
                                                      VETSNET Program Manager 
                                                                              
                                                      VA Office of            
                                                      Information and         
                                                      Technology Principal    
                                                      Associate Deputy        
                                                      Assistant Secretary     
                                                                              
                                                      Chief Financial Officer 
                                                                              
                                                      Director, Compensation  
                                                      and Pension Service     
                                                                              
                                                      Director, Office of     
                                                      Performance Analysis    
                                                      and Integrity           
                                                                              
                                                      Associate Deputy Under  
                                                      Secretary for Policy    
                                                      and Programs            
                                                                              
                                                      Associate Deputy Under  
                                                      Secretary for Field     
                                                      Operations              
                                                                              
                                                      VBA Chief Information   
                                                      Officer                 
VETSNET Executive    Responsible for the delivery  VETSNET Program         
Team                 of the project and acts as    Executive (Special      
                        the review and approval       Assistant to Under      
                        authority for the             Secretary)              
                        Implementation Teams.                                 
                                                      VETSNET Program Manager 
                                                                              
                                                      Contract support        
                                                                              
                                                      Business Architect      
                                                                              
                                                      Technical Architect     
                                                                              
                                                      MITRE support           
VETSNET              Responsible for the           Compensation and        
Implementation Teams day-to-day activities         Pension                 
                        necessary to develop and                              
                        implement the replacement     Office of Resource      
                        system.                       Management              
                                                                              
                                                      Office of Performance   
                                                      Analysis and Integrity  
                                                                              
                                                      Office of Field         
                                                      Operation               
                                                                              
                                                      Office of Information   
                                                      Management, VA Central  
                                                      Office                  
                                                                              
                                                      St. Petersburg Systems  
                                                      Development Center      
                                                                              
                                                      Hines Information       
                                                      Technology Center       
                                                                              
                                                      Austin Automation       
                                                      Center                  

Source: GAO analysis of VBA information.

When the new governance structure was approved in March 2006, the Under
Secretary ensured that those involved in the project gave it high
priority, directing certain key personnel (such as members of the
executive and implementation teams) to make the initiative their primary
responsibility, and other personnel (technical staff that provide support
to other systems) with collateral (non-VETSNET) duties to make the project
their first priority. He also placed limitations on the transfer of
personnel away from the project, recognizing the importance of staff
continuity in successfully completing the initiative. Staff members
assigned project responsibilities could be reassigned (i.e., given
nonpromotion, lateral reassignments) only with approval from the Under
Secretary or his deputy.

By implementing the new governance and organizational structure and
ensuring that the project has priority, VBA partially responded to SEI's
concerns in this area; however, VBA has not yet taken action with regard
to ownership responsibility for total system and process operating costs,
as SEI advised. According to administration officials, the replacement
initiative is an in-house, contractor-assisted development effort, in
which three different contractors provide support for program management,
system development, and testing and validation of requirements. VA
reported VETSNET system costs to the Congress totaling about $89 million
for fiscal years 1996 through 2006, with additional estimated costs for
completion of the initiative in 2009 of about $62.4 million. However,
according to project management officials, these costs do not include
expenditures for in-house development work. This in-house work involves
many VA personnel, as well as travel to various locations for testing and
other project related activities. Thus, considerable costs other than
contract cost have been incurred, which have not been tracked and reported
as costs for the replacement initiative. Without comprehensive tracking
and reporting of costs incurred by the replacement project, the ability of
VBA and the Congress to effectively monitor progress could be impaired.

  VBA Took Action to Address Project Planning and Management

A second major area of overall management concern was project planning. In
particular, the lack of an integrated master schedule for the VETSNET
project was a major concern articulated by SEI, as well as in our prior
work. An integrated project plan and schedule should incorporate all the
critical areas of system development and be used as a means of determining
what needs to be done and when, as well as measuring progress. Such an
integrated schedule should consider all dependencies and include subtasks
so that deadlines are realistic, and it should incorporate review
activities to allow oversight and approval by high-level managers. Among
other things, the program plan should also include capacity requirements
for resources and technical facilities to support development, testing,
user validation, and production.

SEI was specifically concerned that releases with overlapping
functionality were being developed at the same time, with insufficient
time to document or test requirements; this approach constrained resources
and added complexity because of the need to integrate completed
applications and newly developed functionality. In addition, SEI observed
that the VETSNET program suffered from lack of sufficient test facilities
because it did not have enough information to plan for adequate capacity.

In response to these project planning concerns, VETSNET management, with
contractor support, developed an integrated master schedule to guide
development and implementation of the remaining functionalities for the
replacement system. The VETSNET Integrated Master Schedule, finalized in
September 2006, includes an end-to-end plan and a master schedule.
According to VBA, the end-to-end plan documents the end state of the
project from a business perspective, which had not previously been done.
The master schedule identifies the necessary activities to manage and
control the replacement project through completion. The schedule also
describes a new software release process that provides more time to work
on requirements definition and testing, and allows for more
cross-organizational communications to lessen the possibility of not
meeting requirements.

In addition, the new release process includes a series of management
reviews to help control the software development process and ensure that
top management has continuous visibility of project related activities.
These reviews occur at major steps in the system development life cycle
(as described in fig. 1: initiation, preliminary design, and so on). Such
reviews are intended to ensure that the VETSNET Executive Team and the
VETSNET Executive Board agree and accept that the major tasks of each step
have been properly performed.

Figure 1: VETSNET System Life Cycle and Review Concept

Nonetheless, while the Integrated Master Schedule is an important
accomplishment, it may not ensure that the project sufficiently addresses
capacity planning, one of SEI's areas of concern. According to its
assessment, capacity requirements for the fully functional production
system were unclear. Capacity planning is important because program
progress depends on the availability of necessary system capacity to
perform development and testing; adjustments to such capacity take time
and must be planned. If systems do not have adequate capacity to
accommodate workload, interruptions or slowdowns could occur. According to
SEI, capacity adjustments cannot be made instantly, and program progress
will suffer without sufficient attention to resource requirements.

However, the VETSNET Integrated Master Schedule does not identify
activities or resources devoted to capacity planning. According to
officials, the capacity of the corporate environment (that is, corporate
information systems, applications, and networks) is being monitored by
operational teams with responsibility for maintaining this environment.
According to project officials, VETSNET representatives participate in
daily conference calls in which the performance of corporate applications
is discussed, and changes in application performance are reported to the
VETSNET developers for investigation and corrective action. Project
officials reported that when a performance degradation occurred in some
transactions during performance testing, it was determined that additional
computing capacity was needed and would be acquired. One reason why the
occurrence of degradation had not been anticipated by the VETSNET project
was that capacity planning had not taken place. Unless it ensures that
capacity planning and activities are included in the Integrated Master
Schedule, the replacement project may face other unanticipated
degradations that it must react to after the fact, thus jeopardizing the
project's cost, schedule, and performance.

  Conversion Efforts Were Suspended as Advised, but Issues Remain

In its assessment, SEI questioned VBA's approach to developing
functionality while concurrently converting records from the BDN to the
replacement system. It noted that VBA had chosen to complete software
development according to location rather than according to the type of
functionality. Specifically, in 2004, VBA began an effort to remove all
claims activity (both new and existing claims) at one regional office
(Lincoln, Nebraska) from the BDN to the replacement system, developing the
software as necessary to accommodate processing the types of claims
encountered at that site. The intention was to address each regional
office in turn until all sites were converted. According to SEI, this
approach had resulted in the development being stalled by obstacles
arising from the variety of existing claims.14 The contractor advised VBA
to focus first on developing functionality to process original claims and
discontinue efforts to convert existing claims until all the necessary
functionality had been developed, and the replacement system's ability to
handle new cases of any complexity had been proven by actual experience.

14As previously noted, there are variations in types of compensation and
pension benefits, including death pensions for survivors of deceased
wartime veterans, burial benefits, Dependency and Indemnity Compensation
to some surviving parents, and benefits paid to third parties, such as
people to whom a veteran has given power of attorney, medical service
providers, and so on. In addition, different forms of payment must be
accommodated (check or electronic funds transfer).

In accordance with this advice, VBA stopped converting existing records
from the BDN and changed its focus to developing the necessary
functionality to process all new compensation claims. According to the
integrated master schedule, conversion activities are now timed to follow
the release of the needed functionality. That is, according to the
schedule, VBA plans to begin converting each type of record from the BDN
only after the necessary functionality for the replacement system has been
developed and deployed to process that type of record.

In addition, the project is mitigating risk by resuming conversions
beginning with a test phase. Its strategy is first to convert records for
terminated claims--claims that are no longer being paid. Conversion of the
terminated records will be followed by additional conversions of records
for claims receiving payment at Lincoln and Nashville (these two sites are
being used to test system functionality during development). The VETSNET
leadership will consider testing complete with the successful conversion
at these two sites.

However, SEI raised three additional issues with regard to the conversion
of records that VBA has not fully addressed:

           o First, SEI expressed concerns that conversion failures could
           lead to substantial numbers of records being returned to the BDN.
           Because of differences in the database technologies used for the
           old system and the replacement system, certain types of errors in
           BDN records cause conversion to fail (according to SEI,
           approximately 15 percent of all these records are estimated to
           have such errors). If records fail to convert correctly, they may
           need to be returned to the BDN so that benefits can continue to be
           paid. However, this process is not simple and may involve manually
           reentering the records.15

           o Second, SEI observed that VBA was also depending on manual
           processes for determining that records were converted
           successfully, including the use of statistically random samples,
           and that it was aiming to ensure correctness to a confidence level
           of 95 percent. However, in the absence of a straightforward method
           for automatically returning records to the BDN, SEI considered the
           5 percent risk of error unacceptable for conversions of large
           numbers of records.
           o Finally, SEI observed that the lack of automated methods and the
           complexity of the processes meant that conversions required
           careful planning and assurance that adequate staff would be
           available to validate records when the conversions took place.

           However, the VETSNET leadership has not developed any strategy to
           address the possibility that a large number of cases might need to
           be returned to the BDN during the testing phase. For example, it
           has not included this possibility as a risk in its risk management
           plan. The absence of a strategy to address this possibility could
           lead to delays in program execution.

           Further, VBA has not yet decided whether a possible 5 percent
           error rate is acceptable or developed a plan for addressing the
           resulting erroneous records. If VBA does not address these issues
           in its planning, it increases the risk that veterans may not
           receive accurate or timely payments.

           Finally, the VETSNET leadership has not yet developed detailed
           plans that include the scheduled conversions for each regional
           office and identified staff to perform the necessary validation.
           Having such plans would reduce the risk that the conversion
           process could be delayed or fail.
		   
		   VBA Is Improving Software Development Processes, but Improvements
		   Are Not Yet Institutionalized

           In addition to actions addressing the overall management concerns
           identified by SEI, VBA has steps action to improve its software
           development processes in risk management, requirements management,
           defect/change management, and performance measures. SEI described
           weaknesses in all of these areas. The steps taken have generally
           been effective in addressing the identified weaknesses, but VBA
           has not yet institutionalized many of these improvements.
           According to the VETSNET management team, it made a conscious
           decision first to establish the governance, build the
           organization, implement processes to gain control, and gather
           additional information about the project to assist in prioritizing
           the remaining activities. The team also stated that some of the
           processes are no longer VBA's responsibility but are now that of
           the newly realigned Office of Information and Technology.16
           Nonetheless, if VA does not develop and establish documented
           policies and procedures to institutionalize these improvements,
           they may not be maintained through the life of the project or
           available to be applied to other development initiatives.
		   
		     Risk Management Plan Has Been Revised

           Risk management is a process for identifying and assessing risks,
           their impact and status, the probability of their occurrence, and
           mitigation strategies. Effective risk management includes the
           development of a risk management plan and tracking and reporting
           progress against the plan. According to SEI, to the extent that
           risk management existed at all in the replacement program, it was
           conducted on a pro forma basis without real effect on program
           decisions. SEI said that risks and risk mitigation activities
           needed to be incorporated into all aspects of program planning,
           budgeting, scheduling, execution, and review.

           In response to these concerns, VBA has instituted risk management
           activities that, if properly implemented, should mitigate the
           risks associated with the project. Specifically, the VETSNET team,
           with contractor support, developed a risk management plan that was
           adopted in January 2007. The plan includes procedures for
           identifying, validating, analyzing, assessing, developing
           mitigation strategies for, controlling and tracking, reporting,
           and closing risks. It also establishes criteria for assessing the
           severity of the risks and their impact.

           The VETSNET leadership also developed a Risk Registry database,
           and its contractor reviewed and prioritized the open risks. Each
           open risk was evaluated, and a proposed disposition of the risk
           was submitted to VETSNET management. Of the 39 open risks, all but
           3 had been addressed as of January 2007.

           The development documentation for each planned software release
           also includes sections on risk. In accordance with these plans,
           the VETSNET leadership is currently capturing potential risks and
           tracking action items and issues. At weekly status meetings,
           VETSNET leadership reviews Risk Registry reports of open risks.
           According to the contractor, the reports identify each risk and
           provide information on its age, ownership, and severity.

           However, these risk management activities have not yet been
           institutionalized through the definition and establishment of
           associated policies and procedures. If it does not
           institutionalize these improvements, VBA increases the possibility
           that the VETSNET project's improvements in risk management may not
           be maintained through the life of the project.
		   
		     Requirements Management Has Improved

           Requirements management is a process for establishing and
           maintaining a common understanding between the business owners and
           the developers of the requirements to be addressed, as well as
           verifying that the system meets the agreed requirements. SEI's
           report commented that the VETSNET project requirements were not
           stable, and that the business owners (including subject-matter
           experts) and developers were separated by many organizational
           layers, resulting in confusion and delays in development of the
           system. SEI suggested that VA restructure project activities to
           focus on defining an effective requirements process. According to
           SEI, the project needed to ensure that subject-matter experts were
           included in developing requirements and that evaluation criteria
           were established for prioritizing requests for changes to
           requirements. Finally, business owners should confirm that the
           system is meeting organizational needs.

           VBA has instituted requirements management activities that, if
           properly implemented, should help avoid the instability and other
           requirements problems identified by SEI. Specifically, VBA took
           steps to establish a requirements management process and to
           stabilize the requirements. For example, the development release
           process in the Integrated Master Schedule includes a phase for
           requirements identification. In addition, the project has
           established and begun applying evaluation criteria to prioritize
           change requests for its development releases. Further, until all
           claims are completely migrated from the BDN to the replacement
           system, in July 2006, the Under Secretary directed that any
           additional requirements would have to have his approval.

           Responding to SEI's advice regarding the involvement of
           subject-matter experts and business owners, VBA designed the new
           release process to directly involve subject-matter experts in
           requirements workshops. Further, the business teams participate in
           user-acceptance testing.

           However, these requirements management activities have not yet
           been institutionalized through the definition and establishment of
           policies and procedures. Until they are established, VBA runs the
           risk that the improved processes will not be maintained through
           the life of the VETSNET project or used in other software
           development projects.
		   
		     Management Attention Is Being Focused on Major Software Defects

           SEI raised numerous concerns regarding the defect process for the
           replacement system. These concerns for defect management included
           (1) identification of defects, (2) determination of cause, and (3)
           disposition of defects--either by correction or workaround.
           According to SEI guidance, defect management prevents known
           defects from hampering the progress of the program. The management
           process should include clearly identifying and tracking defects,
           analyzing defects to establish their cause, tracking their
           disposition, clearly identifying the rationale for not addressing
           any defects (as well as proposing workarounds), and making
           information on defects and their resolution broadly available.
           SEI's report stated that VBA needed to distinguish defects from
           changes to requirements and develop a process for defect
           management.

           To respond to these concerns and focus program management
           attention on major defects, the VETSNET Executive Team, with
           contractor support, conducted an audit of existing defects and
           revised the defect management process. The audit of the defect
           database determined that the VETSNET database used to capture
           software defects also included change requests; as a result, work
           required to address processes that did not work properly was not
           distinguished from requests for added or changed functionality,
           which would require review and approval before being addressed. To
           address this issue, the team separated defects from change
           requests, and a new severity rating scale was developed. All open
           defects were recategorized to ensure the major defects would
           receive appropriate program management attention. Also, all defect
           categorizations must meet the approval of the VETSNET Business
           Architect and are scheduled for action as dictated by the severity
           level.

           Although these steps address many of SEI's concerns regarding
           VBA's defect management process, more remains to be done before
           the process is institutionalized. The Program Management Office
           has reported that actions to revise the defect management process
           are complete, but the process description is still in draft, and
           policies and procedures have not been fully established. Without
           institutionalized policies and procedures for the defect
           management process, it may not be maintained consistently through
           the life of the project.
		   
		     The Replacement Initiative Is Tracking Certain Performance
			 Measures

           According to SEI, performance measures are the only effective
           mechanism that can provide credible evidence of a program's
           progress. The chosen measures must link directly to the expected
           accomplishments and goals of the system, and they must be applied
           across all activities of the program. In its report, SEI stated
           that although VBA was reporting certain types of performance
           measures, it was not relating these to progress in system
           development. For example, VBA reported the total number of
           veterans paid, but did not provide estimates of how many
           additional veterans would be paid when the system incorporated
           specific functionalities that were under development. SEI
           suggested several measures that would provide more evidence of
           progress, such as increases in the percentage of original claims
           being paid by the replacement system, as well as user satisfaction
           and productivity gains resulting from use of the replacement
           system applications at regional offices.

           In response to these concerns, the replacement project has begun
           tracking a number of the measures suggested by SEI, including

           o increases in the percentage of original claims being paid by the
           replacement system,

           o increases in the percentage of veterans' service representatives
           using the new system,

           o decreases in the percentage of original claims being entered in
           the BDN rather than the replacement system.

           Although these measures provide indications of VA's progress,
           other measures that could demonstrate the effectiveness of the
           replacement system have not been developed. For example, VBA has
           not developed results-oriented measures to capture user
           satisfaction or productivity gains from the system. Without
           measuring user satisfaction, VBA has reduced assurance that the
           replacement system will be accepted by the users. In addition,
           measures of productivity would provide VBA with another indication
           of progress toward meeting business needs.
		   
		   VETSNET Is Currently Processing a Portion of Compensation Claims,
		   but Much Work Remains

           After more than 10 years of effort, including the recent
           management, organizational, and process improvements, VBA has
           achieved critical functionalities needed to process and pay
           certain original compensation claims using the replacement system,
           but it remains far from completing the project. For example, the
           replacement system is currently being used to process a portion of
           the original claims that veterans file for compensation.
           Nonetheless, the system requires further development before it can
           be used to process claims for the full range of compensation and
           pension benefits available to veterans and their dependents. In
           addition, VBA still faces the substantial task of moving
           approximately 3.5 million beneficiaries who are currently being
           served by the BDN to the replacement system.

           As designed, VETSNET consists of five major system applications
           that are used in processing benefits:

           o Share--used to establish claims;17 it records and updates basic
           information about veterans and dependents both in the BDN and the
           replacement system.

           o Modern Award Processing-Development (MAP-D)--used to manage the
           claims development process, including the collection of data to
           support the claims and the tracking of claims.

           o Rating Board Automation 2000 (RBA 2000)--provides laws and
           regulations pertaining to disabilities, which are used by rating
           specialists in evaluating and rating disability claims.

           o Award Processing (Awards)--used to prepare and calculate the
           benefit award based on the rating specialist's determination of
           the claimant's percentage of disability. It is also used to
           authorize the claim for payment.

           o Finance and Accounting System (FAS)--used to develop the actual
           payment record. FAS generates various accounting reports and
           supports generation and audit of benefit payments.

           According to VBA officials, all five of the software applications
           that make up the new system are now being used in VA's 57 regional
           offices to establish and process new compensation claims for
           veterans. As of March 2007, VBA leadership reported that the
           replacement system was providing monthly compensation payments to
           almost 50,000 veterans (out of about 3 million veterans who
           receive such payments). In addition, the replacement system has
           been processing a steadily increasing percentage of all new
           compensation claims completed: this measure was 47 percent in
           January 2007, increasing to 60 percent in February and 83 percent
           in March.

           Nonetheless, considerable work must be accomplished before VBA
           will be able to rely on the replacement system to make payments to
           all compensation and pension beneficiaries. Specifically, while
           all five software applications can now be used to process original
           compensation claims for veterans, two of the applications--Awards
           and FAS--require further development before the system will be
           able to process claims for the full range of benefits available to
           veterans and their dependents. Table 2 shows the status of
           development of all five applications.
		   
15SEI quoted program staff as calling the process "extraordinarily time
consuming," saying that "It takes 5 minutes to get a veteran into VETSNET
and then 5 days to get him back out."

16This realignment was approved on February 27, 2007.

17The functionalities for the Search and Participant Profile, formerly a
separate application, are now in Share.
		   
           Table 2: VETSNET Component Applications and Status of Development

Application Development status                           Deployment status 
Share       Completed                                    Fully deployed at 
                                                            all regional      
                                                            offices.          
MAP-D       Completed                                    Fully deployed at 
                                                            all regional      
                                                            offices.          
RBA 2000    Completed                                    Fully deployed at 
                                                            all regional      
                                                            offices.          
Awards      Functionality to process                     In partial use in 
               third-party/nonveteran payee claims is in    all regional      
               application testing.                         offices.          
                                                                              
               Requirements for processing survivor                           
               benefits are being developed.                                  
                                                                              
               Functionality to process pensions is not yet                   
               under development.                                             
FAS         Functionality to process                     In partial use in 
               third-party/nonveteran payee claims is in    all regional      
               application testing.                         offices.          
                                                                              
               Functionality to generate management reports                   
               is partially in requirements development and                   
               partially in application testing.                              
                                                                              
               Requirements for processing survivor                           
               benefits are being developed.                                  
                                                                              
               Functionality to process pensions is not yet                   
               under development.                                             

           Source: GAO analysis of VBA data.

           According to VBA officials, Awards and FAS do not yet have the
           capability to process original claims for payment to recipients
           other than veterans: that is, the applications do not have the
           functionality to process claims for survivor benefits18 and
           third-party/nonveteran payee claims.19 In addition, further
           development of these applications is needed to process pension
           benefits for qualified veterans and their survivors. Until
           enhancements are made to Awards and FAS, these claims must
           continue to be processed and paid through the BDN. Also, according
           to VBA, FAS does not yet have the capability to generate all the
           necessary accounting reports that support the development of
           benefits payments to claimants.
		   
18Survivor benefits may be paid to survivors of veterans who were eligible
for either compensation or pension benefits, depending on the
circumstances.

19Examples of third-party/nonveteran payees include people to whom a
veteran has given power of attorney, medical service providers designated
to receive payments, and so on.
		   
           As described earlier, VBA now has an Integrated Master Schedule
           that incorporates the activities that VBA needs to manage in order
           to complete the replacement project. According to the schedule,
           the remaining capabilities necessary to process compensation and
           pension claims are to be developed and deployed in three software
           releases, as shown in table 3.

           Table 3: Software Releases Planned to Complete the VETSNET System
		   
                                                              Estimated       
Release number and title Functionality to be attained      completion date 
      1. Complete           Process all compensation claims   August 2007     
      Compensation          and add significant FAS                           
                            functionality (generate                           
                            management reports).                              
      2. Nonincome-Based    Process burial, accrued, and      February 2008   
      Survivor Benefits     survivor claims.                                  
      3. Income-Based       Process nonservice-connected      August 2008     
      Pension               pension (veteran and survivor)                    
                            and parents' Dependent and                        
                            Indemnity Compensation Benefit.                   

                                              Estimated start Estimated       
Phases  Conversions                        date            completion date 
Phase 1 Conversion testing on terminated   March 2007      August 2007     
           records and selected live records                                  
Phase 2 Conversion of live compensation    January 2008    October 2008    
           records                                                            
Phase 3 Conversion of pension records      November 2008   June 2009       

           Source: VBA.

           As table 3 shows, VBA does not expect to complete the development
           of the functionalities needed to process all new compensation and
           pension claims until August 2008. However, according to VBA, the
           estimated completion date is the planned date for completing all
           development and testing, but it is not necessarily the date when
           users will be able to begin using the new system. Before such use
           can begin, other activities need to occur. For example, users must
           receive training, and VETSNET program management must authorize
           the use of the system at each regional office.

           In addition to its remaining software development activities, VBA
           also faces the challenge of converting records for claims
           currently paid by the BDN to the replacement system. Existing
           compensation and pension cases on the BDN number about 3 million
           and about 535,000, respectively. Table 4 shows the phases in which
           VA is planning to perform conversions, according to its Integrated
           Master Schedule.

           Table 4: Conversion Phases

           Source: GAO analysis of VBA data.

           As the table shows, VBA conversion efforts began in March 2007.
           VBA first performed conversion testing on 310,000 terminated (that
           is, inactive) compensation cases so that it could develop and
           apply lessons learned to the conversion of live records. According
           to VETSNET officials, VBA planned to continue testing by
           converting live cases at two regional offices (Lincoln and
           Nashville) that were used as testing sites during development. It
           then plans to perform the conversion of all live compensation
           cases. After the compensation conversion is complete, VBA plans to
           begin efforts to convert pension benefits cases.

           Based on VETSNET project documentation, activities supporting the
           releases have so far been performed on time, consistent with the
           milestones in the recently finalized Integrated Master Schedule.
           For example, VA completed the Project Initiation and Review
           Authorization for Release 1 on September 7, 2006, as scheduled
           (see fig. 1, shown earlier in the report, for the phases of system
           development and the required milestone reviews). It also completed
           the Preliminary Design Review and the Critical Design Review as
           scheduled (on November 20 and December 22, respectively). Planning
           for Release 2 is also on schedule: a kickoff meeting was held on
           January 24, 2007, which established the scope of the release, and
           the Project Initiation and Review Authorization was conducted on
           February 8.
		   
		   Conclusions

           VA has responded to SEI's assessment by making significant changes
           in its approach to the project and its overall management,
           including slowing the pace of development, establishing a new
           governance structure, and ensuring staff resources. However, VBA
           has not yet addressed all the issues raised by the SEI assessment.
           That is, it has not ensured ownership responsibility for total
           system and process operating costs, because it is not currently
           monitoring and reporting in-house expenditures for the project. It
           has not defined processes and resources for capacity planning for
           the project. In addition, VBA has not yet addressed issues related
           to the conversion of records now on the BDN to the replacement
           system. Specifically, it has not addressed the risk that large
           numbers of records may need to be returned to the BDN, decided on
           the degree of confidence it will require that records are
           converted accurately, or developed complete plans for converting
           and validating records. In addition, although VBA has improved key
           processes for managing the software development, these processes
           have not yet been institutionalized in defined policies and
           procedures, and performance measures of productivity and user
           satisfaction have not been developed. VETSNET management has
           stated that it gave priority to other activities, such as
           establishing appropriate governance and organizational structures,
           and that it is still gathering information to assist in
           prioritizing the activities that remain.

           Much work remains to be done to complete the VETSNET initiative.
           Although VBA has substantially increased the number of claims
           being paid by the replacement system, it must not only finish the
           development and deployment of the software, it must also convert
           the over 3.5 million records now on the BDN to the replacement
           system. Addressing the remaining issues identified by SEI would
           improve VBA's chances of successfully completing the replacement
           system and ending reliance on the aged BDN to pay compensation and
           pension benefits.
		   
		   Recommendations for Executive Action

           To enhance the likelihood that the replacement system will be
           successfully completed and implemented, we are recommending that
           the Secretary of Veterans Affairs take the following five actions:

           o Direct the CIO to institute measures to track in-house
           expenditures for the project.

           o Direct the VETSNET project to include activities for capacity
           planning in the VETSNET Integrated Master Schedule and ensure that
           resources are available for these activities.

           o Direct VBA to (1) develop a strategy to address the risk that
           large numbers of records may need to be returned to the BDN; (2)
           determine whether a greater confidence level for accuracy should
           be required in the conversion process; and (3) develop a detailed
           validation plan that includes the scheduled conversions for each
           regional office and the validation team members needed for that
           specific conversion.
           o Direct the CIO to document and incorporate the improved
           processes for managing risks, requirements, and defects into
           specific policy and guidance for the replacement initiative and
           for future use throughout VBA.

           o Direct the replacement project to develop effective
           results-oriented performance measures that show changes in
           efficiency, economy, or improvements in mission performance, as
           well as measures of user satisfaction, and to monitor and report
           on the progress of the initiative according to these measures.
		   
		   Agency Comments and Our Evaluation

           In providing written comments on a draft of this report, the
           Secretary of Veterans Affairs agreed with our conclusions and
           concurred with the report's recommendations. (The department's
           comments are reproduced in app. II.) The comments described
           actions planned that respond to our recommendations, such as
           incorporating processes developed for the VETSNET project in
           standard project management policies, processes, and procedures
           that would be used for all IT projects in the department. In
           addition, the comments provided further information on actions
           already taken, such as details of the records conversion process.
           If the planned actions are properly implemented, they could help
           strengthen the department's management of the replacement system
           project and improve the chances that the system will be
           successfully completed.

           We are sending copies of this report to the Chairman and Ranking
           Minority Member of Committee on Veterans' Affairs. We are also
           sending copies to the Secretary of Veterans Affairs and
           appropriate congressional committees. We will make copies
           available to other interested parties upon request. Copies of this
           report will also be made available at no charge on GAO's Web site
           at http://www.gao.gov .

           Should you or your staff have any questions about this report,
           please contact me at (202) 512-6304 or by e-mail at
           [email protected]. Contact points for our Offices of Congressional
           Relations and Public Affairs may be found on the last page of this
           report. Key contributors to this report are listed in appendix
           III.

           Sincerely yours,

           Valerie C. Melvin
		   Acting Director
		   Human Capital and Management Information Systems
		   
		   Appendix I: Objectives, Scope, and Methodology

           Our objectives were to determine (1) to what extent the Department
           of Veterans Affairs (VA) has followed the course of action
           recommended by the Carnegie Mellon Software Engineering Institute
           (SEI) and addressed the concerns that it raised and (2) the
           current status of the replacement project, the Veterans Service
           Network (VETSNET).

           To determine the actions taken to implement SEI's recommended
           approach and address the concerns it raised, we

           o determined the recommended actions by analyzing the report;

           o compared the concerns identified in the assessment to actions
           planned, actions undertaken but not completed, and actions
           implemented by VA officials or contractors;

           o interviewed contractor, VA, and VETSNET program office officials
           to gain an understanding about processes developed and procedures
           implemented; and

           o obtained and reviewed relevant VA and contractor documents that
           disclosed or validated VA responses to SEI's concerns.

           To determine the status of system development efforts and the
           extent that tasks planned for the initiative were completed, we
           analyzed VA and contractor documentation regarding system
           operations and development, time frames, and activities planned.
           We analyzed VA documents that disclosed costs to date and costs
           planned for completion of the initiative. We did not assess the
           accuracy of the cost data provided to us. We supplemented our
           analyses with interviews of VA and contractor personnel involved
           in the replacement initiative.

           We visited the Nashville and St. Petersburg regional offices to
           observe the replacement system in operation and the processes and
           procedures used to test and validate the replacement system as it
           was being developed and implemented. We analyzed VA documentation
           and relevant evidence from contractors involved in the replacement
           effort to establish the work remaining to complete the project.
           Finally, we interviewed cognizant VA and contractor officials,
           responsible for developing, testing, and implementing the
           replacement system.

           We performed our work at VA offices in Washington, D.C., and at VA
           regional offices in Nashville, Tennessee, and St. Petersburg,
           Florida, from April 2006 to April 2007 in accordance with
           generally accepted government auditing standards.
		   
		   Appendix II: Comments from the Department of Veterans Affairs
		   
		   Appendix III: GAO Contact and Staff Acknowledgments
		   
		   GAO Contact

           Valerie Melvin, (202) 512-6304 or [email protected]
		   
		   Staff Acknowledgments

           In addition to the individual named above, key contributions were
           made to this report by Barbara Oliver, Assistant Director;
           Nabajyoti Barkakati; Barbara Collier; Neil Doherty; Matt Grote;
           Robert Williams; and Charles Youman.
		   
		   GAOï¿½s Mission

           The Government Accountability Office, the audit, evaluation and
           investigative arm of Congress, exists to support Congress in
           meeting its constitutional responsibilities and to help improve
           the performance and accountability of the federal government for
           the American people. GAO examines the use of public funds;
           evaluates federal programs and policies; and provides analyses,
           recommendations, and other assistance to help Congress make
           informed oversight, policy, and funding decisions. GAO's
           commitment to good government is reflected in its core values of
           accountability, integrity, and reliability.
		   
		   Obtaining Copies of GAO Reports and Testimony

           The fastest and easiest way to obtain copies of GAO documents at
           no cost is through GAO's Web site ( www.gao.gov ). Each
           weekday, GAO posts newly released reports, testimony, and
           correspondence on its Web site. To have GAO e-mail you a list of
           newly posted products every afternoon, go to www.gao.gov and
           select "Subscribe to Updates."
		   
		   Order by Mail or Phone

           The first copy of each printed report is free. Additional copies
           are $2 each. A check or money order should be made out to the
           Superintendent of Documents. GAO also accepts VISA and Mastercard.
           Orders for 100 or more copies mailed to a single address are
           discounted 25 percent. Orders should be sent to:

           U.S. Government Accountability Office 441 G Street NW, Room LM
           Washington, D.C. 20548

           To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax:
           (202) 512-6061
		   
		   To Report Fraud, Waste, and Abuse in Federal Programs

           Contact:

           Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail:
           [email protected] Automated answering system: (800) 424-5454 or
           (202) 512-7470
		   
		   Congressional Relations

           Gloria Jarmon, Managing Director, [email protected] (202)
           512-4400 U.S. Government Accountability Office, 441 G Street NW,
           Room 7125 Washington, D.C. 20548
		   
		   Public Affairs

           Paul Anderson, Managing Director, [email protected] (202)
           512-4800 U.S. Government Accountability Office, 441 G Street NW,
           Room 7149 Washington, D.C. 20548

(310758)

www.gao.gov/cgi-bin/getrpt?GAO-07-614 .

To view the full product, including the scope
and methodology, click on the link above.

For more information, contact Valerie Melvin at (202) 512-6304 or
[email protected].

Highlights of [47]GAO-07-614 , a report to the Chairman, Committee on
Veterans' Affairs, House of Representatives

April 2007

VETERANS BENEFITS ADMINISTRATION

Progress Made in Long-Term Effort to Replace Benefits Payment System, but
Challenges Persist

Since 1996, the Veterans Benefit Administration (VBA) has been working on
an initiative to replace its aging system for paying compensation and
pension benefits. In 2005, concerned about the slow pace of development,
VBA contracted with the Software Engineering Institute (SEI) for an
independent evaluation of the project, known as the Veterans Service
Network (VETSNET). SEI advised VBA to continue working on the project at a
reduced pace while addressing management and organization weaknesses that
it determined had hampered the project's progress.

GAO was requested to determine to what extent the VETSNET project has
followed the course of action recommended by SEI and describe the
project's current status.

To perform its review, GAO analyzed project documentation, conducted site
visits, and interviewed key program officials.

[48]What GAO Recommends

To sustain the improved management and software development processes
currently being used by VETSNET project management, GAO is making
recommendations to the Secretary of Veterans Affairs in several areas,
including cost tracking, capacity planning and management, and performance
measures. The Secretary agreed with GAO's recommendations and described
actions planned in response.

VBA is generally following the course of action recommended by SEI by
continuing to work on the replacement initiative at a reduced pace, while
taking action to address identified weaknesses in overall management and
software development processes. For example, VBA established a new
governance structure for the initiative that included senior management
and involved all stakeholders, and it incorporated all critical areas of
system development in an integrated master schedule. However, not all of
SEI's management concerns have been addressed.  For example, SEI advised
VBA to ensure that stakeholders take ownership responsibility for the
project, including the total system and process operating costs; however,
although VBA is tracking costs incurred by contractors, it is not yet
tracking and reporting in-house costs incurred by the project. Further,
although the project has improved its management processes, such as
establishing a process to manage and stabilize system requirements, it has
not yet developed processes for capacity planning and management. This
will be important for ensuring that further VETSNET development does not
lead to delays and slowdowns in processing of benefits. In addition,
although the project has established certain performance measures, it has
not yet established results-oriented measures for productivity and user
satisfaction, both of which will be important for measuring progress.
Finally, the process improvements that VBA has incorporated in the
replacement initiative remain in draft and have not been established
through documented policies and procedures. If VBA does not
institutionalize these improvements, it increases the risk that they may
not be maintained through the life of the project or be available for
application to other development initiatives.

After more than 10 years of effort, including the recent management,
organizational, and process improvements, VBA has developed critical
functionalities needed to process and pay certain original compensation
claims using the replacement system, but it remains far from completing
the project. According to VBA officials, all five of the major software
applications that make up the new system are now being used to establish
and process new compensation claims for veterans. In total, the
replacement system is currently providing monthly compensation payments to
almost 50,000 veterans (out of about 3 million veterans who receive such
payments); the system was used to process about 83 percent of all new
compensation claims completed in March 2007. Nonetheless, the system
requires further development before it can be used to process claims for
the full range of compensation and pension benefits available to veterans
and their dependents.

References

Visible links

  28. http://www.gao.gov/cgi-bin/getrpt?GAO/AIMD-96-90
  29. http://www.gao.gov/cgi-bin/getrpt?GAO/AIMD-97-154
  30. http://www.gao.gov/cgi-bin/getrpt?GAO/T-AIMD-00-321
  31. http://www.gao.gov/cgi-bin/getrpt?GAO-01-550T
  32. http://www.gao.gov/cgi-bin/getrpt?GAO-02-1054T
  33. http://www.gao.gov/cgi-bin/getrpt?GAO-06-905T
  34. http://www.gao.gov/cgi-bin/getrpt?GAO-06-905T
  35. http://www.vba.va.gov/bln/21/index.htm
  36. http://www.gibill.va.gov/
  37. http://www.insurance.va.gov/
  38. http://www.gao.gov/cgi-bin/getrpt?GAO/AIMD-96-90
  47. http://www.gao.gov/cgi-bin/getrpt?GAO-07-614
*** End of document. ***