Software Capability Evaluation: VA's Software Development Process Is
Immature (Letter Report, 06/19/96, GAO/AIMD-96-90).

GAO reviewed the development of computer software at the Veterans
Benefits Administration (VBA) and at the Department of Veterans Affairs'
Austin Automation Center. Neither VBA nor the Austin Automation Center
had satisfied any of the key process areas required for a level 2
capability, as described by the Software Engineering Institute's
five-level software capability model. Under a level 2 capability, basic
project management processes are established to track cost, schedule,
and functionality. Moreover, the necessary process discipline is in
place to repeat earlier successes on projects with similar applications.
The upshot is that the Department of Veterans Affairs (VA) has little
assurance that (1) investments in new software development will achieve
their objectives or (2) software will be delivered consistent with cost
and schedule estimates. GAO recommends that VA delay any major
investment in software development beyond what is needed to sustain
critical day-to-day operations until the repeatable level of process
maturity is achieved. VA should also obtain expert advice on how to
improve the ability of VBA and the Austin Automation Center to develop
high-quality software, develop a strategy to reach the repeatable level
of process maturity, implement that strategy quickly, and ensure that
any future contracts for software development require that the
contractor have a software development capability of at least level 2.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  AIMD-96-90
     TITLE:  Software Capability Evaluation: VA's Software Development 
             Process Is Immature
      DATE:  06/19/96
   SUBJECT:  Research and development contracts
             Veterans benefits
             Computer networks
             Application software
             Contract administration
             Cost control
             Requirements definition
             Information systems
IDENTIFIER:  VA Veterans Integrated Service Network
             VA Benefit Delivery Network
             Software Capability Maturity Model
             
******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved.  Major          **
** divisions and subdivisions of the text, such as Chapters,    **
** Sections, and Appendixes, are identified by double and       **
** single lines.  The numbers on the right end of these lines   **
** indicate the position of each of the subsections in the      **
** document outline.  These numbers do NOT correspond with the  **
** page numbers of the printed product.                         **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                                            **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************


Cover
================================================================ COVER


Report to the Chairman, Subcommittee on Compensation, Pension,
Insurance and Memorial Affairs, Committee on Veterans Affairs, House
of Representatives

June 1996

SOFTWARE CAPABILITY EVALUATION -
VA'S SOFTWARE DEVELOPMENT PROCESS
IS IMMATURE

GAO/AIMD-96-90

VA's Software Capability Evaluation

(511191)


Abbreviations
=============================================================== ABBREV

  AAC - Austin Automation Center
  CMM - Capability Maturity Model
  KPA - key process area
  SCE - software capability evaluation
  SEI - Software Engineering Institute
  SIS - Systems Integration Service
  SQA - software quality assurance
  SQ&C - software quality and control
  VA - Department of Veterans Affairs
  VBA - Veterans Benefits Administration
  VETSNET - Veterans Services Network

Letter
=============================================================== LETTER


B-261652

June 19, 1996

The Honorable Terry Everett
Chairman, Subcommittee on Compensation,
 Pension, Insurance and Memorial Affairs
Committee on Veterans Affairs
House of Representatives

Dear Mr.  Chairman: 

You requested that we review the Department of Veterans Affairs' (VA)
capability for developing and maintaining software for its
computer-based systems.  The Department depends on these systems to
support program operations and assist in managing the achievement of
critical mission objectives.  Our objective was to review software
development processes at the Veterans Benefits Administration (VBA)
and VA's Office of Information Resources Management's Austin
Automation Center (AAC).  The sites and projects were selected by VBA
and VA, respectively, as those that represent their best software
development processes and practices. 


   BACKGROUND
------------------------------------------------------------ Letter :1

VBA is in the process of modernizing many of its older, inefficient
systems and has reportedly spent an estimated $294 million on these
activities between October 1, 1986 and February 29, 1996.  The
modernization program can have a major impact on the efficiency and
accuracy with which over $20 billion in benefits and other services
is paid to our nation's veterans and their dependents.  However, in
the last 6 years some aspects of VBA's service to the veterans have
not improved.  For example, in the past 6 years, VBA's reported
processing time for an original compensation claim rose from 151 days
in fiscal year 1990 to 212 days in fiscal year 1994.  In March 1996
the average time was 156 days. 

Software development is a critical component of this major
modernization initiative.  VBA, with the assistance of contractors,
will be developing software for the Veterans Services Network
(VETSNET) initiative, a replacement for the existing Benefit Delivery
Network.  For efforts like VETSNET to succeed, it is crucial that VBA
have in place a disciplined set of software development processes to
produce high quality software within budget and on schedule. 

VBA relies upon its own staff and contractors to develop and maintain
software that is crucial to its overall operations.  In fiscal year
1995, VBA had 314 full-time equivalents, with payroll expenses of
$20.8 million, devoted to developing and maintaining software
throughout the organization.  It also spent $17.7 million in contract
services in these areas. 


   SCOPE AND METHODOLOGY
------------------------------------------------------------ Letter :2

To evaluate VA's software development capability, version 2.0 of the
Software Engineering Institute's (SEI)\1 software capability
evaluation (SCE) method\2

was used by an SEI-trained team of GAO specialists.  The SCE is a
method for evaluating agencies' and contractors' software development
processes against SEI's five-level software Capability Maturity Model
(CMM), as shown in table 1.  These levels and the key process areas
(KPAs) described within each level define an organization's ability
to develop software, and can be used to improve its software
development processes.  The findings generated from an SCE identify
(1) process strengths that mitigate risks, (2) process weaknesses
that increase risks, and (3) improvement activities that indicate
potential mitigation of risks. 



                                Table 1
                
                      CMM Levels and Descriptions

Level       Name            Description
----------  --------------  ------------------------------------------
5           Optimizing      Continuous process improvement is enabled
                            by quantitative feedback from the process
                            and from piloting innovative ideas and
                            technologies.

4           Managed         Detailed measures of the software process
                            and product quality are collected. Both
                            the software process and products are
                            quantitatively understood and controlled.

3           Defined         The software process for both management
                            and engineering activities is documented,
                            standardized, and integrated into a
                            standard software process for the
                            organization. All projects use an
                            approved, tailored version of the
                            organization's standard software process
                            for developing and maintaining software.

2           Repeatable      Basic project management processes are
                            established to track cost, schedule, and
                            functionality. The necessary process
                            discipline is in place to repeat earlier
                            successes on projects with similar
                            applications.

1           Initial         The software process is characterized as
                            ad hoc, and occasionally even chaotic. Few
                            processes are defined, and success depends
                            on individual effort.
----------------------------------------------------------------------
Source:  Capability Maturity Model for Software, Version 1.1,
(Technical Report CMU/SEI-93-TR-24, February 1993). 

We requested that VA identify for our evaluation those projects using
the best software development processes implemented within VBA and
AAC.  VBA and AAC identified the following sites and projects. 

  -- VBA

Hines, IL

--Compensation & Pension/Financial Management System

--Claims Processing System

--Rating Board Automation

Philadelphia, PA

--Variable Loan Rate

Washington, DC

--Veterans Services Network

  -- AAC

Austin, TX

--On-Line Data Entry

--Personnel Accounting Integrated Data

We evaluated the software development processes used on these
projects, focusing on KPAs necessary to achieve a repeatable
capability.  Organizations that have a repeatable software
development process have been able to significantly improve their
productivity and return on investment.  In contrast, organizations
that have not developed the process discipline necessary to better
manage and control their projects at the repeatable level incur
greater risk of schedule delay, cost overruns, and poor quality
software.\4 These organizations rely solely upon the variable
capabilities of individuals, rather than on institutionalized
processes considered basic to software development. 

According to SEI,\5 processes for a repeatable capability (i.e., CMM
level 2) are considered the most basic in establishing discipline and
control in software development and are crucial steps for any project
to mitigate risks associated with cost, schedule, and quality.  As
shown in table 2, these processes include (1) requirements
management, (2) software project planning, (3) software project
tracking and oversight, (4) software subcontract management, (5)
software quality assurance, and (6) software configuration
management. 



                                Table 2
                
                  CMM Level 2 "Repeatable" Key Process
                           Area Descriptions

CMM Level 2 KPAs                          Description
----------------------------------------  ----------------------------
Requirements management                   Defining, validating, and
                                          prioritizing requirements,
                                          such as functions,
                                          performance, and delivery
                                          dates.

Project planning                          Developing estimates for the
                                          work to be performed,
                                          establishing the necessary
                                          commitments, and defining
                                          the plan to perform the
                                          work.

Project tracking and oversight            Tracking and reviewing
                                          software accomplishments and
                                          results against documented
                                          estimates, commitments, and
                                          plans and adjusting these
                                          based on the actual
                                          accomplishments and results.

Software subcontract management           Selecting qualified
                                          contractors and managing
                                          them effectively.

Software quality assurance                Reviewing and auditing the
                                          software products and
                                          activities to ensure that
                                          they comply with the
                                          applicable processes,
                                          standards, and procedures
                                          and providing the staff and
                                          managers with the results of
                                          their reviews and audits.

Software configuration management         Selecting project baseline
                                          items, such as
                                          specifications;
                                          systematically controlling
                                          these items and changes to
                                          them; and recording and
                                          reporting status and change
                                          activity for these items.
----------------------------------------------------------------------
We conducted our review between August 1995 and February 1996, in
accordance with generally accepted government auditing standards. 


--------------------
\1 SEI is a nationally recognized, federally funded research and
development center established at Carnegie Mellon University in
Pittsburgh, Pennsylvania, to address software development issues.  In
the late 1980's, the Software Engineering Institute, with assistance
from the Mitre Corporation, developed a process maturity framework
that would help organizations improve their software process.  In
general, software process maturity serves as an indicator of the
likely range of cost, schedule, and quality results to be achieved by
projects within a software organization. 

\2 Version 2.0 of the SCE method is based on SEI's Capability
Maturity Model, Version 1.1. 

\3 According to an SEI study ( i.e., Moving on Up:  Data and
Experience Doing CMM-Based Process Improvement, Technical Report
CMU/SEI-95-TR-008, August 1995) of 48 organizations that implemented
software process improvement programs, the time required to increase
process maturity from level 1 to level 2 took an average of 30
months, with a range of 11 months to 58 months. 

\4 Capability Maturity Model for Software, Version 1.1 (Technical
Report CMU/SEI-93-TR-24, February 1993). 

\5 Software Capability Evaluation, Version 2.0, Method Description
(CMU/SEI-94-TR-06, June 1994). 


   RESULTS IN BRIEF
------------------------------------------------------------ Letter :3

VA does not satisfy the criteria for a level 2 (i.e., repeatable)
software development capability on any of the seven projects they
identified as using their most mature processes.  None of the
projects reviewed satisfy any of the criteria (i.e., KPAs) that the
Software Engineering Institute's CMM requires for a repeatable
process.  For example, VBA is extremely weak in the requirements
management, software project planning, and software subcontract
management KPAs with no identifiable strengths or improvement
activities.  As a level 1 organization, VBA cannot reliably develop
and maintain high-quality software on any major project within
existing cost and schedule constraints, placing the VBA modernization
program at risk. 

Similarly, AAC has a level 1 software development capability.  It
does not satisfy any of the criteria (i.e., KPAs) for a level 2
(i.e., repeatable) process.  Austin's weakest KPA is software
subcontract management with no identifiable strengths or improvement
activities. 

However, there are some strengths and improvement activities that
both VBA and AAC can build upon to improve their respective software
development processes.  For example, VBA can build upon its strengths
in the software quality assurance KPA, and its improvement activities
in the software configuration management KPA.  Similarly, AAC can
build upon its strengths in the software quality assurance and
software configuration management KPAS.  Activities of this nature
will help both VBA and AAC move toward a level 2 (i.e., repeatable)
software development capability. 


   VETERANS BENEFITS
   ADMINISTRATION SOFTWARE
   PRACTICES
------------------------------------------------------------ Letter :4

Highlights of our evaluation of VBA's software practices using the
SEI criteria outlined in appendix II follow. 

  -- Requirements Management - The purpose of requirements management
     is to establish a common understanding between the customer and
     the software project of the customer's requirements that will be
     addressed by the software project.  The first goal within this
     KPA states that, "system requirements allocated to software are
     controlled to establish a baseline for software engineering and
     management use." VBA does not manage and control system
     requirements as required by this goal.  Moreover, members of
     software-related groups are not trained in requirements
     management activities.  Also, changes made to software plans,
     work products, and activities resulting from changes to the
     software requirements are not assessed for risk. 

  -- Software Project Planning - The purpose of software project
     planning is to establish reasonable plans for performing the
     software engineering and for managing the software project.  VBA
     projects do not have software development plans, estimates for
     software project costs are not derived using conventional
     industry methods and tools, and VBA is unable to show the
     derivation of the estimates for the size (or changes to the
     size) of the software work products.  Also, individuals involved
     in the software project planning are not trained in estimating
     and planning procedures applicable to their area of
     responsibility. 

  -- Software Project Tracking and Oversight - The purpose of
     software project tracking and oversight is to provide adequate
     visibility into actual progress so that management can take
     effective actions when the software project's performance
     deviates significantly from software plans.  VBA does track
     software project schedules against major milestones; however, as
     mentioned previously, these schedules and milestones are not
     derived using conventional industry methods nor is there a
     comprehensive software plan against which to track activities. 
     Moreover, the size of software work products (or the size of
     changes to software work products) are not tracked, and the
     software risks associated with cost, resource, schedule, and
     technical aspects of the project are not tracked. 

  -- Software Subcontract Management - The purpose of software
     subcontract management is to select qualified software
     subcontractors and manage them effectively.  VBA does not have a
     written organizational policy that describes the process for
     managing software contracts.  Additionally, the software work to
     be contracted is neither defined nor planned according to a
     documented procedure.  Finally, software managers and other
     individuals who are involved in developing, negotiating, and
     managing a software contract are not trained to perform these
     activities. 

  -- Software Quality Assurance - The purpose of software quality
     assurance is to provide management with appropriate visibility
     into the process being used by the software project and of the
     products being built.  VBA has a software quality and control
     (SQ&C) group that has a reporting channel to senior management,
     independent of the project managers.  The SQ&C group also
     performs testing of the software code.  However, the SQ&C group
     does not participate in other software quality assurance (SQA)
     functions, such as the preparation, review, and audit of
     projects' software development plans, standards, procedures, and
     other work products.  Also, projects do not have SQA plans. 

  -- Software Configuration Management - The purpose of software
     configuration management is to establish and maintain the
     integrity of products of the software project throughout the
     project's software life cycle.  VBA has provided formal training
     to its staff in defining software processes.  However, VBA
     cannot effectively control the integrity of its software work
     products because it has no software configuration control board,
     it does not identify software work products to be placed under
     configuration management, and it has no configuration management
     library system to serve as a repository for software work
     products.  VBA has begun improvement activities in this area by
     (1) establishing a software configuration management group and
     (2) drafting a software configuration management procedure. 

Following a presentation of GAO's SCE results to the Chief
Information Officer of VBA, the Director of VBA's Office of
Information Systems forwarded a letter to GAO citing a number of
initiatives that are currently underway to address some of the stated
deficiencies.  Initiatives cited by the VBA include: 

  -- development and distribution of interim configuration management
     procedures;

  -- identification of a library structure to hold all of the work
     products from the development process; and

  -- initiation of several meetings with SEI to discuss the Software
     CMM. 


   AUSTIN AUTOMATION CENTER
   SOFTWARE PRACTICES
------------------------------------------------------------ Letter :5

Similar to VBA, we compared the CMM criteria in appendix II to the
software development practices at AAC.  Summary results of this
evaluation follow. 

  -- Requirements Management - AAC does not create or control a
     requirements baseline for software engineering.  Also, AAC does
     not manage or control requirements.  AAC does have a process for
     negotiating periodic contractual arrangements with customers,
     but this process does not include baselining and controlling
     software requirements. 

  -- Software Project Planning - Although AAC documents its schedule
     estimates for software development projects, there is (1) no
     defined methodology in use for estimating software costs, size,
     or schedule, (2) no derivation of estimates for the size (or
     changes to the size) of software products, and (3) no derivation
     of the estimates for software project costs.  Similarly, AAC
     uses a project planning tool called "MultiTrak".  However,
     projects do not have software development plans. 

  -- Software Project Tracking and Oversight - AAC performs schedule
     tracking at major milestones.  However, the goals for this KPA
     call for (1) the tracking of actual results and performances
     against software plans, (2) the management of corrective actions
     when deviations from the software plan occur, and (3) the
     affected parties to mutually agree to changes in commitments. 
     AAC does not conform to these goals.  For example, AAC does not
     track (1) the software risks associated with cost, resource,
     schedule, and technical aspects of the project and (2) the size
     of software work products (or size of changes to software work
     products). 

  -- Software Subcontract Management - Although the goals for this
     KPA emphasize the selection of qualified software subcontractors
     and managing them effectively, AAC does not (1) have a
     documented procedure that explains how the work to be contracted
     should be defined and planned and (2) ensure that software
     managers and other individuals who are involved in establishing
     a software contract are trained to perform this activity. 

  -- Software Quality Assurance - The goals within this KPA emphasize
     (1) the verification of the adherence of software products and
     activities to applicable standards, procedures, and requirements
     and (2) the reporting of noncompliance issues that cannot be
     resolved within the project to senior management.  AAC has an
     automated data processing system integrity guideline and a
     systems integration service (SIS) group that has a reporting
     channel to senior management and is independent of the project
     managers.  However, projects do not have SQA plans; the SIS
     group does not participate in certain SQA functions, such as the
     preparation, review, and audit of projects' software development
     plans, standards, and procedures; and members of the SIS group
     are not trained to perform their SQA activities. 

  -- Software Configuration Management - AAC performs software (i.e.,
     code only) change control using a tool called "ENDEVOR," and its
     employees are trained in the use of this tool.  However, the
     scope of the goals within this KPA cover all products in the
     entire software life cycle and not just the software code.  AAC
     has not identified software work products (with the exception of
     software code) that need to be placed under configuration
     management, established a configuration management library
     system that can be used as a repository for software work
     products, or established a software configuration control board. 

Unless both VBA and AAC initiate improvement activities within the
various KPAs and accelerate those already underway, they are unlikely
to produce and maintain high-quality software on time and within
budget. 


   CONCLUSIONS AND RECOMMENDATIONS
------------------------------------------------------------ Letter :6

Because VBA and AAC do not satisfy any of the KPAs required for a
level 2 (i.e., repeatable) capability, there is no assurance that (1)
investments made in new software development will achieve their
operational improvement objectives or (2) software will be delivered
consistent with cost and schedule estimates.  To better position VBA
and AAC to develop and maintain their software successfully and to
protect their software investments, we recommend that the Secretary
of Veterans Affairs take the following actions: 

  -- Delay any major investment in software development beyond that
     which is needed to sustain critical day-to-day operations until
     the repeatable level of process maturity is attained. 

  -- Obtain expert advice to assist VBA and AAC in improving their
     ability to develop high-quality software, consistent with
     criteria promulgated by SEI. 

  -- Develop an action plan, within 6 months from the date of this
     letter, that describes a strategy to reach the repeatable level
     of process maturity. 

  -- Implement the action plan expeditiously. 

  -- Ensure that any future contracts for software development
     require the contractor have a software development capability of
     at least CMM level 2. 


   AGENCY COMMENTS AND OUR
   EVALUATION
------------------------------------------------------------ Letter :7

VBA comments responded to its SCE results, and VA comments responded
to the SCE results for AAC. 


      VBA COMMENTS ON GAO
      RECOMMENDATIONS
---------------------------------------------------------- Letter :7.1

In commenting on a draft of this report, the Veterans Benefits
Administration (VBA) agreed with four of our recommendations and
disagreed with one recommendation.  VBA stated that while it agreed
that a repeatable (i.e., level 2) level of process maturity is a goal
that must be attained, it disagreed that "...all software development
beyond that which is day-to-day critical must be curtailed..." VBA
further stated that the payment system replacement projects, the
migration of legacy systems, and other activities to address the
change of century must continue.  While we agree that the software
conversion or development activities required to address issues such
as the change of century or changes to legislation must continue, we
would characterize these as sustaining critical day-to-day
operations.  However, major system development initiatives in support
of major projects such as the system modernization effort, which
involves several system replacement projects and the migration of
legacy systems, and VETSNET, which includes several payment system
replacement projects, should be reassessed for risk of potential
schedule slippage, cost overrun, and shortfall in anticipated system
functions and features.  Shortcomings such as these are more likely
from organizations with a software development maturity rating below
level 2 (i.e., the repeatable level).  Therefore, to minimize
software development risks, we continue to believe that VBA should
delay any major investment in software development unless it is
required to sustain day-to-day operations, until a maturity rating of
level 2 is reached. 

Regarding the remaining four recommendations, we are pleased to see
that VBA is already initiating positive actions, including acquiring
the assistance of the Software Engineering Institute. 


      VA COMMENTS ON SCE RESULTS
      FOR AAC
---------------------------------------------------------- Letter :7.2

VA stated that we did not demonstrate a willingness or flexibility in
relating AAC documentation products, activities, and terms to the SEI
terms.  We reviewed all documentation provided to us by VA including
the documents listed in their comments on our draft report.  As
called for by the SCE methodology, we carefully compared all this
documentation to the SEI CMM criteria.  As stated throughout our
report, we found some strengths but in many cases, VA's documentation
was not commensurate with that called for by the SCE methodology. 
Our comments on the specific key process areas follow. 


         REQUIREMENTS MANAGEMENT
-------------------------------------------------------- Letter :7.2.1

The VA comments stated that the OFM/IRM Business Agreement, dated
September 1994, contains guidelines which mandate the management of
software requirements.  However, in our review of the documentation
listed under requirements management (Enclosure 1:  Documents
Addressing Key Process Area), we found no evidence that these
documents addressed any of the goals of this KPA.  For example, (1)
the allocated requirements are neither managed, controlled, nor
baselined, and (2) no software development plans were developed based
on the allocated requirements. 


         SOFTWARE PROJECT PLANNING
-------------------------------------------------------- Letter :7.2.2

VA feels that the AAC Business Agreement and the negotiated quarterly
contract satisfies this KPA; however, we found that AAC does not
perform a majority of the activities required to meet the goals
within this KPA.  For example, AAC was not able to submit evidence
for estimating software size and cost, nor did AAC demonstrate any
methodology used for estimating schedules. 


         SOFTWARE PROJECT TRACKING
         AND OVERSIGHT
-------------------------------------------------------- Letter :7.2.3

VA stated that project size and risk remain consistent throughout the
development/implementation cycle.  However, AAC did not provide our
SCE team with any evidence validating this assertion and, as
discussed on page 8, AAC does not track this information. 


         SOFTWARE SUBCONTRACT
         MANAGEMENT
-------------------------------------------------------- Letter :7.2.4

VA claims that specific written policies and procedures are followed
when managing software contracts; however, AAC staff interviewed were
unable to provide us with any specific policies or procedures used
for software contracting.  The AAC staff acknowledged that they do
not track (1) software contractor performance at the coding level
(i.e., track functionality only) or (2) contractor produced software
documentation.  Regarding training for software contract management,
VA stated that its COTRs receive training in procurement, project
management, and evaluating contractor performance.  However, there is
no indication that these courses are specific to software
contracting.  In addition, other individuals involved in establishing
the software contract for the projects reviewed had not received
contract management training related to software. 


         SOFTWARE QUALITY ASURANCE
-------------------------------------------------------- Letter :7.2.5

VA states that its ADP System Integrity Guide, dated September 1994,
contains detailed procedures directing the SIS group in specific SQA
functions.  Although this is a good first step, the AAC is still
deficient because it does not have project specific software quality
assurance plans that are implemented for individual projects, as
requuired by this KPA within the CMM.  Furthermore, we were not
provided with any evidence showing that the ADP System Integrity
Guide has been officially issued or whether its use will be mandatory
or discretionary. 


         SOFTWARE CONFIGURATION
         MANAGEMENT
-------------------------------------------------------- Letter :7.2.6

The VA comments do not present any additional evidence that would
help to satisfy the criteria for this KPA.  Specifically,
communication between the SIS, AAC staff, and customer do not
substitute for the rigor and discipline of a software configuration
control board, which VA acknowledged they do not have.  Furthermore,
the placement of software code under configuration management is not
sufficient to satisfy this KPA because other software work
products--such as system design specifications, database
specifications, and computer program specifications--are also
required.  Finally, although the AAC does maintain a library of those
software work products that it does produce, the products are not
maintained under a formal software configuration management
discipline, which would include version control and rigorous
requirements traceability. 


---------------------------------------------------------- Letter :7.3

We are sending copies of this report to the Chairmen and Ranking
Minority Members of the House and Senate Committees on Veterans
Affairs and the House and Senate Committees on Appropriations; the
Secretary of Veterans Affairs; and the Director, Office of Management
and Budget.  Copies will also be made available to other interested
parties upon request. 

This work was performed under the direction of William S.  Franklin,
Director, Information Systems Methodology and Support, who can be
reached at (202) 512-6234.  Other major contributors are listed in
appendix IV. 

Sincerely yours,

Gene L.  Dodaro
Assistant Comptroller General




(See figure in printed edition.)Appendix I
COMMENTS FROM THE VETERANS
BENEFITS ADMINISTRATION
============================================================== Letter 



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)




(See figure in printed edition.)Appendix II
COMMENTS FROM THE DEPARTMENT OF
VETERANS AFFAIRS
============================================================== Letter 

text appears at the end of this appendix. 



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)


The following is GAO's comment on the Department of Veterans Affairs'
May 24, 1996, letter. 

GAO COMMENT

1.  This issue is not addressed in our report. 


LEVEL 2 KEY PROCESS AREA GOALS
========================================================= Appendix III

Level 2 KPA         Purpose                        Goals
------------------  -----------------------------  -----------------------------
Requirements        To establish a common          Goal 1
management          understanding between the      System requirements allocated
                    customer and the software      to software are controlled to
                    project of the customer's      establish a baseline for
                    requirements that will be      software engineering and
                    addressed by the software      management use.
                    project.
                                                   Goal 2
                                                   Software plans, products, and
                                                   activities are kept
                                                   consistent with the system
                                                   requirements allocated to
                                                   software.

Software project    To establish reasonable plans  Goal 1
planning            for performing the software    Software estimates are
                    engineering and for managing   documented for use in
                    the software project.          planning and tracking the
                                                   software project.

                                                   Goal 2
                                                   Software project activities
                                                   and commitments are planned
                                                   and documented.

                                                   Goal 3
                                                   Affected groups and
                                                   individuals agree to their
                                                   commitments related to the
                                                   software project.


Software project    To provide adequate            Goal 1
tracking and        visibility into actual         Actual results and
oversight           progress so that management    performances are tracked
                    can take effective actions     against the software plans.
                    and when the software
                    project's performance          Goal 2
                    deviates significantly from    Corrective actions are taken
                    software plans.                and managed to closure when
                                                   actual results and
                                                   performance deviate
                                                   significantly from the
                                                   software plans.

                                                   Goal 3
                                                   Changes to software
                                                   commitments are agreed to by
                                                   the affected groups and
                                                   individuals.

Software            To select qualified software   Goal 1
subcontract         subcontractors and manage      The organization selects
management          them effectively.              qualified software
                                                   subcontractors.

                                                   Goal 2
                                                   The organization and the
                                                   software subcontractor agree
                                                   to their commitments to each
                                                   other.

                                                   Goal 3
                                                   The organization and the
                                                   software subcontractor
                                                   maintain ongoing
                                                   communications.

                                                   Goal 4
                                                   The organization tracks the
                                                   software subcontractors'
                                                   actual results and
                                                   performance against its
                                                   commitments.

Software quality    To provide management with     Goal 1
assurance           appropriate visibility into    Software quality assurance
                    the process being used by the  activities are planned.
                    software project and of the
                    products being built.          Goal 2
                                                   Adherence of software
                                                   products and activities to
                                                   the applicable standards,
                                                   procedures, and requirements
                                                   is verified objectively.

                                                   Goal 3
                                                   Affected groups and
                                                   individuals are informed of
                                                   software quality assurance
                                                   activities and results.

                                                   Goal 4
                                                   Noncompliance issues that
                                                   cannot be resolved within the
                                                   software project are
                                                   addressed by senior
                                                   management.

Software            To establish and maintain the  Goal 1
configuration       integrity of products of the   Software configuration
management          software project throughout    management activities are
                    the project's software life    planned.
                    cycle.
                                                   Goal 2
                                                   Selected software work
                                                   products are identified,
                                                   controlled, and available.

                                                   Goal 3
                                                   Changes to identified
                                                   software work products are
                                                   controlled.

                                                   Goal 4
                                                   Affected groups and
                                                   individuals are informed of
                                                   the status and content of
                                                   software baselines.
--------------------------------------------------------------------------------

MAJOR CONTRIBUTORS TO THIS REPORT
========================================================== Appendix IV

ACCOUNTING AND INFORMATION
MANAGEMENT DIVISION, WASHINGTON,
D.C. 

David Chao, SCE Team Leader
Gary R.  Austin, SCE Team Member
K.  Alan Merrill, SCE Team Member
Madhav S.  Panwar, SCE Team Member
Keith A.  Rhodes, SCE Team Member
Paul Silverman, SCE Team Member

*** End of document. ***