Veterans Benefits: Training for Claims Processors Needs
Evaluation (31-MAY-01, GAO-01-601).
The Veterans Benefits Administration (VBA) developed a
computer-assisted training program, known as the Training and
Performance Support System (TPSS), to help its employees become
more accurate in processing disability compensation and pension
claims. The program seeks to provide uniform and consistent
training to employees in 57 regional offices. Although VBA's
long-term goal is to attain a 96 percent accuracy rate for claims
processing, VBA reported an accuracy rate of only 59 percent for
fiscal year 2000. This report reviews (1) the status of the TPSS
program's development and implementation and (2) the extent to
which TPSS will meet its objectives. GAO found that despite VBA's
objective to centrally develop a standardized training program,
significant delays in the development of TPSS are hindering the
program's ability to provide standardized training to claims
processing employees. According to VBA's current schedule, the
full development of the program will not be completed until at
least 2004, or about two years later than VBA had planned.
Although VBA provided nine training modules to regional offices
to begin the program, the extent to which the offices implemented
them varied considerably. Many offices reported that workload
pressures prevented them from fully using the modules. GAO found
that TPSS might not fully achieve its objectives. For instance,
TPSS training modules may not be available in time to train new
employees hired to replace employees who are expected to retire
in the future. Furthermore, TPSS may not reduce training time, as
envisioned by VBA.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-01-601
ACCNO: A01029
TITLE: Veterans Benefits: Training for Claims Processors Needs
Evaluation
DATE: 05/31/2001
SUBJECT: Human resources training
Personnel recruiting
Standards and standardization
Veterans benefits
Performance measures
Claims processing
Schedule slippages
VBA Training and Performance Support
System
VA Systematic Individual Performance
Assessment
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Testimony. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-01-601
Report to the Ranking Democratic Member, Committee on Veterans' Affairs,
House of Representatives
United States General Accounting Office
GAO
May 2001 VETERANS' BENEFITS
Training for Claims Processors Needs Evaluation
GAO- 01- 601
Page i GAO- 01- 601 Veterans' Benefits Letter 1
Appendix I Scope and Methodology 16
Appendix II Comments From the Department of Veterans Affairs 17
Tables
Table 1: VBA?s Schedule for Releasing TPSS Modules to Regional Offices 6
Table 2: Number and Percentage of Regional Offices Using TPSS as
of September 30, 2000 8
Abbreviations
TPSS Training and Performance Support System VA Department of Veterans
Affairs VBA Veterans Benefits Administration Contents
Page 1 GAO- 01- 601 Veterans' Benefits
May 31, 2001 The Honorable Lane Evans Ranking Democratic Member Committee on
Veterans? Affairs House of Representatives
Dear Mr. Evans: A top priority of the Veterans Benefits Administration (VBA)
is to improve its employees? accuracy in processing disability compensation
and pension claims. While VBA?s long- term goal is to attain a 96- percent
accuracy rate for claims processing, VBA reported an accuracy rate of only
59 percent for fiscal year 2000. To improve the accuracy rate, VBA?s primary
training initiative is a computer- assisted training program known as the
Training and Performance Support System (TPSS). The program?s purpose is to
provide uniform and consistent training to employees in 57 regional offices
in order to improve the accuracy and consistency of disability claims
processing nationwide. The need to invest in human capital through better
training was emphasized in 1993 when a VBA Blue Ribbon Panel reported that
VBA needed to improve the training for its regional office employees. In
1998, VBA committed itself to develop the TPSS program, and, as of March
2001, had spent or obligated about $18.6 million of the estimated overall
program cost of $32 million.
You asked us to provide you with information on (1) the status of the TPSS
program?s development and implementation and (2) the extent to which TPSS
will meet its objectives. To address these issues, we met with VBA officials
and reviewed VBA documents. We surveyed all regional offices and visited six
of them to determine their experiences with and their views on the TPSS
program. At the six offices we visited, we talked with regional office
officials and employees who had taken TPSS training. We did not evaluate the
effectiveness of any type of training, but we obtained the views of regional
offices on the benefits or adequacy of TPSS. (See app. I for more
information on scope and methodology.) We conducted our review from June
2000 through March 2001 in accordance with generally accepted government
auditing standards.
The development and implementation schedule for the TPSS program is at least
2 years behind schedule. In February 2000, VBA projected that TPSS would be
fully developed and implemented by the middle of calendar year
United States General Accounting Office Washington, DC 20548
Results in Brief
Page 2 GAO- 01- 601 Veterans' Benefits
2002. However, as of March 2001, the earliest projected completion date for
TPSS would be calendar year 2004. As of March 2001, VBA had provided the
regional offices with 9 training modules out of an estimated 50 modules that
would be needed to fully implement TPSS. Data from our survey showed that,
as of October 2000, 8 of the 57 regional offices had not used any of the
available modules to train employees, and the other 49 regional offices had
used the modules to varying extents. Until December 2000, VBA had not
established a requirement that regional offices use the TPSS modules. In
general, the regional offices reported that the most significant hindrance
to using the modules was lack of time for training due to workload
pressures.
Early indications are that TPSS may not fully achieve its objectives. For
example, VBA planned to use TPSS as the means for providing standardized
training to new employees whom VBA currently is hiring or plans to hire in
the near future to replace the wave of employees expected to retire during
the next several years. However, many TPSS training modules will not be
available in time to train new employees. Also, VBA viewed TPSS as the means
for significantly reducing the training period required for employees to
become fully proficient in their jobs. However, in our site visits, regional
office officials said they believe that TPSS will not significantly reduce
the training period and that traditional forms of training, such as local
on- the- job training and coaching and mentoring, will continue to be a
significant and necessary learning source. Furthermore, although VBA
believes that TPSS will improve claims- processing accuracy and consistency,
VBA does not have a formal plan to evaluate the impact of TPSS on the
accuracy rate or on consistency. While it may be difficult to determine the
exact impact of TPSS, avenues are available for VBA to collect data that
could provide indications of whether TPSS is helping to improve accuracy and
consistency.
This report contains recommendations to the Secretary of the Department of
Veterans Affairs (VA) concerning actions needed to provide timely
standardized training and provide indicators of the impact of TPSS on
accuracy and consistency.
VA provided comments on a draft of this report. In its comments, VA agreed
with our recommendations.
VBA?s compensation program pays monthly benefits, based on degree of
disability, to veterans who have service- connected disabilities (injuries
or diseases incurred or aggravated while on active military duty),
regardless Background
Page 3 GAO- 01- 601 Veterans' Benefits
of whether these veterans are employed or have earnings. The pension program
pays monthly benefits, based on financial need, to wartime veterans who have
low incomes and are permanently and totally disabled for reasons not
service- connected. When veterans submit disability claims, VBA?s 57
regional offices must follow newly defined requirements for assisting
veterans in obtaining evidence to support their claims. 1 For a disability
compensation claim, the regional office must (1) obtain the veteran?s
existing service medical records and, if the veteran has furnished
information sufficient to locate such records, other relevant service
records; (2) obtain records of relevant medical treatment or examination of
the veteran provided at VA health- care facilities or at VA?s expense; (3)
obtain any other relevant records held by a federal department or agency;
and (4) provide a medical examination or obtain a medical opinion, if
necessary to make a decision on the claim.
Regional offices have three primary claims- processing positions for which
TPSS is intended to provide training. Veterans Service Representatives
obtain evidence required for veterans? claims. Rating Veterans Service
Representatives (hereafter referred to as Rating Specialists) analyze the
evidence, evaluate service- connected impairments, and assign a rating for
the degree to which the veteran is disabled. Decision Review Officers
attempt to resolve veterans? disagreements with regional office decisions
and can grant benefits based on the same evidence used by the Rating
Specialist. Two of these regional office positions are relatively new- the
Veterans Service Representative position and the Decision Review Officer
position were implemented during 1998- 99 and 2000, respectively.
If a veteran disagrees with a regional office decision, the veteran may file
an appeal asking VA?s Board of Veterans? Appeals to review the decision. 2
The Board makes VA?s final decision on such appeals and can grant or deny
benefits, but in some cases the Board may return (remand) the case to the
regional office to obtain further evidence and reconsider the
1 These requirements are contained in the Veterans Claims Assistance Act of
2000 (P. L. 106475, Nov. 9, 2000). 2 In fiscal year 2000, veterans filed
such appeals in about 5 percent of all initial decisions involving ratings.
Page 4 GAO- 01- 601 Veterans' Benefits
decision. 3 In these cases, the regional office either grants the claim or
returns it to the Board for a final VA decision. 4
Our High- Risk Series states that human capital challenges pose a high risk
for VBA, as well as other agencies across the federal government. 5
Specifically, our High- Risk Series states that VBA?s training and
recruitment programs may not be adequate to ensure a sufficient workforce of
competent claims processors, and the lack of such a workforce could
undermine efforts to reduce current problems of claimsprocessing backlogs
and errors. We have published a human capital selfassessment checklist
providing a systematic approach to identifying and addressing human capital
issues. 6 Our checklist states that an agency should make appropriate
investments in education, training, and other developmental opportunities to
help its employees build the competencies needed to achieve the agency?s
?shared vision?- the mission, vision for the future, core values, goals and
objectives, and strategies that define the agency?s direction and
expectations for itself and its people. Such investments require an explicit
workforce planning strategy, linked to the agency?s strategic and program
planning efforts. These planning efforts should identify, among other
things, the knowledge, skills, and abilities needed to position an agency
for its future. In addition, they should provide an explicit link between
the agency?s training offerings and curriculums and the competencies
identified by the agency for mission accomplishment.
In VA?s Departmental Performance Plan for fiscal year 2001, VBA links the
TPSS program to its strategic goal of providing world- class service to
veterans and their families. 7 In working toward this goal, VBA?s top
3 Not every case returned by the Board indicates that the regional office
made an error. For example, cases may be returned because the veteran
submits new evidence or because changes occur in regulations after an
appealed case is sent to the Board.
4 Veterans have two levels of appeal beyond the Board- first to the U. S.
Court of Appeals for Veterans Claims and then to the Court of Appeals for
the Federal Circuit. 5 High- Risk Series: An Update (GAO- 01- 263, Jan.
2001).
6 Human Capital: A Self- Assessment Checklist for Agency Leaders (GAO/ OCG-
00- 14G, Sept. 2000). 7 The Government Performance and Results Act of 1993
requires VA, as well as other federal agencies, to clearly define its
mission, set goals, and measure performance. VA must submit to the Congress
annual performance plans and annual reports on its success in achieving
program performance goals.
Page 5 GAO- 01- 601 Veterans' Benefits
priority is to improve the technical accuracy of claims- processing, and VBA
states that the TPSS program is one of its most important initiatives for
achieving this objective. 8 To develop TPSS training modules for a specific
claims- processing position, the job and tasks for that position must be
analyzed in detail. After this analysis, VBA estimates that it takes 14 to
16 months to develop each module, which consists of computerassisted
instruction packages and written instructional materials designed to support
cooperative learning by employees who train in groups of two or three. Under
VBA?s direction, contractors develop the modules with the help of subject
matter experts from VBA?s regional and central offices. Before a module is
released, selected regional office employees take the training on a trial
basis to validate its effectiveness. Each training module includes a pretest
for determining whether an employee needs to take the training. Each module
also includes a post- test requiring students to process test cases to
determine their mastery of the tasks and work processes. Using regional
office employees, these pretests and post- tests are also validated before
the modules are released.
The design and development of the TPSS program will take longer than planned
to complete. According to VBA?s current schedule, the development of the
full TPSS program will not be completed until at least calendar year 2004,
or about 2 years later than VBA had planned. Also, based on our survey of
regional offices, 49 of the 57 regional offices had used TPSS, but great
variation existed in the extent to which they had implemented the available
TPSS modules to train employees. In fact, until December 2000, VBA had not
established a requirement that regional offices use the TPSS modules. In
general, the regional offices reported that the most significant hindrance
to using the modules was lack of time for training due to workload
pressures.
In early 2000, VBA projected that the development of the full TPSS program
would be completed during calendar year 2002. 9 However, as of
8 TPSS training is in addition to local regional office training, such as
classroom instruction, on- the- job training, and coaching and mentoring and
VBA training provided via VBA?s Satellite Broadcast Network and the Video
Teleconferencing System shared with the Board of Veterans? Appeals.
9 VBA published the projected schedule in its budget submission for fiscal
year 2001, dated February 2000. TPSS Program
Schedule Has Been Delayed and Regional Office Implementation Has Varied
TPSS Program Is Behind Schedule
Page 6 GAO- 01- 601 Veterans' Benefits
March 2001, the earliest projected completion date for the full program
would be calendar year 2004 (see table 1). As of March 2001, VBA had
provided the regional offices with 9 training modules out of an estimated 50
modules expected to constitute the full TPSS program.
Table 1: VBA?s Schedule for Releasing TPSS Modules to Regional Offices TPSS
modules Estimated completion date shown
in VBA?s fiscal year 2001 budget Actual
release date Planned
release date (as of March
2001) Basic rating modules for new Rating Specialists Sept. 2000
Original compensation Jan. 1999 Original pension Jan. 1999 Original
dependency and indemnity compensation Jan. 1999 Routine future examinations
Dec. 1999 Hospitalization and convalescence reviews Dec. 1999 Reopened
compensation Feb. 2001 New claim for pension Feb. 2001 Reopened dependency
and indemnity compensation Feb. 2001 Due process June 2001 Ancillary
benefits
All basic rating modules were to be completed by Sept. 2000.
June 2002
Advanced rating modules for experienced Rating Specialists and Decision
Review Officers March 2002
Appeals May 1998 Other modules a
All advanced rating modules were to be completed by Mar. 2002. Unknown
Medical modules for Rating Specialists b Sept. 2000
Medical modules (first group) Oct. 2001 Medical modules (second group) May
2002 Medical modules (third group)
All medical modules were to be completed by Sept. 2000.
May 2003
Modules for Veterans Service Representatives Dec. 2001
Original compensation Nov. 2001 Original pension Nov. 2001 Dependency
benefits Nov. 2001 Death pension Nov. 2001 Income adjustments Nov. 2001
Fiduciary issues c Unknown Other modules (approximately 15 modules) d
All modules for Veterans Service Representatives were to be completed by
Dec. 2001.
2002- 2004 a VBA has suspended the development of the other advanced rating
modules until fiscal year 2002. The number of other modules to be developed
has not yet been determined. b VBA has identified a preliminary list of 17
medical modules, such as medical terminology,
musculoskeletal, and organs of sense. c Regional offices have ?Field?
Veterans Service Representatives who specialize in the fiduciary
responsibilities of persons who hold or administer VA benefits in trust for
beneficiaries. The modules for fiduciary issues were to be completed by June
2002, according to the budget for fiscal year 2001.
Page 7 GAO- 01- 601 Veterans' Benefits
d VBA anticipates developing 15 other modules for Veterans Service
Representatives that include such topics as helpless child benefits, aid and
attendance for nursing home patients, and burial benefits.
Source: VBA.
According to VBA, delays in the TPSS program can be attributed to two
primary causes- the creation of new claims- processing positions and
legislative changes in VBA?s duty to assist veterans in obtaining evidence
to support their claims. VBA is waiting for the duties of the new Decision
Review Officer position to stabilize before conducting the task analysis
necessary to develop the training modules for this position. In addition,
after completing the task analysis for the new Veterans Service
Representative position, VBA learned it will need to develop many more
training modules than originally expected because the position encompasses
more tasks and greater complexity than anticipated. 10 Moreover, VBA has
decided to give priority to developing modules for the Veterans Service
Representative position and thus delay the development of training modules
for other categories of employees. 11 Finally, because the Congress enacted
legislative changes in the duty- to- assist requirements in November 2000,
VBA decided to include these new requirements in the remaining basic rating
modules. Doing so delayed releasing those modules to the regional offices.
At the time of our survey in October 2000, the extent to which regional
offices had used the six TPSS modules that were available varied widely. Of
the 57 regional offices, 8 had not used any of the available TPSS modules to
train employees. Of the 49 that had used TPSS, their use varied greatly. For
example, 41 regional offices had used the original compensation module, but
only 11 had used the routine future
10 VBA began a demonstration project in 1996 that led to the adoption of the
TPSS program in 1998. In this demonstration project, VBA developed the
Appeals training module for Rating Specialists and conducted the job and
task analysis necessary to develop the basic rating modules for Rating
Specialists.
11 VBA shifted funding away from the advanced rating modules for Rating
Specialists in order to accelerate the development of modules for Veterans
Service Representatives. VBA also slowed the development of modules for
fiduciary issues. According to VBA officials, resource issues prevent VBA
from developing TPSS modules for all claims- processing positions at the
same time. Beyond the issue of annual funding availability, developing TPSS
modules requires the time of regional office employees for developing and
validating module content and testing. According to VBA, concurrent
development of modules for all claims- processing positions would drain too
many resources from the field. Regional Office
Implementation of TPSS Varied Widely
Page 8 GAO- 01- 601 Veterans' Benefits
examinations module (see table 2). Only 5 regional offices had used all of
the available modules to train employees, and 10 regional offices had used
only one module.
Table 2: Number and Percentage of Regional Offices Using TPSS as of
September 30, 2000
TPSS modules released as of September 30, 2000
Number of regional offices
that had used module
Percentage of 57 regional offices that had
used module
Original compensation 41 72 Appeals 33 58 Original pension 33 58 Original
dependency and indemnity compensation 26 46 Hospitalization and
convalescence 12 21 Routine future exams 11 19
Source: GAO survey.
VBA had strongly encouraged, but not required, regional offices to use TPSS
in their training programs. However, VBA conducted a survey of its own in
October 2000 and found that some regional offices still were not using any
TPSS modules. As a result, in December 2000, VBA issued a letter mandating
the use of TPSS in all regional offices. According to VBA officials, the
December mandate required that regional offices use the available basic
rating modules to train all new Rating Specialists and use future modules
for Veterans Service Representatives to train all new employees filling this
position. VBA officials told us that these new training requirements for
specific types of employees have also been communicated to the regional
offices via nationwide conference calls.
Based on responses to our survey from the 49 regional offices that had used
at least one TPSS module, the most significant hindrance to implementing the
modules was lack of time for training due to workload pressures. About 47
percent of these regional offices said that workload pressures affected
their use of TPSS to at least a moderate extent. As we reported in May 2000,
persistent backlogs of pending claims have resulted in veterans having to
endure long waits to receive decisions on their claims. 12 Consequently,
regional offices face pressure not only to improve
12 Veterans Benefits Administration: Problems and Challenges Facing
Disability Claims Processing (GAO/ T- HEHS/ AIMD- 00- 146, May 2000).
Page 9 GAO- 01- 601 Veterans' Benefits
accuracy in processing claims, but also to reduce the backlog of pending
claims. A contractor evaluating VBA training reported in April 2000 that
?training time is too easily given up to the press of productivity.? 13 The
six modules that had been released at the time of our survey were designed
to require a total of about 24 workdays for an employee to complete.
In 1998, VBA prepared a rationale supporting the adoption of the TPSS
program. 14 In this rationale, VBA outlined several objectives of the TPSS
program related to human capital management, including providing training
for new employees replacing the expected wave of retirees, shortening the
training period for employees, and improving employees? claims- processing
accuracy and consistency. 15 We found that many TPSS modules will not be
available when needed to train new employees and that TPSS may not
substantially reduce the training period for employees. Also, VBA has not
developed a formal plan to evaluate the program?s impact on claims-
processing accuracy or consistency.
In its rationale for the TPSS program, VBA stated that the program would
allow VBA to effectively train employees hired and promoted as replacements
for the expected wave of retiring claims processors. However, given the
slippage in the TPSS schedule, new employees have been and will be working
in these positions before many of the TPSS modules are available for their
training. Of the 4,000 Veterans Service Representatives and Rating
Specialists that were employed as of fiscal year 2000, VBA expects about
1,000 (25 percent) to retire between 2000
13 VBA contracted with the University of Illinois to conduct a three- phase
evaluation of the TPSS program. The first phase, completed in April 1999,
focused on the quality and implementation of the appeals training module.
The second phase, completed in April 2000, attempted, among other things, to
examine whether appeals module training reduced the rate at which the Board
of Veterans? Appeals returned decisions to regional offices for further
development. Researchers concluded they could not isolate the effect of the
appeals module on the return rate. The final report for the third phase of
the TPSS evaluation is not yet available. The third phase continued the
examination of the implementation and effect of the appeals module and also
examined how TPSS training fits into regional office training programs.
14 The rationale for the program was set forth in VBA?s September 1998
Capital Investment Proposal for TPSS. 15 Because of the judgment and
subjectivity involved in many claims, two different Rating Specialists could
review the same case and reach differing decisions that might be considered
?accurate,? but would not be ?consistent.?
TPSS May Not Fully Achieve Some of Its Objectives
Many TPSS Modules Will Not Be Available When Needed to Train New Claims
Processing Employees
Page 10 GAO- 01- 601 Veterans' Benefits
2005. In response, VBA already has hired a total of about 600 new Veterans
Service Representatives and Rating Specialists between fiscal years 1998 and
2000. Additionally, in fiscal year 2001, VBA anticipates hiring 224 Veterans
Service Representatives and 325 Rating Specialists. These new workers need
to be trained in order to carry out the duties of their predecessors.
However, as of late calendar year 2000, VBA did not expect to release any
TPSS modules to train new Veterans Service Representatives until late in
calendar year 2002 or in 2003.
Recognizing that the regional offices had an immediate need to train new
Veterans Service Representatives, the Under Secretary for Benefits indicated
that VBA needed to provide some modules earlier than projected. As a result,
VBA currently plans to release five modules that are more narrowly scoped
than originally planned for Veterans Service Representatives in November
2001 (see table 1). The remaining modules for Veterans Service
Representatives are to be released intermittently during 2002 through 2004.
At the same time, VBA also is exploring alternative means of providing
standardized training to compensate for the delays in developing TPSS
modules. For Veterans Service Representatives, VBA is developing a reference
manual for new hires and also is revising a training guide it has on its
internal computer network to make it more interactive, comprehensive, and
user friendly. VBA expects to complete these efforts in March and April
2001, respectively. Additionally, it is considering using the VBA Satellite
Network and video teleconferencing facilities to provide training for both
Rating Specialists and Veterans Service Representatives. The extent to which
these alternatives may eliminate the need to develop some future TPSS
modules has not been determined.
Furthermore, VBA has established a formal 12- week basic training course for
all new Rating Specialists beginning in March 2001 and for all new Veterans
Service Representatives in April 2001. To provide this training, VBA
established training centers for each of the nine Service Delivery Networks
under which the regional offices are organized. 16 After new trainees have
been given training in specific topics, such as VA
16 VBA has structured its 57 regional offices into nine Service Delivery
Networks. The regional offices that compose each Service Delivery Network
are expected to collaborate, provide mutual support, share resources, use
team- based principles, and share collective responsibility and
accountability for the Service Delivery Network?s overall performance of all
assigned work.
Page 11 GAO- 01- 601 Veterans' Benefits
terminology, by the regional offices, they will alternate between 3 weeks of
intensive training at the training centers and 3- week periods of training
in their regional offices. Training in the regional offices will incorporate
available TPSS modules.
In its rationale for the TPSS program, VBA may have overstated the extent to
which the TPSS modules can be expected to reduce the employees? training
period. VBA assumed that the TPSS basic rating modules would enable new
Rating Specialists to become fully productive in 6 to 8 months rather than 2
years. This assumption presumed that TPSS would significantly reduce the
need for on- the- job training provided by regional offices, which
historically has constituted the majority of training. Although the regional
offices that used the modules generally said in our survey that the TPSS
modules do an adequate job of providing basic knowledge, most of these
regional offices also said that traditional forms of local training will
remain essential or very important.
Regional office officials and employees told us that TPSS cannot by itself
adequately train employees in the many complexities, subjective issues, and
variations in circumstances encountered in individual claims. In a similar
vein, a VBA contractor evaluating TPSS training said that TPSS could not
fully cover the complexity of the work of Rating Specialists. Such in- depth
training can only be gained through on- the- job training and coaching and
mentoring while employees are gaining actual experience in processing
claims. In our site visits, regional office officials said they believe that
achieving full proficiency as a Rating Specialist will require about 2
years, even with TPSS. VBA officials, however, said that such conclusions
may be premature because at the time of our review, the regional offices had
had relatively limited exposure to TPSS. They also said that they believe
the new 12- week training course incorporating TPSS will help new trainees
become proficient faster than in the past. In addition, the officials said
that they plan to evaluate the results of the new 12- week program.
In its rationale for adopting the TPSS program, VBA stated that TPSS would
help improve the accuracy and consistency of claims processing; however, VBA
has no formal plan to assess the impact of TPSS on the accuracy rate or on
consistency. Assessing the specific impact of TPSS on accuracy and
consistency is difficult. While it may not be feasible for VBA to isolate
the effect of TPSS on accuracy or consistency, opportunities exist for VBA
to collect data that could help provide some indications of TPSS May Not
Substantially Shorten the Training Period for Employees to Become Fully
Proficient
Impact of TPSS Can Be Further Evaluated
Page 12 GAO- 01- 601 Veterans' Benefits
the possible extent of such effects. As stated in our self- assessment
checklist for helping agencies address human capital issues, an agency
should make determinations of the impact of its training programs.
Assessing the specific impact of TPSS on the accuracy of claimsprocessing
performance is difficult. In its fiscal year 2001 budget submission, VBA
stated that it is reasonable to anticipate that the TPSS program will
improve the national accuracy rate for disability claims processing by 8
percentage points. However, according to VBA officials, this projected
increase in the accuracy rate is not based on an analysis linking employee
training to claims- processing accuracy, but rather on professional
judgment. The ability to isolate the specific impact of TPSS is difficult
because other initiatives implemented by VBA may also affect claims-
processing performance. For instance, VBA?s new accuracy measurement system
can also affect claims- processing performance because it provides
information on specific types of errors. 17 This information can be used to
correct the causes of errors, thereby helping to improve the overall
accuracy rate. In addition, researchers have found it difficult to link TPSS
training to changes in employees? claims- processing performance. For
example, researchers cautioned that while their findings indicated that TPSS
training did result in knowledge transfer (over 95 percent of the students
who took TPSS module post- tests passed the tests), their evaluation results
could not substantiate whether TPSS training had improved standardization or
increased claims- processing productivity across regional offices.
Using its accuracy measurement system, VBA is able to track overall trends
in the claims- processing accuracy rate, which may provide a general
indicator of whether TPSS is helping to improve accuracy at the national
level. However, it may be possible for VBA to collect data that could
provide further indications of the impact of TPSS. In fiscal year 2001, VBA
is planning to pilot test a new initiative- Systematic Individual
Performance Assessment (SIPA)- to assess performance quality and identify
training needs at the employee level. SIPA will complement the existing
accuracy measurement system by reviewing the quality of a representative
sample of the individual employee?s work products. Such
17 In fiscal year 1999, VBA implemented a new system for measuring the
accuracy of claims processing. Under the new system, the error rate includes
not only incorrect decisions on whether to grant or deny benefits but also
procedural and technical errors such as the failure to include all required
documentation in the case file or to properly notify veterans of decisions.
Impact on Accuracy
Page 13 GAO- 01- 601 Veterans' Benefits
data on individual employees? performance could potentially help provide
indications of whether TPSS is having an impact on accuracy. For example,
VBA could use such data to monitor the accuracy levels of individual
employees as they progress through the TPSS curriculum, as well as compare
the accuracy levels of employees who completed training on certain TPSS
modules with those employees who did not. VBA is considering using data from
SIPA to assess the impact of TPSS, but VBA has not developed a formal
evaluation plan.
VBA in its rationale for the TPSS program stated that the program is
intended to improve claims- processing consistency among the regional
offices. VBA is considering performing assessments to measure
claimsprocessing consistency, but it has not developed a formal evaluation
plan. Such assessments are needed to obtain data that could provide
indications of whether TPSS is helping to improve consistency. In 1997, the
National Academy of Public Administration issued a report that, among other
things, addressed inconsistency in VBA?s claims processing. The report
stated that VBA needed to identify the degree of subjectivity expected for
various medical issues, set consistency standards, and measure the level of
consistency as part of the quality review process or through testing of
control cases in several regional offices. Performing such an assessment
could provide a baseline against which future assessments of consistency
could be compared to measure progress in achieving consistency. While such
assessments would not provide conclusive evidence on the impact of TPSS,
they potentially could provide an indication of whether TPSS is helping to
improve consistency.
While we support VBA?s objective to centrally develop a standardized
training program, significant delays in the development of TPSS are
hindering the program?s ability to achieve its objective of providing
standardized training to claims processing employees. Given the large number
of recent and expected new employees, we support VBA?s effort to expedite
the development of modules aimed at these employees. At the same time, we
support its exploration of alternative ways to provide standardized training
to employees for whom TPSS modules are not yet available. Moreover, because
TPSS is still evolving, opportunities exist to reassess the need for TPSS
modules whose topics are adequately covered through these alternative means.
Furthermore, we recognize that it may not be feasible to measure the exact
extent to which TPSS improves claims- processing accuracy or consistency.
However, opportunities exist for VBA to collect data that could help provide
some indications of the possible extent of such effects. Such information
could serve as an Impact on Consistency
Conclusions
Page 14 GAO- 01- 601 Veterans' Benefits
indicator of the impact of TPSS on accuracy and consistency and help VBA
demonstrate the value of the program to the Congress and to veterans.
To further strengthen VBA?s ability to provide timely, standardized training
to its regional office employees who process disability claims, we recommend
that the Secretary of Veterans Affairs direct the Under Secretary for
Benefits to take the following actions.
As VBA explores alternative means, such as the VBA Satellite Network, to
deliver timely standardized training, it should determine whether the topics
that are covered through such alternative means could be eliminated from
plans for future TPSS modules.
VBA should develop formal plans for evaluating the impact of TPSS on
claims- processing accuracy and consistency. For example, VBA could use
individual performance data obtained from SIPA to provide indications of the
impact of TPSS on accuracy. VBA could also conduct a baseline assessment of
consistency, such as testing control cases in regional offices, and then
conduct periodic assessments to provide indications of whether TPSS is
helping to improve consistency. The information regarding the impact of TPSS
should be included in the annual reports submitted under the Government
Performance and Results Act of 1993.
We received written comments on a draft of this report from VA (see app.
II). In its comments, VA concurred with both of our recommendations. With
regard to our first recommendation, VA stated that VBA will continue
exploring alternative means for providing standardized training and will
evaluate whether training on certain topics is better suited to TPSS or
alternative means such as the VBA Satellite Network. With regard to our
second recommendation, VA stated that when the SIPA program is implemented
in late 2001 and 2002, VBA intends to develop formal plans to use it as a
tool in evaluating the impact of TPSS on accuracy and consistency. We
support these efforts.
In its comments, VA also stated that while some variance exists to the TPSS
schedule produced in fiscal year 1998, VBA has given assurance that it will
meet the original final delivery date in fiscal year 2004. However, in its
fiscal year 2001 budget submission, VBA showed that the TPSS program would
be completed in calendar year 2002. Accordingly, we are reporting that the
TPSS program is behind schedule. Recommendations for
Executive Action Agency Comments and Our Evaluation
Page 15 GAO- 01- 601 Veterans' Benefits
As agreed with your office, unless you publicly announce it contents
earlier, we plan no further distribution of this report until 30 days after
its issue date. At that time, we will send copies of this report to the
Honorable Christopher H. Smith, Chairman of the House Committee on Veterans?
Affairs; the Honorable Arlen Specter, Chairman of the Senate Committee on
Veterans? Affairs; the Honorable John D. Rockefeller IV, Ranking Minority
Member of the Senate Committee on Veterans? Affairs; other appropriate
congressional committees; the Honorable Anthony J. Principi, Secretary of
the Department of Veterans Affairs; and other interested parties. We will
also make copies available to others upon request.
If you have questions about this report, please call me on (202) 512- 7101
or Irene Chu on (202) 512- 7102. Other key contributors were Ira Spears,
Steve Morris, Paul Wright, and Patrick di Battista.
Sincerely yours, Cynthia A. Bascetta Director, Education, Workforce,
and Income Security Issues
Appendix I: Scope and Methodology Page 16 GAO- 01- 601 Veterans' Benefits
To determine regional offices? experiences with and views on the Training
and Performance Support System (TPSS) program, as well as other types of
training, we sent a questionnaire in October 2000 to each of the Veterans
Benefits Administration?s (VBA) 57 regional offices. 1 All 57 offices
responded to our survey. To examine the extent and manner of TPSS usage in
more detail, we visited six regional offices: Des Moines, Iowa; Roanoke,
Virginia; Pittsburgh, Pennsylvania; St. Petersburg, Florida; San Diego,
California; and Seattle, Washington. The six offices we visited were
judgmentally selected to provide a mix of offices based on a combination of
factors, including regional office workloads, Service Delivery Network
affiliation, geographic location, and variations in the regional offices?
implementation and usage of TPSS. While we did not attempt to evaluate the
effectiveness of the TPSS program, or other forms of training, we discussed
the benefits and limitations of these training approaches with officials of
the regional offices we visited and VBA headquarters.
1 Although VBA has an office in Cheyenne, Wyoming, this office reports
administratively through the Denver Regional Office, and for accuracy
measurement, the Cheyenne office is treated as part of the Denver Regional
Office. Therefore, we sent a questionnaire to the Denver Regional Office
only. Appendix I: Scope and Methodology
Appendix II: Comments From the Department of Veterans Affairs
Page 17 GAO- 01- 601 Veterans' Benefits
Appendix II: Comments From the Department of Veterans Affairs
Appendix II: Comments From the Department of Veterans Affairs
Page 18 GAO- 01- 601 Veterans' Benefits
Appendix II: Comments From the Department of Veterans Affairs
Page 19 GAO- 01- 601 Veterans' Benefits (207103)
The first copy of each GAO report is free. Additional copies of reports are
$2 each. A check or money order should be made out to the Superintendent of
Documents. VISA and MasterCard credit cards are also accepted.
Orders for 100 or more copies to be mailed to a single address are
discounted 25 percent.
Orders by mail:
U. S. General Accounting Office P. O. Box 37050 Washington, DC 20013
Orders by visiting:
Room 1100 700 4 th St., NW (corner of 4 th and G Sts. NW) Washington, DC
20013
Orders by phone:
(202) 512- 6000 fax: (202) 512- 6061 TDD (202) 512- 2537
Each day, GAO issues a list of newly available reports and testimony. To
receive facsimile copies of the daily list or any list from the past 30
days, please call (202) 512- 6000 using a touchtone phone. A recorded menu
will provide information on how to obtain these lists.
Orders by Internet
For information on how to access GAO reports on the Internet, send an email
message with ?info? in the body to:
Info@ www. gao. gov or visit GAO?s World Wide Web home page at: http:// www.
gao. gov
Contact one:
Web site: http:// www. gao. gov/ fraudnet/ fraudnet. htm
E- mail: fraudnet@ gao. gov
1- 800- 424- 5454 (automated answering system) Ordering Information
To Report Fraud, Waste, and Abuse in Federal Programs
*** End of document. ***