Veterans' Benefits Claims: Further Improvements Needed in
Claims-Processing Accuracy (Letter Report, 03/01/99, GAO/HEHS-99-35).

Pursuant to a congressional request, GAO reviewed the Department of
Veterans Affairs' (VA) regional offices' (RO) accuracy in processing
disability claims, focusing on: (1) the extent of improvements made by
the Systematic Technical Accuracy Review (STAR) system in measuring
claims-processing accuracy; (2) additional efforts needed to strengthen
the system; and (3) challenges the Veterans Benefits Administration
(VBA) faces in meeting goals for improving claims-processing accuracy.

GAO noted that: (1) the new STAR system represents an important step
forward by VBA in measuring the accuracy of compensation and pension
claims processing; (2) compared with the previous system, STAR focuses
more on RO decisions that are likely to contain processing errors, uses
a stricter method for computing accuracy rates, provides more data on
the performance of organizational levels within VBA, collects more data
on processing errors, and stores more accuracy review results in a
centralized database; (3) even so, VBA can further strengthen STAR's
ability to identify error-prone cases and claims-processing weaknesses
so that it can take corrective actions; (4) VBA needs to better pinpoint
error-prone cases and weaknesses in the development of evidence by
collecting more specific data on the types of medical characteristics
and deficiencies in medical evidence that are most prevalent in
incorrect decisions; (5) VBA can also better address vulnerabilities in
the integrity of accuracy data; (6) STAR reviewers in ROs do not have
sufficient separation of duties or adequate independence to meet
government standards for internal controls or program performance
audits; (7) these shortcomings raise concern about the integrity of STAR
accuracy data, which are a key factor in the performance measurement
system designed by VBA to meet the requirements of the Government
Performance and Results Act (GPRA) of 1993; (8) while such system
improvements are necessary, they alone are not sufficient for VBA to
meet its goal for improving accuracy; (9) using the STAR pilot test's
64-percent accuracy rate as a baseline, VBA's goal is to achieve a
93-percent accuracy rate by fiscal year 2004; (10) VBA faces management
challenges that it must address successfully in order to meet its
accuracy improvement goal; (11) to do this, VBA recognizes that: (a) its
newly implemented performance measurement system must hold program
managers accountable for performance; and (b) the training program under
development must effectively train the current RO workforce as well as
the many new employees who will have to be hired in the coming decade to
replace those who retire; and (12) it is too early to determine whether
VBA's efforts to meet these challenges will be successful.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  HEHS-99-35
     TITLE:  Veterans' Benefits Claims: Further Improvements Needed in 
             Claims-Processing Accuracy
      DATE:  03/01/99
   SUBJECT:  Internal controls
             Veterans disability compensation
             Data integrity
             Claims adjudicators
             Claims processing
             Program evaluation
             Data collection
             Performance measures
IDENTIFIER:  VA Systematic Technical Accuracy Review System
             
******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved.  Major          **
** divisions and subdivisions of the text, such as Chapters,    **
** Sections, and Appendixes, are identified by double and       **
** single lines.  The numbers on the right end of these lines   **
** indicate the position of each of the subsections in the      **
** document outline.  These numbers do NOT correspond with the  **
** page numbers of the printed product.                         **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                                            **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************


Cover
================================================================ COVER


Report to the Ranking Minority Member, Committee on Veterans'
Affairs, House of Representatives

March 1999

VETERANS' BENEFITS CLAIMS -
FURTHER IMPROVEMENTS NEEDED IN
CLAIMS-PROCESSING ACCURACY

GAO/HEHS-99-35

Veterans' Benefits Claims

(105757)


Abbreviations
=============================================================== ABBREV

  ICD - International Classification of Diseases
  RO - regional office
  SDN - service delivery network
  SSA - Social Security Administration
  STAR - Systematic Technical Accuracy Review
  VA - Department of Veterans Affairs
  VBA - Veterans Benefits Administration
  VHA - Veterans Health Administration

Letter
=============================================================== LETTER


B-281315

March 1, 1999

The Honorable Lane Evans
Ranking Minority Member
Committee on Veterans' Affairs
House of Representatives

Dear Mr.  Evans: 

In fiscal year 1997, the Department of Veterans Affairs (VA) paid
about $19 billion in compensation and pension benefits to more than 3
million veterans and their dependents and survivors.  The
compensation program pays monthly benefits to veterans with
service-connected disabilities (injuries or diseases incurred or
aggravated while on active military duty).  Veterans with
service-connected disabilities are entitled to compensation benefits
even if they are working and regardless of the amount they earn.  By
contrast, the pension program pays monthly benefits to wartime
veterans who have low incomes and are permanently and totally
disabled for reasons not connected to their service. 

VA has 58 regional offices (RO) that process veterans' compensation
and pension claims and decide whether to award benefits.  The ROs
develop evidence and adjudicate these claims under program guidance
and policy provided by VA's Veterans Benefits Administration (VBA). 
In recent years, the accuracy of RO claims processing has been the
subject of concern and attention within VA and from the Congress and
veterans' service organizations.  Although VBA had been reporting
until recently that ROs process claims accurately more than 95
percent of the time, questions arose about RO accuracy because, when
dissatisfied veterans appealed ROs' initial decisions, the Board of
Veterans' Appeals during fiscal years 1993-97 reversed about 19
percent of the appealed decisions and remanded about 47 percent back
to ROs for further development and reconsideration.  In fiscal year
1998, VBA pilot tested its new accuracy measurement system, known as
the Systematic Technical Accuracy Review (STAR) system.  Using the
STAR system, VBA found an accuracy rate of only 64 percent for RO
initial decisions, indicating that VBA needs to give more attention
to ensuring that ROs make the correct decision the first time so that
veterans need not make unnecessary appeals or be unnecessarily
delayed in receiving benefits owed them. 

VBA implemented the STAR system nationwide in October 1998.  VBA sees
STAR as an improvement in its ability to measure accuracy and
identify and correct the causes of claims-processing errors.  As
agreed with your office, this report addresses (1) the extent of
improvements made by STAR in measuring claims-processing accuracy,
(2) additional efforts needed to strengthen the system, and (3)
challenges VBA faces in meeting goals for improving claims-processing
accuracy. 

In conducting our review, we spoke with officials of and reviewed
reports and policy guidance by VBA, the Board of Veterans' Appeals,
and 15 ROs.  We also received available data from VBA, the Board, and
the ROs.  Furthermore, we contacted and reviewed documents from
several veterans' service organizations, the National Academy of
Public Administration, the Veterans' Claims Adjudication Commission,
and VA's Office of Inspector General.  We also obtained information
on the quality assurance programs of several other organizations,
including the Social Security Administration (SSA), which administers
the largest federal disability benefits program.  We conducted our
review between October 1997 and December 1998 in accordance with
generally accepted government auditing standards. 


   RESULTS IN BRIEF
------------------------------------------------------------ Letter :1

The new STAR system represents an important step forward by VBA in
measuring the accuracy of compensation and pension claims processing. 
Compared with the previous system, STAR focuses more on RO decisions
that are likely to contain processing errors, uses a stricter method
for computing accuracy rates, provides more data on the performance
of organizational levels within VBA, collects more data on processing
errors, and stores more accuracy review results in a centralized
database for review and analysis. 

Even so, VBA can further strengthen STAR's ability to identify
error-prone cases and claims-processing weaknesses so that it can
take corrective actions.  VBA needs to better pinpoint error-prone
cases and weaknesses in the development of evidence by collecting
more specific data on the types of medical characteristics and
deficiencies in medical evidence that are most prevalent in incorrect
decisions.  VBA can also better address vulnerabilities in the
integrity of accuracy data.  Currently, STAR reviewers in ROs do not
have sufficient separation of duties or adequate independence to meet
government standards for internal controls or program performance
audits.  These shortcomings raise concern about the integrity of STAR
accuracy data, which are a key factor in the performance measurement
system designed by VBA to meet the requirements of the Government
Performance and Results Act of 1993.\1

While such system improvements are necessary, they alone are not
sufficient for VBA to meet its goal for improving accuracy.  Using
the STAR pilot test's 64-percent accuracy rate as a baseline, VBA's
goal is to achieve a 93-percent accuracy rate by fiscal year 2004. 
As acknowledged by VBA, however, it faces management challenges that
it must address successfully in order to meet its accuracy
improvement goal.  To do this, VBA recognizes that (1) its newly
implemented performance measurement system must hold program managers
accountable for performance and (2) the training program under
development must effectively train the current RO workforce as well
as the many new employees who will have to be hired in the coming
decade to replace those who retire.  It is too early to determine
whether VBA's efforts to meet these challenges will be successful. 

This report makes recommendations to (1) further strengthen VBA's
ability to identify error-prone cases by collecting more detailed
data on the human body systems and specific impairments involved in
disability claims as well as data on specific deficiencies in medical
evidence and examinations, (2) implement a system for reviewing
claims-processing accuracy that meets the government's internal
control standard on separation of duties and the program performance
audit standard on organizational independence, and (3) keep the
Congress informed on VBA's progress in establishing stricter employee
accountability and developing more effective training for claims
adjudicators. 


--------------------
\1 The Results Act requires agencies to clearly define their
missions, set goals, measure performance, and make improvements. 
Agencies are required to submit annual reports on their success in
achieving program performance goals for the previous fiscal year. 
The first performance reports for fiscal year 1999 are due by March
31, 2000. 


   BACKGROUND
------------------------------------------------------------ Letter :2

VBA's Compensation and Pension Service, located at VA headquarters,
formulates the policy and guidance used by the RO staff who receive,
develop, and evaluate veterans' compensation and pension claims.  The
compensation program pays monthly benefits to veterans with
service-connected disabilities (injuries or diseases incurred or
aggravated while on active military duty).  Veterans with
service-connected disabilities are entitled to compensation benefits
even if they are working and regardless of the amount they earn.  By
contrast, the pension program pays monthly benefits to wartime
veterans who have low incomes and are permanently and totally
disabled for reasons not connected to their service.  In compensation
cases, the payment amount varies according to degree of disability;
in pension cases, the amount varies according to financial need. 
When veterans are unable to manage their affairs, benefit payments
are made to guardians who serve as their fiduciary representatives. 

Adjudicating an original disability claim involves two basic
functions--"authorization" and "disability rating." Authorization
involves obtaining records from the military services and information
from the veterans, such as medical records and information on income
and dependents.  Disability rating involves establishing whether a
veteran's impairment is service-connected and, if so, evaluating the
veteran's degree of disability.  VBA considers claims requiring a
disability rating to be the core workload of the compensation and
pension program, and as a group, cases requiring a disability rating
are considered to be the most error-prone in the program.  In order
to rate (or evaluate) a veteran's disability, ROs often determine
that they need medical evidence in addition to evidence obtained from
the veteran's physicians and other medical providers.  In such cases,
they send veterans to the Veterans Health Administration (VHA) for
physical or mental examinations by VHA physicians.\2

From the medical evidence, ROs rate a veteran's disability using VA's
Schedule for Rating Disabilities, which lists physical and mental
conditions and assigns a disability rating to each condition.  Under
this schedule, the degree of disability is expressed in 10-percent
increments up to 100-percent disability.  A veteran can also receive
a "zero-percent" disability rating, which means the veteran's
condition is service-connected but not severe enough to qualify for
compensation payments on the basis of the medical criteria specified
in the rating schedule.  If the veteran's condition later worsens, he
or she may reapply, asking VA to increase the rating from zero to 10
percent or more. 

Evaluating the degree of disability for some conditions, such as
mental impairments, can require adjudicators to make subjective
judgments that are not always clear-cut.  For veterans with multiple
impairments, the RO must rate each impairment separately and then
combine them into a composite rating.  After a veteran is placed on
the rolls, his or her condition or circumstances may change in ways
that can result in adjustments to the RO's initial decision.  For
example, a veteran may file a claim for an increase in degree of
disability if his or her medical condition deteriorates.  Or
nonmedical issues may arise that require an adjustment to the initial
decision but do not require a disability rating in order to make the
new decision.  Such cases could arise from changes in the status of
the veteran's dependents or changes in the income of a veteran
receiving pension benefits. 

After the RO notifies the veteran of its decision, the veteran, if
dissatisfied, may ask for a hearing before an RO hearing officer. 
The veteran also may file a notice of disagreement with the RO and
then file an appeal asking for a review of the RO's decision by the
Board of Veterans' Appeals, which makes VA's final decisions on
appeals on behalf of the Secretary.  The Board may conduct a hearing
if the veteran requests one.  In deciding appeals, the Board can
grant benefits (reverse the RO decision), deny benefits (affirm the
RO decision), or remand (or return) the case to the RO to develop
further evidence and reconsider the claim.  After further development
of a remanded claim, the RO either awards the claim or returns it to
the Board for a decision. 

Before 1989, the Board's decisions on appeals were final.  In that
year, however, the Court of Veterans Appeals--established by the
Veterans' Judicial Review Act of 1988 (P.L.  100-687, Nov.  18,
1988)--began to hear cases.  As a result, the Board is no longer the
final step in the claims adjudication process.  When a veteran
disagrees with a decision of the Board, the veteran may now appeal to
the Court, which is independent of VA.  Additionally, either veterans
or VA may appeal decisions of the Court of Veterans Appeals to the
Court of Appeals for the Federal Circuit. 

Since veterans began appealing Board decisions to the Court of
Veterans Appeals, according to a court official, the Court has
remanded more than 4,500 decisions back to the Board for further
development and reconsideration.  According to the same official,
this represents about 59 percent of the Board's decisions that were
appealed to the Court, excluding dismissed cases.  In turn, ROs have
felt the repercussions of these Court decisions as evidenced by
significant increases in the Board's reversals and remands of
appealed RO decisions.  Before the advent of the Court, the Board
historically had annually awarded benefits in 12 to 14 percent of
appealed RO decisions and had annually remanded another 12 to 24
percent back to ROs for further development.\3 However, in the years
since the advent of the Court, the Board has annually awarded
benefits in about 14 to 20 percent of the cases it reviewed and
remanded another 31 to 51 percent back to ROs for further
development.  Despite these increases in awards and remands by the
Board, VBA had continued to report--until STAR was implemented--that
ROs were accurately processing compensation and pension claims more
than 95 percent of the time.  (See app.  I for more details on the
reversal and remand rates of the Court and the Board and on the
accuracy rates reported by VBA.)

VBA considers a disability claim to have been accurately processed if
basic eligibility has been determined correctly, the case file
contains all required medical and nonmedical documentary evidence,
the RO's decision on service-connection and the rating given to each
impairment are correct, the payment amount is correct, and the RO
properly notified the veteran of the outcome of his or her claim. 

Under the accuracy measurement system that was in operation from
fiscal year 1992 through fiscal year 1997, VBA headquarters annually
reviewed approximately 100 cases randomly selected from the cases
completed by each of 57 ROs.\4 These cases were selected from the
entire universe of compensation and pension work products completed
by the ROs.  Using this procedure, VBA produced a national accuracy
rate with a reasonable level of statistical precision.\5 While each
year's sample was too small for VBA to produce accuracy rates for
each RO with a reasonable level of statistical precision, VBA
required each RO to self-review 300 to 900 cases annually, depending
on the size of the RO.  These RO self-reviews were to provide ROs
with information needed to improve quality, not to compute accuracy
rates for measuring performance. 

VBA's new accuracy measurement system, STAR, is part of a customer
service and benefits delivery improvement effort that involves, among
other things, the restructuring of VBA's organization and
accountability systems.  Under the restructuring, VBA has grouped the
ROs into nine service delivery networks (SDN).  An SDN does not have
a centralized office or staff.  Instead, the ROs in each SDN are
expected to closely collaborate with one another, provide mutual
support, share resources, operate according to team-based principles,
and share collective responsibility and accountability for the SDN's
overall performance of all work assigned to it.  In meeting the
requirements of the Results Act, VBA headquarters will measure each
SDN's performance, and each SDN will assess the performance of its
ROs.  This measurement will be made on the basis of five performance
factors:  claims-processing accuracy (as determined by STAR),
timeliness of claims processing, unit cost, customer satisfaction,
and employee satisfaction and development. 


--------------------
\2 VA is currently conducting a pilot test to study the effectiveness
of using private medical providers to perform these examinations. 

\3 Veterans file relatively few appeals with the Board.  In fiscal
year 1997, for example, they filed appeals in only 5.4 percent of all
RO initial decisions. 

\4 Although 58 ROs receive and process claims, the RO in Cheyenne,
Wyoming, reports administratively though the Denver RO; therefore,
cases completed by the Cheyenne RO were included in the universe of
cases from which the Denver RO sample was selected. 

\5 \5 Statistical precision refers to the amount of uncertainty in an
estimate that results from sampling variability at a given level of
confidence.  For example, if a sample that has a 95-percent
confidence level and a precision level of plus or minus 5 percentage
points yields an estimated accuracy rate of 70 percent, this means
that one can be 95-percent confident that the true accuracy rate is
between 65 percent and 75 percent. 


   VBA HAS IMPROVED ITS
   MEASUREMENT OF
   CLAIMS-PROCESSING ACCURACY
------------------------------------------------------------ Letter :3

The new STAR system represents an important step forward by VBA in
measuring the accuracy of compensation and pension claims processing
and in providing data to identify error-prone cases and correct the
causes of errors, including those that result in reversals and
remands by the Board of Veterans' Appeals.  Compared with the
previous accuracy measurement system that VBA had been using since
1992, the STAR system is a step forward because it focuses more on RO
decisions that are likely to contain claims- processing errors, uses
a stricter method for computing accuracy rates, provides more data on
the performance of additional organizational levels within VBA,
collects more data on errors, and stores the results of more accuracy
reviews in a centralized database for further review and analysis. 

Whereas VBA had been reporting more than 95-percent accuracy under
the previous accuracy measurement system, VBA, in its pilot test of
STAR, reported that only 64 percent of veterans' claims were
processed accurately.  A primary reason for this difference is that
the pilot test focused only on the most complex and more error-prone
RO work products, those involving disability rating decisions.  By
contrast, the previous system drew its sample of cases from the
entire universe of RO work products, including those not requiring an
assessment of disability and, therefore, less error-prone.  The newly
implemented STAR system continues to focus on claims that involve
disability ratings, but it also includes a sample of cases that
address issues typically not requiring disability ratings and a
separate sample of cases involving guardianship issues for veterans
unable to represent themselves.  Separate accuracy rates are computed
for each of these two other samples. 

Another reason that the STAR pilot test found an accuracy rate of 64
percent rather than 95 percent as reported under the previous system
is STAR's stricter accuracy rate computation method.  Under the
previous system, VBA categorized each error into one of three areas
of the claims adjudication process:  (1) case control and
development, (2) decision elements, and (3) notification to the
veteran.  Thus, if a case had one error, VBA would record this error
under the appropriate area and show the two other areas as
error-free.  After reviewing all cases, VBA computed separate
accuracy rates for each of the three claims adjudication areas and
then determined an overall accuracy rate by calculating the
arithmetic mean (or average) of the three accuracy rates.  Under
STAR, however, VBA does not compute separate accuracy rates for the
three areas of the claims adjudication process.  If a case has any
errors in any area of the claims adjudication process, the entire
case is counted as incorrect for accuracy rate computation purposes. 
This approach tends to result in a lower accuracy rate than under the
previous system.  (See app.  II for a hypothetical example
demonstrating how STAR's computation method can result in a lower
accuracy rate.)

In addition to focusing more on error-prone RO decisions and using a
stricter accuracy rate computation method, STAR provides accuracy
rates with reasonable statistical precision not only for the nation
as a whole but also for each SDN.\6 Under the previous system, VBA
headquarters had reviewed about 5,700 cases annually.  Its sampling
methodology allowed VBA to produce an accuracy rate with reasonable
statistical precision for the nation as a whole.  Under STAR, VBA
headquarters will review about 7,400 cases annually.  Its sampling
methodology will enable VBA to provide accuracy rates with reasonable
statistical precision for the nation and each SDN for the sample of
cases requiring disability ratings and the sample of cases typically
not requiring such ratings (see app.  II for SDN sample sizes and
statistical precision data).  However, the sample of cases involving
guardianship issues will be too small to provide the same level of
statistical precision. 

VBA originally considered designing STAR so that VBA headquarters
also could produce accuracy rates for each RO but dropped this option
because it would have required VBA headquarters to review an
additional 50,000 cases annually.  Instead, VBA opted to require each
RO to review samples of its own work products using STAR review
procedures.  As in the headquarters review, these RO self-reviews
will produce accuracy rates with reasonable statistical precision for
the sample of cases requiring disability ratings and the sample of
cases typically not requiring such ratings.  However, the sample of
cases involving guardianship issues will be too small to produce
accuracy rates with the same level of statistical precision. 
Nationwide, the ROs will review about 44,000 randomly selected cases
(see app.  II for RO sample sizes and statistical precision data). 
VBA estimates that every 1,000 cases in these samples require about
1.0 full-time equivalent review staff per year. 

STAR is also an improvement over the previous accuracy measurement
system because it provides more precise information on the
inaccuracies it identifies.  Under the previous system, VBA's
database essentially captured only whether a decision did or did not
contain errors.  By contrast, STAR requires reviewers to answer a
standardized series of questions about whether the RO's actions and
decisions were correct or incorrect in various steps of claims
processing.  The reviewers enter their answers to these questions,
along with brief narrative comments, in the STAR database.  In
addition, because the need for further development of evidence is a
primary reason that the Board of Veterans' Appeals remands many cases
to ROs, STAR asks reviewers to identify deficient evidence
categories, such as private medical evidence, VA medical center
records, and service records.  Also, because the Board remands many
cases to ROs to obtain further medical examinations by VHA
physicians, STAR asks reviewers to indicate whether deficiencies in
medical evidence supporting the decision relate to VHA medical
examinations.  These data on deficiencies in evidence are entered in
the STAR database.  The database also identifies cases involving five
special conditions that have medical implications:  prisoner of war,
radiation exposure, Gulf War veteran, Agent Orange exposure, and
posttraumatic stress syndrome. 

Additionally, STAR's database captures the results of accuracy
reviews conducted by both VBA headquarters and the ROs, whereas under
the previous system, VBA's database captured only the results of
accuracy reviews conducted by VBA headquarters.  VBA planned to
implement in February 1999 a new centralized database on its internal
network (intranet) system that will permit both VBA headquarters and
the ROs to input the results of all STAR reviews into the database. 
Capturing RO data will enrich the data available to analyze error
trends, and both VBA headquarters and the ROs will have access to the
full complement of data through the intranet. 


--------------------
\6 See app.  II for more information on statistical precision. 


   VBA CAN STRENGTHEN ITS ABILITY
   TO IDENTIFY ERROR-PRONE CASES
   AND ADDRESS VULNERABILITIES IN
   THE INTEGRITY OF ACCURACY
   REVIEWS
------------------------------------------------------------ Letter :4

Although STAR represents a significant step forward in VBA's ability
to measure claims-processing accuracy and identify error-prone cases,
VBA can take further steps to improve this ability.  These steps
involve collecting additional data for identifying and correcting
error-prone cases and addressing vulnerabilities in the integrity of
accuracy reviews. 


      VBA CAN IMPROVE THE ABILITY
      TO IDENTIFY ERROR-PRONE
      CASES
---------------------------------------------------------- Letter :4.1

Even with the improvements provided by STAR, VBA's ability to
identify error-prone cases and target corrective actions is
constrained by the limited data that it captures on the medical
characteristics of claimants whose claims are processed incorrectly
and on why medical evidence is deficient.  Data captured on
claimants' medical characteristics is currently limited to
identifying whether a veteran was a prisoner of war, served in the
Gulf War, or had posttraumatic stress syndrome, radiation exposure,
or Agent Orange exposure.  More detailed medical characteristics data
could help pinpoint the specific types of claims in which errors
occur.  Also, although STAR captures data on whether medical evidence
and medical examinations are adequate, it does not record statistical
data identifying why reviewers found the evidence or examinations
supporting RO final decisions to be deficient.  Such data also could
help pinpoint the types of corrective actions that need to be taken
to improve the accuracy of RO decisions. 

Limited studies by VBA demonstrate how capturing additional data in
the STAR database on medical issues could help VBA focus on
corrective actions that can reduce claims- processing errors and in
turn reduce remands from the Board of Veterans' Appeals.  In 1996,
VBA and the Board of Veterans' Appeals jointly conducted a limited
study of remanded cases and reported that inadequate medical
examinations were the most frequent reason for remands and that a
majority of the remanded cases involved the need for specialty
examinations, such as orthopedic, psychiatric, neurologic,
audiologic, and ear-nose-throat examinations.  Also, in 1996, the
Milwaukee RO reviewed claims that were awarded by the RO's hearing
officers after the claims were initially denied.  Of the cases in
which the RO's hearing officers reversed the initial decision, the
Milwaukee RO captured data on the specific conditions, such as
orthopedic impairments, that were involved in significant numbers of
cases, and using such data, the RO identified specific corrective
actions.  According to Milwaukee RO officials, this helped reduce the
RO's remand rate from the Board of Veterans' Appeals.  From fiscal
year 1995 to fiscal year 1998, the Milwaukee RO reduced its remand
rate from about 40 percent to about 21 percent, one of the lowest
remand rates in the nation.\7

SSA, which administers the largest federal disability program, has a
quality assurance system that captures detailed data on claimants'
medical characteristics and on weaknesses in evidence.  SSA has found
that such data are helpful in identifying error-prone cases and
targeting corrective actions.  For each case reviewed, SSA's system
captures data on the specific body systems involved, such as
musculoskeletal, respiratory, cardiovascular, and mental systems. 
Further, using codes from the International Classification of
Diseases, SSA's system identifies each claimant's specific
impairments.  Additionally, when medical evidence is judged not
adequate, SSA's system records the specific medical specialty area in
which evidence was lacking, such as orthopedic, psychiatric, and
neurologic areas, and it identifies the specific type of test, study,
or other medical evidence that was lacking. 

Such data, according to an SSA quality assurance official, not only
helps to identify error-prone cases but can pinpoint specific
evidentiary weaknesses for cases involving specific body systems or
impairments.  Also, this official stated that spending resources up
front to capture such data can reduce the need to conduct
time-consuming special studies later to understand why certain types
of cases are being processed incorrectly.  According to the SSA
quality assurance unit, the depth of the data collected from quality
assurance reviews also enables it to assess the implementation of new
or revised policies, perform analyses and make recommendations for
operational and systems corrective actions, and provide broad levels
of management information, such as information by categories of
impairments. 

VBA agrees that the STAR system deployed at the beginning of fiscal
year 1999 provides a sound start for beginning to address
claims-processing accuracy issues.  VBA officials acknowledge,
however, that they realized when STAR was deployed that continuous
improvement should be sought to enhance its effectiveness.  These VBA
officials stated that VBA is open to considering the collection of
additional data in order to enhance STAR. 


--------------------
\7 Remand rates for the 58 ROs ranged from about 19 percent to 59
percent in fiscal year 1998. 


      VBA CAN FURTHER ADDRESS
      VULNERABILITIES IN THE
      INTEGRITY OF PERFORMANCE
      DATA
---------------------------------------------------------- Letter :4.2

To ensure integrity in the operation of government programs,
standards for internal controls call for separation of key duties,
and standards for performance audits call for those who review and
evaluate a program's performance to be organizationally independent
of the program's managers.\8 Under STAR, however, the RO staff who
review the accuracy of RO decisions are themselves responsible for
making such decisions, and they report to RO managers responsible for
claims processing.  Such an arrangement does not meet the standard
for separation of duties, nor does it meet the independence standard. 
Both the RO reviewers and their managers have an inherent
self-interest in having as high an accuracy rate as possible.  This
self-interest derives from the fact that accuracy is one of five
factors that determine RO performance scores, which VBA measures to
meet Results Act requirements.  Thus, without adequate separation of
duties or adequate independence for RO reviewers, the integrity of
both the STAR review process and the resulting accuracy rates and
performance data reported under the Results Act are called into
question. 

The potential effect of impaired objectivity on performance data is
exemplified by findings reported by VA's Inspector General in
September 1998.\9 Because of concern about the accuracy of data used
to meet Results Act requirements, the Inspector General examined the
integrity of certain data used for Results Act reports.  In this
review, the Inspector General found instances in which RO staff had
manipulated data on the timeliness of RO claims processing in order
to make performance appear to be better than it actually was.  The
Inspector General found that weaknesses in internal controls had
contributed to the lack of integrity in the timeliness data reported
under the Results Act.  During our review, some RO staff made
comments on the integrity of accuracy reviews that parallel the
findings of the Inspector General.  These RO staff told us that ROs
are biased against identifying their own errors.  They also stated
that ROs in the past, after selecting samples of cases to review, had
sometimes "sanitized" or fixed problems in the case files before the
cases underwent quality review. 

No data are available to indicate the extent to which RO reviewers
might attempt to overlook errors and sanitize case files to conceal
errors in the approximately 44,000 cases that ROs review annually
under STAR.  However, to the extent that such efforts may occur, the
accuracy rates reported by the ROs would be overstated.  Furthermore,
any attempts by ROs to conceal errors and overstate their accuracy
rates could also result in an overstatement of the accuracy rates
that VBA reports for SDNs and the nation.  This vulnerability in
VBA's data exists because the sample of 7,400 cases that VBA reviews
annually is selected directly from the approximately 44,000 cases
reviewed by the ROs.  VBA reviews its sample of 7,400 cases after the
ROs complete their own reviews of these same cases.  VBA believes
that it can detect most attempts to sanitize case files because such
attempts would likely require extensive backdating of corrected case
file documents, which VBA believes would be difficult to conceal. 
VBA acknowledges, however, that it cannot ensure that it would detect
every such attempt in the cases it reviews.  To the extent that VBA
may not detect all such attempts, the accuracy rates it reports for
SDNs and the nation would be overstated. 

Ensuring the integrity of accuracy data will require that staff who
review claims- processing accuracy neither are responsible for claims
processing nor report to program managers responsible for claims
processing.  VBA stated that resource restrictions prevent
establishing independent accuracy review units either in the ROs or
at VA headquarters; however, unless VBA provides adequate separation
of duties and organizational independence for accuracy reviewers,
potential questions about the integrity of accuracy-related
performance data will likely persist.  By contrast, we found that SSA
has quality assurance units at its headquarters and in each of its 10
regional offices that are organizationally independent of program
management.  The independent quality assurance unit in SSA
headquarters has overall responsibility for assessing disability
claims-processing accuracy.  To do this, it oversees the operation of
the independent regional quality assurance units that review the
accuracy of statistically random samples of the disability decisions
rendered by 54 state agencies that process disability claims for SSA. 

VBA contends that it would be impractical to establish independent
accuracy review units in VBA's 58 ROs, many of which are relatively
small in size.  Establishing independent STAR units in ROs would be
more practical if only a relatively small number of large ROs
processed all compensation and pension claims.  Under the present
structure, however, a more workable long-term solution could involve
establishing an independent headquarters unit responsible for
conducting all reviews used to determine the accuracy rates that go
into the calculation of overall performance scores for VBA
headquarters, SDNs, and ROs. 


--------------------
\8 See U.S.  General Accounting Office, Standards for Internal
Controls in the Federal Government (Washington, D.C.:  1983), and
Government Auditing Standards (Washington, D.C.:  June 1994). 

\9 See Department of Veterans Affairs, Office of Inspector General,
Audit of Data Integrity for Veterans Claims Processing Performance
Measures Used for Reports Required by the Government Performance and
Results Act, Report No.  8R5-B01-147 (Washington, D.C.:  Sept.  22,
1998). 


   VBA FACES CHALLENGES IN MEETING
   ITS GOAL FOR IMPROVING
   CLAIMS-PROCESSING ACCURACY
------------------------------------------------------------ Letter :5

VBA has set a goal of achieving a claims-processing accuracy rate of
93 percent by fiscal year 2004.  This would be almost 30 percentage
points higher than the baseline rate of 64 percent established in the
1998 pilot test of STAR.  VBA acknowledges, however, that the STAR
system on its own cannot ensure that VBA will meet its accuracy goal. 
Beyond any improvements that VBA might make in the STAR system, VBA
acknowledges that there are challenges it must address successfully
in order to meet its goal for improving accuracy.  These challenges
include effectively establishing accountability for accuracy
improvement and developing an effective training program for the
current and future workforce. 


      ESTABLISHING STRICTER
      ACCOUNTABILITY
---------------------------------------------------------- Letter :5.1

In May 1998, VBA identified several root causes of quality problems
in processing disability compensation and pension claims.\10 One such
cause was a lack of employee accountability.  VBA plans to focus on
quality and accountability with a quality assurance system that
provides clear and fair accountability at all organizational levels. 
To accomplish this goal, VBA is implementing the "balanced scorecard"
that scores the performance of VBA headquarters, SDNs, and ROs on the
basis of five performance factors:  claims-processing accuracy (as
determined by STAR), timeliness of claims processing, unit cost,
customer satisfaction, and employee satisfaction and development. 

With the goal of achieving a 93-percent accuracy rate by fiscal year
2004, VBA believes its balanced scorecard approach will, among other
things, drive organizational change, provide employees with feedback
on measures they can influence, and link the performance appraisal
and reward systems to organizational performance measures.  VBA plans
to use the balanced scorecard to give RO managers incentives to work
as teams in their SDNs with a focus on meeting balanced scorecard
performance measures, including accuracy.  The extent to which this
strategy will improve accountability and accuracy cannot yet be
determined. 


--------------------
\10 See VBA, Roadmap to Excellence--Planning the Journey (Washington,
D.C.:  May 1998). 


      DEVELOPING MORE EFFECTIVE
      TRAINING
---------------------------------------------------------- Letter :5.2

In our discussions with RO staff, many stated that VBA had not
provided adequate training for claims adjudicators.  They stated, for
example, that there was confusion in the ROs on how to process cases
because of apparent conflicts between decisions of the Court of
Veterans Appeals and VA's regulations and guidance.  They also stated
that too much of their training was determined locally, resulting in
inconsistent training among the ROs.  VBA acknowledged shortcomings
in training and stated that it had not fared well in preparing its
workforce, with a resultant decline in technical accuracy.  VBA
acknowledged the need for an effective, centralized, and
comprehensive training program that provides the background necessary
for its decisionmakers to render decisions according to the statutes
and regulations mandated for claims adjudication. 

Such training is important not only for current employees but also
for the many new employees whom VBA will have to hire to replace
retiring employees.  According to VBA, it may lose up to 30 percent
of its workforce to retirement by fiscal year 2003.  To develop a
training program for RO staff, VBA plans to identify the necessary
employee skills and work processes for every decisionmaking position,
implement skill certification or credentialing for these positions,
and implement performance-based training connected to measurable
outcomes.  VBA has already developed a computer-based training module
for processing appeals and is working on modules for original
disability claims, service-connected death indemnity benefits, and
pensions.  VBA also plans to produce additional modules, including
one for training RO staff when they first assume disability rating
responsibilities.  Whether these training efforts will enable VBA to
meet its accuracy goal cannot yet be determined. 


   CONCLUSIONS AND RECOMMENDATIONS
------------------------------------------------------------ Letter :6

Although VBA had been reporting until recently that ROs were
processing claims accurately more than 95 percent of the time, the
STAR pilot test in fiscal year 1998 revealed that the accuracy rate
for decisions involving disability ratings was much lower, about 64
percent.\11 This confirmed that VBA needs to give more attention to
ensuring that ROs make the correct decision the first time.  Making
the correct decision the first time would mean that veterans could
avoid having to make unnecessary appeals and would not be
unnecessarily delayed in receiving benefits owed them. 

Although the new STAR system represents genuine improvement in VBA's
ability to measure accuracy and identify error-prone cases, VBA needs
to make further progress in collecting data for identifying difficult
cases, assessing adjudication difficulties, and developing corrective
actions.  Despite its newly implemented STAR system, without further
refinements in the data collected on errors, significant inaccuracies
are likely to persist because VBA is constrained in its ability to
pinpoint error-prone cases and identify corrective actions. 
Moreover, the data produced from STAR reviews will be suspect because
of weaknesses in internal controls and lack of adherence to
performance audit standards.  We believe this can potentially
undermine progress made under STAR. 

To further strengthen VBA's ability to identify error-prone cases,
ensure the integrity of accuracy rate-related performance data
reported under the Results Act, and keep the Congress informed about
VBA's progress in addressing challenges that must be met in order to
improve accuracy, we recommend that the Secretary of the Department
of Veterans Affairs direct the Under Secretary for Benefits to take
the following actions. 

  -- For RO disability decisions found to be in error, revise STAR to
     collect more detailed medical characteristics data, such as the
     human body systems, the specific impairments, and the specific
     deficiencies in medical evidence involved in these disability
     claims, so that VA can identify and focus corrective actions on
     specific problems that RO adjudicators have in correctly
     evaluating certain types of medical conditions or in correctly
     determining whether medical evidence is adequate to make a
     decision. 

  -- Implement a claims-processing accuracy review function that
     meets the government's internal control standard on separation
     of duties and the program performance audit standard on
     organizational independence. 

  -- In the annual Results Act reports, inform the Congress on VBA's
     progress in (1) establishing stricter employee accountability
     for the achievement of performance goals and (2) developing more
     effective training for claims adjudicators. 


--------------------
\11 As mentioned, the lower accuracy rate under STAR is partially
attributable to the fact that STAR computes accuracy rates more
strictly than the previous system, thereby tending to produce lower
accuracy rates (see app.  II). 


   AGENCY COMMENTS AND OUR
   EVALUATION
------------------------------------------------------------ Letter :7

In commenting on our draft report, VA stated that it found the report
to be a fair and balanced appraisal.  VA concurred that its process
for assessing claims accuracy is critical and stated that continued
urgent action is required for VA to meet its own and its
stakeholders' expectations.  VA stated that our recommendations were
generally constructive but had concern about our first two
recommendations. 

The first recommendation in our draft report was that VA "revise STAR
to include the collection of more detailed medical characteristics
data on the human body systems, and specific impairments involved in
disability claims as well as data on specific deficiencies in medical
evidence and examinations." VA interpreted our recommendation to mean
that STAR should collect data on the quality of examinations
conducted by VHA.  However, this was not the intent of our
recommendation.  The intent was for STAR to collect additional data
that would help VA better identify (1) specific types of medical
conditions that RO adjudicators have difficulty evaluating correctly
and (2) specific types of inadequacies in medical evidence that are
most prevalent in incorrect decisions.  This would provide a means
for VA to develop corrective actions addressing the causes of errors
in the evaluation of medical conditions and of failure to collect
adequate medical evidence to make a supportable decision.  We
clarified the recommendation and our discussion of this issue in our
report. 

The second recommendation in our draft report was that VA "implement
a claim processing accuracy review function that meets the
government's internal control standard on separation of duties and
the program performance audit standard on organizational
independence." VA's primary concern about this recommendation was
that current budget constraints make it impractical to adopt
approaches that would fully satisfy these standards--for example,
establishing a single, large centralized review unit to assess all
quality issues, including individual RO quality.  However, while
current budget constraints may present problems in finding ways to
fully meet the standards immediately, we believe meeting these
standards as expeditiously as possible should be a continuing
priority in VA's future planning process.  Until the standards are
met, the integrity of VA's claims-processing accuracy data will
remain questionable.  As VA stated in its comments, "Effective
reviews require an organizational commitment to dedicate the
necessary resources to the review process."

With regard to the second recommendation, VA also stated that while
the STAR system is a compromise reflecting resource constraints, it
has some distinct advantages compared with quality reviews performed
by a consolidated, independent review unit.  VA cited the value of
having reviews performed by local staff in each RO.  Our
recommendation would not preclude local reviews, which we agree are
important.  Even if a single, central unit were established for the
purpose of assessing the degree to which each RO processes claims
accurately, it would still be critical for local RO management to
gather detailed local data on claims processing to understand fully
how to correct local processing problems.  This function, however, is
different from local reviewers conducting accuracy reviews of their
own RO's decisions, which our recommendation is intended to
eliminate. 

VA also stated that it is concerned that a "permanent" independent
review staff would become stagnant.  We disagree because the staff
who perform reviews would not have to be permanently assigned to the
unit but could instead be rotated to avert staff stagnation.  VA
furthermore expressed concern about the cost and increased potential
for losing active case files that would result from mailing many more
thousands of case files from the 58 ROs to a central review site. 
This concern, however, does not negate the need to meet the standards
for separation of duties and organizational independence.  Also, the
concern could potentially be lessened by other measures.  For
example, the Congressional Commission on Servicemembers and Veterans
Transition Assistance in its January 1999 report applauded VBA for
consolidating the administration of its education and loan programs
into fewer than 10 ROs but pointed out that VBA has made no effort to
make a similar consolidation of the adjudication of compensation
claims.\12 If VBA were ever to consolidate the adjudication of claims
into a few relatively large ROs, it would be more practical to locate
an independent STAR unit in each of these ROs to review the accuracy
of cases each one processed.  Each RO STAR unit would then need to
mail to a central review unit only a relatively small random sample
of the cases it reviewed so that the central unit could ensure the
reviews' appropriateness and consistency. 

VA's comments are printed in appendix III. 


As agreed with your office, we plan no further distribution of this
report until 7 days from its date of issue, unless you publicly
announce its contents earlier.  We will then send copies to the
Chairman of the House Committee on Veterans' Affairs, the Secretary
of the Department of Veterans Affairs, other congressional
committees, and others who are interested.  We will also make copies
available to others upon request.  If you have any questions about
this report, please call me at (202) 512-7101 or Irene P.  Chu,
Assistant Director, at (202) 512-7102.  Other major contributors to
this report were Ira B.  Spears, Mark Trapani, Connie D.  Wilson,
Paul C.  Wright, and Deborah L.  Edwards. 

Sincerely yours,

Stephen P.  Backhus
Director, Veterans' Affairs and
 Military Health Care Issues


--------------------
\12 The Commission was established by title VII of the Veterans'
Benefits Improvement Act of 1996 (P.L.  104-275) to examine a broad
range of federal programs that provide transition assistance and
benefits to service members when they leave military service and to
veterans. 


HISTORICAL DATA ON VBA'S
CLAIMS-PROCESSING ACCURACY RATES
AND ON AWARD AND REMAND RATES OF
THE BOARD OF VETERANS' APPEALS
=========================================================== Appendix I

Before the Veterans Benefits Administration (VBA) implemented the
Systematic Technical Accuracy Review (STAR) measurement system, it
reported that regional offices (RO) accurately processed and
adjudicated disability compensation and disability pension claims
more than 95 percent of the time during fiscal years 1993-97 (see
table I.1). 



                         Table I.1
          
               RO National Accuracy Rates for
            Disability Compensation and Pension
                Claims, Fiscal Years 1993-97

                Proper
               control
                   and                  Proper
            developmen  Correctnes  notificati  Overall RO
Fiscal            t of     s of RO       on to    accuracy
year            claims   decisions    veterans        rate
----------  ----------  ----------  ----------  ----------
1993             97.5%       94.5%       93.4%       95.2%
1994              97.7        95.6        95.2        96.2
1995              97.2        95.4        95.9        96.2
1996              97.5        96.5        96.8        96.9
1997              98.0        96.6        96.9        97.2
----------------------------------------------------------
Source:  VBA. 

The validity of such high accuracy rates, however, seemed
inconsistent with the results of decisions made by the Board of
Veterans' Appeals when veterans appealed unfavorable RO decisions. 
The Board of Veterans' Appeals awarded benefits or remanded cases for
further development more than 60 percent of the time when veterans
appealed RO decisions during fiscal years 1993-97 (see table I.2). 



                                        Table I.2
                         
                         Decisions by Board of Veterans' Appeals
                            Resulting in Awards or Remands of
                           Appealed Disability Compensation and
                           Pension Cases, Fiscal Years 1993-97

                               Total awards by Board           Total remands by Board
                           ------------------------------  ------------------------------
                    Total
               disability                      Percent of                      Percent of
Fiscal     decisions made                           total                           total
year             by Board          Number       decisions          Number       decisions
---------  --------------  --------------  --------------  --------------  --------------
1993               22,924           4,026            17.6          10,350            45.1
1994               19,343           3,474            18.0           9,583            49.5
1995               24,834           4,921            19.8          12,073            48.6
1996               29,818           6,137            20.6          13,357            44.8
1997               37,936           6,627            17.5          17,783            46.9
-----------------------------------------------------------------------------------------
Source:  Annual reports of the Chairman of the Board of Veterans'
Appeals, fiscal years 1993-97. 

Only a small proportion of RO decisions are appealed to the Board. 
For example, in fiscal year 1997, veterans filed notices of
disagreement in about 14 percent of the disability compensation
claims processed by ROs (see table I.3).  The number of cases
appealed, however, is less than the number of cases in which veterans
file a notice of disagreement with VA.  In some cases, after notices
of disagreement are filed, ROs award the benefits sought, or some
veterans decide not to continue with their appeals if the RO again
denies benefits at this point.\13 In fiscal year 1997, the Board
received initial substantive appeals equivalent to about 5 percent of
all disability compensation claims processed by ROs. 



                               Table I.3
                
                Disability Compensation Claim Decisions
                 Appealed by Veterans, Fiscal Year 1997

                                                        Number  Percen
                                                            of    t of
Type of action                                           cases  claims
------------------------------------------------------  ------  ------
Disability compensation claims processed by ROs         486,42   100.0
 (original and reopened)                                     5
Notices of disagreement filed with ROs                  66,566    13.7
Initial substantive appeals filed requesting Board of   26,033     5.4
 Veterans' Appeals review
----------------------------------------------------------------------
Source:  VBA. 


--------------------
\13 VBA does not maintain data on the number of these cases for which
benefits are awarded. 


SAMPLE SIZES AND ACCURACY RATE
COMPUTATION METHODS FOR
COMPENSATION AND PENSION CASES
========================================================== Appendix II


   SAMPLE SIZES
-------------------------------------------------------- Appendix II:1

Under the pre-STAR accuracy measurement system, VBA annually reviewed
approximately 5,700 compensation and pension cases, or approximately
100 cases randomly selected from the cases completed by each of 57
ROs.\14 These cases were selected from the entire universe of
compensation and pension work products completed by the ROs.  Using
this procedure, VBA annually produced, with a reasonable level of
statistical precision, a national accuracy rate for the entire body
of compensation and pension work done by the ROs during the prior
year.  The sample of approximately 100 cases selected for each RO was
too small to produce accuracy rates for each RO with a reasonable
level of statistical precision.  However, VBA required each RO to
self-review a sample of 300 to 900 cases annually, depending on the
size of the RO.  These RO self-reviews were intended to provide the
RO with information needed to improve quality, not to compute
accuracy rates for VBA to measure performance. 

Under STAR, VBA annually reviews 7,371 compensation and pension cases
for the nine service delivery networks (SDN), and the 57 ROs
self-review about 44,000 cases.  These cases are made up of three
separate samples:  (1) rating-related work products; (2)
authorization work products that require significant development,
review, and administrative decision or award action but may not
involve any rating-related action; and (3) principal guardianship
files, referred to as fiduciary cases.  (See table II.1 for SDN and
RO sample sizes.) For rating-related work products and authorization
work products that typically do not require rating-related action,
the sampling methodology will allow VBA to produce accuracy rates
with a reasonable level of statistical precision for the nation and
each SDN.  However, the sample of fiduciary cases is too small to
provide accuracy rates with the same level of statistical precision. 
Similarly, for cases that are self-reviewed by ROs, the sampling
methodology will allow each RO to produce accuracy rates with a
reasonable level of statistical precision for rating-related work
products and authorization work products typically not requiring
ratings.  Again, however, the sample of fiduciary cases is too small
to provide accuracy rates with the same level of statistical
precision. 



                                        Table II.1
                         
                           Annual Sample Sizes Under VBA's STAR
                               Accuracy Measurement System

                        Samples reviewed by VBA         Samples self-reviewed by each of
                    headquarters for each of 9 SDNs                  57 ROs
                   ----------------------------------  ----------------------------------
Types of cases         Sample  Statistical                 Sample  Statistical
reviewed                 size  precision\a                 size\b  precision\a
-----------------  ----------  ----------------------  ----------  ----------------------
Total for each\c          819  Not applicable             645-855  Not applicable

Cases requiring a         354  +/-5 percentage points     300-352  +/-5 percentage points
rating

Cases typically           325  +/-5 percentage points     285-323  +/-5 percentage points
not requiring a
rating

Fiduciary cases           140  +/-8 percentage points      60-180  +/-7 to +/-13
                                                                   percentage points\d

National total\e        7,371  Not applicable              44,175  Not applicable

Cases requiring a       3,186  +/-2 percentage points      19,388  Not applicable
rating

Cases typically         2,925  +/-2 percentage points      17,947  Not applicable
not requiring a
rating

Fiduciary cases         1,260  +/-3 percentage points       6,840  Not applicable
-----------------------------------------------------------------------------------------
\a Statistical precision refers to the amount of uncertainty in an
estimate that results from sampling variability at a given level of
confidence.  These precision levels were calculated at the 95-percent
confidence level.  For example, an estimated accuracy rate of 70
percent at a precision level of plus or minus 5 percentage points
means that one is 95-percent confident that the true accuracy rate is
between 65 percent and 75 percent. 

\b The range in sample sizes stems from the varying size of caseloads
among ROs.  The ROs with the smallest caseloads, for example, have
the smallest sample size to review. 

\c The totals in this row represent total sample size for each SDN
and each RO. 

\d Precision is dependent on sample size.  Sampling errors range from
plus or minus 7 percentage points for the sample of 180 cases to plus
or minus 13 percentage points for the sample of 60. 

\e The totals in this row represent national total sample size for 9
SDNs and 57 ROs. 

Source:  VBA. 


--------------------
\14 Although 58 ROs receive and process claims, the RO in Cheyenne,
Wyoming, reports administratively through the Denver RO; therefore,
cases completed by the Cheyenne RO are included in the universe of
cases from which the Denver RO sample is selected. 


   ACCURACY RATE COMPUTATION
   METHODS
-------------------------------------------------------- Appendix II:2

For each case reviewed under the previous accuracy measurement
system, VBA categorized each error into one of three areas of the
claims adjudication process:  (1) control and development of the
claim, (2) decision elements, and (3) notification to the veteran. 
Thus, for example, if a case had only one error, VBA would record
this error under the appropriate area of the claims adjudication
process and would show the two other areas as error-free for that
case.  After all cases were reviewed, VBA would compute an accuracy
rate for each of the three areas in the claims adjudication process. 
To arrive at an overall accuracy rate for the three areas combined,
VBA computed their arithmetic mean (or average).  For example, table
II.2 shows a hypothetical outcome for accuracy reviews of 10 cases. 
Under the control and development area, one case has an error (a
90-percent accuracy rate); under the decision element area, two cases
have errors (an 80-percent accuracy rate); and under the notification
area, one case has an error (a 90-percent accuracy rate).  For this
sample of 10 cases as a whole, the overall accuracy rate is the
average of these three accuracy rates, or 86.6 percent. 



                         Table II.2
          
            Hypothetical Computation of Accuracy
             Rates Under the Pre-STAR Accuracy
                     Measurement System

                      Errors found in accuracy review
                  ----------------------------------------
                  Area 1:
                  Control and   Area 2:       Area 3:
Hypothetical      development   Decision      Notification
case              of claim      elements      to veteran
----------------  ------------  ------------  ------------
1                 X             X

2

3

4

5

6                               X

7

8

9

10                                            X

Total cases       1             2             1
with errors

Accuracy rate     9/10 = 90%    8/10 = 80%    9/10 = 90%
for sample of 10
cases\a
----------------------------------------------------------
\a Overall average accuracy rate:  (90% + 80% + 90%) / 3 = 86.6%. 

For each case reviewed under STAR, however, VBA does not compute
separate accuracy rates for the three areas of the claims
adjudication process.  If a case has any errors in any area of the
claims adjudication process, the entire case is counted as incorrect
for accuracy rate computation purposes.  This approach tends to
result in a lower accuracy rate than under the previous system.  For
example, in the hypothetical sample of 10 cases shown in table II.2,
3 cases would be counted as incorrect under STAR because they contain
at least one processing error, and the resultant accuracy rate for
the sample would be only 70 percent (7 out of 10 cases with no errors
= 70-percent accuracy rate), compared with the overall accuracy rate
of 86.6 percent calculated under the previous system. 



(See figure in printed edition.)Appendix III COMMENTS FROM THE
DEPARTMENT OF VETERANS AFFAIRS



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)


*** End of document. ***