Veterans' Employment and Training Service: Proposed Performance  
Measurement System Improved, But Further Changes Needed 	 
(15-MAY-01, GAO-01-580).					 
								 
This report reviews the Department of Labor's Veterans' 	 
Employment and Training Service (VETS) performance measurement	 
system. Specifically, this report reviews (1) VETS' proposed	 
performance measures, including possible areas of concern	 
regarding the measures, (2) the data source VETS proposes to use 
in the new system, and (3) other measurement issues that would	 
effect the comparability of states' performance data. GAO found  
that VETS' proposed performance measures would improve		 
performance accountability over the current system, but certain  
aspects of the new measures raise concerns that VETS should	 
address. VETS' strategic plan suggests that states focus their	 
efforts on providing staff-assisted services to veterans,	 
including case management. Yet, none of the proposed measures	 
specifically gauge the success of these services. In addition,	 
VETS' proposal includes one measure--the number of federal	 
contractor jobs listed with local employment offices--that is not
only process-oriented but also focuses on outcomes that are	 
beyond the control of staff serving veterans. VETS proposes that 
all states use a single data source--Unemployment Insurance wage 
records--to identify veterans who get jobs. Using these data will
greatly improve the comparability and reliability of the new	 
measures. While using these data will improve some aspects of	 
data collection, the data present some challenges. States	 
generally do not have access to wage records from other states	 
and, therefore, should find ways to track individuals who receive
services in one state but get a job in another. Other issues that
affect the comparability of states' performance-related data	 
should be considered. For example, states vary in whether they	 
register and count, for performance reporting purposes, job	 
seekers who use only self-service tools, such as internet-based  
job listings.							 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-01-580 					        
    ACCNO:   A01001						        
  TITLE:     Veterans' Employment and Training Service: Proposed      
             Performance Measurement System Improved, But Further Changes     
             Needed                                                           
     DATE:   05/15/2001 
  SUBJECT:   Veterans employment programs			 
	     Performance measures				 
	     Strategic planning 				 
	     Reporting requirements				 
	     Data collection					 
	     Federal/state relations				 
	     Program evaluation 				 
	     VA Disabled Veteran Outreach Program		 
	     VA Local Veterans Employment Reps			 
	     Program						 
								 
	     Unemployment Insurance Program			 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Testimony.                                               **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-01-580
     
VETERANS' EMPLOYMENT AND TRAINING SERVICE

Proposed Performance Measurement System Improved, But Further Changes Needed

Report to the Chairman, Committee on Veterans' Affairs, House of
Representatives

United States General Accounting Office

GAO

May 2001 GAO- 01- 580

Page i GAO- 01- 580 VETS' Performance Measures and Data Letter 1

Appendix I Comparison of VETS, ES, and WIA Performance Measures 22

Appendix II States? Entered- Employment Rates for Veterans in Program Year
1999 24

Appendix III Comments From the Department of Labor 26

Tables

Table 1: VETS? Current Performance Measures and Standards 6 Table 2: VETS?
Current Performance Measures Compared With

Proposed Measures 10 Table 3: A Comparison of Entered- Employment Rates by

Registration Policy 17 Abbreviations DVOP Disabled Veterans? Outreach
Program DVOPS Disabled Veterans? Outreach Program Specialists ES Employment
Services ETA Employment and Training Administration GPRA Government
Performance and Results Act of 1993 LVER Local Veterans? Employment
Representatives UI Unemployment Insurance VETS Veterans? Employment and
Training Service WIA Workforce Investment Act WRIS Wage Record Interchange
System Contents

Page 1 GAO- 01- 580 VETS' Performance Measures and Data

May 15, 2001 The Honorable Christopher H. Smith Chairman, Committee on
Veterans? Affairs House of Representatives

Dear Mr. Chairman: The Department of Labor?s Veterans? Employment and
Training Service (VETS) administers programs and activities designed to help
veterans obtain employment and training assistance. Recently, policymakers
have focused increasing attention on VETS and its programs, advocating
changes to the structure and administration of the program and in the way it
assesses program performance. For example, in 1999, the Congressional
Commission on Servicemembers and Veterans Transition Assistance recommended
that the Congress establish effective outcome measures for VETS. In
addition, legislation was introduced during the 106th Congress that would
restructure the program and require a new, comprehensive performance
accountability system. 1 This focus on reform comes at a time when other
federally funded employment and training programs are changing the way they
provide services and measure performance. For example, the Workforce
Investment Act (WIA), which provides employment and training assistance for
youth, adults, and dislocated workers through one- stop centers, recently
established outcome measures that are similar to those that VETS now
proposes. With these new measures, VETS has the opportunity to bring its
performance management system in line with those of other key employment and
training programs.

Because of the Committee?s interest in improving the way employment services
are provided to veterans, you asked us to review VETS? performance
measurement system. Specifically, you asked us to review (1) VETS? proposed
performance measures, including possible areas of concern regarding the
measures; (2) the data source VETS proposes to use in the new system; and
(3) any other measurement issues that may affect the comparability of
states? performance data. Our review is based on

1 H. R. 4765, 106th Cong. (2000). This bill also provides for incentive
grants based on state performance in carrying out veterans? employment,
training, and placement services.

United States General Accounting Office Washington, DC 20548

Page 2 GAO- 01- 580 VETS' Performance Measures and Data

interviews and discussions we had with over 45 officials in 15 states, 2
interviews with VETS officials, and a review of relevant documents,
including VETS? 2000- 2005 draft strategic plan as of March 2001. We
conducted our review from October 2000 to April 2001 in accordance with
generally accepted government auditing standards.

VETS? proposed performance measures improve performance accountability over
the current system, but certain aspects of the new measures raise concerns
that VETS should address. The measures currently in place focus largely on
the process- that is, the number of services provided to veterans. Under the
proposed system, more focus is given to measuring the outcomes veterans
achieve from the program. In addition, under the proposed system, VETS no
longer requires states to compare services provided to veterans with those
provided to nonveterans when establishing expected levels of performance.
The proposed measures are also more closely aligned with those of WIA,
making it easier for service providers to report on outcomes. While the
proposed measures improve the way in which VETS will measure program
performance, a few areas of concern remain. VETS? strategic plan suggests
that states focus their efforts on providing staff- assisted services to
veterans, including case management. Yet, none of the proposed measures
specifically gauge the success of these services and may instead encourage
staff to focus their efforts on assisting veterans who may more easily find
jobs. In addition, VETS? proposal includes one measure- the number of
federal contractor jobs listed with local employment offices- that is not
only processoriented but also focuses on outcomes that are beyond the
control of staff serving veterans. We are recommending that VETS establish a
performance measure that more effectively gauges the success of these staff-
assisted services and that VETS clearly define its target populations. In
addition, we are recommending that VETS eliminate the measure related to the
number of federal contractor jobs listed.

2 We chose to interview officials from these 15 states because they were
familiar with performance measurement issues. The states included:
California, Colorado, Connecticut, Florida, Illinois, Indiana, Michigan,
Missouri, Montana, New Jersey, Oregon, Pennsylvania, Tennessee, Texas, and
Washington. Officials from these states participated in a workgroup that was
convened by Labor?s Office of Workforce Security in January 2000. It was
tasked with developing labor exchange performance measures, revising the
data collection and reporting systems provided under the Wagner- Peyser Act,
and developing procedures for establishing levels of performance for the
delivery of labor exchange services. Results in Brief

Page 3 GAO- 01- 580 VETS' Performance Measures and Data

VETS proposes that all states use a single data source- Unemployment
Insurance (UI) wage records- to identify veterans who get jobs. Using these
data will greatly improve the comparability and reliability of the new
measures, because it will standardize the data states use to report their
performance. Currently, the data used by VETS are not comparable across
states, in part, because states use a variety of data sources for
performance- reporting purposes. In addition, we found evidence that using
wage record data will help state staff better identify those veterans who
find work after receiving services. While using these data will improve some
aspects of data collection, the data present some challenges. States
generally do not have access to wage records from other states and,
therefore, should find ways to track individuals who receive services in one
state but get a job in another. In addition, VETS? programs should identify,
if necessary, those veterans finding jobs in categories not covered by wage
records, such as self- employed workers and federal employees.

As VETS finalizes its performance- reporting requirements, other issues that
affect the comparability of states? performance- related data should be
considered. For example, states vary in whether they register and count, for
performance reporting purposes, job seekers who use only self- service
tools, such as internet- based job listings. In addition, states differ in
how long a veteran remains registered with the state after seeking services.
These differences in state registration procedures affect how states
calculate the number of veterans who get jobs and make comparing performance
across states less reliable. In order to have performance data that are
comparable, we are recommending that VETS standardize the way states
register and count veterans for reporting purposes.

VETS administers national programs intended to (1) ensure that veterans
receive priority in employment and training opportunities from the
employment service; (2) assist veterans, reservists, and National Guard
members in securing employment; and (3) protect veterans? employment rights
and benefits. The key elements of VETS? services include enforcing veterans?
preference and reemployment rights and securing employment and training
services. VETS? programs are among those federal programs whose services
have been affected by WIA and other legislative changes aimed at
streamlining services and holding programs accountable for their results.
Background

Page 4 GAO- 01- 580 VETS' Performance Measures and Data

VETS carries out its responsibilities through a nationwide network that
includes representation in each of the Department of Labor?s 10 regions and
staff in each state. The Office of the Assistant Secretary for Veterans?
Employment and Training administers VETS? activities through regional
administrators and a VETS director in each state. These federally paid VETS
staff are the link between VETS and the states? employment service system,
which is overseen by Labor?s Employment and Training Administration (ETA).
VETS funds two primary veterans? employment assistance grants to states- the
Disabled Veterans? Outreach Program (DVOP) and the Local Veterans?
Employment Representatives (LVER). Fiscal year 2001 appropriation for VETS
was about $183 million, including $81.6 million for DVOP specialists (DVOPS)
and $77.2 million for LVER staff. These funds paid for 1,327 DVOP positions
and 1,206 LVER positions.

The roles of the DVOPS and LVERs have been separately defined in two
statutes. LVERs were first authorized under the original GI bill (the
Servicemen?s Readjustment Act of 1944) and DVOP specialists were authorized
by the Veterans? Rehabilitation and Education Amendments of 1980. A key
responsibility of a DVOP is to develop job and job training opportunities
for veterans through contacts with employers, especially small- and medium-
size private sector employers. LVERs are to provide program oversight of
local employment service offices to ensure that veterans receive maximum
employment and training opportunities from the entire local office staff. In
addition, DVOPS and LVERs traditionally have provided services that include

 locating veterans who need services,

 networking in the community for employment and training programs,

 bringing together veterans looking for work and employers seeking to fill
job openings,

 making referrals to support services, and

 providing case management for those veterans in need of more intensive
services.

Increasingly, however, veterans are accessing services on their own, through
tools such as internet- based job listings or resume writing software.

As part of the DVOP and LVER grant agreements, states must provide or ensure
that veterans receive priority at every point where public employment and
training services are available. The DVOP and LVER programs give priority to
the needs of disabled veterans and veterans who VETS? Programs

Page 5 GAO- 01- 580 VETS' Performance Measures and Data

served during the Vietnam era. States? employment service systems are
expected to give priority to veterans over nonveterans. Generally, this
means that local employment offices are to offer or provide all services to
veterans before offering or providing those services to nonveterans. To
monitor the states? programs, VETS has been using a set of measures that
evaluates states? performance in five dimensions: (1) veterans placed in
training, (2) those receiving counseling, (3) those receiving services, (4)
those entering employment, and (5) those obtaining federal contractor jobs.
These measures primarily count the number of services that veterans receive
and compare the totals with similar services provided to nonveterans. To
ensure priority service to veterans, VETS expects levels of performance for
services provided to veterans to be higher than levels for nonveterans. For
example, veterans and other eligibles must be placed in or obtain employment
at a rate 15 percent higher than that achieved by nonveterans. (See table 1
for VETS? specific performance standards.)

Page 6 GAO- 01- 580 VETS' Performance Measures and Data

Table 1: VETS? Current Performance Measures and Standards Measure and
category of veteran

Standard for veterans over nonveterans

(Percent) Placed/ obtained employment

Veterans and other eligibles 15 Vietnam- era veterans 20 Disabled veterans
25

FCJL placements a

Vietnam- era veterans 20 Special disabled veterans b 25

Number counseled

Veterans and other eligibles 15 Vietnam- era veterans 20 Disabled veterans
25

Number placed in training

Veterans and other eligibles 15 Vietnam- era veterans 20 Disabled veterans
25

Number receiving a reportable service

Veterans and other eligibles 15 Vietnam- era veterans 20 Disabled veterans
25

a Veterans placed in jobs with federal contractors. b A special disabled
veteran is (1) a veteran who is entitled to compensation (or who, but for
the receipt of military retired pay, would be entitled to compensation)
under laws administered by the Department of Veterans Affairs for a
disability rated at 30 percent or more or (2) a person who was discharged or
released from active duty because of a service- connected disability.

Source: Veterans? Employment and Training Service, Department of Labor.

To report on performance, VETS currently relies on the Employment and
Training Administration?s 9002 system to aggregate data reported by states
on veterans and nonveterans who register with state Employment Services (ES)
offices, track the services provided to them (such as counseling or job
referral), and gather information on their employment outcomes. The 9002
system also collects information such as the registrants? employment status,
level of education (e. g., high school, postsecondary degree/ certificate),
and basic demographic information, such as age and race.

Page 7 GAO- 01- 580 VETS' Performance Measures and Data

Over the past several years, the Congress has taken steps to streamline and
integrate services provided by federally funded employment and training
programs. WIA, which the Congress passed in 1998, requires states and
localities to use a one- stop center structure to provide access to most
employment and training services in a single location. WIA requires about 17
categories of programs, including VETS and ES programs, to provide services
through the one- stop center. However, because DVOP and LVER staff can
provide assistance only to veterans, and because their roles in one- stop
centers are not specifically addressed in WIA, it is unclear how they will
function with regard to one- stop centers. According to VETS officials, this
lack of clarity has been addressed. Agreements made with each state on
planned services to veterans now include provisions on how DVOPS and LVERs
will be integrated into the one- stop delivery system.

In addition to changing the way services are provided, programs are now
increasingly held accountable for their results. Through the Government
Performance and Results Act of 1993 (GPRA), the Congress seeks to improve
the efficiency, effectiveness, and public accountability of federal agencies
as well as improve congressional decision making. GPRA does so, in part, by
promoting a focus on what the program achieves rather than tracking program
activities. GPRA outlines a series of steps in which agencies are required
to identify their goals, measure performance, and report on the degree to
which those goals were met. Executive branch agencies were required to
submit the first of their strategic plans to the Office of Management and
Budget and the Congress in September 1997. Although not required by GPRA,
Labor?s component agencies, such as VETS, have prepared their own strategic
and performance plans at the direction of the Secretary of Labor.

To address the goals of GPRA and in response to recommendations by us and
other groups, such as the Congressional Commission on Servicemembers and
Veterans Transition Assistance, 3 VETS is currently developing a new system
to measure the performance of its programs. Over the last several years,
VETS conducted pilot programs in about eight states that tested some new
performance measures and the use of new

3 See Report of the Congressional Commission on Servicemembers and Veterans
Transition Assistance, January 14, 1999, Congressional Commission on
Servicemembers and Veterans Transition Assistance, Arlington, VA.
Legislative and Regulatory

Changes Affecting VETS

Page 8 GAO- 01- 580 VETS' Performance Measures and Data

data to support these measures. VETS officials told us that they anticipate
implementing their new performance measurement system in July 2001.

VETS? proposed performance measures are a significant improvement over
current measures, but certain aspects of these measures raise concerns that
VETS may need to address. The proposed measures include an (1) entered-
employment rate, (2) employment rate following staffassisted services, (3)
employment retention rate, and (4) increase in the number of federal
contractor job openings listed. These measures are an improvement over
current measures because they

 focus more on what the programs achieve and less on the number of services
they provide,

 no longer use the level of services provided to nonveterans as the
standard for services that must be provided to veterans,

 adjust expected state performance to economic conditions within states,
and

 establish two measures that are already collected for WIA- funded services
and proposed for ES.

However, even with these improvements, the proposed measures continue to
send a mixed message to staff about where to place their service priorities.
In addition, the proposed measures include a redefined measure for tracking
federal contractor job openings, but the measure is processoriented and
outside the scope of the work of DVOPS and LVERs.

The proposed performance measures improve accountability because they place
more emphasis on employment- related outcomes by eliminating process-
oriented measures- measures that simply track services provided to veterans.
Current process measures that VETS eliminated from the proposed performance
system include the number of veterans referred to counseling, the number
placed in training, and the number receiving certain other services, such as
job referrals. As we noted in past reports, these process- oriented measures
are activity- and volume- driven and focus efforts on the number of services
provided, not on the outcomes veterans VETS? Proposed

Measures Hold Promise, But Some Concerns Remain

Proposed Measures Are An Improvement Over the Current Ones

Page 9 GAO- 01- 580 VETS' Performance Measures and Data

achieve. 4 These measures offer states little incentive to provide services
to those veterans who are only marginally prepared for work and who may need
more intensive services requiring more staff time. The VETS? proposal still
includes one process- oriented measure that simply reflects the percentage
increase in the number of federal contractor job openings listed with the
public labor exchange but adds two outcome- oriented measures- job retention
after 6 months and the employment rate following staff- assisted services. 5
The VETS? proposal also retains an outcome measure that is in the current
system- the entered- employment rate. (See table 2.)

4 See, Veterans? Employment and Training Service: Focusing on Program
Results to Improve Agency Performance (GAO/ T- HEHS- 97- 129, May 7, 1997)
and Veterans? Employment and Training Service: Better Planning Needed to
Address Future Needs (GAO/ T- HEHS- 00- 206, Sept. 27, 2000). 5 VETS
considers this measure as process- oriented ?with an emphasis on outcomes.?
However, for this report, we classified the measure as outcome- based
because it reports an employment rate rather than only reporting a count of
services.

Page 10 GAO- 01- 580 VETS' Performance Measures and Data

Table 2: VETS? Current Performance Measures Compared With Proposed Measures
Current measures Processoriented Outcomeoriented

Entered- employment rate:

The percentage of all registered veterans who were placed in or obtained
employment.

X Number of veterans placed in training:

A count of the veterans placed in training. X Number of veterans receiving
counseling:

A count of the veterans who received counseling services. X Number of
veterans receiving some reportable service:

A count of the veterans who received at least one reportable service.

X Federal contractor jobs filled by Vietnam and special disabled veterans:

A count of the veterans who were placed in jobs listed on the federal
contractor job list.

X Proposed measures Process Outcomeoriented

oriented Entered- employment rate:

The percentage of all registered veterans who were placed in or obtained
employment.

X Employment rate following receipt of staff- assisted services:

The percentage of registered veterans who are employed after receiving some
form of staff- assisted labor exchange services.

X Employment retention rate at 6 months:

Of the veterans who had entered employment following registration, the
percentage of those who continued to earn wages 6 months after entering
employment.

X Federal contractor job openings listed with the public labor exchange:

The percentage increase in the number of federal contractor job openings
listed annually with the public labor exchange from one program year to the
next.

X

Source: Veterans? Employment and Training Service, Department of Labor.

The proposed performance measures also improve the way VETS establishes the
level of performance that states are expected to achieve. VETS no longer
requires states to compare the level of services provided to veterans with
those provided to nonveterans. 6 In past reports, we have pointed out that
the use of these relative standards results in states with

6 While states will no longer be required to compare the level of services
given to veterans and nonveterans, VETS is required to report annually to
the Congress on the job placement rate of veterans compared with the rate
for nonveterans. 38 U. S. C. sect. 4107.

Page 11 GAO- 01- 580 VETS' Performance Measures and Data

poor levels of service to nonveterans being held to lower standards for
service to veterans than states with better overall performance. 7 For
example, in program year 1999, Rhode Island reported an enteredemployment
rate of 5.49 percent for nonveterans. Because VETS requires states to ensure
that they achieve an entered- employment rate for veterans that is 15
percent higher than that for nonveterans, Rhode Island?s 1999 expected
performance level was 6.32 percent of registered veterans entering
employment- a low level of performance. Under the proposed system, VETS will
negotiate performance levels annually with each state based on that state?s
past performance, using guidelines similar to those used for WIA. 8 VETS
will also be able to adjust these levels based on economic conditions within
each state, such as the unemployment rate, the rate of job creation or loss,
or other factors. 9

The proposed performance measures are also similar to those established
under WIA, making it easier for service providers to achieve WIA?s goal of
integrating and streamlining employment and training services. In the
current environment, many of the programs that provide services through the
one- stop centers have their own unique performance measures and program
definitions, requiring multiple systems and multiple data collection efforts
to track a single client. In the proposed system, VETS has made an effort to
align its performance measures with those of WIA. In fact, two of the five
proposed measures- entered- employment rate and employment retention- are
nearly identical to WIA?s and to those proposed for ES. If VETS aligns the
measures with those of WIA and ES, local offices will be more readily able
to establish integrated data systems that will minimize the data collection
burden on service providers and clients. (See app. I for a comparison of the
WIA performance measures with those proposed for VETS and ES.)

7 See GAO/ T- HEHS- 97- 129. 8 VETS is planning to use WIA?s negotiation
process to establish expected performance levels for labor exchange
services. VETS proposes that states use 2 years of data if possible, but not
less than 1 year in determining trends for performance and factors that may
influence performance.

9 VETS? proposal mentions the following as possible factors for states to
consider when negotiating expected levels of performance: the unemployment
rate, the rate of job creation/ loss and new business start- ups,
availability of transportation and daycare, pursuit of new or enhanced
employment partnerships, natural disasters, and state legislation.

Page 12 GAO- 01- 580 VETS' Performance Measures and Data

While the proposed performance measures are an improvement over those
currently in place, there are issues with these measures that VETS should
address. First, a comparison of the performance measures with the strategic
plan indicates that VETS is sending a mixed message to states about what
services to provide and to whom. The strategic plan suggests that states
focus their efforts on providing staff- assisted services to veterans,
including case management. Yet, none of the proposed measures specifically
gauges whether more staff- intensive services are helping veterans get jobs.
VETS? proposal includes a measure that tracks employment outcomes following
staff- assisted services. However, this measure is broadly defined, and the
list of staff- assisted services includes nearly all services provided to
veterans. 10 This makes the outcomes achieved for the staff- assisted
measure nearly identical to those reported for the more general ?entered-
employment rate.? In addition, as VETS has defined it, staff- assisted
services include many services that might not be considered ?intensive,?
such as referral to a job and job search activities. Because the definition
is so broadly defined, a veteran who only attended a job search workshop
would be counted the same as a veteran who received more intensive services,
such as testing and employability planning. Both would be counted in the
more general entered- employment rate measure, as well as the staff-
assisted service measure. A stricter definition for staff- assisted services
that includes only those services that are generally considered staff-
intensive would allow VETS to more accurately assess the success of those
services and help to clarify the goals of the program.

Second, VETS is sending a mixed message about which groups of veterans to
target for services. As we noted in past reports and testimonies, VETS has
inconsistently identified various ?targeted? groups of veterans it plans to
help. 11 In its strategic plan, VETS identifies two broad veteran groups
that should be targeted to receive special attention-( 1) disabled veterans

10 VETS uses the ETA definition of staff- assisted services. Staff- assisted
services include (a) referral to a job; (b) placement in training; (c)
assessment services, including an assessment interview, testing, counseling,
and employability planning; (d) career guidance; (e) job search activities,
including resume assistance, job search workshops, job finding clubs,
providing specific labor market information and job search planning; (f)
federal bonding program; (g) job development contacts; (h) tax credit
eligibility determination; (i) referral to other services, including skills
training, educational services, and supportive services; and (j) any other
service requiring expenditure of time. Application taking and/ or
registration services are not included as staff- assisted services.

11 See GAO/ T- HEHS- 00- 206. Concerns Remain That

VETS Should Address

Page 13 GAO- 01- 580 VETS' Performance Measures and Data

and (2) all veterans and other eligible persons. And consistent with this,
VETS proposes that expected performance levels be negotiated separately for
each of these same two groups. Yet, the strategic plan also suggests that,
when providing services to all veterans, special attention should be given
to meeting the needs of certain other target groups, some of which might
require more intensive services to become employed. The groups targeted for
special attention include (1) veterans who have significant barriers to
employment, (2) veterans who served on active duty during a war (or campaign
or expedition in which a campaign badge has been authorized), and (3)
veterans recently separated from military service. In reviewing VETS?
proposed measures and the plan for negotiating performance levels, staff may
be confused as to where they should place their service priorities. It is
unclear what steps VETS will take to ensure that DVOPS and LVERs are
provided ample opportunity and encouragement to focus attention on the
portion of the ?all veterans? group who may require more staff time to be
successful in getting a job.

Last, VETS? proposal also continues to include a performance measure related
to federal contractor job openings listed with the state?s ES office.
However, in its proposal, VETS has changed the measure. Under the current
system, VETS tracks the number of Vietnam- era and special disabled veterans
who were placed in jobs listed by federal contractors- an outcome measure.
Now, under the proposed system, VETS will track the increase in the number
of federal contractor jobs listed with the state?s ES office- a process-
oriented measure. 12 This new measure ultimately holds DVOPS and LVERs
accountable for the number of federal contractors in a given state or local
area, not for veteran placements with those contractors. The presence of
federal contractors in a given state or local area is unpredictable and is
determined by the federal agencies awarding contracts. Furthermore,
according to state officials that we talked with, the federal contractor
measure should be eliminated altogether because it is the responsibility of
contractors to list their job openings. In addition, it is the Office of
Federal Contract Compliance that is responsible for ensuring that all
companies conducting business with

12 Any contractor or subcontractor with a contract of $25,000 or more with
the federal government must take affirmative action to hire and promote
qualified special disabled veterans, veterans of the Vietnam- era, or any
other veterans who served on active duty during a war or in a campaign or
expedition for which a campaign badge has been authorized. Contractors and
subcontractors with job openings, other than executive or top management
jobs, must list them with the nearest state employment office. Veterans
cited above receive priority for referral to federal contractor job openings
listed at those offices.

Page 14 GAO- 01- 580 VETS' Performance Measures and Data

the federal government list their jobs with state ES offices and take
affirmative action to hire qualified veterans.

The proposed data for the new measures will greatly improve the
comparability and reliability of these measures, but this change will bring
some challenges that VETS will need to address. Consistent with WIA and ES,
VETS is proposing that all states use UI wage records to identify veterans
who get jobs. UI wage records contain the earnings of each employee reported
quarterly by employers to state UI agencies. 13 Currently, the data VETS
uses are not comparable across states, in part, because states use different
data sources to report employment- related outcomes. Using a single,
standardized source for collecting data will improve VETS? ability to
compare performance across states. UI wage records will also provide state
officials with a better means to identify veterans who get jobs than does
the traditional follow- up method of telephoning veterans and/ or employers
to verify employment. However, states cannot readily access wage records
from other states, wage records do not cover certain types of employment,
and these data are not available until 3 to 9 months after an individual
gets a job.

Using a single data source will help to standardize the way in which states
collect data on veterans, thereby making it easier to compare performance
across states. Currently, states are using various data sources for
performance- reporting purposes. While almost all of the states in our
review used a combination of data sources to determine whether or not a
veteran got a job, most of the states relied substantially on one data
source, but that source differed among states. For example, in program year
1999

 7 of the 15 states that we contacted relied to a large extent on wage
record data to determine whether a veteran got a job or not;

 7 others relied, for the most part, on telephone calls and letters to
veterans and employers to determine a veteran?s employment status; and

13 Each calendar quarter, employers in a state provide wage information on
their employees to their state?s UI agency or some other state agency. The
information contained in wage records varies from state to state. However,
all wage records contain at least the following information: the calendar
quarter that the wages were reported in, the employee?s social security
number, wages paid to the employee in that quarter, and employer
information. Proposed Data Source

for New Measures Is An Improvement But Will Bring Some Challenges

Proposed Data Source Will Help To Ensure Comparability and Reliability
Across States

Page 15 GAO- 01- 580 VETS' Performance Measures and Data

 one state relied primarily on its new hire database for employment data.
14

In addition to making state data more comparable, we found evidence that
states currently using wage records have been able to better identify those
veterans who get jobs after receiving services. A recent study found that UI
wage records more accurately identified how many veterans got jobs after
receiving DVOP, LVER, or ES services. 15 Using UI wage records, this study
tracked veterans who registered with the Maryland Job Service during program
year 1997 and found an entered- employment rate that ranged from 65 percent
to 82 percent, depending on the way the study defined a registrant. In that
same program year, Maryland reported to VETS an entered- employment rate of
31 percent, which was based on staff telephoning veterans and employers to
verify employment. In addition, most states in our review that are now using
UI wage records, either as their primary data source or to augment other
data sources, reported higher employment rates in program year 1999 for
veterans they served than that year?s national average of 30 percent. (See
app. II for a list of all states and their respective entered- employment
rates for program years 1996- 1999.) By comparison, all but one of the
states that relied either on manual follow- up or the new hire database
reported an employment rate below the national average.

Another benefit of using UI wage records is that staff assisting veterans
will be relying on data already available rather than collecting additional
information from veterans or employers. Relying on these already reported
data would require less staff time from DVOP, LVER, and ES staff, freeing
them to focus more on providing job- related services to veterans. State
officials told us that relying on manual follow- up, such as

14 The Personal Responsibility and Work Opportunity Reconciliation Act of
1996 established the National Directory of New Hires and State Directories
of New Hires. The National Directory is maintained by the Social Security
Administration on behalf of the Department of Health and Human Services,
Office of Child Support Enforcement. States maintain their own State
Directories of New Hires and generally supply data for the National
Directory. Information in these directories includes: new hire information,
such as name, address, and social security number of the employee and the
name, address, federal identification number of the employer; in some
states, wage information; and UI claim information.

15 Proposed New Entered Employment Patterns of Veteran Wagner- Peyser
Registrants in the State of Maryland, by Robert Cook, BETAH Associates; and
Edward Davin and Karin Willner, DynCorp (Apr. 12, 2000).

Page 16 GAO- 01- 580 VETS' Performance Measures and Data

telephone calls, has been labor- intensive and has diverted staff attention
away from providing appropriate assistance to veterans. 16

While UI wage records offer advantages over the current data collection
system, some challenges need to be addressed. First, states should find ways
to identify interstate job placements. Because the UI wage record system
resides within each state, states generally do not have access to wage
records from other states, making it difficult to track individuals who
receive services in one state but get a job in another. Currently, there is
no national system in place that facilitates data sharing among states.
However, in response to WIA requirements, states are developing an
interstate UI wage record information sharing system, known as the Wage
Record Interchange System (WRIS). The system is designed to minimize the
burden on state unemployment insurance programs in responding to requests
for wage record data, to ensure the security of the transactions involving
individual wage records, and to produce the results at a low cost per
record. In addition, some states have entered into agreements with
neighboring states to share wage information in support of WIA. These
efforts should help VETS as well.

Second, states should find ways to identify those veterans finding jobs in
categories not covered by UI wage records. UI wage records cover about 94
percent of wage and salary workers, but certain employment categories are
not covered, such as self- employed persons, most independent contractors,
military personnel, federal government workers, railroad employees, some
part- time employees of nonprofit institutions, and employees of religious
orders. Therefore, the UI system will not be able to track and count
veterans who get these types of jobs. This is an issue for WIA as well, and
states are beginning to assess the extent to which this issue will affect
their ability to accurately determine the outcome of WIAfunded programs.

16 See draft report, ?Measuring Employment and Income for Low- Income
Populations with Administrative and Survey Data,? V. Joseph Holtz,
University of California at Los Angeles and John Karl Scholz, University of
Wisconsin, June 30, 2000, for a discussion of the benefits of UI data
compared to contacting program participants for performance reporting. UI
Data Presents Some

Challenges

Page 17 GAO- 01- 580 VETS' Performance Measures and Data

There are other issues not related to the use of UI wage records that VETS
should consider as it finalizes its performance- reporting requirements.
VETS? proposed performance system does not standardize how states report
veterans or nonveterans who use self- service activities, making it
difficult to reliably assess nationwide performance. In an environment in
which self- service is becoming more common, we found that states vary in
whether they register veteran job seekers who access self- service tools,
such as internet- based job listings or resume writing software. For
example, some states allow job seekers greater access to job listings
without requiring that they register, while others have more restrictions on
who can access job lists. Table 3 shows how such differences can affect
entered- employment rates. In this example, 100 veterans enter the
employment service for assistance. In both cases, 40 veterans ultimately get
jobs after receiving identical services. In one case, the placement rate is
40 percent and in the other, 50 percent- a 10 percentage point difference.
This difference results from counting all job seekers in one case and only
those requiring staff assistance in the other. As a result of the different
ways states currently count veterans and report outcomes, the entered-
employment rate measure is not consistently calculated across states, and
nationwide comparisons are misleading.

Table 3: A Comparison of Entered- Employment Rates by Registration Policy
All veterans required to register Veterans accessing self- service do not
have to register

Veterans registered Number of

veterans who get

jobs Number of

veterans with jobs counted

in enteredemployment rate

Veterans registered Number of

veterans who get jobs

Number of veterans with jobs counted in

enteredemployment rate

40 Veterans use selfservice 40 10 10 40 Veterans

use selfservice 0 10 0

60 Veterans require staff assistance

60 30 30 60 Veterans require staff assistance

60 30 30

Total 100 40 40 Total 60 40 30 Reported Entered- Employment Rate: 40/ 100 =
40% Reported Entered- Employment Rate: 30/ 60 = 50%

Source: GAO analysis.

VETS? proposed performance system does not standardize how long a veteran or
nonveteran remains registered after seeking services for performance-
reporting purposes. We found that states differ in how long they keep
veterans registered. This difference affects the calculation of the Other
Measurement Issues

Affect Comparability of States? Performance Data

Page 18 GAO- 01- 580 VETS' Performance Measures and Data

entered- employment rate (i. e., the number of veterans that get jobs),
making performance comparisons across states less reliable. Many of the
states we contacted count individuals as registered who have received a
service in the last 6 months. However, two states only count those as
registered who have received a service in the last 3 months, while two
others count only those who received a service in the last 2 months. And in
one state, anyone who has received a service from the state?s employment
office since 1998 is counted as a registrant when determining the
enteredemployment rate. States with shorter registration periods may be able
to report a higher entered- employment rate than states with longer
registration periods.

VETS is improving its performance measurement system by proposing new
measures that are more outcome- oriented than its current measures and by
requiring that all states use wage record data to improve the comparability
and reliability of reported program performance. While these changes move
VETS a step closer to implementing an effective accountability system, they
may not go far enough. VETS continues to send a mixed message to states
about what services to provide and to whom. As presently defined, two of the
proposed measures- the enteredemployment rate and the employment rate
following staff- assisted services- may provide nearly identical results,
and neither helps VETS to monitor whether more intensive services are being
provided to veterans or whether these services are successful. VETS also
continues to inconsistently identify the groups of veterans that it wants
states to help. In addition, VETS maintains a measure related to federal
contractors- one that is beyond the control of DVOPS and LVERs.

Furthermore, in its proposed system, VETS allows states to decide which
veterans to include in its performance reports. This results in data
inconsistencies that make state- to- state comparisons unreliable. Without
clear and consistent direction from VETS? planning documents and performance
measures, staff assisting veterans will be uncertain where to place their
priorities. In addition, without stricter guidelines for how to count
veterans, VETS will be unable to accurately assess program performance
nationwide. Unless further modifications are made, VETS will be unable to
fully determine whether its programs and services are fulfilling its
mission. Conclusions

Page 19 GAO- 01- 580 VETS' Performance Measures and Data

In order to establish a more effective performance management system, we
recommend that the Secretary of Labor direct VETS to do the following:

 Redefine staff- assisted services to include only those that may be
considered staff intensive, such as case management, so that VETS will be
able to evaluate the success of intensive staff- assisted services.

 Clearly define target populations so that staff assisting veterans know
where to place their priorities. If staff are to focus on assisting veterans
who need more assistance, VETS should provide incentives and opportunities
to do so through appropriate performance measures or negotiated levels of
performance.

 Eliminate the measure related to federal contractor jobs so that staff are
not held accountable for the number of federal contractors in a state or
local area or for the failure of contractors to list their jobs with ES
offices.

 Establish and communicate guidelines that standardize how to count
veterans for performance- reporting purposes so that VETS will be able to
assess program performance nationwide.

We provided VETS with the opportunity to comment on a draft of this report.
Formal comments from VETS appear in appendix III. In addition to the
comments discussed below, VETS provided technical comments that we
incorporated where appropriate.

VETS generally agreed with our findings and two of our recommendations but
disagreed with the other two recommendations. VETS acknowledged that its
current strategic plan (Nov. 2000) sends a mixed message to the states about
which groups of veterans staff should target for special attention. VETS
noted that it is revising its strategic and annual plans to reflect a more
consistent message about what services to provide and to whom. VETS also
explained that it is developing new performance standards specific to DVOP
and LVER staff that will clarify the role they play in providing services to
veterans. According to VETS officials, states will have the option of using
these specific standards or developing their own. When developing these
standards, VETS will need to ensure that the specific standards developed
for DVOPS and LVERs are consistent with the message in the revised strategic
plan and that together they provide a coherent strategy as to where staff
should place their service priorities.

VETS disagreed with our recommendation for a revised definition of the
performance measure related to staff- assisted services. VETS said that any
veteran receiving staff- assisted services may require a multitude of the
services cited in the definition- any one of which or combination thereof
may require extensive staff time. We disagree that any one of these
Recommendations

Agency Comments and Our Evaluation

Page 20 GAO- 01- 580 VETS' Performance Measures and Data

services necessarily requires extensive staff time. As noted in our report,
a veteran may be counted as receiving staff- assisted services after
receiving only a job referral or labor market information- services that by
themselves would not involve extensive staff resources. Moreover, we
continue to believe that the broadly defined staff- assisted service measure
will likely not report outcomes substantially different from those reported
for the more general entered- employment rate measure. As noted in our
report, a stricter definition for staff- assisted services that includes
only those services generally considered to be staff- intensive would allow
VETS to more accurately assess outcomes associated with those services.

VETS disagreed with our recommendation to discontinue the measure related to
jobs listed by federal contractors. However, VETS agreed to reconsider the
suitability of this specific measure after public comments have been
received. As we noted in our report, the presence of federal contractors in
a given state or local area is determined by the federal agencies awarding
contracts. In addition, state officials told us that it is the
responsibility of the contractors, not DVOP and LVER staff, to list their
job openings with employment services. Current law requires the Secretary of
Labor to report annually to the Congress on the number of federal contractor
positions listed and the number of veterans receiving job priority through
this program. This information could be collected in absence of a specific
performance measure.

With regard to our recommendation that VETS establish guidelines that
standardize how states count veterans for performance- reporting purposes,
VETS said that it will be working with ETA to determine how states can
uniformly report veterans and nonveterans that use self- service activities.
In addition, VETS noted that the revised ETA 9002 report will provide
uniform instructions on how long individuals remain registered in the
system.

We are sending copies of this report to the Honorable Elaine L. Chao,
Secretary of Labor; appropriate congressional committees; and other
interested parties. We will also make copies available to others upon
request.

Page 21 GAO- 01- 580 VETS' Performance Measures and Data

If you or your staff have questions about this report, please contact me on
(202) 512- 7215 or Dianne Blank on (202) 512- 5654. Individuals making key
contributions to this report include Elizabeth Morrison and Amanda
Ahlstrand.

Sincerely yours, Sigurd R. Nilsen, Director Education, Workforce and

Income Security Issues

Appendix I: Comparison of VETS, ES, and WIA Performance Measures

Page 22 GAO- 01- 580 VETS' Performance Measures and Data

Similar to Workforce Investment Act (WIA) programs, the Employment Service
(ES) and the Veterans? Employment and Training Service (VETS) are proposing
that their programs use Unemployment Insurance wage records to report on
performance measures. Each calendar quarter, employers submit wage record
data to their state?s UI agency or some other state agency. The following
table compares the proposed performance measures of VETS and ES and those
used by WIA?s adult and dislocated worker programs.

VETS proposed performance measures

ES proposed performance measures WIA performance measures (adult and

dislocated worker programs) Entered- employment rate:

The percentage of all registered veterans who got a job in the 1st or 2nd
quarter after registration.

Entered- employment rate:

The percentage of workers who got a job in the 1 st or 2nd quarter after
registration.

Entered- employment rate:

The percentage of workers who got a job by the end of the 1st quarter after
exit.

Employment retention rate at 6 months:

Of the veterans who got a job after registration, the percentage who were
still earning wages in the 2 nd quarter after getting a job.

Employment retention rate at 6 months:

The percentage of workers who continued to earn wages in the 2 nd quarter
after the 1st quarter in which there were earned wages.

Employment retention rate:

Of those who had a job in the 1st quarter after exit, the percentage of
workers who have a job in the 3rd quarter after exit.

Employment rate following receipt of staff- assisted services:

Of the veterans who received staffassisted services, the percentage who got
a job in the 1st or 2nd quarter after registration. a

No measure No measure

Federal contractor job openings listed with the public labor exchange:

The percentage increase in the number of federal contractor Job openings
listed annually with the public labor exchange from one program year to the
next.

No measure No measure No measure Employer customer satisfaction:

Average of three survey questions on employers? satisfaction with services
received.

Employer customer satisfaction:

Average of three survey questions on employers? satisfaction with services
received. No measure Job seeker customer satisfaction:

Average of three survey questions on job seekers? satisfaction with services
received.

Job seeker customer satisfaction:

Average of three survey questions on job seekers? satisfaction with services
received.

Appendix I: Comparison of VETS, ES, and WIA Performance Measures

Appendix I: Comparison of VETS, ES, and WIA Performance Measures

Page 23 GAO- 01- 580 VETS' Performance Measures and Data

VETS proposed performance measures

ES proposed performance measures WIA performance measures (adult and

dislocated worker programs)

No measure No measure Earnings change (adults only):

The difference between total post- program earnings (from the 2nd and 3rd
quarters after exiting the WIA program) and the total pre- program earnings
(from the 2nd and 3rd quarters prior to entering the WIA program) divided by
the number of participants leaving the program. No measure No measure
Earnings replacement rate (dislocated

workers only):

Total post- program earnings (in the 2nd and 3rd quarters after exit)
divided by predislocation earnings (in the 2nd and 3rd quarters prior to
dislocation). a Staff- assisted services include: (a) referral to a job; (b)
placement in training; (c) assessment

services, including an assessment interview, testing, counseling and
employability planning; (d) career guidance; (e) job search activities,
including resume assistance, job search workshops, job finding clubs,
specific labor market information and job search planning; (f) federal
bonding program; (g) job development contacts; (h) tax credit eligibility
determination; (i) referral to other services, including skills training,
educational services and supportive services; and (j) any other service
requiring expenditure of time. Application taking and/ or registration
services are not included as staffassisted services.

Source: Veterans? Employment and Training Service, U. S. Department of
Labor; U. S. Department of Labor Training and Employment Information Notice
Number 13- 000, ?Consultation Paper on Labor Exchange Performance
Measurement System;? and U. S. Department of Labor Training and Employment
Guidance Letter Number 7- 99, ?Core and Customer Satisfaction Performance
Measures for the Workforce Investment System.?

Appendix II: States? EnteredEmployment Rates for Veterans in Program Year
1999

Page 24 GAO- 01- 580 VETS' Performance Measures and Data

Entered- employment rate State 1999 1998 1997 1996

Alabama 38.4 40.4 38.9 34.3 Alaska 22.6 20.8 25.8 22.9 Arizona 32.4 29.4
21.4 18.9 Arkansas 30.3 29.5 25.6 31.8 California 5. 2 3.7 15.0 18.5
Colorado 30.4 28.7 27.9 26.1 Connecticut 27.4 23.7 21.2 22.2 Delaware 17.1
12.1 11.8 12.8 District of Columbia a 20.3 17.0 15.1 9. 5 Florida 24.5 19.4
21.2 20.5 Georgia 38.1 34.5 30.0 26.1 Hawaii 22.2 15.5 16.8 14.5 Idaho 30.5
30.0 29.0 30.3 Illinois 35.8 34.5 30.4 28.9 Indiana 21.9 14.1 18.8 16.7 Iowa
46.1 48.6 44.4 45.2 Kansas 23.3 23.7 26.6 23.8 Kentucky 26.7 28.7 24.8 25.0
Louisiana 29.3 15.5 18.3 16.1 Maine 22.7 24.0 22.6 13.6 Maryland 34.0 31.1
27.6 25.8 Massachusetts 42.2 35.5 31.2 25.2 Michigan 17.0 18.4 6. 8 8.7
Minnesota 32.9 18.4 20.2 20.9 Mississippi 33.3 31.8 30.7 30.5 Missouri 33.9
24.9 32.5 30.9 Montana 33.1 30.2 29.2 31.7 Nebraska 25.5 24.1 26.5 26.3
Nevada 26.9 24.2 27.6 28.5 New Hampshire 37.1 36.6 27.4 23.6 New Jersey 34.4
35.6 39.9 40.0 New Mexico 30.9 30.5 29.8 17.7 New York 20.5 20.7 19.7 18.2
North Carolina 44.7 44.7 38.7 38.5 North Dakota 50.9 48.5 47.5 38.8 Ohio
18.5 16.1 18.5 15.8 Oklahoma 45.7 41.8 44.1 44.2 Oregon 36.2 33.8 28.9 33.0
Pennsylvania 33.6 26.2 23.0 21.8 Puerto Rico a 13.8 17.2 15.6 18.1 Rhode
Island 15.6 12.1 7. 3 8.9 South Carolina 36.8 35.7 32.8 30.7 South Dakota
61.3 58.1 44.2 40.6 Tennessee 54.4 68.9 47.2 20.3

Appendix II: States? Entered- Employment Rates for Veterans in Program Year
1999

Appendix II: States? EnteredEmployment Rates for Veterans in Program Year
1999

Page 25 GAO- 01- 580 VETS' Performance Measures and Data

Entered- employment rate State 1999 1998 1997 1996

Texas 45.8 36.1 38.0 35.3 Utah 45.7 33.5 41.7 45.8 Vermont 25.3 17.6 18.4
18.0 Virgin Islands a 30.3 23.0 15.3 17.6 Virginia 32.3 23.2 18.8 14.1
Washington 17.4 20.8 25.0 24.7 West Virginia 16.4 15.4 13.8 15.1 Wisconsin
42.6 43.8 44.4 10.9 Wyoming 32.3 29.0 28.8 28.3 a Shown as states for this
report.

Source: Prepared by GAO from data provided by the Veterans? Employment and
Training Service for program years 1996 through1999.

Appendix III: Comments From the Department of Labor

Page 26 GAO- 01- 580 VETS' Performance Measures and Data

Appendix III: Comments From the Department of Labor

Appendix III: Comments From the Department of Labor

Page 27 GAO- 01- 580 VETS' Performance Measures and Data

Now on p. 4.

Appendix III: Comments From the Department of Labor

Page 28 GAO- 01- 580 VETS' Performance Measures and Data

Now on p. 12. Now on p. 7.

Appendix III: Comments From the Department of Labor

Page 29 GAO- 01- 580 VETS' Performance Measures and Data

Now on p. 12.

Appendix III: Comments From the Department of Labor

Page 30 GAO- 01- 580 VETS' Performance Measures and Data

Now on p. 17.

(130000)

The first copy of each GAO report is free. Additional copies of reports are
$2 each. A check or money order should be made out to the Superintendent of
Documents. VISA and MasterCard credit cards are also accepted.

Orders for 100 or more copies to be mailed to a single address are
discounted 25 percent.

Orders by mail:

U. S. General Accounting Office P. O. Box 37050 Washington, DC 20013

Orders by visiting:

Room 1100 700 4 th St., NW (corner of 4 th and G Sts. NW) Washington, DC
20013

Orders by phone:

(202) 512- 6000 fax: (202) 512- 6061 TDD (202) 512- 2537

Each day, GAO issues a list of newly available reports and testimony. To
receive facsimile copies of the daily list or any list from the past 30
days, please call (202) 512- 6000 using a touchtone phone. A recorded menu
will provide information on how to obtain these lists.

Orders by Internet

For information on how to access GAO reports on the Internet, send an email
message with ?info? in the body to:

Info@ www. gao. gov or visit GAO?s World Wide Web home page at: http:// www.
gao. gov

Contact one:

 Web site: http:// www. gao. gov/ fraudnet/ fraudnet. htm

 E- mail: fraudnet@ gao. gov

 1- 800- 424- 5454 (automated answering system) Ordering Information

To Report Fraud, Waste, and Abuse in Federal Programs
*** End of document. ***