Veterans' Employment and Training Service: Labor Could Improve	 
Information on Reemployment Services, Outcomes, and Program	 
Impact (24-MAY-07, GAO-07-594). 				 
                                                                 
In 2002, Congress enacted the Jobs for Veterans Act (JVA), which 
modified two Department of Labor (Labor) programs that		 
specifically target veteran job seekers: the Disabled Veterans'  
Outreach Program (DVOP) and the Local Veterans' Employment	 
Representative (LVER) program. However, questions have been	 
raised about the adequacy of performance information on services 
to veterans by these and other employment programs. In this	 
report, GAO examined (1) the extent to which DVOP and LVER	 
performance information reflects services and outcomes for	 
veterans; (2) the extent to which performance information on	 
veterans paints a clear picture of their use of one-stop	 
services; and (3) what Labor is doing to improve the quality of  
performance data and better understand program impact and	 
outcomes for veterans.						 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-07-594 					        
    ACCNO:   A69908						        
  TITLE:     Veterans' Employment and Training Service: Labor Could   
Improve Information on Reemployment Services, Outcomes, and	 
Program Impact							 
     DATE:   05/24/2007 
  SUBJECT:   Data collection					 
	     Employment assistance programs			 
	     Performance measures				 
	     Program evaluation 				 
	     Program management 				 
	     Reporting requirements				 
	     State-administered programs			 
	     Strategic planning 				 
	     Training utilization				 
	     Veterans benefits					 
	     Veterans education 				 
	     Veterans employment programs			 
	     Performance management				 
	     DOL Disabled Veterans' Outreach Program		 
	     DOL Local Veterans' Employment			 
	     Representative Program				 
                                                                 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-07-594

   

     * [1]Results in Brief
     * [2]Background

          * [3]Jobs for Veterans Act Reforms
          * [4]Changes to the DVOP and LVER Performance Information
          * [5]Data Reporting
          * [6]The Role of Impact Evaluations in Assessing Program Outcomes

     * [7]DVOP and LVER Performance Information Is Weakened by Several

          * [8]The New Performance Measures Provide Information on Some Out
          * [9]Performance Measures and the Data Reported to Labor Do Not R
          * [10]Performance Data Are Not Comparable over Time or across Stat

     * [11]Available Data Paint an Unclear Picture of Veterans' Use of

          * [12]Veterans Receive Services from Multiple Programs in the One-
          * [13]Data on Veterans' Services Are Well Integrated across Some b
          * [14]Different Veteran Definitions Complicate Efforts to Standard
          * [15]Labor Has No Means of Evaluating Priority of Service for Vet

     * [16]Labor Has Taken Steps to Better Understand Veterans' Outcome

          * [17]Labor Has Developed Some Processes to Enhance the Quality of
          * [18]Labor's Proposed Integrated Data System Could Improve the Un
          * [19]Little Is Known about the Impact of One-Stop Services, inclu

     * [20]Conclusions
     * [21]Recommendations for Executive Action
     * [22]Agency Comments and Our Evaluation
     * [23]Survey
     * [24]Site Visits
     * [25]Research and Experts
     * [26]GAO Contact
     * [27]Staff Acknowledgments
     * [28]GAO's Mission
     * [29]Obtaining Copies of GAO Reports and Testimony

          * [30]Order by Mail or Phone

     * [31]To Report Fraud, Waste, and Abuse in Federal Programs
     * [32]Congressional Relations
     * [33]Public Affairs

Report to Congressional Committees

United States Government Accountability Office

GAO

May 2007

VETERANS' EMPLOYMENT AND TRAINING SERVICE

Labor Could Improve Information on Reemployment Services, Outcomes, and
Program Impact

GAO-07-594

Contents

Letter 1

Results in Brief 3
Background 5
DVOP and LVER Performance Information Is Weakened by Several Factors 11
Available Data Paint an Unclear Picture of Veterans' Use of One-Stop
Services 18
Labor Has Taken Steps to Better Understand Veterans' Outcomes, but the
Programs' Impact Remains Unknown 25
Conclusions 31
Recommendations for Executive Action 31
Agency Comments and Our Evaluation 32
Appendix I Objectives, Scope, and Methodology 34
Appendix II Summary of State Performance in the DVOP and LVER Programs for
Benchmark Program Year 2005 and Negotiated Goals for Program Year 2006 37
Appendix III Summary of State-Negotiated Goals and Performance for
Veterans in the Employment Service, Program Year 2005 42
Appendix IV Comments from the Department of Labor 44
Appendix V GAO Contact and Staff Acknowledgments 46
Related GAO Products 47

Tables

Table 1: Selected Responsibilities of DVOP Specialists and LVER Staff 8
Table 2: Common Measures Adopted by Labor's Employment and Training
Programs 9
Table 3: Performance Measures for the DVOP and LVER Programs 12
Table 4: DVOP Specialists and LVER Staff Served Similar Proportions of
Recently Separated and Disabled Participants in Program Year 2005 14
Table 5: Differences between Veteran Definitions in Employment and
Training Programs 22
Table 6: States and Local Areas in Our Study 36
Table 7: Summary of State Performance in the DVOP Program for Benchmark
Program Year 2005 and Negotiated Goals for Program Year 2006 38
Table 8: Summary of State Performance in the LVER Program for Benchmark
Program Year 2005 and Negotiated Goals for Program Year 2006 40
Table 9: State-Negotiated Goals and Performance for Veterans in the
Employment Service, Program Year 2005 42

Figures

Figure 1: A Large Proportion of Disabled and Recently Separated Veterans
Are Not Captured in Performance Measures for Those Populations (Program
Year 2005) 15
Figure 2: Changes to Performance Information over Time for the DVOP and
LVER Programs 17
Figure 3: Veterans Can Access Multiple Employment Services in the One-Stop
System 19
Figure 4: Many States Report That DVOP Specialists or LVER Staff See Half
or Fewer of Veterans That Come into One-Stop Centers 20
Figure 5: Percentage of States Responding that Different Definitions
Affect the Coordination of Services to Veterans 23
Figure 6: Most State Workforce Agencies Report That Their State Systems
Take Certain Steps to Ensure the Accuracy and Reliability of Employment
Service, DVOP, and LVER Data 28

Abbreviations

DVOP Disabled Veterans' Outreach Program
EER entered employment rate
EMILE ETA Management Information and Longitudinal Evaluation system
ERR employment retention rate
ETA Employment and Training Administration
JVA Jobs for Veterans Act
LVER Local Veterans' Employment Representative
NASWA National Association of State Workforce Agencies
OMB Office of Management and Budget
TAA Trade Adjustment Assistance
UI Unemployment Insurance
VA Department of Veterans Affairs
VETS Veterans' Employment and Training Service
WIA Workforce Investment Act
WIASRD Workforce Investment Act Standardized Record Data
WISPR Workforce Investment Streamlined Performance Reporting System
WRIS Wage Record Interchange System

This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.

United States Government Accountability Office
Washington, DC 20548

May 24, 2007

The Honorable Daniel K. Akaka
Chairman
The Honorable Larry E. Craig
Ranking Member
Committee on Veterans' Affairs
U.S. Senate 

The Honorable Bob Filner
Chairman
Committee on Veterans' Affairs
House of Representatives

Approximately 700,000 veterans are unemployed in any given month, and
roughly 200,000 service members leave active duty and transition to
civilian life each year, according to the Department of Labor (Labor).
While their unemployment rate is similar to that of the general
population, some veterans have special needs in transitioning to the
civilian labor market. Labor has long provided assistance to veterans to
help them succeed in this endeavor. In 2002, Congress enacted the Jobs for
Veterans Act (JVA) to improve employment and training services for
veterans by better integrating them into the comprehensive service
delivery system, or one-stop system, created under the Workforce
Investment Act (WIA) of 1998. JVA modified two Labor programs that
specifically target veteran job seekers and that are administered by the
Veterans' Employment and Training Service (VETS)--the Disabled Veterans'
Outreach Program (DVOP) and the Local Veterans' Employment Representative
(LVER) program--and consolidated the funding into a single grant to
states. These two programs were funded at about $155 million in fiscal
year 2006, and served over 715,000 veterans nationwide in the most recent
program year. JVA addressed concerns raised by some that the programs were
overly prescriptive and did not provide states the flexibility to
determine the best way to serve veteran job seekers. Instead, JVA
identified broad roles and responsibilities of DVOP and LVER staff while
giving states flexibility to determine the number of staff for each
program. In addition, the law required the implementation of a
comprehensive performance accountability system that included performance
measures for the two programs--consistent with those under WIA--and
enhanced accountability for veteran services in the one-stop system.
Moreover, many veterans receive reemployment services from programs other
than Approximately 700,000 veterans are unemployed in any given month, and
roughly 200,000 service members leave active duty and transition to
civilian life each year, according to the Department of Labor (Labor).
While their unemployment rate is similar to that of the general
population, some veterans have special needs in transitioning to the
civilian labor market. Labor has long provided assistance to veterans to
help them succeed in this endeavor. In 2002, Congress enacted the Jobs for
Veterans Act (JVA) to improve employment and training services for
veterans by better integrating them into the comprehensive service
delivery system, or one-stop system, created under the Workforce
Investment Act (WIA) of 1998. JVA modified two Labor programs that
specifically target veteran job seekers and that are administered by the
Veterans' Employment and Training Service (VETS)--the Disabled Veterans'
Outreach Program (DVOP) and the Local Veterans' Employment Representative
(LVER) program--and consolidated the funding into a single grant to
states. These two programs were funded at about $155 million in fiscal
year 2006, and served over 715,000 veterans nationwide in the most recent
program year. JVA addressed concerns raised by some that the programs were
overly prescriptive and did not provide states the flexibility to
determine the best way to serve veteran job seekers. Instead, JVA
identified broad roles and responsibilities of DVOP and LVER staff while
giving states flexibility to determine the number of staff for each
program. In addition, the law required the implementation of a
comprehensive performance accountability system that included performance
measures for the two programs--consistent with those under WIA--and
enhanced accountability for veteran services in the one-stop system.
Moreover, many veterans receive reemployment services from programs other
than the DVOP and LVER, and JVA stipulated that veterans served by these
other programs be given preference over nonveterans, as long as they meet
the eligibility requirements. These programs--administered by a separate
office within Labor, the Employment and Training Administration
(ETA)--include the WIA Adult and Dislocated Worker programs, the
Wagner-Peyser-funded Employment Service, and other key employment
programs.

To implement JVA, VETS adopted a set of common performance measures in
July 2005, similar to those adopted by other employment and training
programs. Each quarter, states report a wide range of data to Labor on
demographics, services, and outcomes for veterans in the DVOP and LVER
programs. States are specifically held accountable for certain
employment-related outcomes for all veterans--whether veteran participants
get and keep jobs--and separately, for the outcomes of some disabled and
recently separated veterans. JVA requires Labor to include information in
its annual report to Congress on whether veterans are receiving priority
and are being fully served by employment and training programs. However,
questions have been raised about whether available performance information
accurately reflects services and outcomes for veterans. Moreover, outcome
information alone cannot be used to evaluate the effectiveness--or
impact--of employment services for veterans. Program outcomes may be
attributable to factors other than a program's services, such as local
labor market conditions. In view of these concerns and the changes to the
performance accountability system resulting from the Jobs for Veterans
Act, you requested that we review the performance information collected on
employment and training programs that serve veterans. Specifically, we
examined (1) the extent to which DVOP and LVER performance information
reflects services and outcomes for veterans served by these programs; (2)
the extent to which performance information on veterans served by other
key programs paints a clear picture of veterans' use of one-stop services
and (3) what Labor is doing to improve the quality of performance data and
better understand outcomes for veteran job seekers.

To determine the extent to which available performance information
reflects services and outcomes for veteran job seekers, we administered a
nationwide Web-based survey to state workforce administrators in all 50
states and the District of Columbia. We received responses from all 51
administrators. In addition, we analyzed performance data from the
relevant programs and reviewed Labor's program guidance. To better
understand state and local variation in employment programs serving
veterans, we conducted site visits to three states: New Hampshire,
California, and Tennessee. We selected these states based on a range of
criteria, including geographic dispersion, state size and veteran
demographics, recent state performance in veterans' programs, and
recommendations by Labor and the National Association of State Workforce
Agencies. Our site visits included interviews with state workforce agency
officials, the federal Director of Veterans' Employment and Training for
each state, one-stop managers, and DVOP and LVER program staff.
Additionally, we interviewed Labor officials from both VETS and ETA. As
part of this effort, we conducted interviews with all six Regional
Administrators of Veterans' Employment and Training to help understand
regional variation in the programs. Finally, we reviewed literature on
attributes of successful performance measures, including our previous
work.^1 For a detailed discussion of our scope and methodology, see
appendix I. We conducted our review from May 2006 to April 2007 in
accordance with generally accepted government auditing standards.

Results in Brief

Performance measures and data for the DVOP and LVER programs generally
reflect veterans' services and outcomes, but are weakened by several
factors. The new performance measures, in place since July 2005, provide
information on some outcomes for veterans, but have not been fully
implemented. For example, states are held accountable for helping veterans
get and keep jobs, but are not yet held accountable for veterans' average
earnings, as they are for other employment and training programs.
Additionally, the current performance measures do not include a method to
weight the successful outcomes of veterans who are harder to serve, such
as those with barriers to employment, as required by JVA. Labor developed
but has not implemented such a method--which could better reflect the
difficulty of serving these veterans--because the system was seen as too
complicated for field staff. In addition, neither the performance measures
nor the data reported to Labor reflect the full range of services that
DVOP and LVER staff provide to veterans. For example, the current measures
hold only DVOP specialists accountable for disabled veterans' employment
and retention, but LVER staff also serve a large number of disabled
veterans who are not counted in those measures. Similarly, only LVER staff
are measured on outcomes for recently separated veterans, although DVOP
specialists serve roughly the same number of recently separated veterans.
Also, while states collect a wide range of performance data on services
and outcomes for veterans, the data reported to Labor do not currently
include information on outreach to employers, a key program activity.
Furthermore, it is difficult to assess outcomes over time, in part because
of frequent changes in states' reporting requirements that prevent
establishing reliable trend data.

^1See the following GAO reports: The Results Act: An Evaluator's Guide to
Assessing Agency Annual Performance Plans, [34]GAO/GGD-10.1.20 (Washington
D.C.: April 1998); Agencies' Annual Performance Plans under the Results
Act: An Assessment Guide to Facilitate Congressional Decisionmaking,
GAO/GGD/AIMD-10.1.18 (Washington, D.C: February 1998); Executive Guide:
Effectively Implementing the Government Performance and Results Act,
GAO/GGD-96-118 (Washington, D.C.: June 1996).

Labor's data on services and outcomes for veteran job seekers paint an
unclear picture of veterans' use of employment and training services in
the one-stop system. Despite the shared use of common performance
measures, key employment and training programs vary in the extent to which
their data on veteran participants are integrated or shared with other
programs. As a result, many states may not know how many veterans they
serve through the one-stop system. For example, most states reported that
veteran job seekers receive initial assistance from the Employment
Service, which uses the same reporting system as the DVOP and LVER
programs and produces separate quarterly reports on services and outcomes
for veterans. However, states use a different reporting system for WIA,
and fewer than half the states told us that they do not match WIA and
Employment Service records to obtain an unduplicated count of veterans
served by those programs. In addition, Labor and some state officials we
surveyed reported that statutory differences in the way veterans are
defined for purposes of program eligibility make it difficult to
standardize data across employment programs. For example, a veteran is
generally defined for the DVOP and LVER programs as an individual who
served on active duty for more than 180 days, while WIA does not specify a
length of time in service. About half of states claimed that the
conflicting definitions in various employment programs complicate data
entry, referrals to other programs, and the implementation of priority of
service. Moreover, Labor has no method of gauging the extent to which
priority of service for veterans has been implemented in various
employment programs, despite JVA's requirement that the agency include
this information in its annual report to Congress.

Labor has taken some steps to improve the quality of performance data and
better understand veterans' services and outcomes, but the overall impact
of employment services for veterans is unknown. Labor has developed
processes to enhance data quality. For example, ETA requires states to
validate some key performance data reported for Employment Service and
WIA-funded programs. In addition, ETA and VETS have begun to work together
on issues of data quality by, for example, conducting joint monitoring
visits to states that include an examination of the data. Furthermore,
Labor's planned implementation of an integrated data reporting system
could greatly enhance the understanding of veterans' services and
outcomes. The new system is designed to integrate and expand data
reporting across multiple employment programs, including the Employment
Service, WIA, Trade Adjustment Assistance (TAA), and the DVOP and LVER
programs. Using this system, for the first time, Labor and states would be
able to track individual veterans' progress through different programs in
the one-stop system. However, according to a Labor official, states have
expressed concerns about the time needed to implement such a system, and
the timeline for implementation remains unclear. Furthermore, while
outcome information on veteran job seekers helps assess whether
individuals are achieving their intended goals--such as obtaining
employment--it cannot measure whether the outcomes are a direct result of
program participation, rather than external factors. Labor has sponsored
research on services to veterans. However, it has not conducted an impact
evaluation, as required under WIA, to assess the effectiveness of one-stop
services. Such a study should include impacts for key participant groups,
including veterans. We recommended in a prior report that Labor take steps
to conduct such an evaluation, but there has been no action to date.

To improve performance information on veteran job seekers, we are making a
number of recommendations to Labor regarding changes to the performance
measures for the DVOP and LVER programs, standardization of the different
veteran definitions in employment programs, guidance on Labor's proposed
integrated data-reporting system, and inclusion of veterans' services in
an impact evaluation. In its comments, Labor generally concurred with our
findings, conclusions, and recommendations.

Background

The nation's veteran job seekers receive employment and training services
from programs overseen by two agencies within Labor--the Veterans'
Employment and Training Service and the Employment and Training
Administration. General employment services fall under the purview of ETA,
which administers the Wagner-Peyser-funded Employment Service program,
providing a national system of public employment services to all
individuals seeking employment--including veterans. Thus, those veterans
considered job ready and not in need of intensive services may be served
by Employment Service staff and receive such services as assessment,
counseling, job readiness evaluation, and placement. ETA carries out its
Employment Service program through workforce agencies in each state. In
fiscal year 2006, the Employment Service program provided a total of about
$716 million to states.

While ETA administers programs that serve the general population,
including veterans, VETS administers the DVOP and LVER programs, which
focus exclusively on serving veterans, often providing more intensive
services than the Employment Service does. Like ETA, VETS carries out its
responsibilities through a nationwide network that includes representation
in each of Labor's six regions and staff in each state. The Office of the
Assistant Secretary for VETS administers the agency's activities through
regional administrators and state directors. The DVOP specialists and LVER
staff, whose positions are funded by VETS, are part of states' public
employment services. In fiscal year 2006, the DVOP and LVER programs were
funded at about $155 million. In the most recent program year--program
year 2005, which spanned July 1, 2005, to June 30, 2006--the Employment
Service, together with the DVOP and LVER programs, reported serving about
1.32 million veterans nationwide, of whom over 715,000 were served by DVOP
specialists and LVER staff.

The Employment Service and the DVOP and LVER programs are mandatory
partners in the one-stop system under WIA--where services are provided by
a range of employment and training programs in a single location.
Veterans, along with other eligible job seekers, may receive services from
other mandatory one-stop partners, such as WIA-funded training or Trade
Adjustment Assistance.^2 Additionally, job seekers, including veterans,
may use the one-stop centers' computers and other resources without staff
assistance, and in many places may access one-stop services online from
home. Department of Veterans Affairs (VA) programs are not mandatory
partners in the one-stop system, but do participate at some locations.

Jobs for Veterans Act Reforms

In 2002, the Jobs for Veterans Act amended Title 38 of the U.S. Code--
which governs the DVOP and LVER programs--and by doing so, introduced an
array of reforms to the way employment and training services are provided
to veterans.^3 JVA sought to address concerns that the programs were
overly prescriptive by providing states with enhanced flexibility to
determine the best way to serve veteran job seekers. Among its reforms,
JVA

^2The Trade Adjustment Assistance program is the primary federal
employment and training program serving workers from the manufacturing
sector who are dislocated due to trade.

           o redefined the DVOP specialist and LVER staff roles but gave
           states flexibility in deciding their duties;
           o established a single state grant and a new funding formula that
           allowed states to determine the mix of DVOP specialists and LVER
           staff;
           o required a comprehensive performance accountability system
           consistent with WIA performance measures;
           o required that veterans receive priority over other job seekers
           in all Labor job training programs, not just the Employment
           Service;^4 and
           o required that VETS include information in its annual report to
           Congress on employment services to veterans throughout the
           one-stop system.

JVA identified broad roles and responsibilities of DVOP specialists and
LVER staff. For example, DVOP specialists are to focus on providing
intensive services to eligible veterans, giving priority to disabled
veterans and those with other barriers to employment. LVER staff are to
focus on conducting outreach to employers to assist veterans in gaining
employment, as well as facilitating employment, training, and placement
services given to veterans. State workforce agencies receive a single
veterans' program grant to fund both programs; the amount each state
receives is determined in part by the size of the veteran population
within each state. State agencies then decide how to distribute the amount
they receive between the two programs. Table 1 lists selected
responsibilities of DVOP specialists and LVER staff as set forth in Labor
guidance.

^3Prior to JVA, Title 38 provided that there was to be one DVOP for each
7,400 veterans in a state and prescribed 11 functions for DVOP to carry
out in providing services to eligible veterans. Similarly, Title 38
formerly provided that, in any fiscal year, funding should be available
for 1,600 full-time LVER staff and prescribed 13 functions to be performed
by the LVER staff.

^4Veterans were required to receive priority of service in the Employment
Service before JVA was enacted.

Table 1: Selected Responsibilities of DVOP Specialists and LVER Staff

                    DVOP specialists           LVER staff                     
Unique              o Facilitate intensive     o Promote veterans' skills  
responsibilities    services to veterans       and experience and advocate 
                       with special employment    for veterans with employers 
                       and training needs         through activities such as  
                       o Target services to       job fairs                   
                       disabled veterans and      o Facilitate and maintain   
                       veterans with other        regular contact with        
                       barriers to employment,    employers                   
                       such as homeless                                       
                       veterans                                               
Shared           Both DVOP specialists and LVER staff may
responsibilities                            
                       o Provide a full range of employment and training
                       services to veterans    
                       o Facilitate employment workshops for those leaving
                       the military            

Source: GAO analysis of Labor data.

JVA also stipulated that veteran job seekers must receive priority over
other job seekers in any job training program administered by Labor.
Labor's guidance requires states to explain how veterans will be given
priority and how veterans' services will be provided through the state's
one-stop system. For programs that target particular populations, such as
seniors or low-income individuals, veterans' priority is applied after any
other mandatory eligibility provisions are met.

Changes to the DVOP and LVER Performance Information

Like other Labor employment and training programs, the DVOP and LVER
programs have experienced changes both in the way outcomes are tracked and
in the measures used to assess performance. Specifically, in 1998, WIA
required that states use automated unemployment insurance wage records to
track employment-related outcomes. Formerly, to obtain data on outcomes,
states relied on a manual follow-up process using administrative records
or contacts with job seekers. To conform to WIA, VETS moved from such a
manual follow-up system to the new automated process in 2002.

The measures that Labor uses to assess performance in the DVOP and LVER
programs have also changed over time, gradually reflecting more emphasis
on outcome-based measures. Before passage of the JVA in 2002, for example,
some of the measures used for the DVOP and LVER programs focused more on
services received--such as the number of veterans in training or receiving
counseling--than on outcomes achieved. In 2002, JVA required that Labor
develop a comprehensive performance accountability system and required
that the new system measure performance in a way that is consistent with
WIA. In 2003, VETS adopted performance measures for the DVOP and LVER
programs based on those then used in WIA. In 2005, in response to an
Office of Management and Budget (OMB) initiative, Labor began requiring
states to implement common performance measures for its employment and
training programs, including the DVOP and LVER programs, the Employment
Service, and WIA. OMB established a set of common measures to be applied
to most federally funded job training programs that share similar goals.
Labor further defined the common measures for all of its Employment and
Training Administration programs, applying three measures to each of its
adult programs (see table 2).

Table 2: Common Measures Adopted by Labor's Employment and Training
Programs

Common measure          Definition                                         
Entered employment rate The number of participants employed in the first   
                           quarter after exiting the program, divided by the  
                           total number of participants who exit the program  
                           during the quarter.                                
Employment retention    Of those participants who are employed in the      
rate                    first quarter after exiting the program, the       
                           number employed in both the second and third       
                           quarters after exit, divided by the number of      
                           participants employed in the first quarter after   
                           the quarter of exit.                               
Average earnings        Of those participants who are employed after the   
                           first, second, and third quarter after the exit    
                           quarter, total earnings in the second quarter plus 
                           total earnings in the third quarter after the exit 
                           quarter divided by the number of adult             
                           participants who exit during the quarter.          

Source: Labor.

In applying the common measures to its programs, VETS also developed
additional measures to emphasize outcomes for disabled veterans in the
DVOP program and outcomes for recently separated veterans in the LVER
program.

Data Reporting

Labor collects performance data for the DVOP and LVER programs on a
quarterly basis from state workforce agencies. The state agencies use
report formats developed by Labor to provide detailed tabulations of
aggregate information on the characteristics of veteran participants,
services, and outcomes for the two programs, including data showing
states' performance using the common measures. The state agencies provide
this information to Labor in three separate reports: one for the DVOP
program, one for the LVER program, and one representing an unduplicated
count for both programs.

Furthermore, Labor collects additional information on veterans who
participate in other Labor programs. For example, ETA collects performance
data for the Employment Service on all participants on a quarterly basis
from state workforce agencies, and these reports break out services and
outcomes for veteran participants. States submit their quarterly reports
for the Employment Service and the DVOP and LVER programs through the same
Labor reporting system.

The Role of Impact Evaluations in Assessing Program Outcomes

Information on the services a program has provided and the outcomes
obtained by program participants are necessary to assess program impacts.
However, this information is not sufficient to measure program
impacts--the outcomes may be due to other external factors such as local
labor market conditions. While impact evaluations allow one to isolate a
program's effect on the outcomes of participants, there are several
approaches to conducting such evaluations. The experimental method is
often considered the most rigorous method for conducting impact
evaluations. In the experimental method, participants are randomly
assigned to two groups--one that receives a program service (or treatment)
and one that does not (control group). The resulting outcome data on both
groups are compared and the difference in outcomes between the groups is
taken to demonstrate the programs impact. However, it is not always
feasible to use the experimental method for assessing program impacts.
Alternatively, researchers may use a quasi-experimental approach in which
program participation is not randomly assigned. One approach, often called
a comparison group study, compares outcome data for individuals who
participated in the program with data on others who did not participate
for various reasons. In a comparison group study, it is important to find
ways to minimize, or statistically control for, any differences between
the two groups. According to OMB, well-matched comparison group studies,
under certain circumstances, can approach the rigor of the experimental
method, and it recommends considering this method if random assignment is
not feasible or appropriate.

Under WIA, Labor was required to conduct at least one impact evaluation of
program services by 2005. In a 2004 report, we found that Labor had not
yet begun such an evaluation, and recommended that the agency comply with
this statutory requirement and help federal, state, and local policy
makers understand what services are most effective for improving
employment-related outcomes.^5

DVOP and LVER Performance Information Is Weakened by Several Factors

The DVOP and LVER programs' performance information is weakened by several
factors, including implementation challenges and frequent changes to
performance reporting requirements. In July 2005, Labor implemented new
performance measures, which provide information on some outcomes for
veterans. However, not all performance measures have been fully
implemented. Additionally, neither the performance measures nor the data
reported to Labor reflect the full range of services that DVOP specialists
and LVER staff provide to veteran job seekers. Furthermore, it is
difficult to assess outcomes over time or across states because of
frequent changes in states' reporting requirements that prevent
establishing reliable trend data.

The New Performance Measures Provide Information on Some Outcomes for Veterans,
but Have Not Been Fully Implemented

In July 2005, the DVOP and LVER programs adopted the Office of Management
and Budget's common measures, along with other employment programs,
including WIA and the Employment Service. Specifically, states implemented
measures that track whether veterans obtain and keep jobs after receiving
services through these programs, but they have not yet implemented a
measure to track veterans' earnings. States are held accountable for four
separate measures in each program that focus on outcomes attained by
veterans (see table 3). For the DVOP program, states are held accountable
for employment and retention for all veterans served by the program, as
well as for disabled veterans. For the LVER program, states are assessed
on employment and retention for all veterans, as well as for recently
separated veterans. Currently, all states collect and report data to Labor
for calculating performance attainment and negotiating state goals for
these eight measures.^6

5GAO, Workforce Investment Act: States and Local Areas Have Developed
Strategies to Assess Performance, but Labor Could Do More to Help,
[35]GAO-04-657 (Washington, D.C.: June 1, 2004).

^6See appendix II for a state-by-state list of negotiated and achieved
performance goals for the programs.

Table 3: Performance Measures for the DVOP and LVER Programs

                                                                Recently
                                                Disabled        separated
Program Type of measure      All veterans    veterans        veterans
DVOP    Entered employment       SQRT          SQRT               
           rate                                                      
           Employment retention     SQRT          SQRT               
           rate                                                      
LVER    Entered employment       SQRT                          SQRT
           rate                                                      
           Employment retention     SQRT                          SQRT
           rate                                                      

Source: Labor.

However, states are not yet held accountable for an additional common
measure--veterans' average earnings--in either the DVOP or the LVER
programs. Other employment and training programs, such as WIA and the
Employment Service, include an average earnings measure for which states
are accountable. For the DVOP and LVER programs, however, calculating the
average earnings was not as straightforward as Labor had anticipated. A
VETS official told us that the agency will calculate baseline data for
average earnings during the current program year, but Labor will not
establish goals and states will not be held accountable for their
performance on this measure until the following year--program year
2007--at the earliest.

Furthermore, Labor has not adopted a system to give more weight to
successful outcomes for veterans who have substantial barriers to
employment, such as a disability. JVA required Labor to weight performance
measures to provide special consideration to veterans requiring intensive
services, as well as disabled veterans. Such a weighting system would
compensate for the fact that veterans with barriers to employment may need
more assistance than others in finding jobs. It would also provide an
incentive for program staff to help veterans with severe barriers to
employment. For example, if a veteran has a disability and requires
intensive case management services, his or her successful outcomes would
have a greater effect on a state's overall performance than those of other
veterans with fewer barriers. Following JVA's enactment, Labor formed a
work group to develop a weighting system for the DVOP and LVER performance
measures. On the basis of the group's work, the agency issued guidance to
introduce the weighted measures to states in June 2003, with the
expectation of implementing them soon after. However, after further
review, a Labor official told us the agency did not implement the weights
in order to give states time to fully implement other reporting changes.
At this time, it is not clear whether Labor will implement this system in
the future.

Performance Measures and the Data Reported to Labor Do Not Reflect the Full
Range of Staff Services

Although DVOP specialists and LVER staff perform similar duties for all
types of veterans in most states, the current performance measures hold
the two programs accountable for different groups of veterans. JVA and
Labor's guidance outline the key responsibilities and target populations
for DVOP specialists and LVER staff, but also allow for some flexibility
in their roles and responsibilities. Both DVOP and LVER staff are expected
to serve the general veteran population, but DVOP specialists are also
expected to target their services toward veterans who have greater
barriers to employment and need intensive case management, including
disabled veterans. JVA specifies that LVER staff focus on conducting
outreach to employers and assisting all veteran job seekers. In addition,
Labor has recently added the expectation that LVER staff focus their
responsibilities on assisting recently separated veterans. As a result of
these expectations, Labor separately holds DVOP specialists accountable
for the outcomes achieved by the disabled veterans they serve, and LVER
staff for the outcomes of the recently separated veterans they serve.

In practice, however, both programs' staff serve similar veteran
populations. In program year 2005, for example, 14 percent of veterans
served by the DVOP program were disabled and 21 percent were recently
separated. For the LVER program, 10 percent of veterans served were
disabled and 19 percent were recently separated (see table 4). States
acknowledged this similarity in our survey as well. Over a third of states
responded that DVOP and LVER staff are equally likely to serve disabled
veterans, while about half of states responded that the two programs'
staff are equally likely to serve recently separated veterans.

Table 4: DVOP Specialists and LVER Staff Served Similar Proportions of
Recently Separated and Disabled Participants in Program Year 2005

                                         Veterans
             Recently separated         Disabled                     
              Number Percentage      Number Percentage  Total served 
Served by             78,843 21  52,331                        14  382,144 
DVOPs                                                                      
Served by             78,379 19  41,847                        10  408,837 
LVERs                                                                      

Source: Labor.

Note: These numbers include 49 states, the District of Columbia, and
Puerto Rico. They do not include Pennsylvania because it is piloting a new
reporting system and currently not submitting the standard quarterly
reports for the DVOP and LVER programs.

In addition to finding similarity in populations served, we also found
some similarity in activities carried out by DVOP and LVER staff. States
reported that the three activities on which DVOP specialists spend the
most time include

           o providing intensive case management services,
           o conducting an initial assessment or interview, and
           o assisting with job search activities.

The top three activities LVER staff perform include

           o conducting outreach to employers,
           o assisting with job search activities, and
           o conducting an initial assessment or interview.

This division of duties appears to reflect the different focuses of the
two programs, as well as the flexibility under JVA for states to decide on
staff duties. However, almost 85 percent of states responded that DVOP
specialists conduct outreach to employers, a focus of the LVER program.
Additionally, almost 60 percent of states responded that LVER staff
provide intensive services, a primary focus of the DVOP program.

In our site visits, we found that this similarity in staff roles and
target populations exists in part because some one-stop centers have only
a single DVOP specialist or LVER staff on duty at any given time. In these
particular one-stop centers, the same employee is responsible for serving
all groups of veterans and carrying out job roles for both programs. Even
in centers with more than one staff person, veterans tend to be served by
whichever staff person is available at that time. Program staff in several
centers told us that recently separated veterans were not specifically
directed to LVER staff for services, nor were disabled veterans directed
to DVOP staff. This sharing of duties may be due, in part, to changes in
staffing levels. More than half of states reported a decrease in the
number of full-time DVOP specialists or LVER staff over the last 2 years,
and most attributed this decline to the size of their state grant for the
programs.

Nevertheless, this similarity in roles and populations served causes the
current performance measures to present an incomplete view of outcomes for
disabled and recently separated veterans in the DVOP and LVER programs.
The large numbers of disabled veterans served by the LVER program and
recently separated veterans served by the DVOP program are not included in
the set of measures that focus on the outcomes of those populations (see
fig. 1).

Figure 1: A Large Proportion of Disabled and Recently Separated Veterans
Are Not Captured in Performance Measures for Those Populations (Program
Year 2005)

Beyond the measures for disabled and recently separated veterans, having
separate measures for the DVOP and LVER programs obscures the overall
picture of outcomes to veterans, given the similarity between many of the
program activities and the reality of how the programs operate. According
to our survey, almost half the states would like Labor to consolidate the
performance measures for the DVOP and LVER programs.

While the performance measures present an incomplete view of the outcomes
for veterans, the data that states are required to report to Labor do not
reflect the full range of staff services. Labor requires states to report
a wide range of data for the DVOP and LVER programs, including information
on veteran characteristics--such as age and disability status--and staff
services provided--such as intensive services and referrals to other
programs. However, Labor does not require data on employer outreach
activities, despite JVA's designation of employer outreach as a primary
job responsibility of LVER staff. Consequently, Labor and states cannot
formally monitor the extent to which staff perform this activity. Some
states, however, collect these data for their own use. According to our
survey, almost half of states currently collect employer-related
information for the DVOP and LVER programs, and over 75 percent of states
reported that it would be helpful to collect these data.

In addition, even though the data reported to Labor generally reflect
services and outcomes for veterans, these data are aggregate tallies and
do not show services provided to individual veterans. For example, each
state's quarterly reports reflect the sum of all services provided and do
not show the number of services provided per veteran or per staff person.
The current data are useful to provide an overall picture of the programs'
volume and operations. However, these data provide little information
about services received by individual veterans or delivered by particular
veteran staff.

Performance Data Are Not Comparable over Time or across States

In recent years, reporting requirements for the DVOP and LVER programs
have undergone several significant changes. These changes have moved the
performance accountability system closer to those of other employment and
training programs. At the same time, the changes have resulted in a lack
of reliable trend data. In July 2002, the DVOP and LVER programs changed
from using administrative follow-up to determine veterans' employment
outcomes to obtaining information from Unemployment Insurance (UI) wage
records. In doing so, Labor changed its method of calculating outcomes for
veterans in the DVOP and LVER programs.^7 Then, in July 2005, Labor
applied the common measures to these two programs, refining and
standardizing the application of UI wage records to determine outcomes.
Under the old system, Labor calculated entered employment and employment
retention rates based on the number of veterans who participated in the
programs. However, under the new system, Labor calculated these rates
based on how many veterans terminate services and exit the programs.
Although these changes have standardized the performance measures across
programs, they have also prevented Labor and states from developing
consistent, comparable data over the past 5 years. As a result, Labor does
not have reliable historic data for either program. Figure 2 illustrates
the various changes to the DVOP and LVER programs' performance reporting
requirements.

^7Because UI wage records only contain aggregate quarterly earnings for
individuals, it was necessary for Labor to change the way the measures
were calculated.

Figure 2: Changes to Performance Information over Time for the DVOP and
LVER Programs

Furthermore, the instability in data collection and reporting has left
Labor unable to establish a national veterans' entered employment
standard, as required by JVA. Labor anticipates that it will need at least
3 years of stable data to establish the national standard. Once it is
established, all states will be held accountable to the same minimum goal
for veterans' entered employment. However, it is unclear when Labor will
have sufficient data to establish this standard because states continue to
experience difficulty adjusting to the numerous changes. According to our
survey, over 70 percent of states reported that frequent changes to
performance reporting requirements have been either a great or very great
challenge.

The data also vary somewhat state by state. For example, the application
of wage records to calculate veteran outcomes across state lines is no
longer consistent across states. The Wage Record Interchange System (WRIS)
allows states to share UI wage records and account for job seekers who
participate in one state's employment programs but get jobs in another
state. In recent years, all states but one participated in WRIS, which was
operated by the nonprofit National Association of State Workforce
Agencies. In July 2006, Labor assumed responsibility for administering
WRIS. However, many states have withdrawn, in part because of a perceived
conflict of interest between ETA's role in enforcing federal law and the
states' role in protecting the confidentiality of their data. As of March
2007, only 30 states were participating in the program, and it is unknown
if and when the other states will enter the data-sharing agreement. As a
result, DVOP and LVER performance information in almost half the states
will not include employment outcomes for veterans who found jobs outside
the states in which they received services. In addition, other reasons
contribute to data variation by state. Labor allows states flexibility in
choosing data collection software, which has resulted in some states
adapting more quickly than others to the recent changes, depending on
their software capabilities. Several Labor officials told us that because
of differences in software capabilities, some states' data may be more
reliable than others'.

Available Data Paint an Unclear Picture of Veterans' Use of One-Stop Services

Labor's data on veteran job seekers paint an unclear picture of their use
of employment and training services in the one-stop system, despite the
shared use of common performance measures across programs. Although many
veterans use employment services other than those provided by the DVOP and
LVER programs, key employment programs vary in how well their data on
veteran participants are integrated or shared with other programs. As a
result, many states may not know how many veterans use one-stop services.
In addition, statutory differences in the way veterans are defined for
purposes of program eligibility make it difficult to standardize data
across employment programs. Moreover, Labor has no means of assessing
whether priority of service for veterans has been implemented in various
employment programs.

Veterans Receive Services from Multiple Programs in the One-Stop System

Many veteran job seekers receive employment services from the DVOP and
LVER programs. However, some veterans--often the more job-ready--only use
one-stop services aimed at the general population, such as the Employment
Service and WIA programs. In addition, some veterans use services focused
on other subsets of job seekers--such as TAA (see fig. 3). As a result,
performance information on many veterans is collected and reported
elsewhere in the one-stop system. In fact, 20 states reported that about
half or fewer of veteran job seekers who access employment programs
receive services from a DVOP specialist or LVER staff, according to our
survey (see fig. 4). In addition, some veterans obtain services from more
than one employment program in the one-stop system, all of which use the
common measures to assess their performance.

Figure 3: Veterans Can Access Multiple Employment Services in the One-Stop
System

Note: The figure reflects only the services provided by some of the
mandatory partners in the one-stop system. Other optional partners may
participate, but the specific optional partners vary by location.

Figure 4: Many States Report That DVOP Specialists or LVER Staff See Half
or Fewer of Veterans That Come into One-Stop Centers

Data on Veterans' Services Are Well Integrated across Some but Not All Programs
in the One-Stop System

Performance data on veteran job seekers are well integrated or shared
across some key employment and training programs, but not others, despite
the mutual use of common measures. As a result, many states may not know
how many veterans they serve through the one-stop system. Data on veterans
who access the Employment Service are completely integrated with data from
the DVOP and LVER programs--they share the same reporting system, and DVOP
and LVER data are a subset of Employment Service data. According to our
survey, veteran job seekers in most states receive initial assistance from
the Employment Service when they access the one-stop system. If they are
subsequently referred to the DVOP and LVER programs, all of their
information is housed in the same system and an unduplicated count of
veterans served between these programs can be obtained. In addition,
states are held accountable for meeting separate goals in the Employment
Service for veterans and disabled veterans (see app. III). Labor considers
these measures to reflect veterans' outcomes for the entire one-stop
system, as they constitute outcomes for all veterans who access the
Employment Service, DVOP, and LVER programs. Furthermore, they are the
best approximation of a total count of veterans who access the one-stop
system that the current data will allow.

On the other hand, data on veterans served by other one-stop programs are
not well integrated. States report data to Labor on WIA participants who
exit the programs, including veterans, using the Workforce Investment Act
Standardized Record Data (WIASRD) system. Although WIASRD contains
sufficient information to produce separate veteran outcome data for WIA
programs, states are not required to produce separate veteran reports and
are not accountable for meeting veteran goals in those programs. In
addition, fewer than half the states reported that they routinely match
WIA and Employment Service records to attain an unduplicated count of
veterans served by those programs. Consequently, veterans who access two
different employment services may be counted twice in some cases. Data for
TAA participants are reported to Labor in yet another data system, which
does allow states to report on the veteran status of participants, but
Labor officials told us the agency does not currently use veteran outcomes
from that program for any purpose.^8 VETS does not include the veteran
outcome data from WIA or TAA in its annual report to Congress, and Labor
officials told us they are exploring ways to better use the data.

In addition, data are not always collected on job seekers who use
Employment Service or WIA resources without assistance from program staff.
These self-assisted job seekers--including veterans--access services such
as labor market or career information either in one-stop centers or on
home computers, but do not receive active assistance from program staff.
Historically, some states have collected information on these job seekers,
while others have not. In our survey, 73 percent of states reported that
they capture information on all veterans who receive self-assisted
services through the Employment Service, while 82 percent of states
reported doing so for all veterans who receive self-assisted WIA services.
Labor has encouraged--but not mandated--states to collect information on
this group of job seekers, but agency officials acknowledged that states
continue to vary in how they report such data. Labor officials have
expressed concern that requiring veterans who receive self-assisted
services to register for the programs might discourage some of them from
pursuing the services they need.

^8For more information on TAA outcome data, see GAO, Trade Adjustment
Assistance: Labor Should Take Action to Ensure Performance Data Are
Complete, Accurate, and Accessible, [36]GAO-06-496 (Washington, D.C.: Apr.
25, 2006).

Different Veteran Definitions Complicate Efforts to Standardize Data and
Implement Priority of Service

Labor and some state officials we surveyed reported that statutory
differences in the definitions of veterans for various employment programs
make it difficult to standardize data across programs. For the purposes of
the DVOP and LVER programs, an eligible veteran is statutorily defined as
an individual who served on active duty for more than 180 days.^9 Labor
also uses this definition for the Employment Service. WIA, on the other
hand, does not specify a length of time in service for a person to be
considered a veteran. Moreover, to qualify as a recently separated veteran
in the DVOP and LVER programs, a veteran must have left active duty in the
last 3 years. By contrast, WIA defines recently separated as having left
active duty in the last 4 years (see table 5).

Table 5: Differences between Veteran Definitions in Employment and
Training Programs

                    Length of time in    Recently separated                   
                    service              veteran            Statutory source  
DVOP, LVER, and  Served on active     Up to 3 years      38 U.S.C. 4211    
Employment       duty for a period of after discharge or                   
Service programs more than 180 days   release from                         
                                         active duty                          
WIA programs     No requirement for   Up to 4 years      Workforce         
                    time in service      after discharge or Investment Act of 
                                         release from       1998 (29 U.S.C.   
                                         active duty        2801)             

Source: GAO analysis.

Note: To be eligible, all programs require that veterans be discharged or
released from duty under conditions other than dishonorable.

These inconsistent definitions have been difficult for Labor and states to
reconcile with the concept of seamless service delivery and have caused
some confusion for states as they implement priority of service throughout
the one-stop system. While JVA requires that veterans receive priority
over other job seekers in Labor-funded employment and training programs,
it does not define a veteran for purposes of the priority requirement.
Labor has interpreted JVA's provisions to mean that while veterans are to
receive preference in the programs after any other statutory eligibility
requirements are met, each program must use its own statutory definition
of a veteran in applying that preference. Labor officials told us that one
state applied for a waiver in 2006 to use a single definition of veterans
for all of its employment and training programs, but Labor's Solicitor's
Office orally denied the request. In our survey, approximately half of all
states reported that the conflicting veteran definitions in various
employment programs complicate data entry, referrals to other programs,
and the implementation of priority of service. In addition, about a third
of the states claimed that the definitions created gaps in services for
veteran clients as they moved among employment programs (see fig. 5). For
example, if a veteran receives services from WIA and is subsequently
referred to the DVOP program but is found ineligible, he or she may become
discouraged and stop seeking services altogether.

^9Veterans discharged because of service-connected disabilities and
members of reserve components who have served on active duty during a war
or designated campaign are not subject to a 180-day requirement.

Figure 5: Percentage of States Responding that Different Definitions
Affect the Coordination of Services to Veterans

Almost half of states shared their concerns about different definitions by
providing additional comments in our survey, many of which cited the
difficulty of providing priority of service under these circumstances. For
example, one state responded that different definitions often lead to
inappropriate referrals, resulting in poor customer service and
frustration for program participants and service providers. Other states
focused on the burden that competing definitions placed on data collection
and reporting. For example, one state responded that the issue has made it
difficult to integrate the state's Employment Service and WIA data
systems, because the different definitions could lead to invalidating the
veteran numbers on reports for those programs. Another state cited the
difficulty in assessing how many veterans were served by the state,
highlighting the complexity of producing an unduplicated count of veterans
served by different programs that do not share a single definition. States
also cited challenges in dealing with other agencies that are not mandated
partners in the one-stop system. For example, two states mentioned that
some staff of other agencies' programs may hesitate to refer participants
to the DVOP and LVER programs because they are unsure about participant
eligibility. An expert on veterans' issues in the states concurred that
the different eligibility criteria for veterans has been a problem for
states and told us that a common veteran definition for employment and
training programs would be an improvement.

Labor Has No Means of Evaluating Priority of Service for Veterans

Despite JVA's mandate, Labor has not produced information on the extent to
which veterans receive priority of service in all qualified employment and
training programs. Specifically, JVA required Labor to evaluate and report
on whether veterans are receiving priority of service and are fully served
by its employment programs, as well as whether the representation of
veterans in such programs is in proportion to their participation in the
labor force. In its fiscal year 2005 report, Labor stated that the
participation rate for veterans in its adult programs was approximately
8.4 percent--slightly higher than veterans' participation rates in the
U.S. workforce. In addition, the agency reported that outcomes for
veterans served in these programs closely mirrored those of all job
seekers in the programs. However, Labor has no method of gauging how--and
how consistently--priority of service is actually applied. Labor officials
told us that the highly devolved workforce development system makes it
very difficult to evaluate priority of service, because different programs
have multiple access points and diverse eligibility criteria that prevent
Labor from applying a simple measurement technique to each.

States reported that implementing priority of service has been
challenging, as has holding one-stop partner programs accountable for
serving veterans. To supplement federal guidance on this issue, at least
one state has developed its own guidance for implementing and measuring
priority of service. Some Regional Directors of VETS told us they
encouraged the use of that state's guidance as a model for assessing
priority of service for states in their own regions. We do not know when
Labor will develop further guidance on the issue. However, in December
2006, Congress passed the Veterans Benefits, Health Care, and Information
Technology Act of 2006, which included a requirement that Labor release
regulations on implementing priority of service within 2 years. In
addition, the agency has begun planning a study of priority of service in
response to our prior report.^10 According to Labor officials, the study
will combine a survey of participants with a process evaluation and an
analysis of outcomes. Labor does not yet know when the study will get
under way.

Labor Has Taken Steps to Better Understand Veterans' Outcomes, but the Programs'
Impact Remains Unknown

Labor has taken some steps to improve the quality of performance data and
better understand veterans' services and outcomes, but the overall impact
of employment services for veterans is unknown. Labor has developed some
processes to enhance data quality. For example, Labor's ETA requires
states to validate some data in key programs. Furthermore, Labor plans to
implement an integrated data-reporting system that would allow Labor and
states to track individual veterans' progress through different programs
in the one-stop system. Additionally, the new system would expand data
collection by, for example, collecting more data on services to employers.
However, states have raised concerns about the challenge of meeting the
system's planned implementation date, and the timeline for implementation
remains unclear. Furthermore, while performance information helps assess
whether individuals are achieving their intended outcomes--such as
obtaining employment--it cannot measure whether the outcomes are a direct
result of program participation, rather than external factors. To measure
the effects of a program, it is necessary to conduct an impact evaluation
that would seek to assess whether the program itself led to participant
outcomes. Labor has sponsored research on services to veterans. However,
it has not conducted an impact evaluation, as required under WIA, to
assess the effectiveness of one-stop services. Such a study should include
impacts for key participant groups, including veterans. We recommended in
2004 that Labor take steps to conduct such an evaluation, but there has
been no action to date.

Labor Has Developed Some Processes to Enhance the Quality of Performance Data

Labor has taken some steps to improve the quality of performance data and
enhance the understanding of veterans' services and outcomes. To address
data quality concerns, ETA has developed processes requiring states to
validate certain data reported for participants in WIA and
Wagner-Peyser-funded Employment Service programs. However, while these
programs serve veterans, participant records are randomly selected in both
programs from the total participant population and, therefore, may not
include the records of veteran participants.

^10GAO, Veterans' Employment and Training Service: Labor Actions Needed to
Improve Accountability and Help States Implement Reforms to Veterans'
Employment Services, [37]GAO-06-176 (Washington, D.C.: Dec. 30, 2005).

Both the WIA data validation process, developed in 2004, and the
Employment Service process, developed in 2003, involve two types of data
validation, although the WIA process is more intensive, according to Labor
officials. Both processes involve (1) data element validation--comparing
randomly sampled participant records to source files, and (2) report
validation--assessing whether states' software accurately calculated
performance outcomes. While element validation in WIA is conducted on-site
with hardcopy source documentation, the Employment Service data validation
process is performed centrally and electronically, because Employment
Service records are generally electronic.^11 The Employment Service
element validation process checks for duplicate or invalid entries in
source files by, for example, checking for inconsistencies among various
veteran-related fields, such as veteran status and disabled veteran.
However, the Employment Service element validation process cannot check
the underlying accuracy of the data, because there is no hard copy
documentation to prove whether a participant is in fact a veteran.

Labor officials told us that the Employment Service data validation
process has been helpful in raising awareness among states about the
importance of data quality and that some states have come to see it as a
useful tool. Additionally, states responding to our survey generally
agreed that it has been effective--38 states, or about 75 percent, rated
the Employment Service data validation process as effective in ensuring
the accuracy of veteran job seekers' information. For example, according
to one respondent, review of the data validation results is used as a
management tool, to highlight successes and to alert staff to weaknesses.
Nevertheless, some states have expressed concerns about the data
validation processes. Concerns about the process were also raised by state
officials in all 3 of the states we visited. For example, officials in 2
of the 3 states noted that they had experienced difficulties adjusting to
frequent changes in software before the results were due to Labor. On our
survey, 2 states said that the sample size was too small to be meaningful,
and 4 states expressed concerns about the fact that the process does not
verify the accuracy of the data in source files. These concerns are
similar to those we identified in a previous report that addressed the WIA
process.^12

11For more information on the WIA data validation process, see GAO,
Workforce Investment Act: Labor and States Have Taken Actions to Improve
Data Quality, but Additional Steps Are Needed, [38]GAO-06-82 (Washington,
D.C.: Nov. 14, 2005).

Additionally, Labor has taken steps to address data quality as a part of
its routine monitoring and technical assistance. Specifically, beginning
in 2004, ETA regional staff have incorporated a data quality component
into compliance visits to state offices, which are generally conducted
once or twice a year, according to Labor officials. Data validation is
just one component of these compliance visits, which typically do not
focus on veterans' data as a separate issue. To support this effort, Labor
officials told us that ETA has amended its monitoring guide for these
visits to include a section on data validation. According to Labor
officials, these visits have been useful in identifying problems and
corrective actions. Moreover, ETA and VETS have recently collaborated on a
few of these compliance visits.^13 Labor officials said they believed this
joint monitoring was beneficial, and expect those efforts to be a model
for future joint visits.

There are several other forms of management reviews that generally focus
on services to veterans but also offer a chance to review data. For
example, VETS regional and state-based staff conduct site visits as part
of their routine monitoring, which focus primarily on services to veterans
but which can include reviewing performance information as well.
Additionally, VETS has required a series of annual assessments--of the
program for each state overall, and self-assessments by DVOP specialists,
LVER staff, and one-stop managers--that address data issues to a limited
extent. State directors use performance data to substantiate services
described in the self-assessment. For example, according to one official
we spoke with, to confirm a LVER staff's claim of travel to several job
fairs, the director can consult the one-stop's travel log to substantiate
whether the LVER staff actually made the trips.

Beyond the steps Labor has taken, state workforce agencies also perform
functions that affect performance data on services to veterans. Most
states responding to our survey reported that they have taken certain
steps to ensure the accuracy and reliability of data for the Employment
Service, DVOP, and LVER programs, such as having their systems perform
automated checks for inconsistencies in data or for duplicate veteran
files (see fig. 6).

^12 [39]GAO-06-82 .

^13According to Labor officials, joint compliance visits have been
conducted in three states to date: Arizona, Oregon, and Washington.

Figure 6: Most State Workforce Agencies Report That Their State Systems
Take Certain Steps to Ensure the Accuracy and Reliability of Employment
Service, DVOP, and LVER Data

Labor's Proposed Integrated Data System Could Improve the Understanding of
Services and Outcomes for Veterans

Since 2004, Labor has been planning to implement an integrated data
reporting system that could greatly enhance the understanding of veterans'
services and outcomes. In 2004, Labor first proposed a single, streamlined
reporting system, known as the ETA Management Information and Longitudinal
Evaluation system (EMILE) that would have replaced reporting systems for
several Labor programs. Labor substantially modified this system's design
in response to concerns raised by state and local agencies about the
burden and cost of the new system, as well as the challenge of meeting the
implementation deadline. The modified system, now called the Workforce
Investment Streamlined Performance Reporting System (WISPR), was planned
with a July 2007 implementation date.

WISPR has been designed to both integrate and expand data reporting. If
implemented, the system would integrate data reporting by using
standardized reporting requirements across the Employment Service, DVOP
and LVER, WIA, and TAA programs, and ultimately replace their preexisting
reporting systems with a single reporting structure. Additionally, it
would rely on a standardized set of data elements and quarterly reports to
provide data on participant characteristics and services provided, as well
as performance outcomes based on the common measures. Its integrated
design would, for the first time, allow Labor and states to track
individual veterans' progress through the one-stop system. In addition,
the system would expand data collection and reporting in two key areas:
the services that LVER staff provide to employers, a key aspect of the
LVER role on which Labor currently collects no data, and estimates of the
population of veterans who access the one-stop system but ultimately
receive limited or no services from one-stop staff.

As with EMILE, however, concerns have been raised about challenges in
implementing the new system, and at present, the timeline for WISPR's
implementation remains unclear. Some of the comments received by OMB
during the official comment period noted the challenge of a July 2007
implementation date, according to a Labor official. While states will have
a 2-year period to consolidate reporting on the full range of programs,
they are expected to begin collecting and reporting data in the new format
immediately. As of December 2006, 39 entities, including state workforce
agencies, local agencies and unions, had submitted comments reflecting
their concerns about WISPR to the Office of Management and Budget (OMB).
Of the 20 states that submitted comments, 14 noted that a July 2007
implementation date would represent a challenge. For example, some of them
expressed the view that Labor had underestimated the time states would
need to revise policy, reprogram systems, and retrain staff. In addition,
some states expressed concerns about their ability to provide data on
services to employers. Moreover, two states expressed the concern that
meeting Labor's planned implementation date would have adverse
consequences, such as compromised data quality or cost overruns. OMB's
official review will address the time needed to build the reporting
system's technical infrastructure, and will play a key role in deciding
the system's final implementation schedule, according to a Labor official.
States and local areas will need enough time to fully meet the
requirements of this expanded data collection.

Little Is Known about the Impact of One-Stop Services, including Those to
Veterans

Although Labor has improved its outcome data on job seekers who
participate in its programs, these data alone cannot measure whether
outcomes are a direct result of program participation, rather than
external factors. For example, local labor market conditions may affect an
individual's ability to find a job as much or more than participation in
an employment and training program. To measure the effects of a program,
it is necessary to conduct an impact evaluation that would seek to assess
whether the program itself led to participant outcomes. Labor has not
conducted an impact evaluation of one-stop services, including those to
veterans. However, the department did sponsor a study, issued in 2003,
that examined the relationship between services provided to certain groups
of veterans and employment and earnings outcomes.^14 This study employed a
number of data sources and statistical techniques to learn more about how
veterans were using one-stop services. However, while this study provided
some useful information, it could not determine that these services caused
positive outcomes for veteran job seekers. In addition, the study relied
on data from 8 states and its findings could not be generalized to the
national population of veteran job seekers.

Since the full implementation of WIA in 2000--in which the one-stop system
became the required means to provide employment and training services,
including those to veterans--Labor has not made evaluating the impact of
those services a research priority. While WIA required one such evaluation
by 2005, Labor has declined to fund one in prior budgets. In a 2004
report, we recommended that Labor comply with the requirements of WIA and
conduct an impact evaluation of WIA services to better understand what
services are most effective for improving employment-related outcomes.^15
In response to our report, Labor cited the need for program stability and
proposed delaying an impact evaluation of WIA until any changes that might
be included in reauthorization legislation had been implemented. While
efforts to reauthorize WIA began in 2003, they have stalled and it is not
clear at this time when they will be complete. Furthermore, OMB has also
found Labor's evaluations of WIA services to be lacking. In response, in
its 2008 budget proposal, Labor identified an assessment of WIA's impact
on employment, retention, and earnings outcomes for participants as an
effort the agency would begin. According to Labor officials, the agency
has not yet begun to design the study. Such a study should include impacts
for key participant groups, including veterans. To do so would require a
sufficient sample of veterans to allow such analysis.

^14Battelle Memorial Institute, Assessment of Unemployed Veterans' Needs
for the Department of Labor's Veterans' Employment and Training Service,
PSC Contract No. 282-98-0019, Task Order No. 7, Battelle Project Number
FG465407 (Arlington, Virginia: November 30, 2003).

^15GAO, Workforce Investment Act: States and Local Areas Have Developed
Strategies to Assess Performance, but Labor Could Do More to Help,
[40]GAO-04-657 (Washington, D.C.: June 1, 2004).

Conclusions

At a time when the nation's attention is focused on those who have served
their country, it is vital that Congress and the Administration are able
to make informed decisions about programs that help veterans find and keep
jobs in the civilian labor market. Frequent changes in Labor's performance
accountability system have hampered Labor's ability to produce consistent
and meaningful performance information on veteran job seekers. States and
local areas have had difficulty implementing the constant changes to
performance information, which introduce error and make it difficult to
identify trends that would give Congress a better idea of the programs'
achievements. While the anticipated transition to a new reporting system
represents a promising advance in Labor's ability to track the outcomes of
veterans in the one-stop system, states will need time to effectively
implement the changes to avoid compromising the potential benefits--such
as improved data quality--of the system.

Furthermore, the current separate performance measures for the DVOP and
LVER programs do not account for the considerable similarity in veteran
populations served by DVOP specialists and LVER staff, and thus do not
provide an accurate picture of outcomes for veterans served by these two
programs. Using the existing measures, Labor also cannot ensure that
performance outcomes give more weight to services for veterans with
greater barriers to employment. In addition, different veteran definitions
in other programs could make it difficult to analyze services to veterans
throughout the one-stop system. Further, Labor cannot provide assurance
that veterans are appropriately given service priority by programs in the
one-stop system, or that services to veterans are truly effective. The
federal government spends about $155 million each year on the DVOP and
LVER programs alone, not counting the amounts spent on veterans who use
other one-stop programs, but there is no information on whether these
programs have an impact in helping this important population. Establishing
a means to gauge the programs' impact would require a considerable
investment of time and money, but would contribute greatly to the
understanding of whether current employment and training services are
meeting veterans' needs. Furthermore, we continue to urge Labor to meet
WIA requirements and our 2004 recommendation to conduct an impact
evaluation of one-stop services.

Recommendations for Executive Action

To provide a better picture of services and outcomes for veteran job
seekers, improve program reporting, and facilitate priority of service, we
recommend that the Secretary of Labor

           o ensure that states are given adequate direction and sufficient
           time to implement ETA's planned integrated data reporting system
           and make necessary changes;

           o consolidate all performance measures for the DVOP and LVER
           programs, including those for disabled and recently separated
           veterans;

           o comply with JVA's requirement to implement a weighting system
           for the DVOP and LVER performance measures that takes into account
           the difficulty of serving veterans with particular barriers to
           employment;

           o develop legislative proposals for appropriate changes to the
           definitions of veterans across employment and training programs to
           ensure consistency; and

           o ensure that Labor moves forward with an impact evaluation for
           the one-stop system under WIA as we recommended in 2004, and that
           the evaluation's sampling methodology includes veterans in
           sufficient numbers to allow analysis of the impact of services to
           veterans in the one-stop system, including those served by the
           DVOP and LVER programs.

Agency Comments and Our Evaluation

We provided a draft of this report to Labor for review and comment. In its
comments, Labor generally concurred with our findings, conclusions, and
recommendations and expressed appreciation that the report acknowledges
the steps the agency has taken to improve the quality of performance data
and better understand outcomes for veterans. Labor noted that it is
considering adopting a different approach to measuring outcomes for the
DVOP and LVER programs by program year 2008--one that may take into
account the similar veteran populations served, as well as outreach to
employers. As it develops this new approach, Labor reported that it will
also introduce a system of weighted measures that will emphasize services
to veterans with barriers to employment. These changes will coincide with
the implementation of Labor's proposed integrated data system, WISPR.
Labor also noted that it would work with states and grantees to ensure a
smooth transition to the new system. In addition, Labor stated that it
intends to pursue a WIA impact evaluation, which will allow for analysis
of services to sub-populations, including veterans. Labor reported that
our recommendation to develop proposals for changing veteran definitions
across employment and training programs must be evaluated with the input
of other agencies. Labor also provided technical comments that we
incorporated where appropriate. Labor's comments are reproduced in full in
appendix IV.

We will send copies of this report to the Secretary of Labor, relevant
congressional committees, and other interested parties and will make
copies available to others upon request. In addition, the report will be
available at no charge on GAO's Web site at http://www.gao.gov .

A list of related GAO products is included at the end of this report. If
you or your staff have any questions about this report, please contact me
at (202) 512-7215 or at [email protected]. Contact points for our Offices of
Congressional Relations and Public Affairs may be found on the last page
of this report. Other contacts and staff acknowledgments are listed in
appendix V.

Sigurd R. Nilsen, Director
Education, Workforce, and Income Security Issues

Appendix I: Objectives, Scope, and Methodology

The objectives of this report were to determine (1) the extent to which
DVOP and LVER performance information reflects services and outcomes for
veterans served by these programs, (2) the extent to which performance
information on veterans served by other key programs is comprehensive and
well integrated across programs in the one-stop system, and (3) what Labor
is doing to improve the quality of performance data and better understand
outcomes for veteran job seekers.

To address these objectives, we

           o conducted a nationwide Web-based survey to state workforce
           administrators in the 50 states and the District of Columbia;
           o conducted site visits to 3 states, during which we interviewed
           state and federal officials, one-stop managers, and program staff;
           o interviewed Labor officials from both the Veterans' Employment
           and Training Service (VETS) and the Employment and Training
           Administration (ETA);
           o analyzed relevant performance data from ETA and VETS; and
           o reviewed our previous work on attributes of successful
           performance measures.

We conducted our work in accordance with generally accepted government
auditing standards between May 2006 and April 2007.

Survey

To obtain further information on our objectives, we surveyed state
workforce administrators from November 15 to December 27, 2006. The survey
addressed all three objectives and included questions about performance
information for the DVOP and LVER programs, integration of data across
employment programs serving veterans, and efforts to ensure data quality.

We developed the survey based on knowledge obtained during our preliminary
research. This included a literature review and initial interviews with
officials from the Department of Labor, the National Association of State
Workforce Agencies (NASWA), and the state of New Hampshire, where we
conducted our initial site visit. We then obtained a list of state
workforce administrators from NASWA. We asked state administrators to
provide information on the DVOP and LVER programs' capacity, other
programs within the one-stop system that serve veteran job seekers,
performance measures and data; and challenges to managing the programs. To
determine whether respondents would understand the questions as intended,
we pretested the survey with state officials in 5 states. We then made
changes to the questions based on comments we received during the
pretests.

The survey was conducted using self-administered electronic Web-based
questionnaires. We sent notification of the survey to the 50 states and
the District of Columbia in November 2006 and followed up with e-mail
messages and telephone calls as necessary during November and December.
All 51 recipients submitted their responses by the end of December 2006,
providing us with a response rate of 100 percent. We did not independently
verify information obtained through the survey. During our data analysis
we held three follow-up conversations to fill in gaps from incomplete
survey information.

Because this survey was not a sample survey, there are no sampling errors.
However, the practical difficulties of conducting any survey may introduce
errors, commonly referred to as nonsampling errors. For example,
difficulties in how a particular question is interpreted, in the sources
of information that are available to respondents, or how the data are
entered into a database can introduce unwanted variability into the survey
results. We took steps during survey development, data collection, and
data analysis to minimize these nonsampling errors. For example, we
pretested the questionnaire to ensure that questions were clear and
understandable. Since this was a Web-based survey in which respondents
entered their responses directly into out database, there was little
possibility of data entry error. During data analysis, a second,
independent analyst checked all computer programming. Also, to the extent
possible, we compared both closed and open ended survey responses with our
site visit observations. While survey results are also subject to
different types of systematic errors or bias, we do not have reason to
believe that respondents falsely reported any information for this survey.

Site Visits

To obtain a detailed understanding of how veteran job seekers are served
by the one-stop system and how their information is captured, we conducted
visits to three states: New Hampshire, California, and Tennessee. We
selected these states based on a range of selection criteria, including
geographic dispersion, state size and veteran demographics, recent state
performance in veterans' programs, and recommendations by Labor and NASWA.
Our site visits at the state level included interviews with state
workforce agency officials and state directors of Veterans' Employment and
Training. We also chose two local one-stops in each state and met with
local managers and veteran program staff (see table 6).

Table 6: States and Local Areas in Our Study

State site visits Local area                                   City        
New Hampshire     New Hampshire Works                          Concord     
                     New Hampshire Works             Manchester   
California        Mark Sanders Sacramento Midtown One-Stop     Sacramento  
                     Career Center                                            
                     San Diego South Metro Career    San Diego    
                     Center                                       
Tennessee         Clarksville/Montgomery County Career Center  Clarksville 
                     Middle Tennessee Career Center  Murfreesboro 

Source: GAO.

During each interview, we used standard interview protocols to obtain
detailed and comparable information. In our interview with state workforce
officials, we discussed the role of the state workforce agency in
administering veterans' employment and training programs, details about
the programs serving veteran job seekers, views on the current performance
accountability system, and information about data collection and
validation. In our interviews with the state directors and their staff, we
discussed their oversight roles and responsibilities, relationship with
the state workforce agency, and views on the current performance
accountability system and data collection. At the local one-stops, we
discussed the coordination of veteran staff with other programs within the
one-stop system, priority of service, and data collection and reporting.
In each state, we also received a tutorial of the state's data collection
software. We conducted our site visits between July and November 2006.

Research and Experts

As part of our work, we interviewed officials of ETA and VETS, including
all six Regional Administrators of VETS. We conducted these telephone
interviews in the following locations: Boston, Atlanta, Dallas, Chicago,
San Francisco, and Philadelphia. During each interview, we obtained
information on regional differences in administering the DVOP and LVER
programs, views on the current performance measures, and information on
Labor's monitoring role in each state.

We also analyzed performance data from the DVOP, LVER, and Employment
Service programs and reviewed Labor's guidance. In addition, we reviewed
relevant literature, including our past work on attributes of successful
performance measures. We also interviewed representatives of NASWA and two
private-sector staffing agencies.

Appendix II: Summary of State Performance in the DVOP and LVER Programs
for Benchmark Program Year 2005 and Negotiated Goals for Program Year 2006

The following tables include

           o baseline performance data by state for the DVOP and LVER
           programs from benchmark program year 2005 (July 1, 2005-June 30,
           2006) and

           o negotiated goals by state for the following year, program year
           2006.

Labor and states did not negotiate goals for the DVOP or LVER programs for
program year 2005, which was a baseline year for performance under the new
common measures. Four performance measures contribute to each program's
performance. For the DVOP program, there is one set of measures for all
veterans and one set for disabled veterans. For the LVER program, there is
a set of measures for all veterans and another set for recently separated
veterans. Each set of measures includes

           o entered employment rate (EER): the number of participants who
           are employed in the first quarter after the exit quarter divided
           by the number of participants who exit during the quarter and
           o employment retention rate (ERR): the number of participants who
           are employed in both the second and third quarters after the exit
           quarter divided by the number of adult participants who exit
           during the quarter.

These figures were provided by the Department of Labor. GAO has not
verified the accuracy or reliability of these data.

Table 7: Summary of State Performance in the DVOP Program for Benchmark
Program Year 2005 and Negotiated Goals for Program Year 2006

                      Disabled Veterans Outreach Program (DVOP)
                            All veterans                        Disabled veterans
                  2005  2006     2005                        2006                
              achieved  goal achieved    2006          2005  goal        2005    2006
State              EER   EER      ERR goalERR   achievedEER   EER achievedERR goalERR
Alabama               64%     64%         79%      81%      56%      56%     77%  79% 
Alaska                52%     52%         78%      78%      47%      50%     79%  78% 
Arizona               57%     58%         80%      80%      53%      57%     75%  77% 
Arkansas              66%     66%         81%      80%      59%      55%     78%  70% 
California            53%     54%         74%      74%      49%      47%     72%  73% 
Colorado              59%     63%         75%      76%      55%      58%     74%  72% 
Connecticut           55%     52%         77%      75%      43%      46%     77%  75% 
Delaware              60%     60%         76%      83%      55%      63%     87%  88% 
District of           58%     60%         75%      75%      44%      60%     83%  78% 
Columbia                                                                              
Florida               61%     59%         80%      79%      56%      56%     78%  77% 
Georgia               70%     70%         81%      80%      62%      62%     82%  80% 
Hawaii                53%     52%         77%      72%      54%      47%     73%  76% 
Idaho                 67%     67%         84%      83%      59%      59%     79%  80% 
Illinois              59%     54%         84%      81%      56%      53%     86%  85% 
Indiana               69%     59%         84%      80%      58%      50%     84%  79% 
Iowa                  66%     75%         83%      87%      62%      72%     82%  86% 
Kansas                68%     69%         83%      83%      64%      64%     84%  83% 
Kentucky              66%     64%         82%      80%      59%      57%     82%  80% 
Louisiana             59%     57%         76%      74%      53%      58%     74%  74% 
Maine                 64%     73%         80%      81%      63%      73%     80%  77% 
Maryland              57%     64%         79%      83%      54%      52%     84%  82% 
Massachusetts         57%     59%         80%      80%      48%      48%     75%  81% 
Michigan              57%     59%         78%      81%      54%      55%     78%  78% 
Minnesota             59%     62%         82%      81%      49%      55%     80%  81% 
Mississippi           49%     60%         57%      82%      48%      56%     59%  79% 
Missouri              61%     64%         78%      81%      54%      58%     77%  80% 
Montana               61%     70%         75%      83%      56%      65%     80%  83% 
Nebraska              66%     68%         81%      76%      58%      62%     78%  81% 
Nevada                72%     71%         80%      79%      68%      67%     80%  79% 
New Hampshire         56%     64%         81%      81%      54%      60%     81%  80% 
New Jersey            50%     55%         73%      80%      42%      60%     73%  73% 
New Mexico            40%     57%         80%      73%      37%      56%     76%  80% 
New York              61%     61%         78%      80%      56%      56%     77%  89% 
North                 57%     55%         72%      79%      53%      52%     71%  76% 
Carolina                                                                              
North Dakota          73%     69%         76%      89%      56%      55%     77%  79% 
Ohio                  65%     60%         84%      82%      58%      55%     81%  82% 
Oklahoma              64%     58%         80%      77%      60%      55%     76%  77% 
Oregon                55%     56%         78%      76%      42%      46%     74%  76% 
Pennsylvania          60%     60%         83%      77%      47%      56%     80%  77% 
Puerto Rico           11%     52%          8%      51%       6%      50%      9%  51% 
Rhode Island          45%     58%         35%      68%      33%      58%     38%  68% 
South                 68%     63%         83%      82%      64%      59%     81%  82% 
Carolina                                                                              
South Dakota          68%     70%         83%      80%      64%      67%     86%  80% 
Tennessee             67%     67%         80%      80%      62%      67%     76%  80% 
Texas                 68%     63%         85%      82%      55%      55%     86%  82% 
Utah                  57%     57%         82%      79%      53%      53%     80%  79% 
Vermont               79%     75%         87%      73%      82%      77%     72%  87% 
Virginia              76%     70%         92%      87%      71%      66%     91%  73% 
Washington            71%     68%         85%      86%      67%      64%     85%  86% 
West Virginia         65%     66%         84%      88%      56%      60%     74%  85% 
Wisconsin             65%     75%         89%      85%      56%      70%     87%  88% 
Wyoming               66%     54%         81%      80%      66%      52%     80%  80% 

Source: Labor.

Note: All years refer to program years.

Table 8: Summary of State Performance in the LVER Program for Benchmark
Program Year 2005 and Negotiated Goals for Program Year 2006

                   Local Veterans Employment Representative (LVER)
                                                               Recently separated
                            All veterans                            veterans
                  2005  2006     2005                        2006                
              achieved  goal achieved    2006          2005  goal        2005    2006
State              EER   EER      ERR goalERR   achievedEER   EER achievedERR goalERR
Alabama               63%     62%         78%      82%      66%      61%     75%  80% 
Alaska                49%     50%         75%      79%      67%      50%     60%  81% 
Arizona               55%     51%         79%      70%      55%      51%     77%  70% 
Arkansas              68%     66%         82%      82%      61%      64%     82%  82% 
California            53%     55%         74%      76%      52%      55%     77%  78% 
Colorado              62%     64%         76%      76%      61%      63%     72%  74% 
Connecticut           55%     52%         75%      76%      64%      56%     71%  76% 
Delaware              65%     67%         79%      76%      66%      77%     77%  77% 
District of           59%     60%         76%      75%      56%      60%     79%  76% 
Columbia                                                                              
Florida               59%     58%         80%      79%      60%      58%     79%  79% 
Georgia               69%     69%         81%      80%      68%      65%     79%  79% 
Hawaii                47%     49%         76%      77%      40%      41%     67%  75% 
Idaho                 72%     71%         85%      84%      78%      74%     85%  84% 
Illinois              61%     58%         85%      83%      60%      56%     84%  81% 
Indiana               70%     60%         84%      81%      74%      64%     83%  77% 
Iowa                  69%     74%         84%      85%      69%      76%     84%  85% 
Kansas                66%     66%         82%      83%      76%      75%     84%  84% 
Kentucky              67%     65%         83%      80%      65%      64%     81%  80% 
Louisiana             61%     57%         73%      74%      62%      57%     70%  74% 
Maine                 64%     73%         81%      81%      65%      73%     77%  81% 
Maryland              58%     69%         78%      82%      60%      76%     88%  90% 
Massachusetts         58%     59%         79%      81%      59%      61%     71%  75% 
Michigan              57%     59%         78%      81%      61%      61%     77%  79% 
Minnesota             69%     63%         83%      81%      50%      50%     71%  81% 
Mississippi           50%     60%         61%      82%      46%      56%     58%  80% 
Missouri              64%     66%         81%      82%      69%      70%     80%  82% 
Montana               53%     70%         71%      83%      53%      65%     72%  83% 
Nebraska              63%     65%         81%      82%      63%      64%     81%  82% 
Nevada                73%     73%         83%      79%      77%      76%     83%  80% 
New Hampshire         60%     64%         83%      83%      61%      64%     80%  80% 
New Jersey            51%     55%         77%      80%      55%      65%     76%  80% 
New Mexico            42%     68%         82%      81%      41%      56%     77%  78% 
New York              64%     64%         80%      80%      58%      59%     78%  80% 
North                 55%     55%         73%      80%      49%      50%     68%  78% 
Carolina                                                                              
North Dakota          72%     71%         83%      81%      82%      65%     83%  82% 
Ohio                  67%     60%         85%      83%      71%      60%     90%  85% 
Oklahoma              69%     59%         81%      82%      73%      60%     80%  79% 
Oregon                57%     58%         77%      79%      57%      62%     78%  78% 
Pennsylvania          62%     60%         85%      81%      58%      60%     83%  81% 
Puerto Rico            0%     52%         10%      73%       0%      50%      9%  51% 
Rhode Island          33%     58%         47%      69%      35%      59%     40%  57% 
South                 66%     61%         83%      83%      70%      63%     82%  83% 
Carolina                                                                              
South Dakota          74%     72%         85%      68%      57%      69%     86%  76% 
Tennessee             68%     67%         82%      82%      68%      67%     79%  80% 
Texas                 66%     63%         84%      82%      84%      64%     87%  82% 
Utah                  67%     61%         85%      79%      70%      61%     82%  79% 
Vermont               74%     73%         82%      82%      77%      72%     82%  74% 
Virginia              75%     69%         92%      88%      69%      60%     85%  81% 
Washington            66%     65%         85%      84%      74%      70%     87%  87% 
West Virginia         66%     65%         86%      68%      70%      70%     79%  73% 
Wisconsin             64%     71%         89%      84%      70%      88%     85%  81% 
Wyoming               68%     62%         77%      82%      67%      62%     82%  82% 

Source: Labor.

Note: All years refer to program years.

Appendix III: Summary of State-Negotiated Goals and Performance for
Veterans in the Employment Service, Program Year 2005

This table illustrates the negotiated goals and performance achieved by
each state for program year 2005 for veterans in the Wagner-Peyser-funded
Employment Service. It includes the entered employment and employment
retention rates for all veterans and disabled veterans within the
Employment Service, including those in the DVOP and LVER programs.

These figures were provided by the Department of Labor. GAO has not
verified the accuracy or reliability of these data.

Table 9: State-Negotiated Goals and Performance for Veterans in the
Employment Service, Program Year 2005

                                   Employment Service
                            All veterans                         Disabled veterans
              2005     2005 2005                               2005  2005        
              goal achieved goal                      2005 achieved  goal        
State          EER      EER  ERR 2005achievedERR   goalEER      EER   ERR 2005achievedERR
Alabama             60%    62%               76%    78%      54%   53%       76%      77% 
Alaska              58%    55%               77%    73%      48%   53%       75%      75% 
Arizona             58%    59%               74%    80%      54%   56%       68%      78% 
Arkansas            53%    67%               70%    81%      53%   63%       70%      78% 
California          53%    52%               70%    78%      49%   47%       68%      77% 
Colorado            60%    59%               79%    76%      59%   55%       78%      75% 
Connecticut         57%    56%               66%    77%      53%   49%       66%      77% 
Delaware            55%    59%               75%    78%      50%   55%       73%      83% 
District of         58%    58%               75%    75%      54%   59%       73%      71% 
Columbia                                                                                  
Florida             55%    59%               72%    79%      54%   55%       73%      78% 
Georgia             65%    68%               72%    81%      65%   61%       72%      81% 
Hawaii              46%    49%               70%    77%      41%   53%       68%      76% 
Idaho               50%    71%               71%    83%      43%   65%       70%      81% 
Illinois            44%    63%               72%    85%      40%   59%       72%      85% 
Indiana             54%    68%               75%    84%      46%   59%       73%      84% 
Iowa                58%    67%               72%    83%      56%   62%       70%      83% 
Kansas              58%    66%               72%    83%      56%   62%       72%      82% 
Kentucky            63%    67%               72%    82%      58%   59%       72%      81% 
Louisiana           59%    62%               60%    77%      60%   61%       60%      75% 
Maine               58%    64%               70%    80%      58%   62%       68%      78% 
Maryland            58%    59%               77%    79%      53%   54%       77%      78% 
Massachusetts       57%    58%               78%    79%      48%   47%       72%      74% 
Michigan            48%    57%               73%    79%      43%   51%       71%      77% 
Minnesota           49%    61%               77%    82%      48%   54%       71%      80% 
Mississippi         62%    49%               72%    61%      55%   48%       72%      61% 
Missouri            58%    62%               72%    79%      52%   58%       70%      78% 
Montana             46%    44%               80%    75%      46%   39%       79%      80% 
Nebraska            59%    63%               76%    81%      55%   54%       74%      79% 
Nevada              58%    72%               72%    82%      54%   68%       72%      81% 
New Hampshire       67%    58%               67%    82%      62%   58%       66%      82% 
New Jersey          55%    51%               74%    75%      59%   43%       72%      74% 
New Mexico          58%    45%               75%    79%      54%   42%       73%      77% 
New York            56%    61%               72%    79%      52%   58%       72%      78% 
North               58%    54%               72%    74%      54%   49%       72%      75% 
Carolina                                                                                  
North Dakota        63%    70%               78%    82%      59%   60%       75%      83% 
Ohio                57%    65%               78%    84%      51%   59%       74%      82% 
Oklahoma            60%    64%               72%    80%      54%   56%       65%      77% 
Oregon              55%    57%               75%    80%      48%   46%       73%      76% 
Pennsylvania        59%    NDA               73%    NDA      52%   NDA       73%      NDA 
Puerto Rico         NDP    NDA               NDP    NDA      NDP   NDA       NDP      NDA 
Rhode Island        58%    DNV               72%    DNV      52%   DNV       72%      DNV 
South               59%    67%               73%    82%      53%   64%       73%      81% 
Carolina                                                                                  
South Dakota        64%    69%               78%    80%      58%   66%       78%      82% 
Tennessee           53%    66%               70%    81%      49%   61%       70%      79% 
Texas               56%    65%               80%    85%      50%   55%       77%      86% 
Utah                57%    66%               75%    84%      51%   59%       74%      82% 
Vermont             59%    73%               73%    82%      54%   78%       73%      76% 
Virginia            56%    76%               72%    92%      52%   70%       75%      91% 
Washington          50%    70%               80%    85%      44%   67%       79%      85% 
West Virginia       58%    66%               72%    85%      58%   61%       72%      81% 
Wisconsin           52%    63%               76%    89%      46%   56%       73%      87% 
Wyoming             58%    NDA               75%    NDA      48%   NDA       78%      NDA 

Source: Labor.

Notes: NDA = No data available / NDP = No data provided / DNV = Data not
valid.

All years refer to program years.

Appendix IV: Comments from the Department of Labor

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Sigurd R. Nilsen (202) 512-7215 or [email protected]

Staff Acknowledgments

Dianne Blank, Assistant Director Rebecca Woiwode, Analyst-in-Charge

Chris Morehouse and Beth Faraguna made significant contributions to this
report in all facets of the work. In addition, Walter Vance assisted in
the design of the national survey; Gloria Hernandez-Saunders helped with
data analysis; Meeta Engle lent subject matter expertise; Jessica Botsford
and Richard Burkard provided legal support; and Charlie Willson provided
writing assistance.

Related GAO Products

Trade Adjustment Assistance: Labor Should Take Action to Ensure
Performance Data Are Complete, Accurate, and Accessible. [43]GAO-06-496 .
Washington, D.C.: April 25, 2006.

Veterans' Employment and Training Service: Greater Accountability and
Other Labor Actions Needed to Better Serve Veterans. [44]GAO-06-357T .
Washington, D.C.: February 2, 2006.

Veterans' Employment and Training Service: Labor Actions Needed to Improve
Accountability and Help States Implement Reforms to Veterans' Employment
Services. [45]GAO-06-176 . Washington, D.C.: December 30, 2005.

Workforce Investment Act: Labor and States Have Taken Actions to Improve
Data Quality, but Additional Steps Are Needed. [46]GAO-06-82 . Washington,
D.C.: November 14, 2005.

Veterans' Employment and Training Service: Preliminary Observations on
Changes to Veterans' Employment Programs. [47]GAO-05-662T . Washington,
D.C.: May 12, 2005.

Performance Measurement and Evaluation: Definitions and Relationships.
[48]GAO-05-739SP . Washington, D.C.: May 2005.

Workforce Investment Act: States and Local Areas Have Developed Strategies
to Assess Performance, but Labor Could Do More to Help. [49]GAO-04-657 .
Washington, D.C.: June 1, 2004.

Performance Reporting: Few Agencies Reported on the Completeness and
Reliability of Performance Data. [50]GAO-02-372 . Washington, D.C.: April
26, 2002.

Veterans' Employment and Training Service: Flexibility and Accountability
Needed to Improve Service to Veterans. [51]GAO-01-928 . Washington, D.C.:
September 12, 2001.

Veterans' Employment and Training Service: Proposed Performance
Measurement System Improved, but Further Changes Needed. [52]GAO-01-580 .
Washington, D.C.: May 15, 2001.

Veterans' Employment and Training Service: Strategic and Performance Plans
Lack Vision and Clarity. [53]GAO/T-HEHS-99-177 . Washington, D.C.: July
29, 1999.

Veterans' Employment and Training Service: Assessment of the Fiscal Year
1999 Performance Plan. [54]GAO/HEHS-98-240R . Washington, D.C.: September
30, 1998.

Veterans' Employment and Training: Services Provided by Labor Department
Programs. [55]GAO/HEHS-98-7 . Washington, D.C.: October 17, 1997.

The Results Act: An Evaluator's Guide to Assessing Agency Annual
Performance Plans. [56]GAO/GGD-10.1.20. Washington, D.C.: April 1998.

Agencies' Annual Performance Plans under the Results Act: An Assessment
Guide to Facilitate Congressional Decisionmaking. GAO/GDD/AIMD-10.1.18.
Washington, D.C.: February 1998.

Executive Guide: Effectively Implementing the Government Performance and
Results Act. GAO/GGD-96-118 . Washington, D.C.: June 1996.

(130581)

GAO's Mission

The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost
is through GAO's Web site ( www.gao.gov ). Each weekday, GAO posts
newly released reports, testimony, and correspondence on its Web site. To
have GAO e-mail you a list of newly posted products every afternoon, go to
www.gao.gov and select "Subscribe to Updates."

Order by Mail or Phone

The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:

U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548

To order by Phone: Voice: (202) 512-6000
TDD: (202) 512-2537
Fax: (202) 512-6061

To Report Fraud, Waste, and Abuse in Federal Programs

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: [email protected]
Automated answering system: (800) 424-5454 or (202) 512-7470

Congressional Relations

Gloria Jarmon, Managing Director, [email protected] (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125 Washington,
D.C. 20548

Public Affairs

Paul Anderson, Managing Director, [email protected] (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548

www.gao.gov/cgi-bin/getrpt?GAO-07-594 .

To view the full product, including the scope
and methodology, click on the link above.

For more information, contact Sigurd Nilsen at (202) 512-7215 or
[email protected].

Highlights of [64]GAO-07-594 , a report to congressional committees

May 2007

VETERANS'EMPLOYMENT AND TRAINING SERVICE

Labor Could Improve Information on Reemployment Services, Outcomes, and
Program Impact

In 2002, Congress enacted the Jobs for Veterans Act (JVA), which modified
two Department of Labor (Labor) programs that specifically target veteran
job seekers: the Disabled Veterans' Outreach Program (DVOP) and the Local
Veterans' Employment Representative (LVER) program. However, questions
have been raised about the adequacy of performance information on services
to veterans by these and other employment programs. In this report, GAO
examined (1) the extent to which DVOP and LVER performance information
reflects services and outcomes for veterans; (2) the extent to which
performance information on veterans paints a clear picture of their use of
one-stop services; and (3) what Labor is doing to improve the quality of
performance data and better understand program impact and outcomes for
veterans.

[65]What GAO Recommends

GAO is making a number of recommendations to improve the performance
measurement system for the DVOP and LVER programs and to better understand
services and their impact for job seekers in the one-stop system,
including veterans.

Labor generally agreed with the report's findings, conclusions, and
recommendations.

Performance information for the DVOP and LVER programs provides some sense
of services and outcomes for veterans, but is weakened by several factors.
In July 2005, Labor adopted new performance measures for the programs, but
not all have been fully implemented. For example, states are held
accountable for helping veterans get and keep jobs, but are not yet held
accountable for their average earnings once employed, as they are for
other programs. Additionally, having separate performance measures for the
DVOP and LVER programs fails to acknowledge the similarity of the
populations they serve and duties they perform. Furthermore, it is
difficult to assess outcomes over time or across states because of
frequent changes in reporting requirements that prevent establishing
reliable trend data.

Labor's data on veteran job seekers paint an unclear picture of their use
of other employment and training services in the one-stop system, despite
the use of common performance measures across programs. Although many
veterans use services other than those provided by the DVOP and LVER
programs, key employment programs vary in how well their data on veteran
job seekers are shared across programs, making it difficult to know how
many veterans are served. In addition, statutory differences in the
definitions of veterans hinder efforts to standardize data across
employment programs. Moreover, Labor has no means of assessing whether
priority of service for veterans has been implemented in various
employment programs.

Labor has taken some steps to improve the quality of performance data and
better understand outcomes for veterans. For example, Labor requires
states to validate key performance data. Labor has also planned an
integrated data reporting system that would track individual veterans'
progress through the one-stop system. However, states have raised concerns
about the timelines and its current implementation date is unclear.
Furthermore, while outcome information on veterans is helpful, it cannot
measure whether the outcomes are due to the program or other factors.
While Labor has sponsored research on services to veterans, it has not yet
conducted the impact evaluation required by law to assess the
effectiveness of one-stop services.

Percentage of Disabled and Recently Separated Veterans Served by the DVOP
and LVER Programs, Program Year 2005

References

Visible links
  34. http://www.gao.gov/cgi-bin/getrpt?GAO/GGD-10.1.20 and http://www.gao.gov/cgi-bin/getrpt?GAO/GGD/AIMD-10.1.18 and http://www.gao.gov/cgi-bin/getrpt?GAO/GGD-96-118
  35. http://www.gao.gov/cgi-bin/getrpt?GAO-04-657
  36. http://www.gao.gov/cgi-bin/getrpt?GAO-06-496
  37. http://www.gao.gov/cgi-bin/getrpt?GAO-06-176
  38. http://www.gao.gov/cgi-bin/getrpt?GAO-06-82
  39. http://www.gao.gov/cgi-bin/getrpt?GAO-06-82
  40. http://www.gao.gov/cgi-bin/getrpt?GAO-04-657
  43. http://www.gao.gov/cgi-bin/getrpt?GAO-06-496
  44. http://www.gao.gov/cgi-bin/getrpt?GAO-06-357T
  45. http://www.gao.gov/cgi-bin/getrpt?GAO-06-176
  46. http://www.gao.gov/cgi-bin/getrpt?GAO-06-82
  47. http://www.gao.gov/cgi-bin/getrpt?GAO-05-662T
  48. http://www.gao.gov/cgi-bin/getrpt?GAO-05-739SP
  49. http://www.gao.gov/cgi-bin/getrpt?GAO-04-657
  50. http://www.gao.gov/cgi-bin/getrpt?GAO-02-372
  51. http://www.gao.gov/cgi-bin/getrpt?GAO-01-928
  52. http://www.gao.gov/cgi-bin/getrpt?GAO-01-580
  53. http://www.gao.gov/cgi-bin/getrpt?GAO/T-HEHS-99-177
  54. http://www.gao.gov/cgi-bin/getrpt?GAO/HEHS-98-240R
  55. http://www.gao.gov/cgi-bin/getrpt?GAO/HEHS-98-7
  56. http://www.gao.gov/cgi-bin/getrpt?GAO/GGD-10.1.20 and http://www.gao.gov/cgi-bin/getrpt?GAO/GDD/AIMD-10.1.18 and http://www.gao.gov/cgi-bin/getrpt?GAO/GGD-96-118
  64. http://www.gao.gov/cgi-bin/getrpt?GAO-07-594
*** End of document. ***