Veterans' Employment and Training Service: Labor Could Improve
Information on Reemployment Services, Outcomes, and Program
Impact (24-MAY-07, GAO-07-594).
In 2002, Congress enacted the Jobs for Veterans Act (JVA), which
modified two Department of Labor (Labor) programs that
specifically target veteran job seekers: the Disabled Veterans'
Outreach Program (DVOP) and the Local Veterans' Employment
Representative (LVER) program. However, questions have been
raised about the adequacy of performance information on services
to veterans by these and other employment programs. In this
report, GAO examined (1) the extent to which DVOP and LVER
performance information reflects services and outcomes for
veterans; (2) the extent to which performance information on
veterans paints a clear picture of their use of one-stop
services; and (3) what Labor is doing to improve the quality of
performance data and better understand program impact and
outcomes for veterans.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-07-594
ACCNO: A69908
TITLE: Veterans' Employment and Training Service: Labor Could
Improve Information on Reemployment Services, Outcomes, and
Program Impact
DATE: 05/24/2007
SUBJECT: Data collection
Employment assistance programs
Performance measures
Program evaluation
Program management
Reporting requirements
State-administered programs
Strategic planning
Training utilization
Veterans benefits
Veterans education
Veterans employment programs
Performance management
DOL Disabled Veterans' Outreach Program
DOL Local Veterans' Employment
Representative Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-07-594
* [1]Results in Brief
* [2]Background
* [3]Jobs for Veterans Act Reforms
* [4]Changes to the DVOP and LVER Performance Information
* [5]Data Reporting
* [6]The Role of Impact Evaluations in Assessing Program Outcomes
* [7]DVOP and LVER Performance Information Is Weakened by Several
* [8]The New Performance Measures Provide Information on Some Out
* [9]Performance Measures and the Data Reported to Labor Do Not R
* [10]Performance Data Are Not Comparable over Time or across Stat
* [11]Available Data Paint an Unclear Picture of Veterans' Use of
* [12]Veterans Receive Services from Multiple Programs in the One-
* [13]Data on Veterans' Services Are Well Integrated across Some b
* [14]Different Veteran Definitions Complicate Efforts to Standard
* [15]Labor Has No Means of Evaluating Priority of Service for Vet
* [16]Labor Has Taken Steps to Better Understand Veterans' Outcome
* [17]Labor Has Developed Some Processes to Enhance the Quality of
* [18]Labor's Proposed Integrated Data System Could Improve the Un
* [19]Little Is Known about the Impact of One-Stop Services, inclu
* [20]Conclusions
* [21]Recommendations for Executive Action
* [22]Agency Comments and Our Evaluation
* [23]Survey
* [24]Site Visits
* [25]Research and Experts
* [26]GAO Contact
* [27]Staff Acknowledgments
* [28]GAO's Mission
* [29]Obtaining Copies of GAO Reports and Testimony
* [30]Order by Mail or Phone
* [31]To Report Fraud, Waste, and Abuse in Federal Programs
* [32]Congressional Relations
* [33]Public Affairs
Report to Congressional Committees
United States Government Accountability Office
GAO
May 2007
VETERANS' EMPLOYMENT AND TRAINING SERVICE
Labor Could Improve Information on Reemployment Services, Outcomes, and
Program Impact
GAO-07-594
Contents
Letter 1
Results in Brief 3
Background 5
DVOP and LVER Performance Information Is Weakened by Several Factors 11
Available Data Paint an Unclear Picture of Veterans' Use of One-Stop
Services 18
Labor Has Taken Steps to Better Understand Veterans' Outcomes, but the
Programs' Impact Remains Unknown 25
Conclusions 31
Recommendations for Executive Action 31
Agency Comments and Our Evaluation 32
Appendix I Objectives, Scope, and Methodology 34
Appendix II Summary of State Performance in the DVOP and LVER Programs for
Benchmark Program Year 2005 and Negotiated Goals for Program Year 2006 37
Appendix III Summary of State-Negotiated Goals and Performance for
Veterans in the Employment Service, Program Year 2005 42
Appendix IV Comments from the Department of Labor 44
Appendix V GAO Contact and Staff Acknowledgments 46
Related GAO Products 47
Tables
Table 1: Selected Responsibilities of DVOP Specialists and LVER Staff 8
Table 2: Common Measures Adopted by Labor's Employment and Training
Programs 9
Table 3: Performance Measures for the DVOP and LVER Programs 12
Table 4: DVOP Specialists and LVER Staff Served Similar Proportions of
Recently Separated and Disabled Participants in Program Year 2005 14
Table 5: Differences between Veteran Definitions in Employment and
Training Programs 22
Table 6: States and Local Areas in Our Study 36
Table 7: Summary of State Performance in the DVOP Program for Benchmark
Program Year 2005 and Negotiated Goals for Program Year 2006 38
Table 8: Summary of State Performance in the LVER Program for Benchmark
Program Year 2005 and Negotiated Goals for Program Year 2006 40
Table 9: State-Negotiated Goals and Performance for Veterans in the
Employment Service, Program Year 2005 42
Figures
Figure 1: A Large Proportion of Disabled and Recently Separated Veterans
Are Not Captured in Performance Measures for Those Populations (Program
Year 2005) 15
Figure 2: Changes to Performance Information over Time for the DVOP and
LVER Programs 17
Figure 3: Veterans Can Access Multiple Employment Services in the One-Stop
System 19
Figure 4: Many States Report That DVOP Specialists or LVER Staff See Half
or Fewer of Veterans That Come into One-Stop Centers 20
Figure 5: Percentage of States Responding that Different Definitions
Affect the Coordination of Services to Veterans 23
Figure 6: Most State Workforce Agencies Report That Their State Systems
Take Certain Steps to Ensure the Accuracy and Reliability of Employment
Service, DVOP, and LVER Data 28
Abbreviations
DVOP Disabled Veterans' Outreach Program
EER entered employment rate
EMILE ETA Management Information and Longitudinal Evaluation system
ERR employment retention rate
ETA Employment and Training Administration
JVA Jobs for Veterans Act
LVER Local Veterans' Employment Representative
NASWA National Association of State Workforce Agencies
OMB Office of Management and Budget
TAA Trade Adjustment Assistance
UI Unemployment Insurance
VA Department of Veterans Affairs
VETS Veterans' Employment and Training Service
WIA Workforce Investment Act
WIASRD Workforce Investment Act Standardized Record Data
WISPR Workforce Investment Streamlined Performance Reporting System
WRIS Wage Record Interchange System
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
United States Government Accountability Office
Washington, DC 20548
May 24, 2007
The Honorable Daniel K. Akaka
Chairman
The Honorable Larry E. Craig
Ranking Member
Committee on Veterans' Affairs
U.S. Senate
The Honorable Bob Filner
Chairman
Committee on Veterans' Affairs
House of Representatives
Approximately 700,000 veterans are unemployed in any given month, and
roughly 200,000 service members leave active duty and transition to
civilian life each year, according to the Department of Labor (Labor).
While their unemployment rate is similar to that of the general
population, some veterans have special needs in transitioning to the
civilian labor market. Labor has long provided assistance to veterans to
help them succeed in this endeavor. In 2002, Congress enacted the Jobs for
Veterans Act (JVA) to improve employment and training services for
veterans by better integrating them into the comprehensive service
delivery system, or one-stop system, created under the Workforce
Investment Act (WIA) of 1998. JVA modified two Labor programs that
specifically target veteran job seekers and that are administered by the
Veterans' Employment and Training Service (VETS)--the Disabled Veterans'
Outreach Program (DVOP) and the Local Veterans' Employment Representative
(LVER) program--and consolidated the funding into a single grant to
states. These two programs were funded at about $155 million in fiscal
year 2006, and served over 715,000 veterans nationwide in the most recent
program year. JVA addressed concerns raised by some that the programs were
overly prescriptive and did not provide states the flexibility to
determine the best way to serve veteran job seekers. Instead, JVA
identified broad roles and responsibilities of DVOP and LVER staff while
giving states flexibility to determine the number of staff for each
program. In addition, the law required the implementation of a
comprehensive performance accountability system that included performance
measures for the two programs--consistent with those under WIA--and
enhanced accountability for veteran services in the one-stop system.
Moreover, many veterans receive reemployment services from programs other
than Approximately 700,000 veterans are unemployed in any given month, and
roughly 200,000 service members leave active duty and transition to
civilian life each year, according to the Department of Labor (Labor).
While their unemployment rate is similar to that of the general
population, some veterans have special needs in transitioning to the
civilian labor market. Labor has long provided assistance to veterans to
help them succeed in this endeavor. In 2002, Congress enacted the Jobs for
Veterans Act (JVA) to improve employment and training services for
veterans by better integrating them into the comprehensive service
delivery system, or one-stop system, created under the Workforce
Investment Act (WIA) of 1998. JVA modified two Labor programs that
specifically target veteran job seekers and that are administered by the
Veterans' Employment and Training Service (VETS)--the Disabled Veterans'
Outreach Program (DVOP) and the Local Veterans' Employment Representative
(LVER) program--and consolidated the funding into a single grant to
states. These two programs were funded at about $155 million in fiscal
year 2006, and served over 715,000 veterans nationwide in the most recent
program year. JVA addressed concerns raised by some that the programs were
overly prescriptive and did not provide states the flexibility to
determine the best way to serve veteran job seekers. Instead, JVA
identified broad roles and responsibilities of DVOP and LVER staff while
giving states flexibility to determine the number of staff for each
program. In addition, the law required the implementation of a
comprehensive performance accountability system that included performance
measures for the two programs--consistent with those under WIA--and
enhanced accountability for veteran services in the one-stop system.
Moreover, many veterans receive reemployment services from programs other
than the DVOP and LVER, and JVA stipulated that veterans served by these
other programs be given preference over nonveterans, as long as they meet
the eligibility requirements. These programs--administered by a separate
office within Labor, the Employment and Training Administration
(ETA)--include the WIA Adult and Dislocated Worker programs, the
Wagner-Peyser-funded Employment Service, and other key employment
programs.
To implement JVA, VETS adopted a set of common performance measures in
July 2005, similar to those adopted by other employment and training
programs. Each quarter, states report a wide range of data to Labor on
demographics, services, and outcomes for veterans in the DVOP and LVER
programs. States are specifically held accountable for certain
employment-related outcomes for all veterans--whether veteran participants
get and keep jobs--and separately, for the outcomes of some disabled and
recently separated veterans. JVA requires Labor to include information in
its annual report to Congress on whether veterans are receiving priority
and are being fully served by employment and training programs. However,
questions have been raised about whether available performance information
accurately reflects services and outcomes for veterans. Moreover, outcome
information alone cannot be used to evaluate the effectiveness--or
impact--of employment services for veterans. Program outcomes may be
attributable to factors other than a program's services, such as local
labor market conditions. In view of these concerns and the changes to the
performance accountability system resulting from the Jobs for Veterans
Act, you requested that we review the performance information collected on
employment and training programs that serve veterans. Specifically, we
examined (1) the extent to which DVOP and LVER performance information
reflects services and outcomes for veterans served by these programs; (2)
the extent to which performance information on veterans served by other
key programs paints a clear picture of veterans' use of one-stop services
and (3) what Labor is doing to improve the quality of performance data and
better understand outcomes for veteran job seekers.
To determine the extent to which available performance information
reflects services and outcomes for veteran job seekers, we administered a
nationwide Web-based survey to state workforce administrators in all 50
states and the District of Columbia. We received responses from all 51
administrators. In addition, we analyzed performance data from the
relevant programs and reviewed Labor's program guidance. To better
understand state and local variation in employment programs serving
veterans, we conducted site visits to three states: New Hampshire,
California, and Tennessee. We selected these states based on a range of
criteria, including geographic dispersion, state size and veteran
demographics, recent state performance in veterans' programs, and
recommendations by Labor and the National Association of State Workforce
Agencies. Our site visits included interviews with state workforce agency
officials, the federal Director of Veterans' Employment and Training for
each state, one-stop managers, and DVOP and LVER program staff.
Additionally, we interviewed Labor officials from both VETS and ETA. As
part of this effort, we conducted interviews with all six Regional
Administrators of Veterans' Employment and Training to help understand
regional variation in the programs. Finally, we reviewed literature on
attributes of successful performance measures, including our previous
work.^1 For a detailed discussion of our scope and methodology, see
appendix I. We conducted our review from May 2006 to April 2007 in
accordance with generally accepted government auditing standards.
Results in Brief
Performance measures and data for the DVOP and LVER programs generally
reflect veterans' services and outcomes, but are weakened by several
factors. The new performance measures, in place since July 2005, provide
information on some outcomes for veterans, but have not been fully
implemented. For example, states are held accountable for helping veterans
get and keep jobs, but are not yet held accountable for veterans' average
earnings, as they are for other employment and training programs.
Additionally, the current performance measures do not include a method to
weight the successful outcomes of veterans who are harder to serve, such
as those with barriers to employment, as required by JVA. Labor developed
but has not implemented such a method--which could better reflect the
difficulty of serving these veterans--because the system was seen as too
complicated for field staff. In addition, neither the performance measures
nor the data reported to Labor reflect the full range of services that
DVOP and LVER staff provide to veterans. For example, the current measures
hold only DVOP specialists accountable for disabled veterans' employment
and retention, but LVER staff also serve a large number of disabled
veterans who are not counted in those measures. Similarly, only LVER staff
are measured on outcomes for recently separated veterans, although DVOP
specialists serve roughly the same number of recently separated veterans.
Also, while states collect a wide range of performance data on services
and outcomes for veterans, the data reported to Labor do not currently
include information on outreach to employers, a key program activity.
Furthermore, it is difficult to assess outcomes over time, in part because
of frequent changes in states' reporting requirements that prevent
establishing reliable trend data.
^1See the following GAO reports: The Results Act: An Evaluator's Guide to
Assessing Agency Annual Performance Plans, [34]GAO/GGD-10.1.20 (Washington
D.C.: April 1998); Agencies' Annual Performance Plans under the Results
Act: An Assessment Guide to Facilitate Congressional Decisionmaking,
GAO/GGD/AIMD-10.1.18 (Washington, D.C: February 1998); Executive Guide:
Effectively Implementing the Government Performance and Results Act,
GAO/GGD-96-118 (Washington, D.C.: June 1996).
Labor's data on services and outcomes for veteran job seekers paint an
unclear picture of veterans' use of employment and training services in
the one-stop system. Despite the shared use of common performance
measures, key employment and training programs vary in the extent to which
their data on veteran participants are integrated or shared with other
programs. As a result, many states may not know how many veterans they
serve through the one-stop system. For example, most states reported that
veteran job seekers receive initial assistance from the Employment
Service, which uses the same reporting system as the DVOP and LVER
programs and produces separate quarterly reports on services and outcomes
for veterans. However, states use a different reporting system for WIA,
and fewer than half the states told us that they do not match WIA and
Employment Service records to obtain an unduplicated count of veterans
served by those programs. In addition, Labor and some state officials we
surveyed reported that statutory differences in the way veterans are
defined for purposes of program eligibility make it difficult to
standardize data across employment programs. For example, a veteran is
generally defined for the DVOP and LVER programs as an individual who
served on active duty for more than 180 days, while WIA does not specify a
length of time in service. About half of states claimed that the
conflicting definitions in various employment programs complicate data
entry, referrals to other programs, and the implementation of priority of
service. Moreover, Labor has no method of gauging the extent to which
priority of service for veterans has been implemented in various
employment programs, despite JVA's requirement that the agency include
this information in its annual report to Congress.
Labor has taken some steps to improve the quality of performance data and
better understand veterans' services and outcomes, but the overall impact
of employment services for veterans is unknown. Labor has developed
processes to enhance data quality. For example, ETA requires states to
validate some key performance data reported for Employment Service and
WIA-funded programs. In addition, ETA and VETS have begun to work together
on issues of data quality by, for example, conducting joint monitoring
visits to states that include an examination of the data. Furthermore,
Labor's planned implementation of an integrated data reporting system
could greatly enhance the understanding of veterans' services and
outcomes. The new system is designed to integrate and expand data
reporting across multiple employment programs, including the Employment
Service, WIA, Trade Adjustment Assistance (TAA), and the DVOP and LVER
programs. Using this system, for the first time, Labor and states would be
able to track individual veterans' progress through different programs in
the one-stop system. However, according to a Labor official, states have
expressed concerns about the time needed to implement such a system, and
the timeline for implementation remains unclear. Furthermore, while
outcome information on veteran job seekers helps assess whether
individuals are achieving their intended goals--such as obtaining
employment--it cannot measure whether the outcomes are a direct result of
program participation, rather than external factors. Labor has sponsored
research on services to veterans. However, it has not conducted an impact
evaluation, as required under WIA, to assess the effectiveness of one-stop
services. Such a study should include impacts for key participant groups,
including veterans. We recommended in a prior report that Labor take steps
to conduct such an evaluation, but there has been no action to date.
To improve performance information on veteran job seekers, we are making a
number of recommendations to Labor regarding changes to the performance
measures for the DVOP and LVER programs, standardization of the different
veteran definitions in employment programs, guidance on Labor's proposed
integrated data-reporting system, and inclusion of veterans' services in
an impact evaluation. In its comments, Labor generally concurred with our
findings, conclusions, and recommendations.
Background
The nation's veteran job seekers receive employment and training services
from programs overseen by two agencies within Labor--the Veterans'
Employment and Training Service and the Employment and Training
Administration. General employment services fall under the purview of ETA,
which administers the Wagner-Peyser-funded Employment Service program,
providing a national system of public employment services to all
individuals seeking employment--including veterans. Thus, those veterans
considered job ready and not in need of intensive services may be served
by Employment Service staff and receive such services as assessment,
counseling, job readiness evaluation, and placement. ETA carries out its
Employment Service program through workforce agencies in each state. In
fiscal year 2006, the Employment Service program provided a total of about
$716 million to states.
While ETA administers programs that serve the general population,
including veterans, VETS administers the DVOP and LVER programs, which
focus exclusively on serving veterans, often providing more intensive
services than the Employment Service does. Like ETA, VETS carries out its
responsibilities through a nationwide network that includes representation
in each of Labor's six regions and staff in each state. The Office of the
Assistant Secretary for VETS administers the agency's activities through
regional administrators and state directors. The DVOP specialists and LVER
staff, whose positions are funded by VETS, are part of states' public
employment services. In fiscal year 2006, the DVOP and LVER programs were
funded at about $155 million. In the most recent program year--program
year 2005, which spanned July 1, 2005, to June 30, 2006--the Employment
Service, together with the DVOP and LVER programs, reported serving about
1.32 million veterans nationwide, of whom over 715,000 were served by DVOP
specialists and LVER staff.
The Employment Service and the DVOP and LVER programs are mandatory
partners in the one-stop system under WIA--where services are provided by
a range of employment and training programs in a single location.
Veterans, along with other eligible job seekers, may receive services from
other mandatory one-stop partners, such as WIA-funded training or Trade
Adjustment Assistance.^2 Additionally, job seekers, including veterans,
may use the one-stop centers' computers and other resources without staff
assistance, and in many places may access one-stop services online from
home. Department of Veterans Affairs (VA) programs are not mandatory
partners in the one-stop system, but do participate at some locations.
Jobs for Veterans Act Reforms
In 2002, the Jobs for Veterans Act amended Title 38 of the U.S. Code--
which governs the DVOP and LVER programs--and by doing so, introduced an
array of reforms to the way employment and training services are provided
to veterans.^3 JVA sought to address concerns that the programs were
overly prescriptive by providing states with enhanced flexibility to
determine the best way to serve veteran job seekers. Among its reforms,
JVA
^2The Trade Adjustment Assistance program is the primary federal
employment and training program serving workers from the manufacturing
sector who are dislocated due to trade.
o redefined the DVOP specialist and LVER staff roles but gave
states flexibility in deciding their duties;
o established a single state grant and a new funding formula that
allowed states to determine the mix of DVOP specialists and LVER
staff;
o required a comprehensive performance accountability system
consistent with WIA performance measures;
o required that veterans receive priority over other job seekers
in all Labor job training programs, not just the Employment
Service;^4 and
o required that VETS include information in its annual report to
Congress on employment services to veterans throughout the
one-stop system.
JVA identified broad roles and responsibilities of DVOP specialists and
LVER staff. For example, DVOP specialists are to focus on providing
intensive services to eligible veterans, giving priority to disabled
veterans and those with other barriers to employment. LVER staff are to
focus on conducting outreach to employers to assist veterans in gaining
employment, as well as facilitating employment, training, and placement
services given to veterans. State workforce agencies receive a single
veterans' program grant to fund both programs; the amount each state
receives is determined in part by the size of the veteran population
within each state. State agencies then decide how to distribute the amount
they receive between the two programs. Table 1 lists selected
responsibilities of DVOP specialists and LVER staff as set forth in Labor
guidance.
^3Prior to JVA, Title 38 provided that there was to be one DVOP for each
7,400 veterans in a state and prescribed 11 functions for DVOP to carry
out in providing services to eligible veterans. Similarly, Title 38
formerly provided that, in any fiscal year, funding should be available
for 1,600 full-time LVER staff and prescribed 13 functions to be performed
by the LVER staff.
^4Veterans were required to receive priority of service in the Employment
Service before JVA was enacted.
Table 1: Selected Responsibilities of DVOP Specialists and LVER Staff
DVOP specialists LVER staff
Unique o Facilitate intensive o Promote veterans' skills
responsibilities services to veterans and experience and advocate
with special employment for veterans with employers
and training needs through activities such as
o Target services to job fairs
disabled veterans and o Facilitate and maintain
veterans with other regular contact with
barriers to employment, employers
such as homeless
veterans
Shared Both DVOP specialists and LVER staff may
responsibilities
o Provide a full range of employment and training
services to veterans
o Facilitate employment workshops for those leaving
the military
Source: GAO analysis of Labor data.
JVA also stipulated that veteran job seekers must receive priority over
other job seekers in any job training program administered by Labor.
Labor's guidance requires states to explain how veterans will be given
priority and how veterans' services will be provided through the state's
one-stop system. For programs that target particular populations, such as
seniors or low-income individuals, veterans' priority is applied after any
other mandatory eligibility provisions are met.
Changes to the DVOP and LVER Performance Information
Like other Labor employment and training programs, the DVOP and LVER
programs have experienced changes both in the way outcomes are tracked and
in the measures used to assess performance. Specifically, in 1998, WIA
required that states use automated unemployment insurance wage records to
track employment-related outcomes. Formerly, to obtain data on outcomes,
states relied on a manual follow-up process using administrative records
or contacts with job seekers. To conform to WIA, VETS moved from such a
manual follow-up system to the new automated process in 2002.
The measures that Labor uses to assess performance in the DVOP and LVER
programs have also changed over time, gradually reflecting more emphasis
on outcome-based measures. Before passage of the JVA in 2002, for example,
some of the measures used for the DVOP and LVER programs focused more on
services received--such as the number of veterans in training or receiving
counseling--than on outcomes achieved. In 2002, JVA required that Labor
develop a comprehensive performance accountability system and required
that the new system measure performance in a way that is consistent with
WIA. In 2003, VETS adopted performance measures for the DVOP and LVER
programs based on those then used in WIA. In 2005, in response to an
Office of Management and Budget (OMB) initiative, Labor began requiring
states to implement common performance measures for its employment and
training programs, including the DVOP and LVER programs, the Employment
Service, and WIA. OMB established a set of common measures to be applied
to most federally funded job training programs that share similar goals.
Labor further defined the common measures for all of its Employment and
Training Administration programs, applying three measures to each of its
adult programs (see table 2).
Table 2: Common Measures Adopted by Labor's Employment and Training
Programs
Common measure Definition
Entered employment rate The number of participants employed in the first
quarter after exiting the program, divided by the
total number of participants who exit the program
during the quarter.
Employment retention Of those participants who are employed in the
rate first quarter after exiting the program, the
number employed in both the second and third
quarters after exit, divided by the number of
participants employed in the first quarter after
the quarter of exit.
Average earnings Of those participants who are employed after the
first, second, and third quarter after the exit
quarter, total earnings in the second quarter plus
total earnings in the third quarter after the exit
quarter divided by the number of adult
participants who exit during the quarter.
Source: Labor.
In applying the common measures to its programs, VETS also developed
additional measures to emphasize outcomes for disabled veterans in the
DVOP program and outcomes for recently separated veterans in the LVER
program.
Data Reporting
Labor collects performance data for the DVOP and LVER programs on a
quarterly basis from state workforce agencies. The state agencies use
report formats developed by Labor to provide detailed tabulations of
aggregate information on the characteristics of veteran participants,
services, and outcomes for the two programs, including data showing
states' performance using the common measures. The state agencies provide
this information to Labor in three separate reports: one for the DVOP
program, one for the LVER program, and one representing an unduplicated
count for both programs.
Furthermore, Labor collects additional information on veterans who
participate in other Labor programs. For example, ETA collects performance
data for the Employment Service on all participants on a quarterly basis
from state workforce agencies, and these reports break out services and
outcomes for veteran participants. States submit their quarterly reports
for the Employment Service and the DVOP and LVER programs through the same
Labor reporting system.
The Role of Impact Evaluations in Assessing Program Outcomes
Information on the services a program has provided and the outcomes
obtained by program participants are necessary to assess program impacts.
However, this information is not sufficient to measure program
impacts--the outcomes may be due to other external factors such as local
labor market conditions. While impact evaluations allow one to isolate a
program's effect on the outcomes of participants, there are several
approaches to conducting such evaluations. The experimental method is
often considered the most rigorous method for conducting impact
evaluations. In the experimental method, participants are randomly
assigned to two groups--one that receives a program service (or treatment)
and one that does not (control group). The resulting outcome data on both
groups are compared and the difference in outcomes between the groups is
taken to demonstrate the programs impact. However, it is not always
feasible to use the experimental method for assessing program impacts.
Alternatively, researchers may use a quasi-experimental approach in which
program participation is not randomly assigned. One approach, often called
a comparison group study, compares outcome data for individuals who
participated in the program with data on others who did not participate
for various reasons. In a comparison group study, it is important to find
ways to minimize, or statistically control for, any differences between
the two groups. According to OMB, well-matched comparison group studies,
under certain circumstances, can approach the rigor of the experimental
method, and it recommends considering this method if random assignment is
not feasible or appropriate.
Under WIA, Labor was required to conduct at least one impact evaluation of
program services by 2005. In a 2004 report, we found that Labor had not
yet begun such an evaluation, and recommended that the agency comply with
this statutory requirement and help federal, state, and local policy
makers understand what services are most effective for improving
employment-related outcomes.^5
DVOP and LVER Performance Information Is Weakened by Several Factors
The DVOP and LVER programs' performance information is weakened by several
factors, including implementation challenges and frequent changes to
performance reporting requirements. In July 2005, Labor implemented new
performance measures, which provide information on some outcomes for
veterans. However, not all performance measures have been fully
implemented. Additionally, neither the performance measures nor the data
reported to Labor reflect the full range of services that DVOP specialists
and LVER staff provide to veteran job seekers. Furthermore, it is
difficult to assess outcomes over time or across states because of
frequent changes in states' reporting requirements that prevent
establishing reliable trend data.
The New Performance Measures Provide Information on Some Outcomes for Veterans,
but Have Not Been Fully Implemented
In July 2005, the DVOP and LVER programs adopted the Office of Management
and Budget's common measures, along with other employment programs,
including WIA and the Employment Service. Specifically, states implemented
measures that track whether veterans obtain and keep jobs after receiving
services through these programs, but they have not yet implemented a
measure to track veterans' earnings. States are held accountable for four
separate measures in each program that focus on outcomes attained by
veterans (see table 3). For the DVOP program, states are held accountable
for employment and retention for all veterans served by the program, as
well as for disabled veterans. For the LVER program, states are assessed
on employment and retention for all veterans, as well as for recently
separated veterans. Currently, all states collect and report data to Labor
for calculating performance attainment and negotiating state goals for
these eight measures.^6
5GAO, Workforce Investment Act: States and Local Areas Have Developed
Strategies to Assess Performance, but Labor Could Do More to Help,
[35]GAO-04-657 (Washington, D.C.: June 1, 2004).
^6See appendix II for a state-by-state list of negotiated and achieved
performance goals for the programs.
Table 3: Performance Measures for the DVOP and LVER Programs
Recently
Disabled separated
Program Type of measure All veterans veterans veterans
DVOP Entered employment SQRT SQRT
rate
Employment retention SQRT SQRT
rate
LVER Entered employment SQRT SQRT
rate
Employment retention SQRT SQRT
rate
Source: Labor.
However, states are not yet held accountable for an additional common
measure--veterans' average earnings--in either the DVOP or the LVER
programs. Other employment and training programs, such as WIA and the
Employment Service, include an average earnings measure for which states
are accountable. For the DVOP and LVER programs, however, calculating the
average earnings was not as straightforward as Labor had anticipated. A
VETS official told us that the agency will calculate baseline data for
average earnings during the current program year, but Labor will not
establish goals and states will not be held accountable for their
performance on this measure until the following year--program year
2007--at the earliest.
Furthermore, Labor has not adopted a system to give more weight to
successful outcomes for veterans who have substantial barriers to
employment, such as a disability. JVA required Labor to weight performance
measures to provide special consideration to veterans requiring intensive
services, as well as disabled veterans. Such a weighting system would
compensate for the fact that veterans with barriers to employment may need
more assistance than others in finding jobs. It would also provide an
incentive for program staff to help veterans with severe barriers to
employment. For example, if a veteran has a disability and requires
intensive case management services, his or her successful outcomes would
have a greater effect on a state's overall performance than those of other
veterans with fewer barriers. Following JVA's enactment, Labor formed a
work group to develop a weighting system for the DVOP and LVER performance
measures. On the basis of the group's work, the agency issued guidance to
introduce the weighted measures to states in June 2003, with the
expectation of implementing them soon after. However, after further
review, a Labor official told us the agency did not implement the weights
in order to give states time to fully implement other reporting changes.
At this time, it is not clear whether Labor will implement this system in
the future.
Performance Measures and the Data Reported to Labor Do Not Reflect the Full
Range of Staff Services
Although DVOP specialists and LVER staff perform similar duties for all
types of veterans in most states, the current performance measures hold
the two programs accountable for different groups of veterans. JVA and
Labor's guidance outline the key responsibilities and target populations
for DVOP specialists and LVER staff, but also allow for some flexibility
in their roles and responsibilities. Both DVOP and LVER staff are expected
to serve the general veteran population, but DVOP specialists are also
expected to target their services toward veterans who have greater
barriers to employment and need intensive case management, including
disabled veterans. JVA specifies that LVER staff focus on conducting
outreach to employers and assisting all veteran job seekers. In addition,
Labor has recently added the expectation that LVER staff focus their
responsibilities on assisting recently separated veterans. As a result of
these expectations, Labor separately holds DVOP specialists accountable
for the outcomes achieved by the disabled veterans they serve, and LVER
staff for the outcomes of the recently separated veterans they serve.
In practice, however, both programs' staff serve similar veteran
populations. In program year 2005, for example, 14 percent of veterans
served by the DVOP program were disabled and 21 percent were recently
separated. For the LVER program, 10 percent of veterans served were
disabled and 19 percent were recently separated (see table 4). States
acknowledged this similarity in our survey as well. Over a third of states
responded that DVOP and LVER staff are equally likely to serve disabled
veterans, while about half of states responded that the two programs'
staff are equally likely to serve recently separated veterans.
Table 4: DVOP Specialists and LVER Staff Served Similar Proportions of
Recently Separated and Disabled Participants in Program Year 2005
Veterans
Recently separated Disabled
Number Percentage Number Percentage Total served
Served by 78,843 21 52,331 14 382,144
DVOPs
Served by 78,379 19 41,847 10 408,837
LVERs
Source: Labor.
Note: These numbers include 49 states, the District of Columbia, and
Puerto Rico. They do not include Pennsylvania because it is piloting a new
reporting system and currently not submitting the standard quarterly
reports for the DVOP and LVER programs.
In addition to finding similarity in populations served, we also found
some similarity in activities carried out by DVOP and LVER staff. States
reported that the three activities on which DVOP specialists spend the
most time include
o providing intensive case management services,
o conducting an initial assessment or interview, and
o assisting with job search activities.
The top three activities LVER staff perform include
o conducting outreach to employers,
o assisting with job search activities, and
o conducting an initial assessment or interview.
This division of duties appears to reflect the different focuses of the
two programs, as well as the flexibility under JVA for states to decide on
staff duties. However, almost 85 percent of states responded that DVOP
specialists conduct outreach to employers, a focus of the LVER program.
Additionally, almost 60 percent of states responded that LVER staff
provide intensive services, a primary focus of the DVOP program.
In our site visits, we found that this similarity in staff roles and
target populations exists in part because some one-stop centers have only
a single DVOP specialist or LVER staff on duty at any given time. In these
particular one-stop centers, the same employee is responsible for serving
all groups of veterans and carrying out job roles for both programs. Even
in centers with more than one staff person, veterans tend to be served by
whichever staff person is available at that time. Program staff in several
centers told us that recently separated veterans were not specifically
directed to LVER staff for services, nor were disabled veterans directed
to DVOP staff. This sharing of duties may be due, in part, to changes in
staffing levels. More than half of states reported a decrease in the
number of full-time DVOP specialists or LVER staff over the last 2 years,
and most attributed this decline to the size of their state grant for the
programs.
Nevertheless, this similarity in roles and populations served causes the
current performance measures to present an incomplete view of outcomes for
disabled and recently separated veterans in the DVOP and LVER programs.
The large numbers of disabled veterans served by the LVER program and
recently separated veterans served by the DVOP program are not included in
the set of measures that focus on the outcomes of those populations (see
fig. 1).
Figure 1: A Large Proportion of Disabled and Recently Separated Veterans
Are Not Captured in Performance Measures for Those Populations (Program
Year 2005)
Beyond the measures for disabled and recently separated veterans, having
separate measures for the DVOP and LVER programs obscures the overall
picture of outcomes to veterans, given the similarity between many of the
program activities and the reality of how the programs operate. According
to our survey, almost half the states would like Labor to consolidate the
performance measures for the DVOP and LVER programs.
While the performance measures present an incomplete view of the outcomes
for veterans, the data that states are required to report to Labor do not
reflect the full range of staff services. Labor requires states to report
a wide range of data for the DVOP and LVER programs, including information
on veteran characteristics--such as age and disability status--and staff
services provided--such as intensive services and referrals to other
programs. However, Labor does not require data on employer outreach
activities, despite JVA's designation of employer outreach as a primary
job responsibility of LVER staff. Consequently, Labor and states cannot
formally monitor the extent to which staff perform this activity. Some
states, however, collect these data for their own use. According to our
survey, almost half of states currently collect employer-related
information for the DVOP and LVER programs, and over 75 percent of states
reported that it would be helpful to collect these data.
In addition, even though the data reported to Labor generally reflect
services and outcomes for veterans, these data are aggregate tallies and
do not show services provided to individual veterans. For example, each
state's quarterly reports reflect the sum of all services provided and do
not show the number of services provided per veteran or per staff person.
The current data are useful to provide an overall picture of the programs'
volume and operations. However, these data provide little information
about services received by individual veterans or delivered by particular
veteran staff.
Performance Data Are Not Comparable over Time or across States
In recent years, reporting requirements for the DVOP and LVER programs
have undergone several significant changes. These changes have moved the
performance accountability system closer to those of other employment and
training programs. At the same time, the changes have resulted in a lack
of reliable trend data. In July 2002, the DVOP and LVER programs changed
from using administrative follow-up to determine veterans' employment
outcomes to obtaining information from Unemployment Insurance (UI) wage
records. In doing so, Labor changed its method of calculating outcomes for
veterans in the DVOP and LVER programs.^7 Then, in July 2005, Labor
applied the common measures to these two programs, refining and
standardizing the application of UI wage records to determine outcomes.
Under the old system, Labor calculated entered employment and employment
retention rates based on the number of veterans who participated in the
programs. However, under the new system, Labor calculated these rates
based on how many veterans terminate services and exit the programs.
Although these changes have standardized the performance measures across
programs, they have also prevented Labor and states from developing
consistent, comparable data over the past 5 years. As a result, Labor does
not have reliable historic data for either program. Figure 2 illustrates
the various changes to the DVOP and LVER programs' performance reporting
requirements.
^7Because UI wage records only contain aggregate quarterly earnings for
individuals, it was necessary for Labor to change the way the measures
were calculated.
Figure 2: Changes to Performance Information over Time for the DVOP and
LVER Programs
Furthermore, the instability in data collection and reporting has left
Labor unable to establish a national veterans' entered employment
standard, as required by JVA. Labor anticipates that it will need at least
3 years of stable data to establish the national standard. Once it is
established, all states will be held accountable to the same minimum goal
for veterans' entered employment. However, it is unclear when Labor will
have sufficient data to establish this standard because states continue to
experience difficulty adjusting to the numerous changes. According to our
survey, over 70 percent of states reported that frequent changes to
performance reporting requirements have been either a great or very great
challenge.
The data also vary somewhat state by state. For example, the application
of wage records to calculate veteran outcomes across state lines is no
longer consistent across states. The Wage Record Interchange System (WRIS)
allows states to share UI wage records and account for job seekers who
participate in one state's employment programs but get jobs in another
state. In recent years, all states but one participated in WRIS, which was
operated by the nonprofit National Association of State Workforce
Agencies. In July 2006, Labor assumed responsibility for administering
WRIS. However, many states have withdrawn, in part because of a perceived
conflict of interest between ETA's role in enforcing federal law and the
states' role in protecting the confidentiality of their data. As of March
2007, only 30 states were participating in the program, and it is unknown
if and when the other states will enter the data-sharing agreement. As a
result, DVOP and LVER performance information in almost half the states
will not include employment outcomes for veterans who found jobs outside
the states in which they received services. In addition, other reasons
contribute to data variation by state. Labor allows states flexibility in
choosing data collection software, which has resulted in some states
adapting more quickly than others to the recent changes, depending on
their software capabilities. Several Labor officials told us that because
of differences in software capabilities, some states' data may be more
reliable than others'.
Available Data Paint an Unclear Picture of Veterans' Use of One-Stop Services
Labor's data on veteran job seekers paint an unclear picture of their use
of employment and training services in the one-stop system, despite the
shared use of common performance measures across programs. Although many
veterans use employment services other than those provided by the DVOP and
LVER programs, key employment programs vary in how well their data on
veteran participants are integrated or shared with other programs. As a
result, many states may not know how many veterans use one-stop services.
In addition, statutory differences in the way veterans are defined for
purposes of program eligibility make it difficult to standardize data
across employment programs. Moreover, Labor has no means of assessing
whether priority of service for veterans has been implemented in various
employment programs.
Veterans Receive Services from Multiple Programs in the One-Stop System
Many veteran job seekers receive employment services from the DVOP and
LVER programs. However, some veterans--often the more job-ready--only use
one-stop services aimed at the general population, such as the Employment
Service and WIA programs. In addition, some veterans use services focused
on other subsets of job seekers--such as TAA (see fig. 3). As a result,
performance information on many veterans is collected and reported
elsewhere in the one-stop system. In fact, 20 states reported that about
half or fewer of veteran job seekers who access employment programs
receive services from a DVOP specialist or LVER staff, according to our
survey (see fig. 4). In addition, some veterans obtain services from more
than one employment program in the one-stop system, all of which use the
common measures to assess their performance.
Figure 3: Veterans Can Access Multiple Employment Services in the One-Stop
System
Note: The figure reflects only the services provided by some of the
mandatory partners in the one-stop system. Other optional partners may
participate, but the specific optional partners vary by location.
Figure 4: Many States Report That DVOP Specialists or LVER Staff See Half
or Fewer of Veterans That Come into One-Stop Centers
Data on Veterans' Services Are Well Integrated across Some but Not All Programs
in the One-Stop System
Performance data on veteran job seekers are well integrated or shared
across some key employment and training programs, but not others, despite
the mutual use of common measures. As a result, many states may not know
how many veterans they serve through the one-stop system. Data on veterans
who access the Employment Service are completely integrated with data from
the DVOP and LVER programs--they share the same reporting system, and DVOP
and LVER data are a subset of Employment Service data. According to our
survey, veteran job seekers in most states receive initial assistance from
the Employment Service when they access the one-stop system. If they are
subsequently referred to the DVOP and LVER programs, all of their
information is housed in the same system and an unduplicated count of
veterans served between these programs can be obtained. In addition,
states are held accountable for meeting separate goals in the Employment
Service for veterans and disabled veterans (see app. III). Labor considers
these measures to reflect veterans' outcomes for the entire one-stop
system, as they constitute outcomes for all veterans who access the
Employment Service, DVOP, and LVER programs. Furthermore, they are the
best approximation of a total count of veterans who access the one-stop
system that the current data will allow.
On the other hand, data on veterans served by other one-stop programs are
not well integrated. States report data to Labor on WIA participants who
exit the programs, including veterans, using the Workforce Investment Act
Standardized Record Data (WIASRD) system. Although WIASRD contains
sufficient information to produce separate veteran outcome data for WIA
programs, states are not required to produce separate veteran reports and
are not accountable for meeting veteran goals in those programs. In
addition, fewer than half the states reported that they routinely match
WIA and Employment Service records to attain an unduplicated count of
veterans served by those programs. Consequently, veterans who access two
different employment services may be counted twice in some cases. Data for
TAA participants are reported to Labor in yet another data system, which
does allow states to report on the veteran status of participants, but
Labor officials told us the agency does not currently use veteran outcomes
from that program for any purpose.^8 VETS does not include the veteran
outcome data from WIA or TAA in its annual report to Congress, and Labor
officials told us they are exploring ways to better use the data.
In addition, data are not always collected on job seekers who use
Employment Service or WIA resources without assistance from program staff.
These self-assisted job seekers--including veterans--access services such
as labor market or career information either in one-stop centers or on
home computers, but do not receive active assistance from program staff.
Historically, some states have collected information on these job seekers,
while others have not. In our survey, 73 percent of states reported that
they capture information on all veterans who receive self-assisted
services through the Employment Service, while 82 percent of states
reported doing so for all veterans who receive self-assisted WIA services.
Labor has encouraged--but not mandated--states to collect information on
this group of job seekers, but agency officials acknowledged that states
continue to vary in how they report such data. Labor officials have
expressed concern that requiring veterans who receive self-assisted
services to register for the programs might discourage some of them from
pursuing the services they need.
^8For more information on TAA outcome data, see GAO, Trade Adjustment
Assistance: Labor Should Take Action to Ensure Performance Data Are
Complete, Accurate, and Accessible, [36]GAO-06-496 (Washington, D.C.: Apr.
25, 2006).
Different Veteran Definitions Complicate Efforts to Standardize Data and
Implement Priority of Service
Labor and some state officials we surveyed reported that statutory
differences in the definitions of veterans for various employment programs
make it difficult to standardize data across programs. For the purposes of
the DVOP and LVER programs, an eligible veteran is statutorily defined as
an individual who served on active duty for more than 180 days.^9 Labor
also uses this definition for the Employment Service. WIA, on the other
hand, does not specify a length of time in service for a person to be
considered a veteran. Moreover, to qualify as a recently separated veteran
in the DVOP and LVER programs, a veteran must have left active duty in the
last 3 years. By contrast, WIA defines recently separated as having left
active duty in the last 4 years (see table 5).
Table 5: Differences between Veteran Definitions in Employment and
Training Programs
Length of time in Recently separated
service veteran Statutory source
DVOP, LVER, and Served on active Up to 3 years 38 U.S.C. 4211
Employment duty for a period of after discharge or
Service programs more than 180 days release from
active duty
WIA programs No requirement for Up to 4 years Workforce
time in service after discharge or Investment Act of
release from 1998 (29 U.S.C.
active duty 2801)
Source: GAO analysis.
Note: To be eligible, all programs require that veterans be discharged or
released from duty under conditions other than dishonorable.
These inconsistent definitions have been difficult for Labor and states to
reconcile with the concept of seamless service delivery and have caused
some confusion for states as they implement priority of service throughout
the one-stop system. While JVA requires that veterans receive priority
over other job seekers in Labor-funded employment and training programs,
it does not define a veteran for purposes of the priority requirement.
Labor has interpreted JVA's provisions to mean that while veterans are to
receive preference in the programs after any other statutory eligibility
requirements are met, each program must use its own statutory definition
of a veteran in applying that preference. Labor officials told us that one
state applied for a waiver in 2006 to use a single definition of veterans
for all of its employment and training programs, but Labor's Solicitor's
Office orally denied the request. In our survey, approximately half of all
states reported that the conflicting veteran definitions in various
employment programs complicate data entry, referrals to other programs,
and the implementation of priority of service. In addition, about a third
of the states claimed that the definitions created gaps in services for
veteran clients as they moved among employment programs (see fig. 5). For
example, if a veteran receives services from WIA and is subsequently
referred to the DVOP program but is found ineligible, he or she may become
discouraged and stop seeking services altogether.
^9Veterans discharged because of service-connected disabilities and
members of reserve components who have served on active duty during a war
or designated campaign are not subject to a 180-day requirement.
Figure 5: Percentage of States Responding that Different Definitions
Affect the Coordination of Services to Veterans
Almost half of states shared their concerns about different definitions by
providing additional comments in our survey, many of which cited the
difficulty of providing priority of service under these circumstances. For
example, one state responded that different definitions often lead to
inappropriate referrals, resulting in poor customer service and
frustration for program participants and service providers. Other states
focused on the burden that competing definitions placed on data collection
and reporting. For example, one state responded that the issue has made it
difficult to integrate the state's Employment Service and WIA data
systems, because the different definitions could lead to invalidating the
veteran numbers on reports for those programs. Another state cited the
difficulty in assessing how many veterans were served by the state,
highlighting the complexity of producing an unduplicated count of veterans
served by different programs that do not share a single definition. States
also cited challenges in dealing with other agencies that are not mandated
partners in the one-stop system. For example, two states mentioned that
some staff of other agencies' programs may hesitate to refer participants
to the DVOP and LVER programs because they are unsure about participant
eligibility. An expert on veterans' issues in the states concurred that
the different eligibility criteria for veterans has been a problem for
states and told us that a common veteran definition for employment and
training programs would be an improvement.
Labor Has No Means of Evaluating Priority of Service for Veterans
Despite JVA's mandate, Labor has not produced information on the extent to
which veterans receive priority of service in all qualified employment and
training programs. Specifically, JVA required Labor to evaluate and report
on whether veterans are receiving priority of service and are fully served
by its employment programs, as well as whether the representation of
veterans in such programs is in proportion to their participation in the
labor force. In its fiscal year 2005 report, Labor stated that the
participation rate for veterans in its adult programs was approximately
8.4 percent--slightly higher than veterans' participation rates in the
U.S. workforce. In addition, the agency reported that outcomes for
veterans served in these programs closely mirrored those of all job
seekers in the programs. However, Labor has no method of gauging how--and
how consistently--priority of service is actually applied. Labor officials
told us that the highly devolved workforce development system makes it
very difficult to evaluate priority of service, because different programs
have multiple access points and diverse eligibility criteria that prevent
Labor from applying a simple measurement technique to each.
States reported that implementing priority of service has been
challenging, as has holding one-stop partner programs accountable for
serving veterans. To supplement federal guidance on this issue, at least
one state has developed its own guidance for implementing and measuring
priority of service. Some Regional Directors of VETS told us they
encouraged the use of that state's guidance as a model for assessing
priority of service for states in their own regions. We do not know when
Labor will develop further guidance on the issue. However, in December
2006, Congress passed the Veterans Benefits, Health Care, and Information
Technology Act of 2006, which included a requirement that Labor release
regulations on implementing priority of service within 2 years. In
addition, the agency has begun planning a study of priority of service in
response to our prior report.^10 According to Labor officials, the study
will combine a survey of participants with a process evaluation and an
analysis of outcomes. Labor does not yet know when the study will get
under way.
Labor Has Taken Steps to Better Understand Veterans' Outcomes, but the Programs'
Impact Remains Unknown
Labor has taken some steps to improve the quality of performance data and
better understand veterans' services and outcomes, but the overall impact
of employment services for veterans is unknown. Labor has developed some
processes to enhance data quality. For example, Labor's ETA requires
states to validate some data in key programs. Furthermore, Labor plans to
implement an integrated data-reporting system that would allow Labor and
states to track individual veterans' progress through different programs
in the one-stop system. Additionally, the new system would expand data
collection by, for example, collecting more data on services to employers.
However, states have raised concerns about the challenge of meeting the
system's planned implementation date, and the timeline for implementation
remains unclear. Furthermore, while performance information helps assess
whether individuals are achieving their intended outcomes--such as
obtaining employment--it cannot measure whether the outcomes are a direct
result of program participation, rather than external factors. To measure
the effects of a program, it is necessary to conduct an impact evaluation
that would seek to assess whether the program itself led to participant
outcomes. Labor has sponsored research on services to veterans. However,
it has not conducted an impact evaluation, as required under WIA, to
assess the effectiveness of one-stop services. Such a study should include
impacts for key participant groups, including veterans. We recommended in
2004 that Labor take steps to conduct such an evaluation, but there has
been no action to date.
Labor Has Developed Some Processes to Enhance the Quality of Performance Data
Labor has taken some steps to improve the quality of performance data and
enhance the understanding of veterans' services and outcomes. To address
data quality concerns, ETA has developed processes requiring states to
validate certain data reported for participants in WIA and
Wagner-Peyser-funded Employment Service programs. However, while these
programs serve veterans, participant records are randomly selected in both
programs from the total participant population and, therefore, may not
include the records of veteran participants.
^10GAO, Veterans' Employment and Training Service: Labor Actions Needed to
Improve Accountability and Help States Implement Reforms to Veterans'
Employment Services, [37]GAO-06-176 (Washington, D.C.: Dec. 30, 2005).
Both the WIA data validation process, developed in 2004, and the
Employment Service process, developed in 2003, involve two types of data
validation, although the WIA process is more intensive, according to Labor
officials. Both processes involve (1) data element validation--comparing
randomly sampled participant records to source files, and (2) report
validation--assessing whether states' software accurately calculated
performance outcomes. While element validation in WIA is conducted on-site
with hardcopy source documentation, the Employment Service data validation
process is performed centrally and electronically, because Employment
Service records are generally electronic.^11 The Employment Service
element validation process checks for duplicate or invalid entries in
source files by, for example, checking for inconsistencies among various
veteran-related fields, such as veteran status and disabled veteran.
However, the Employment Service element validation process cannot check
the underlying accuracy of the data, because there is no hard copy
documentation to prove whether a participant is in fact a veteran.
Labor officials told us that the Employment Service data validation
process has been helpful in raising awareness among states about the
importance of data quality and that some states have come to see it as a
useful tool. Additionally, states responding to our survey generally
agreed that it has been effective--38 states, or about 75 percent, rated
the Employment Service data validation process as effective in ensuring
the accuracy of veteran job seekers' information. For example, according
to one respondent, review of the data validation results is used as a
management tool, to highlight successes and to alert staff to weaknesses.
Nevertheless, some states have expressed concerns about the data
validation processes. Concerns about the process were also raised by state
officials in all 3 of the states we visited. For example, officials in 2
of the 3 states noted that they had experienced difficulties adjusting to
frequent changes in software before the results were due to Labor. On our
survey, 2 states said that the sample size was too small to be meaningful,
and 4 states expressed concerns about the fact that the process does not
verify the accuracy of the data in source files. These concerns are
similar to those we identified in a previous report that addressed the WIA
process.^12
11For more information on the WIA data validation process, see GAO,
Workforce Investment Act: Labor and States Have Taken Actions to Improve
Data Quality, but Additional Steps Are Needed, [38]GAO-06-82 (Washington,
D.C.: Nov. 14, 2005).
Additionally, Labor has taken steps to address data quality as a part of
its routine monitoring and technical assistance. Specifically, beginning
in 2004, ETA regional staff have incorporated a data quality component
into compliance visits to state offices, which are generally conducted
once or twice a year, according to Labor officials. Data validation is
just one component of these compliance visits, which typically do not
focus on veterans' data as a separate issue. To support this effort, Labor
officials told us that ETA has amended its monitoring guide for these
visits to include a section on data validation. According to Labor
officials, these visits have been useful in identifying problems and
corrective actions. Moreover, ETA and VETS have recently collaborated on a
few of these compliance visits.^13 Labor officials said they believed this
joint monitoring was beneficial, and expect those efforts to be a model
for future joint visits.
There are several other forms of management reviews that generally focus
on services to veterans but also offer a chance to review data. For
example, VETS regional and state-based staff conduct site visits as part
of their routine monitoring, which focus primarily on services to veterans
but which can include reviewing performance information as well.
Additionally, VETS has required a series of annual assessments--of the
program for each state overall, and self-assessments by DVOP specialists,
LVER staff, and one-stop managers--that address data issues to a limited
extent. State directors use performance data to substantiate services
described in the self-assessment. For example, according to one official
we spoke with, to confirm a LVER staff's claim of travel to several job
fairs, the director can consult the one-stop's travel log to substantiate
whether the LVER staff actually made the trips.
Beyond the steps Labor has taken, state workforce agencies also perform
functions that affect performance data on services to veterans. Most
states responding to our survey reported that they have taken certain
steps to ensure the accuracy and reliability of data for the Employment
Service, DVOP, and LVER programs, such as having their systems perform
automated checks for inconsistencies in data or for duplicate veteran
files (see fig. 6).
^12 [39]GAO-06-82 .
^13According to Labor officials, joint compliance visits have been
conducted in three states to date: Arizona, Oregon, and Washington.
Figure 6: Most State Workforce Agencies Report That Their State Systems
Take Certain Steps to Ensure the Accuracy and Reliability of Employment
Service, DVOP, and LVER Data
Labor's Proposed Integrated Data System Could Improve the Understanding of
Services and Outcomes for Veterans
Since 2004, Labor has been planning to implement an integrated data
reporting system that could greatly enhance the understanding of veterans'
services and outcomes. In 2004, Labor first proposed a single, streamlined
reporting system, known as the ETA Management Information and Longitudinal
Evaluation system (EMILE) that would have replaced reporting systems for
several Labor programs. Labor substantially modified this system's design
in response to concerns raised by state and local agencies about the
burden and cost of the new system, as well as the challenge of meeting the
implementation deadline. The modified system, now called the Workforce
Investment Streamlined Performance Reporting System (WISPR), was planned
with a July 2007 implementation date.
WISPR has been designed to both integrate and expand data reporting. If
implemented, the system would integrate data reporting by using
standardized reporting requirements across the Employment Service, DVOP
and LVER, WIA, and TAA programs, and ultimately replace their preexisting
reporting systems with a single reporting structure. Additionally, it
would rely on a standardized set of data elements and quarterly reports to
provide data on participant characteristics and services provided, as well
as performance outcomes based on the common measures. Its integrated
design would, for the first time, allow Labor and states to track
individual veterans' progress through the one-stop system. In addition,
the system would expand data collection and reporting in two key areas:
the services that LVER staff provide to employers, a key aspect of the
LVER role on which Labor currently collects no data, and estimates of the
population of veterans who access the one-stop system but ultimately
receive limited or no services from one-stop staff.
As with EMILE, however, concerns have been raised about challenges in
implementing the new system, and at present, the timeline for WISPR's
implementation remains unclear. Some of the comments received by OMB
during the official comment period noted the challenge of a July 2007
implementation date, according to a Labor official. While states will have
a 2-year period to consolidate reporting on the full range of programs,
they are expected to begin collecting and reporting data in the new format
immediately. As of December 2006, 39 entities, including state workforce
agencies, local agencies and unions, had submitted comments reflecting
their concerns about WISPR to the Office of Management and Budget (OMB).
Of the 20 states that submitted comments, 14 noted that a July 2007
implementation date would represent a challenge. For example, some of them
expressed the view that Labor had underestimated the time states would
need to revise policy, reprogram systems, and retrain staff. In addition,
some states expressed concerns about their ability to provide data on
services to employers. Moreover, two states expressed the concern that
meeting Labor's planned implementation date would have adverse
consequences, such as compromised data quality or cost overruns. OMB's
official review will address the time needed to build the reporting
system's technical infrastructure, and will play a key role in deciding
the system's final implementation schedule, according to a Labor official.
States and local areas will need enough time to fully meet the
requirements of this expanded data collection.
Little Is Known about the Impact of One-Stop Services, including Those to
Veterans
Although Labor has improved its outcome data on job seekers who
participate in its programs, these data alone cannot measure whether
outcomes are a direct result of program participation, rather than
external factors. For example, local labor market conditions may affect an
individual's ability to find a job as much or more than participation in
an employment and training program. To measure the effects of a program,
it is necessary to conduct an impact evaluation that would seek to assess
whether the program itself led to participant outcomes. Labor has not
conducted an impact evaluation of one-stop services, including those to
veterans. However, the department did sponsor a study, issued in 2003,
that examined the relationship between services provided to certain groups
of veterans and employment and earnings outcomes.^14 This study employed a
number of data sources and statistical techniques to learn more about how
veterans were using one-stop services. However, while this study provided
some useful information, it could not determine that these services caused
positive outcomes for veteran job seekers. In addition, the study relied
on data from 8 states and its findings could not be generalized to the
national population of veteran job seekers.
Since the full implementation of WIA in 2000--in which the one-stop system
became the required means to provide employment and training services,
including those to veterans--Labor has not made evaluating the impact of
those services a research priority. While WIA required one such evaluation
by 2005, Labor has declined to fund one in prior budgets. In a 2004
report, we recommended that Labor comply with the requirements of WIA and
conduct an impact evaluation of WIA services to better understand what
services are most effective for improving employment-related outcomes.^15
In response to our report, Labor cited the need for program stability and
proposed delaying an impact evaluation of WIA until any changes that might
be included in reauthorization legislation had been implemented. While
efforts to reauthorize WIA began in 2003, they have stalled and it is not
clear at this time when they will be complete. Furthermore, OMB has also
found Labor's evaluations of WIA services to be lacking. In response, in
its 2008 budget proposal, Labor identified an assessment of WIA's impact
on employment, retention, and earnings outcomes for participants as an
effort the agency would begin. According to Labor officials, the agency
has not yet begun to design the study. Such a study should include impacts
for key participant groups, including veterans. To do so would require a
sufficient sample of veterans to allow such analysis.
^14Battelle Memorial Institute, Assessment of Unemployed Veterans' Needs
for the Department of Labor's Veterans' Employment and Training Service,
PSC Contract No. 282-98-0019, Task Order No. 7, Battelle Project Number
FG465407 (Arlington, Virginia: November 30, 2003).
^15GAO, Workforce Investment Act: States and Local Areas Have Developed
Strategies to Assess Performance, but Labor Could Do More to Help,
[40]GAO-04-657 (Washington, D.C.: June 1, 2004).
Conclusions
At a time when the nation's attention is focused on those who have served
their country, it is vital that Congress and the Administration are able
to make informed decisions about programs that help veterans find and keep
jobs in the civilian labor market. Frequent changes in Labor's performance
accountability system have hampered Labor's ability to produce consistent
and meaningful performance information on veteran job seekers. States and
local areas have had difficulty implementing the constant changes to
performance information, which introduce error and make it difficult to
identify trends that would give Congress a better idea of the programs'
achievements. While the anticipated transition to a new reporting system
represents a promising advance in Labor's ability to track the outcomes of
veterans in the one-stop system, states will need time to effectively
implement the changes to avoid compromising the potential benefits--such
as improved data quality--of the system.
Furthermore, the current separate performance measures for the DVOP and
LVER programs do not account for the considerable similarity in veteran
populations served by DVOP specialists and LVER staff, and thus do not
provide an accurate picture of outcomes for veterans served by these two
programs. Using the existing measures, Labor also cannot ensure that
performance outcomes give more weight to services for veterans with
greater barriers to employment. In addition, different veteran definitions
in other programs could make it difficult to analyze services to veterans
throughout the one-stop system. Further, Labor cannot provide assurance
that veterans are appropriately given service priority by programs in the
one-stop system, or that services to veterans are truly effective. The
federal government spends about $155 million each year on the DVOP and
LVER programs alone, not counting the amounts spent on veterans who use
other one-stop programs, but there is no information on whether these
programs have an impact in helping this important population. Establishing
a means to gauge the programs' impact would require a considerable
investment of time and money, but would contribute greatly to the
understanding of whether current employment and training services are
meeting veterans' needs. Furthermore, we continue to urge Labor to meet
WIA requirements and our 2004 recommendation to conduct an impact
evaluation of one-stop services.
Recommendations for Executive Action
To provide a better picture of services and outcomes for veteran job
seekers, improve program reporting, and facilitate priority of service, we
recommend that the Secretary of Labor
o ensure that states are given adequate direction and sufficient
time to implement ETA's planned integrated data reporting system
and make necessary changes;
o consolidate all performance measures for the DVOP and LVER
programs, including those for disabled and recently separated
veterans;
o comply with JVA's requirement to implement a weighting system
for the DVOP and LVER performance measures that takes into account
the difficulty of serving veterans with particular barriers to
employment;
o develop legislative proposals for appropriate changes to the
definitions of veterans across employment and training programs to
ensure consistency; and
o ensure that Labor moves forward with an impact evaluation for
the one-stop system under WIA as we recommended in 2004, and that
the evaluation's sampling methodology includes veterans in
sufficient numbers to allow analysis of the impact of services to
veterans in the one-stop system, including those served by the
DVOP and LVER programs.
Agency Comments and Our Evaluation
We provided a draft of this report to Labor for review and comment. In its
comments, Labor generally concurred with our findings, conclusions, and
recommendations and expressed appreciation that the report acknowledges
the steps the agency has taken to improve the quality of performance data
and better understand outcomes for veterans. Labor noted that it is
considering adopting a different approach to measuring outcomes for the
DVOP and LVER programs by program year 2008--one that may take into
account the similar veteran populations served, as well as outreach to
employers. As it develops this new approach, Labor reported that it will
also introduce a system of weighted measures that will emphasize services
to veterans with barriers to employment. These changes will coincide with
the implementation of Labor's proposed integrated data system, WISPR.
Labor also noted that it would work with states and grantees to ensure a
smooth transition to the new system. In addition, Labor stated that it
intends to pursue a WIA impact evaluation, which will allow for analysis
of services to sub-populations, including veterans. Labor reported that
our recommendation to develop proposals for changing veteran definitions
across employment and training programs must be evaluated with the input
of other agencies. Labor also provided technical comments that we
incorporated where appropriate. Labor's comments are reproduced in full in
appendix IV.
We will send copies of this report to the Secretary of Labor, relevant
congressional committees, and other interested parties and will make
copies available to others upon request. In addition, the report will be
available at no charge on GAO's Web site at http://www.gao.gov .
A list of related GAO products is included at the end of this report. If
you or your staff have any questions about this report, please contact me
at (202) 512-7215 or at [email protected]. Contact points for our Offices of
Congressional Relations and Public Affairs may be found on the last page
of this report. Other contacts and staff acknowledgments are listed in
appendix V.
Sigurd R. Nilsen, Director
Education, Workforce, and Income Security Issues
Appendix I: Objectives, Scope, and Methodology
The objectives of this report were to determine (1) the extent to which
DVOP and LVER performance information reflects services and outcomes for
veterans served by these programs, (2) the extent to which performance
information on veterans served by other key programs is comprehensive and
well integrated across programs in the one-stop system, and (3) what Labor
is doing to improve the quality of performance data and better understand
outcomes for veteran job seekers.
To address these objectives, we
o conducted a nationwide Web-based survey to state workforce
administrators in the 50 states and the District of Columbia;
o conducted site visits to 3 states, during which we interviewed
state and federal officials, one-stop managers, and program staff;
o interviewed Labor officials from both the Veterans' Employment
and Training Service (VETS) and the Employment and Training
Administration (ETA);
o analyzed relevant performance data from ETA and VETS; and
o reviewed our previous work on attributes of successful
performance measures.
We conducted our work in accordance with generally accepted government
auditing standards between May 2006 and April 2007.
Survey
To obtain further information on our objectives, we surveyed state
workforce administrators from November 15 to December 27, 2006. The survey
addressed all three objectives and included questions about performance
information for the DVOP and LVER programs, integration of data across
employment programs serving veterans, and efforts to ensure data quality.
We developed the survey based on knowledge obtained during our preliminary
research. This included a literature review and initial interviews with
officials from the Department of Labor, the National Association of State
Workforce Agencies (NASWA), and the state of New Hampshire, where we
conducted our initial site visit. We then obtained a list of state
workforce administrators from NASWA. We asked state administrators to
provide information on the DVOP and LVER programs' capacity, other
programs within the one-stop system that serve veteran job seekers,
performance measures and data; and challenges to managing the programs. To
determine whether respondents would understand the questions as intended,
we pretested the survey with state officials in 5 states. We then made
changes to the questions based on comments we received during the
pretests.
The survey was conducted using self-administered electronic Web-based
questionnaires. We sent notification of the survey to the 50 states and
the District of Columbia in November 2006 and followed up with e-mail
messages and telephone calls as necessary during November and December.
All 51 recipients submitted their responses by the end of December 2006,
providing us with a response rate of 100 percent. We did not independently
verify information obtained through the survey. During our data analysis
we held three follow-up conversations to fill in gaps from incomplete
survey information.
Because this survey was not a sample survey, there are no sampling errors.
However, the practical difficulties of conducting any survey may introduce
errors, commonly referred to as nonsampling errors. For example,
difficulties in how a particular question is interpreted, in the sources
of information that are available to respondents, or how the data are
entered into a database can introduce unwanted variability into the survey
results. We took steps during survey development, data collection, and
data analysis to minimize these nonsampling errors. For example, we
pretested the questionnaire to ensure that questions were clear and
understandable. Since this was a Web-based survey in which respondents
entered their responses directly into out database, there was little
possibility of data entry error. During data analysis, a second,
independent analyst checked all computer programming. Also, to the extent
possible, we compared both closed and open ended survey responses with our
site visit observations. While survey results are also subject to
different types of systematic errors or bias, we do not have reason to
believe that respondents falsely reported any information for this survey.
Site Visits
To obtain a detailed understanding of how veteran job seekers are served
by the one-stop system and how their information is captured, we conducted
visits to three states: New Hampshire, California, and Tennessee. We
selected these states based on a range of selection criteria, including
geographic dispersion, state size and veteran demographics, recent state
performance in veterans' programs, and recommendations by Labor and NASWA.
Our site visits at the state level included interviews with state
workforce agency officials and state directors of Veterans' Employment and
Training. We also chose two local one-stops in each state and met with
local managers and veteran program staff (see table 6).
Table 6: States and Local Areas in Our Study
State site visits Local area City
New Hampshire New Hampshire Works Concord
New Hampshire Works Manchester
California Mark Sanders Sacramento Midtown One-Stop Sacramento
Career Center
San Diego South Metro Career San Diego
Center
Tennessee Clarksville/Montgomery County Career Center Clarksville
Middle Tennessee Career Center Murfreesboro
Source: GAO.
During each interview, we used standard interview protocols to obtain
detailed and comparable information. In our interview with state workforce
officials, we discussed the role of the state workforce agency in
administering veterans' employment and training programs, details about
the programs serving veteran job seekers, views on the current performance
accountability system, and information about data collection and
validation. In our interviews with the state directors and their staff, we
discussed their oversight roles and responsibilities, relationship with
the state workforce agency, and views on the current performance
accountability system and data collection. At the local one-stops, we
discussed the coordination of veteran staff with other programs within the
one-stop system, priority of service, and data collection and reporting.
In each state, we also received a tutorial of the state's data collection
software. We conducted our site visits between July and November 2006.
Research and Experts
As part of our work, we interviewed officials of ETA and VETS, including
all six Regional Administrators of VETS. We conducted these telephone
interviews in the following locations: Boston, Atlanta, Dallas, Chicago,
San Francisco, and Philadelphia. During each interview, we obtained
information on regional differences in administering the DVOP and LVER
programs, views on the current performance measures, and information on
Labor's monitoring role in each state.
We also analyzed performance data from the DVOP, LVER, and Employment
Service programs and reviewed Labor's guidance. In addition, we reviewed
relevant literature, including our past work on attributes of successful
performance measures. We also interviewed representatives of NASWA and two
private-sector staffing agencies.
Appendix II: Summary of State Performance in the DVOP and LVER Programs
for Benchmark Program Year 2005 and Negotiated Goals for Program Year 2006
The following tables include
o baseline performance data by state for the DVOP and LVER
programs from benchmark program year 2005 (July 1, 2005-June 30,
2006) and
o negotiated goals by state for the following year, program year
2006.
Labor and states did not negotiate goals for the DVOP or LVER programs for
program year 2005, which was a baseline year for performance under the new
common measures. Four performance measures contribute to each program's
performance. For the DVOP program, there is one set of measures for all
veterans and one set for disabled veterans. For the LVER program, there is
a set of measures for all veterans and another set for recently separated
veterans. Each set of measures includes
o entered employment rate (EER): the number of participants who
are employed in the first quarter after the exit quarter divided
by the number of participants who exit during the quarter and
o employment retention rate (ERR): the number of participants who
are employed in both the second and third quarters after the exit
quarter divided by the number of adult participants who exit
during the quarter.
These figures were provided by the Department of Labor. GAO has not
verified the accuracy or reliability of these data.
Table 7: Summary of State Performance in the DVOP Program for Benchmark
Program Year 2005 and Negotiated Goals for Program Year 2006
Disabled Veterans Outreach Program (DVOP)
All veterans Disabled veterans
2005 2006 2005 2006
achieved goal achieved 2006 2005 goal 2005 2006
State EER EER ERR goalERR achievedEER EER achievedERR goalERR
Alabama 64% 64% 79% 81% 56% 56% 77% 79%
Alaska 52% 52% 78% 78% 47% 50% 79% 78%
Arizona 57% 58% 80% 80% 53% 57% 75% 77%
Arkansas 66% 66% 81% 80% 59% 55% 78% 70%
California 53% 54% 74% 74% 49% 47% 72% 73%
Colorado 59% 63% 75% 76% 55% 58% 74% 72%
Connecticut 55% 52% 77% 75% 43% 46% 77% 75%
Delaware 60% 60% 76% 83% 55% 63% 87% 88%
District of 58% 60% 75% 75% 44% 60% 83% 78%
Columbia
Florida 61% 59% 80% 79% 56% 56% 78% 77%
Georgia 70% 70% 81% 80% 62% 62% 82% 80%
Hawaii 53% 52% 77% 72% 54% 47% 73% 76%
Idaho 67% 67% 84% 83% 59% 59% 79% 80%
Illinois 59% 54% 84% 81% 56% 53% 86% 85%
Indiana 69% 59% 84% 80% 58% 50% 84% 79%
Iowa 66% 75% 83% 87% 62% 72% 82% 86%
Kansas 68% 69% 83% 83% 64% 64% 84% 83%
Kentucky 66% 64% 82% 80% 59% 57% 82% 80%
Louisiana 59% 57% 76% 74% 53% 58% 74% 74%
Maine 64% 73% 80% 81% 63% 73% 80% 77%
Maryland 57% 64% 79% 83% 54% 52% 84% 82%
Massachusetts 57% 59% 80% 80% 48% 48% 75% 81%
Michigan 57% 59% 78% 81% 54% 55% 78% 78%
Minnesota 59% 62% 82% 81% 49% 55% 80% 81%
Mississippi 49% 60% 57% 82% 48% 56% 59% 79%
Missouri 61% 64% 78% 81% 54% 58% 77% 80%
Montana 61% 70% 75% 83% 56% 65% 80% 83%
Nebraska 66% 68% 81% 76% 58% 62% 78% 81%
Nevada 72% 71% 80% 79% 68% 67% 80% 79%
New Hampshire 56% 64% 81% 81% 54% 60% 81% 80%
New Jersey 50% 55% 73% 80% 42% 60% 73% 73%
New Mexico 40% 57% 80% 73% 37% 56% 76% 80%
New York 61% 61% 78% 80% 56% 56% 77% 89%
North 57% 55% 72% 79% 53% 52% 71% 76%
Carolina
North Dakota 73% 69% 76% 89% 56% 55% 77% 79%
Ohio 65% 60% 84% 82% 58% 55% 81% 82%
Oklahoma 64% 58% 80% 77% 60% 55% 76% 77%
Oregon 55% 56% 78% 76% 42% 46% 74% 76%
Pennsylvania 60% 60% 83% 77% 47% 56% 80% 77%
Puerto Rico 11% 52% 8% 51% 6% 50% 9% 51%
Rhode Island 45% 58% 35% 68% 33% 58% 38% 68%
South 68% 63% 83% 82% 64% 59% 81% 82%
Carolina
South Dakota 68% 70% 83% 80% 64% 67% 86% 80%
Tennessee 67% 67% 80% 80% 62% 67% 76% 80%
Texas 68% 63% 85% 82% 55% 55% 86% 82%
Utah 57% 57% 82% 79% 53% 53% 80% 79%
Vermont 79% 75% 87% 73% 82% 77% 72% 87%
Virginia 76% 70% 92% 87% 71% 66% 91% 73%
Washington 71% 68% 85% 86% 67% 64% 85% 86%
West Virginia 65% 66% 84% 88% 56% 60% 74% 85%
Wisconsin 65% 75% 89% 85% 56% 70% 87% 88%
Wyoming 66% 54% 81% 80% 66% 52% 80% 80%
Source: Labor.
Note: All years refer to program years.
Table 8: Summary of State Performance in the LVER Program for Benchmark
Program Year 2005 and Negotiated Goals for Program Year 2006
Local Veterans Employment Representative (LVER)
Recently separated
All veterans veterans
2005 2006 2005 2006
achieved goal achieved 2006 2005 goal 2005 2006
State EER EER ERR goalERR achievedEER EER achievedERR goalERR
Alabama 63% 62% 78% 82% 66% 61% 75% 80%
Alaska 49% 50% 75% 79% 67% 50% 60% 81%
Arizona 55% 51% 79% 70% 55% 51% 77% 70%
Arkansas 68% 66% 82% 82% 61% 64% 82% 82%
California 53% 55% 74% 76% 52% 55% 77% 78%
Colorado 62% 64% 76% 76% 61% 63% 72% 74%
Connecticut 55% 52% 75% 76% 64% 56% 71% 76%
Delaware 65% 67% 79% 76% 66% 77% 77% 77%
District of 59% 60% 76% 75% 56% 60% 79% 76%
Columbia
Florida 59% 58% 80% 79% 60% 58% 79% 79%
Georgia 69% 69% 81% 80% 68% 65% 79% 79%
Hawaii 47% 49% 76% 77% 40% 41% 67% 75%
Idaho 72% 71% 85% 84% 78% 74% 85% 84%
Illinois 61% 58% 85% 83% 60% 56% 84% 81%
Indiana 70% 60% 84% 81% 74% 64% 83% 77%
Iowa 69% 74% 84% 85% 69% 76% 84% 85%
Kansas 66% 66% 82% 83% 76% 75% 84% 84%
Kentucky 67% 65% 83% 80% 65% 64% 81% 80%
Louisiana 61% 57% 73% 74% 62% 57% 70% 74%
Maine 64% 73% 81% 81% 65% 73% 77% 81%
Maryland 58% 69% 78% 82% 60% 76% 88% 90%
Massachusetts 58% 59% 79% 81% 59% 61% 71% 75%
Michigan 57% 59% 78% 81% 61% 61% 77% 79%
Minnesota 69% 63% 83% 81% 50% 50% 71% 81%
Mississippi 50% 60% 61% 82% 46% 56% 58% 80%
Missouri 64% 66% 81% 82% 69% 70% 80% 82%
Montana 53% 70% 71% 83% 53% 65% 72% 83%
Nebraska 63% 65% 81% 82% 63% 64% 81% 82%
Nevada 73% 73% 83% 79% 77% 76% 83% 80%
New Hampshire 60% 64% 83% 83% 61% 64% 80% 80%
New Jersey 51% 55% 77% 80% 55% 65% 76% 80%
New Mexico 42% 68% 82% 81% 41% 56% 77% 78%
New York 64% 64% 80% 80% 58% 59% 78% 80%
North 55% 55% 73% 80% 49% 50% 68% 78%
Carolina
North Dakota 72% 71% 83% 81% 82% 65% 83% 82%
Ohio 67% 60% 85% 83% 71% 60% 90% 85%
Oklahoma 69% 59% 81% 82% 73% 60% 80% 79%
Oregon 57% 58% 77% 79% 57% 62% 78% 78%
Pennsylvania 62% 60% 85% 81% 58% 60% 83% 81%
Puerto Rico 0% 52% 10% 73% 0% 50% 9% 51%
Rhode Island 33% 58% 47% 69% 35% 59% 40% 57%
South 66% 61% 83% 83% 70% 63% 82% 83%
Carolina
South Dakota 74% 72% 85% 68% 57% 69% 86% 76%
Tennessee 68% 67% 82% 82% 68% 67% 79% 80%
Texas 66% 63% 84% 82% 84% 64% 87% 82%
Utah 67% 61% 85% 79% 70% 61% 82% 79%
Vermont 74% 73% 82% 82% 77% 72% 82% 74%
Virginia 75% 69% 92% 88% 69% 60% 85% 81%
Washington 66% 65% 85% 84% 74% 70% 87% 87%
West Virginia 66% 65% 86% 68% 70% 70% 79% 73%
Wisconsin 64% 71% 89% 84% 70% 88% 85% 81%
Wyoming 68% 62% 77% 82% 67% 62% 82% 82%
Source: Labor.
Note: All years refer to program years.
Appendix III: Summary of State-Negotiated Goals and Performance for
Veterans in the Employment Service, Program Year 2005
This table illustrates the negotiated goals and performance achieved by
each state for program year 2005 for veterans in the Wagner-Peyser-funded
Employment Service. It includes the entered employment and employment
retention rates for all veterans and disabled veterans within the
Employment Service, including those in the DVOP and LVER programs.
These figures were provided by the Department of Labor. GAO has not
verified the accuracy or reliability of these data.
Table 9: State-Negotiated Goals and Performance for Veterans in the
Employment Service, Program Year 2005
Employment Service
All veterans Disabled veterans
2005 2005 2005 2005 2005
goal achieved goal 2005 achieved goal
State EER EER ERR 2005achievedERR goalEER EER ERR 2005achievedERR
Alabama 60% 62% 76% 78% 54% 53% 76% 77%
Alaska 58% 55% 77% 73% 48% 53% 75% 75%
Arizona 58% 59% 74% 80% 54% 56% 68% 78%
Arkansas 53% 67% 70% 81% 53% 63% 70% 78%
California 53% 52% 70% 78% 49% 47% 68% 77%
Colorado 60% 59% 79% 76% 59% 55% 78% 75%
Connecticut 57% 56% 66% 77% 53% 49% 66% 77%
Delaware 55% 59% 75% 78% 50% 55% 73% 83%
District of 58% 58% 75% 75% 54% 59% 73% 71%
Columbia
Florida 55% 59% 72% 79% 54% 55% 73% 78%
Georgia 65% 68% 72% 81% 65% 61% 72% 81%
Hawaii 46% 49% 70% 77% 41% 53% 68% 76%
Idaho 50% 71% 71% 83% 43% 65% 70% 81%
Illinois 44% 63% 72% 85% 40% 59% 72% 85%
Indiana 54% 68% 75% 84% 46% 59% 73% 84%
Iowa 58% 67% 72% 83% 56% 62% 70% 83%
Kansas 58% 66% 72% 83% 56% 62% 72% 82%
Kentucky 63% 67% 72% 82% 58% 59% 72% 81%
Louisiana 59% 62% 60% 77% 60% 61% 60% 75%
Maine 58% 64% 70% 80% 58% 62% 68% 78%
Maryland 58% 59% 77% 79% 53% 54% 77% 78%
Massachusetts 57% 58% 78% 79% 48% 47% 72% 74%
Michigan 48% 57% 73% 79% 43% 51% 71% 77%
Minnesota 49% 61% 77% 82% 48% 54% 71% 80%
Mississippi 62% 49% 72% 61% 55% 48% 72% 61%
Missouri 58% 62% 72% 79% 52% 58% 70% 78%
Montana 46% 44% 80% 75% 46% 39% 79% 80%
Nebraska 59% 63% 76% 81% 55% 54% 74% 79%
Nevada 58% 72% 72% 82% 54% 68% 72% 81%
New Hampshire 67% 58% 67% 82% 62% 58% 66% 82%
New Jersey 55% 51% 74% 75% 59% 43% 72% 74%
New Mexico 58% 45% 75% 79% 54% 42% 73% 77%
New York 56% 61% 72% 79% 52% 58% 72% 78%
North 58% 54% 72% 74% 54% 49% 72% 75%
Carolina
North Dakota 63% 70% 78% 82% 59% 60% 75% 83%
Ohio 57% 65% 78% 84% 51% 59% 74% 82%
Oklahoma 60% 64% 72% 80% 54% 56% 65% 77%
Oregon 55% 57% 75% 80% 48% 46% 73% 76%
Pennsylvania 59% NDA 73% NDA 52% NDA 73% NDA
Puerto Rico NDP NDA NDP NDA NDP NDA NDP NDA
Rhode Island 58% DNV 72% DNV 52% DNV 72% DNV
South 59% 67% 73% 82% 53% 64% 73% 81%
Carolina
South Dakota 64% 69% 78% 80% 58% 66% 78% 82%
Tennessee 53% 66% 70% 81% 49% 61% 70% 79%
Texas 56% 65% 80% 85% 50% 55% 77% 86%
Utah 57% 66% 75% 84% 51% 59% 74% 82%
Vermont 59% 73% 73% 82% 54% 78% 73% 76%
Virginia 56% 76% 72% 92% 52% 70% 75% 91%
Washington 50% 70% 80% 85% 44% 67% 79% 85%
West Virginia 58% 66% 72% 85% 58% 61% 72% 81%
Wisconsin 52% 63% 76% 89% 46% 56% 73% 87%
Wyoming 58% NDA 75% NDA 48% NDA 78% NDA
Source: Labor.
Notes: NDA = No data available / NDP = No data provided / DNV = Data not
valid.
All years refer to program years.
Appendix IV: Comments from the Department of Labor
Appendix V: GAO Contact and Staff Acknowledgments
GAO Contact
Sigurd R. Nilsen (202) 512-7215 or [email protected]
Staff Acknowledgments
Dianne Blank, Assistant Director Rebecca Woiwode, Analyst-in-Charge
Chris Morehouse and Beth Faraguna made significant contributions to this
report in all facets of the work. In addition, Walter Vance assisted in
the design of the national survey; Gloria Hernandez-Saunders helped with
data analysis; Meeta Engle lent subject matter expertise; Jessica Botsford
and Richard Burkard provided legal support; and Charlie Willson provided
writing assistance.
Related GAO Products
Trade Adjustment Assistance: Labor Should Take Action to Ensure
Performance Data Are Complete, Accurate, and Accessible. [43]GAO-06-496 .
Washington, D.C.: April 25, 2006.
Veterans' Employment and Training Service: Greater Accountability and
Other Labor Actions Needed to Better Serve Veterans. [44]GAO-06-357T .
Washington, D.C.: February 2, 2006.
Veterans' Employment and Training Service: Labor Actions Needed to Improve
Accountability and Help States Implement Reforms to Veterans' Employment
Services. [45]GAO-06-176 . Washington, D.C.: December 30, 2005.
Workforce Investment Act: Labor and States Have Taken Actions to Improve
Data Quality, but Additional Steps Are Needed. [46]GAO-06-82 . Washington,
D.C.: November 14, 2005.
Veterans' Employment and Training Service: Preliminary Observations on
Changes to Veterans' Employment Programs. [47]GAO-05-662T . Washington,
D.C.: May 12, 2005.
Performance Measurement and Evaluation: Definitions and Relationships.
[48]GAO-05-739SP . Washington, D.C.: May 2005.
Workforce Investment Act: States and Local Areas Have Developed Strategies
to Assess Performance, but Labor Could Do More to Help. [49]GAO-04-657 .
Washington, D.C.: June 1, 2004.
Performance Reporting: Few Agencies Reported on the Completeness and
Reliability of Performance Data. [50]GAO-02-372 . Washington, D.C.: April
26, 2002.
Veterans' Employment and Training Service: Flexibility and Accountability
Needed to Improve Service to Veterans. [51]GAO-01-928 . Washington, D.C.:
September 12, 2001.
Veterans' Employment and Training Service: Proposed Performance
Measurement System Improved, but Further Changes Needed. [52]GAO-01-580 .
Washington, D.C.: May 15, 2001.
Veterans' Employment and Training Service: Strategic and Performance Plans
Lack Vision and Clarity. [53]GAO/T-HEHS-99-177 . Washington, D.C.: July
29, 1999.
Veterans' Employment and Training Service: Assessment of the Fiscal Year
1999 Performance Plan. [54]GAO/HEHS-98-240R . Washington, D.C.: September
30, 1998.
Veterans' Employment and Training: Services Provided by Labor Department
Programs. [55]GAO/HEHS-98-7 . Washington, D.C.: October 17, 1997.
The Results Act: An Evaluator's Guide to Assessing Agency Annual
Performance Plans. [56]GAO/GGD-10.1.20. Washington, D.C.: April 1998.
Agencies' Annual Performance Plans under the Results Act: An Assessment
Guide to Facilitate Congressional Decisionmaking. GAO/GDD/AIMD-10.1.18.
Washington, D.C.: February 1998.
Executive Guide: Effectively Implementing the Government Performance and
Results Act. GAO/GGD-96-118 . Washington, D.C.: June 1996.
(130581)
GAO's Mission
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony
The fastest and easiest way to obtain copies of GAO documents at no cost
is through GAO's Web site ( www.gao.gov ). Each weekday, GAO posts
newly released reports, testimony, and correspondence on its Web site. To
have GAO e-mail you a list of newly posted products every afternoon, go to
www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548
To order by Phone: Voice: (202) 512-6000
TDD: (202) 512-2537
Fax: (202) 512-6061
To Report Fraud, Waste, and Abuse in Federal Programs
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: [email protected]
Automated answering system: (800) 424-5454 or (202) 512-7470
Congressional Relations
Gloria Jarmon, Managing Director, [email protected] (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125 Washington,
D.C. 20548
Public Affairs
Paul Anderson, Managing Director, [email protected] (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
www.gao.gov/cgi-bin/getrpt?GAO-07-594 .
To view the full product, including the scope
and methodology, click on the link above.
For more information, contact Sigurd Nilsen at (202) 512-7215 or
[email protected].
Highlights of [64]GAO-07-594 , a report to congressional committees
May 2007
VETERANS'EMPLOYMENT AND TRAINING SERVICE
Labor Could Improve Information on Reemployment Services, Outcomes, and
Program Impact
In 2002, Congress enacted the Jobs for Veterans Act (JVA), which modified
two Department of Labor (Labor) programs that specifically target veteran
job seekers: the Disabled Veterans' Outreach Program (DVOP) and the Local
Veterans' Employment Representative (LVER) program. However, questions
have been raised about the adequacy of performance information on services
to veterans by these and other employment programs. In this report, GAO
examined (1) the extent to which DVOP and LVER performance information
reflects services and outcomes for veterans; (2) the extent to which
performance information on veterans paints a clear picture of their use of
one-stop services; and (3) what Labor is doing to improve the quality of
performance data and better understand program impact and outcomes for
veterans.
[65]What GAO Recommends
GAO is making a number of recommendations to improve the performance
measurement system for the DVOP and LVER programs and to better understand
services and their impact for job seekers in the one-stop system,
including veterans.
Labor generally agreed with the report's findings, conclusions, and
recommendations.
Performance information for the DVOP and LVER programs provides some sense
of services and outcomes for veterans, but is weakened by several factors.
In July 2005, Labor adopted new performance measures for the programs, but
not all have been fully implemented. For example, states are held
accountable for helping veterans get and keep jobs, but are not yet held
accountable for their average earnings once employed, as they are for
other programs. Additionally, having separate performance measures for the
DVOP and LVER programs fails to acknowledge the similarity of the
populations they serve and duties they perform. Furthermore, it is
difficult to assess outcomes over time or across states because of
frequent changes in reporting requirements that prevent establishing
reliable trend data.
Labor's data on veteran job seekers paint an unclear picture of their use
of other employment and training services in the one-stop system, despite
the use of common performance measures across programs. Although many
veterans use services other than those provided by the DVOP and LVER
programs, key employment programs vary in how well their data on veteran
job seekers are shared across programs, making it difficult to know how
many veterans are served. In addition, statutory differences in the
definitions of veterans hinder efforts to standardize data across
employment programs. Moreover, Labor has no means of assessing whether
priority of service for veterans has been implemented in various
employment programs.
Labor has taken some steps to improve the quality of performance data and
better understand outcomes for veterans. For example, Labor requires
states to validate key performance data. Labor has also planned an
integrated data reporting system that would track individual veterans'
progress through the one-stop system. However, states have raised concerns
about the timelines and its current implementation date is unclear.
Furthermore, while outcome information on veterans is helpful, it cannot
measure whether the outcomes are due to the program or other factors.
While Labor has sponsored research on services to veterans, it has not yet
conducted the impact evaluation required by law to assess the
effectiveness of one-stop services.
Percentage of Disabled and Recently Separated Veterans Served by the DVOP
and LVER Programs, Program Year 2005
References
Visible links
34. http://www.gao.gov/cgi-bin/getrpt?GAO/GGD-10.1.20 and http://www.gao.gov/cgi-bin/getrpt?GAO/GGD/AIMD-10.1.18 and http://www.gao.gov/cgi-bin/getrpt?GAO/GGD-96-118
35. http://www.gao.gov/cgi-bin/getrpt?GAO-04-657
36. http://www.gao.gov/cgi-bin/getrpt?GAO-06-496
37. http://www.gao.gov/cgi-bin/getrpt?GAO-06-176
38. http://www.gao.gov/cgi-bin/getrpt?GAO-06-82
39. http://www.gao.gov/cgi-bin/getrpt?GAO-06-82
40. http://www.gao.gov/cgi-bin/getrpt?GAO-04-657
43. http://www.gao.gov/cgi-bin/getrpt?GAO-06-496
44. http://www.gao.gov/cgi-bin/getrpt?GAO-06-357T
45. http://www.gao.gov/cgi-bin/getrpt?GAO-06-176
46. http://www.gao.gov/cgi-bin/getrpt?GAO-06-82
47. http://www.gao.gov/cgi-bin/getrpt?GAO-05-662T
48. http://www.gao.gov/cgi-bin/getrpt?GAO-05-739SP
49. http://www.gao.gov/cgi-bin/getrpt?GAO-04-657
50. http://www.gao.gov/cgi-bin/getrpt?GAO-02-372
51. http://www.gao.gov/cgi-bin/getrpt?GAO-01-928
52. http://www.gao.gov/cgi-bin/getrpt?GAO-01-580
53. http://www.gao.gov/cgi-bin/getrpt?GAO/T-HEHS-99-177
54. http://www.gao.gov/cgi-bin/getrpt?GAO/HEHS-98-240R
55. http://www.gao.gov/cgi-bin/getrpt?GAO/HEHS-98-7
56. http://www.gao.gov/cgi-bin/getrpt?GAO/GGD-10.1.20 and http://www.gao.gov/cgi-bin/getrpt?GAO/GDD/AIMD-10.1.18 and http://www.gao.gov/cgi-bin/getrpt?GAO/GGD-96-118
64. http://www.gao.gov/cgi-bin/getrpt?GAO-07-594
*** End of document. ***