Workforce Investment Act: Improvements Needed in Performance
Measures to Provide a More Accurate Picture of WIA's
Effectiveness (01-FEB-02, GAO-02-275).
Congress passed the Workforce Investment Act (WIA) in 1998 to
bring most federally funded employment and training services into
a single, one one-stop center system. GAO assessed three programs
that provide service through this system. States and localities
have begun to implement the new performance measurement system
for the three WIA-funded programs but report several challenges.
States had to change the way they collected and reported
performance data. They also faced challenges in implementing
these measures due to their complexity and the resource demands
created by new measures. Some developed new procedures to obtain
access to sensitive records. The performance levels are of
particular concern to state and local officials because failure
to meet them can result in financial sanctions. As a result,
states may be choosing to serve only those job seekers who are
most likely to be successful. Even when fully implemented,
performance measures may not provide a true picture of WIA-funded
program performance because data are neither comparable across
states nor timely. The measures include many of the indicators
relevant to an employment and training program, such as getting
and keeping jobs and increasing wages and skills. Although
measures exist to gauge the performance of the three WIA-funded
programs, there are no measures to gauge the performance of the
one-stop system as a whole. At least 17 programs provide services
through the one-stop system, and most have their own performance
measures. Although these performance measures may be used for
assessing outcomes for individual programs, they cannot be used
to measure the success of the overall system.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-02-275
ACCNO: A02658
TITLE: Workforce Investment Act: Improvements Needed in
Performance Measures to Provide a More Accurate Picture of WIA's
Effectiveness
DATE: 02/01/2002
SUBJECT: Federal/state relations
Labor force
Labor law
Performance measures
State/local relations
Data collection
Reporting requirements
Employment or training programs
Program evaluation
Adult and Dislocated Worker Program
DOL Adult Education and Literacy Program
Dept. of Education Vocational Education
Program
DOL Employment Service Program
DOL Job Corps Program
DOL One-Stop Planning and Implementation
Grant Program
HUD Youthbuild Program
Veterans' Employment and Training
Service
Wage Record Interchange System
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Testimony. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-02-275
United States General Accounting Office
GAO
Report to Congressional Requesters
February 2002
WORKFORCE INVESTMENT ACT
Improvements Needed in Performance Measures to Provide a More Accurate
Picture of WIA's Effectiveness
GAO-02-275
Contents
Letter
Results in Brief
Background
States and Localities Are Taking Action to Implement WIA
Performance Measures but Face Challenges in Doing So Performance Measures
May Not Accurately Assess Performance of the Three WIA-Funded Programs
Existing Performance Measures Fail to Gauge Overall One-Stop
Performance Conclusions Recommendations for Executive Action Agency Comments
1
2 4
9
17
22 27 29 30
Appendix I WIA Performance Measures for Adults, Dislocated Workers, and
Older Youth
Appendix II Comments from the Department of Labor
Appendix III GAO Contacts and Staff Acknowledgments 37
GAO Contacts 37 Staff Acknowledgments 37
Related GAO Products
Tables
Table 1: Performance Measures for the Three WIA-Funded
Programs 6 Table 2. WIA's Mandatory Programs and Related Federal Agencies 8
Table 3: Comparing Wage Replacement Rates for Two Job Seekers 16
Figures
Figure 1: Timing of Data Collection by Quarter for Selected Performance
Measures 11
Figure 2: Oregon's Bean Counter 12
Figure 3: Number of Months it Takes to Receive UI Data for Job Placement 21
Figure 4: Comparison of Performance Measures for 17 One-Stop Programs 24
Abbreviations
GAO General Accounting Office
GPRA Government Performance and Results Act
JTPA Job Training Partnership Act
MDA Mississippi Development Authority
MIS management information system
OMB Office of Management and Budget
TEGL Training and Employment Guidance Letter
UI Unemployment Insurance
WIA Workforce Investment Act
WRIS Wage Record Interchange System
United States General Accounting Office Washington, DC 20548
February 1, 2002
The Honorable Edward M. Kennedy Chairman, Committee on Health, Education,
Labor and Pensions United States Senate
The Honorable James M. Jeffords United States Senate
The Congress passed the Workforce Investment Act (WIA) in 1998 to begin
unifying a fragmented employment and training system and to better serve job
seekers and employers. To create a more comprehensive workforce investment
system, WIA requires states and localities to bring together most federally
funded employment and training services into a single system, called the
one-stop center system. Seventeen programs across four federal
agencies-programs such as the Employment Service (Wagner-Peyser) and Adult
Education and Literacy-must provide services through this one-stop system.1
Three of these programs, whose funding is authorized by WIA under Title I to
provide services to adults, dislocated workers, and youth, replace those
previously funded under the Job Training Partnership Act (JTPA).2 These
three new WIA programs, authorized at about $3.7 billion in fiscal year
2001, have performance measures established under WIA that states and
localities must track in order to demonstrate the effectiveness of the
programs. These performance measures gauge program results in the areas of
job placement, employment retention, and earnings change, as well as skill
attainment and customer satisfaction. States are held accountable by the
U.S. Department of Labor for their performance in these areas. If they fail
to meet their expected performance levels, they may suffer financial
sanctions; if they meet or exceed their levels, they may be eligible to
1In addition to the Department of Labor, which oversees the implementation
of WIA, programs funded through the Departments of Education, Health and
Human Services, and Housing and Urban Development are required to provide
services for certain of their programs through this one-stop center system.
2While WIA reauthorizes or amends other existing programs, including Adult
Education and Literacy, Employment Service (Wagner-Peyser), and Vocational
Rehabilitation, the performance measures apply only to the adult, dislocated
worker, and youth programs under Title I, which, for this report, we will
refer to as "WIA-funded programs."
receive additional funds, called incentive grants. 3 In order to make
objective decisions about which states receive incentives and sanctions,
WIA seeks to ensure that states collect and report comparable
performance data on all participants.
In an effort to assess the effectiveness of WIA's performance measures
and whether they will yield useful information that can clearly
demonstrate performance under WIA, we assessed (1) the progress states
and localities have made and the issues they have faced in implementing
performance measures for the three WIA-funded programs; (2) how useful
the WIA performance measures are in accurately gauging the performance
of the three WIA-funded programs; and (3) beyond gauging the
performance of the three WIA-funded programs, how well the
performance of the one-stop system is being measured.
Our review is based on a survey of WIA program administrators in all 50
states; visits to Florida, Indiana, Missouri, Oregon, and Pennsylvania,
where we interviewed state and local workforce investment officials;
interviews with Labor officials and with national associations
representing
state and local workforce development officials; and a review of
relevant
documents. We selected the five states for site visits based on a
variety of
factors, including their experience using unemployment insurance (UI)
wage record data to measure outcomes, the status of their data
collection
systems, and their progress in developing additional measures for state
and local use. Our review focused on measures for Title I Adult,
Dislocated
Worker, and Older Youth programs. We did not review the performance
measures for younger youth (age 14-18).4 We conducted our work between
December 2000 and August 2001 in accordance with generally accepted
government auditing standards.
Results in Brief States and localities have taken action to implement the new performance
measurement system for the three WIA-funded programs but reported
confronting a number of challenges in doing so. As part of their
3States must also meet performance levels for programs authorized under the
Adult Education and Family Literacy Act and Carl D. Perkins Vocational and
Applied Technology Education Act (Perkins) to be eligible for incentive
grants.
4We also did not review measures for eligible training providers, which were
reviewed in our study, U.S. General Accounting Office, Workforce Investment
Act: Better Guidance Needed to Address Concerns Over New Requirements,
GAO-02-72 (Washington, D.C. : 2001).
implementation efforts, states had to change the way they collected and
reported performance data from the way they were collected for previous
programs. Most states decided to develop new automated data systems, but 15
states do not have their systems completely in place. The timing of the
guidance from Labor on how to report the data--not issued until well into
the first year of implementation--delayed the development of these systems.
States and localities also faced challenges in implementing these measures
due to their complexity and the resource demands created by new measures;
some had to develop new procedures to obtain access to sensitive records.
For example, a new measure to assess customer satisfaction for job seekers
and employers requires states to implement complicated procedures to conduct
a customer satisfaction survey. In addition, state officials expressed
concern that the levels of performance that they are expected to achieve are
too high. Because the program was in its first year, there were little or no
data that could be used as a point of comparison, or baseline, for
establishing performance levels. Furthermore, state and local officials
believed that the established performance levels did not take into account
local economic conditions. The performance levels are of particular concern
to state and local officials because failure to meet them can result in
financial sanctions. As a result, states may be choosing to serve only those
job seekers who are most likely to be successful. For example, to meet their
performance levels to replace earnings for workers who have been dislocated
from their jobs, local staff might provide WIA services only to job seekers
who have the best prospect of finding work that can match or surpass their
previous earnings.
Even when fully implemented, WIA performance measures may still not provide
a true picture of WIA-funded program performance largely because data are
not comparable across states or timely. The measures include many of the
indicators relevant to an employment and training program, such as getting
and keeping jobs and increasing wages and skills. However, data may not
accurately gauge performance and are not comparable across states for a
variety of reasons. For example, Labor's guidance on enrolling job seekers-a
process called registration-does not provide clear direction on when to
start collecting performance data on participants, and we found that states
and localities differed on whom they tracked and when. In addition, UI wage
records, required in order to track performance outcomes, are not easily
used to identify employment across state lines, and some states are more
successful than others in obtaining this information. Only seven states are
currently using a clearinghouse-called the Wage Record Interchange System
(WRIS)-that allows states to exchange UI data. Many of the remaining states
are
reluctant to participate, often citing cost concerns. In addition, the time
lag associated with UI data becoming available (as much as 9 months or
longer) results in states not being able to use the data to manage their
programs in the short term.
While there are measures designed to gauge the performance of the three
WIA-funded programs, there are no measures to gauge the performance of the
one-stop system as a whole. At least 17 programs provide services through
the one-stop system and most have their own performance measures. Although
these performance measures may be used for assessing outcomes for individual
programs, they cannot be used to measure the success of the overall system.
For example, no program has a measure to track job seekers who use only
self-service or informational activities offered through the one-stop, which
may constitute a large proportion of job seekers. Not knowing how many job
seekers use the one-stop's services limits the one-stop's ability to assess
its impact. Furthermore, state and local officials told us that having
multiple performance measures has impeded coordination among programs. There
has been limited progress in developing overall performance measures for the
one-stop system. Labor convened a working group to develop additional
indicators of the one-stop system's performance, but they have not yet
issued them.
To provide the Congress and the public with a more accurate picture of WIA
performance, we are making recommendations to Labor in this report to
postpone the implementation of financial sanctions, to expedite the
development of criteria for renegotiating performance levels, to more
clearly define policies and measures, and to develop ways to address the
challenges in using UI data. In its written comments, Labor concurred with
our findings and recommendations.
Labor required states to implement major provisions of WIA Title I by July
1, 2000, although some states began implementing provisions of WIA as early
as July 1999. Services provided under WIA represent a marked change from
those provided under the previous program, allowing for a greater array of
services to the general public. WIA is designed to provide for greater
accountability over what existed previously: it established new performance
measures and a new requirement to use UI data to track and report on the
achievements of the three WIA-funded programs. WIA also requires that many
federal programs work together to provide employment and training services
through the one-stop system.
Background
WIA-Funded Services Represent a Change from those Funded under JTPA
Program services provided under WIA represent a marked change from those
provided under JTPA. When WIA was enacted in 1998, it replaced the JTPA
programs for economically disadvantaged adults and youth and for dislocated
workers with three new programs-WIA Adult, Dislocated Worker, and Youth-that
provide a broader range of services to the general public, no longer using
income to determine eligibility for all program services.5 The newly
authorized WIA programs6 no longer focus exclusively on training but provide
for three tiers, or levels, of service for adults and dislocated workers:
core, intensive, and training. Core services include basic services such as
job searches and labor market information. These activities may be
self-service or require some staff assistance. Intensive services include
such activities as comprehensive assessment and case management-activities
that require greater staff involvement. Training services include such
activities as occupational skills or on-the-job training. These tiers of
WIA-funded services are provided sequentially. That is, in order to receive
intensive services, job seekers must first receive at least one core
service; to receive training services, a job seeker must first receive at
least one core service and then at least one intensive service. Key to
moving from core to a higher level of services is that the services are
needed to help job seekers become self-sufficient. Labor's guidance provides
for monitoring and tracking to begin when job seekers receive core services
that require significant staff assistance. Job seekers who receive core
services that are self-service in nature are not included in the performance
measures.
WIA Performance Measures Are Designed to Increase Accountability for Three
WIA-Funded Programs
WIA is designed to provide for greater accountability than the
accountability provided for under JTPA. It does so by establishing new
performance measures and a new requirement to use UI data to track and
report on the achievements of the three WIA-funded programs. According to
Labor, performance data collected from the states in support of the measures
are intended to be comparable across states in order to maintain objectivity
in determining incentives and sanctions. They are also intended to provide
information to support Labor's performance goals under the Government
Performance and Results Act (GPRA)7 and for program
5WIA's Youth program uses low income as an eligibility requirement.
6Authorized through fiscal year 2003, WIA's three new programs had a budget
authority of about $3.7 billion in fiscal year 2001.
7 GPRA was intended to focus government decisionmaking, management, and
accountability on the results and outcomes achieved by federal programs.
evaluation. Some of the measures that relate to adults, dislocated workers,
and older youth are similar to those used under JTPA, including job
placement, job retention, and wage gains or replacement. Attainment of a
credential8-a degree or certification of skills or training completed-and
customer satisfaction for both job seekers and employers are new under WIA.
(See table 1 for a complete list of the WIA performance measures and
appendix I for a more complete explanation of the performance measures
discussed in this report.)
Table 1: Performance Measures for the Three WIA-Funded Programs
WIA funding stream Performance measure
Adult 1. Entered employment rate
2. Employment retention rate at 6 months
3. Average earnings change in 6 months
4. Entered employment and credential rate Dislocated worker 5. Entered
employment rate
6. Employment retention rate at 6 months
7. Earnings replacement rate in 6 months
8. Entered employment and credential rate Older youth 9. Entered employment
rate (age 19-21) 10 Employment retention rate at 6 months
11. Average earnings change in 6 months
12. Entered employment/education/training and credential rate
Younger youth 13. Skill attainment rate
(age 14-18) 14. Diploma or equivalent attainment
15. Placement and retention rate
Customer satisfaction 16. Customer satisfaction for participants
17. Customer satisfaction for employers
Source: U.S. Department of Labor Employment and Training Administration,
Training and Employment Guidance Letter No. 7-99 (Mar. 3, 2000).
In contrast to JTPA, for which data on outcomes were obtained through
follow-ups with job seekers, WIA requires states to use UI wage records to
track outcomes. According to Labor's guidance, if a program participant does
not appear in the UI wage records, states may use supplemental data sources,
such as follow-ups with participants and employers, to track entered
employment, retention, and credential attainment. However, only UI wage
records may be used to calculate earnings change and replacement.
8Guidance from Labor defines a credential as a nationally recognized degree
or certificate or a recognized state/locally defined credential.
Unlike JTPA, which established expected performance levels using a computer
model, WIA requires states to negotiate with Labor to establish expected
performance levels for each measure. States, in turn, must negotiate
performance levels with each local area. The law requires that these
negotiations take into account differences in economic conditions,
participant characteristics, and services provided. To derive equitable
performance levels, Labor and the states use historical data to develop
their estimates of expected performance levels. These estimates provide the
basis for negotiations.
WIA holds states accountable for achieving their performance levels by tying
those levels to financial sanctions and incentive funding. States that meet
their performance levels under WIA are eligible to receive incentive grants
that may generally range from $750,000 to $3 million.9 To be eligible for an
incentive grant, states must also meet levels established under the
Department of Education's Vocational Education (Perkins Act) and Adult
Education and Literacy programs. States that do not meet their performance
levels under WIA are subject to sanctions.10 If a state fails to meet its
performance levels for 1 year, Labor provides technical assistance, if
requested. If a state fails to meet its performance levels for 2 consecutive
years, it may be subject to up to a 5-percent reduction in its annual WIA
formula grant. Under JTPA, the most stringent sanction was the possible
reorganization of the local service delivery organization.
9According to Labor's guidance, states must achieve a cumulative score of
100 percent or more in each program area-adult, dislocated worker, and
youth-and for customer satisfaction and at least 80 percent of the
negotiated performance level for all 17 measures. Failing to achieve the
80-percent level for any one performance measure disqualifies a state from
being considered for an incentive grant.
10According to Labor, while the agency has sole responsibility for
administering sanctions under WIA, awarding incentives is a joint effort
involving Labor and the Department of Education.
WIA Requires that Many Federal Programs Work Together to Provide Services
through the
One-Stop System
In addition to establishing the three new programs, WIA requires that states
use the one-stop center system to provide services for these and many other
employment and training programs. This system was developed by states prior
to WIA through One-Stop Planning and Implementation Grants from Labor. About
17 programs funded through four federal agencies are now required to provide
services through the one-stop center under WIA.11 Table 2 shows the programs
that WIA requires to provide services through the one-stop centers (termed
mandatory programs) and the related federal agency.
Table 2. WIA's Mandatory Programs and Related Federal Agencies
Federal agency Mandatory programs
Department of Labor WIA adult WIA dislocated worker WIA youth Employment
Service (Wagner-Peyser) Trade adjustment assistance programs Veterans'
employment and training programs Unemployment Insurance Job Corps
Welfare-to-Work grant-funded programs Senior Community Service Employment
Program Employment and training for migrant and
seasonal farm workers Employment and training for Native Americans
Department of Education Vocational Rehabilitation Program Adult Education
and Literacy Vocational Education (Perkins Act)
Department of Health and Human Community Services Block Grant
Services
Department of Housing and Urban HUD-administered employment and training
Development (HUD)
Under WIA, employers are expected to play a key role in establishing
regional workforce development policies, deciding how services should be
provided in the one-stop, and overseeing one-stop operations. Employers, who
are encouraged to use the one-stop system to fill their job vacancies, are
also seen as key one-stop customers under WIA.
11In addition, three other categories of programs are required to provide
services through the one-stop center: Youth Opportunity Grants;
demonstration, pilot, multiservice, research, and multistate projects; and
national emergency grants. Because they are of limited scope, we did not
include them in our total.
States and Localities Are Taking Action to Implement WIA Performance
Measures but Face Challenges in Doing So
States and localities are taking action to implement performance measures
for the three WIA-funded programs, but they have confronted several
challenges in doing so. To implement these measures, states and localities
had to change the way they collected and reported performance data. Most
states we surveyed had to create new automated data systems to collect and
report WIA data. Many state systems, however, are still not completely in
place. The lack of final guidance from Labor on how to report the data
slowed the development of these systems. States and localities also faced
challenges in implementing the measures due to their complexity and the
resource demands they created, and some had to develop new procedures to
obtain UI wage records. In addition, states faced a new negotiation process
with Labor to set performance levels for each measure. Many states believe
these levels are too high because little or no baseline data were used, and
the negotiations did not sufficiently account for differences in economic
conditions and populations served. Under WIA, performance levels are now
tied to incentives and sanctions so that states can be financially rewarded
if they meet them or penalized if they do not. States reported that the need
to meet these performance levels may lead local staff to focus WIA-funded
services on job seekers who are most likely to succeed in their job search
or who are most able to make wage gains.
Management Information Systems Are Still Being Developed to Collect and
Report WIA Data
As part of implementing WIA performance measures, states had to develop
automated data systems to track the activities of individual WIA
participants and report on performance.12 Based on our survey, most states
developed a new automated data system, or management information system
(MIS), to collect and report WIA performance data at the state level. The
remaining states adapted their previous data collection systems used under
JTPA. However, 15 states, regardless of whether they were developing a new
system or adapting their existing system, reported that, as of August 2001,
they did not have their system completely in place. All states expect to
have completed their systems by July 2002. In some states, local areas do
not use the state MIS to collect local WIA performance information. In these
states, local areas must develop their own systems, taking time and
resources to do so.
12WIA requires states to establish and operate management information
systems based on guidelines established by Labor, which are designed to
promote the efficient collection of information.
The lack of timely reporting guidance slowed the development of the data
systems. Final guidance on how states must report their performance to Labor
was issued to the states in March 2001-8 months after the states were
required to implement major provisions of WIA and begin collecting data.13
Lack of final guidance resulted in delays and costly program changes as
states and local areas developed and adjusted their final systems. For
example, one local area we visited decided to continue using its old system
and delayed the development of a new system pending final guidance because
it would be too costly and time-consuming to develop a system that might
need to be changed. All states, regardless of whether or not they had
implemented a new system, had to make changes in their automated MIS systems
to accommodate the final guidance.
Complex Performance Measures Stymied State and Local Efforts to Implement
Them
States and localities reported that the complexity of WIA's new performance
measures made them difficult and time-consuming to implement. Many of the
states that we surveyed commented that the measures were hard to follow
because the calculations for the measures are complex and sometimes
confusing, specifically who to include in the measures, when to collect the
data for the measures, and how to calculate the measures.14
* Knowing who to include in the measures. It is difficult to know whether or
not a job seeker should be counted in the measure for a program, even if he
or she is served by the program. For example, a participant in the adult
program who is already employed must be included in the retention and wage
gain measures but cannot be counted in the entered employment measure. Yet,
for the dislocated worker program, the entered employment measure can
include those who may be employed when they enter the program.15
* Knowing when to measure performance for participants. The data for
different performance measures can be collected in different
13U.S. Department of Labor Employment and Training Administration, Training
and Employment Guidance Letter No. 14-00 (Mar. 5, 2001).
14Calculations for the measures are found in U.S. Department of Labor
Employment and Training Administration, Training and Employment Guidance
Letter (TEGL) No. 7-99 (Mar. 3, 2000).
15Dislocated workers may include those who have obtained interim employment
after being displaced.
quarters of the program year. For example, for customer satisfaction, data
can be collected at two points in time, depending on how the participant
exited the program (see fig. 1). Data on entered employment for participants
in the adult program are collected in the third quarter after exit;
retention data are collected in the fifth quarter after exit. 16 Data on
earnings change and replacement are collected at two points in time:
pre-program earnings data can be collected at registration, and post-program
earnings data are collected in the fifth quarter after exit.17 If a program
participant does not appear in the UI wage records, local staff can collect
supplemental data to establish employment for the participant, but this must
be recorded within 30 days after the WIA participant is found missing from
the wage records. For entered employment, staff can collect supplemental
data in the fourth quarter after a participant leaves (or exits) the
program, but for retention, staff can collect it in the sixth quarter after
exit. Supplemental data cannot be used to measure earnings change and
replacement. Because the timing of data collection is complex and can be
confusing, one local area in Oregon developed a tool it calls the "bean
counter" to help local staff determine when to follow-up with participants
so their performance counts in the calculations (see fig. 2).
Figure 1: Timing of Data Collection by Quarter for Selected Performance
Measures
Program Year 1999 Program Year 2000 Program Year 2001
Source: Source: U.S. Department of Labor TEGL 7-99.
16 States must wait two full quarters after exit before collecting data
because of delays in available wage records.
17Data for pre-program earnings may be collected at registration or at any
time until the fifth quarter after exit.
Figure 2: Oregon's Bean Counter
Source: Worksystems Inc.
* Knowing how to calculate the measures. In order to calculate the measures,
states must account for a variety of factors. The type and combination of
these factors determine the calculations that will be used. For example, in
calculating some of the measures for the adult program, states must consider
(1) whether the job seeker is employed at registration, (2) whether he or
she is employed at both the first and third quarters after exit, and (3)
whether the data source used to confirm employment was UI records or
supplemental data. This information in various combinations results in 14
different ways that adult participants can be grouped in order to calculate
the measures.
In addition to noting the complexity of the measures, state and local
officials said that the new measures taxed their resources. States had to
develop procedures to collect data for the new customer satisfaction measure
in compliance with detailed guidance from Labor. The guidance calls for
states to conduct a telephone survey from a random sample large enough to
obtain 500 completed surveys from both participants and employers. Because
this guidance changed over time, states had to revise their procedures
accordingly. For example, a revision to the guidance issued in October 2001
required states to maintain an up-to-date list of participants' names and
addresses from which to sample-a requirement that was originally voluntary.
One indication of states' progress in implementing these measures may be
reflected in their ability to submit complete quarterly reports.18 Quarterly
reports require data on all 17 performance measures. For the quarterly
report that was due in May 2001, all states submitted their reports, but,
according to Labor, only 16 states were able to provide data for all 17
performance measures. For the quarterly report that was due in August
2001-more than 1 year after WIA implementation-all states submitted their
reports, but only 23 could provide data on employer customer satisfaction.19
According to Labor, states could not fully report on customer satisfaction
because they have not yet fully implemented procedures to measure it. One
state had to compile the data manually because its MIS was not fully
operational.
States Adopt New Procedures to Access Sensitive Information
WIA's new requirement that states use UI wage records to measure outcomes
has led states to adopt new procedures to access these and other sensitive
records. Unlike JTPA, which relied on surveys of participants to collect
information on employment and earnings, WIA requires UI wage records to be
used as the primary data source of employment and wage information-and the
only data source for some measures, according to Labor's guidance. To obtain
employment and earnings information, states match information collected on
individual WIA participants against state UI wage records. To access UI data
from the state agency that oversees the UI database, some states had to
establish data-sharing agreements. In Mississippi, for example, the agency
18As authorized by WIA, Labor requires states to submit quarterly summary
reports reflecting the state's performance levels and other activities.
19The 23 states are a subset of the 53 states and territories that report
WIA data to Labor.
responsible for overseeing WIA-the Mississippi Development Authority
(MDA)-had to make arrangements with the agency that oversees the UI data-the
Mississippi Employment Security Commission-to have them match the wage
records and provide the results to MDA. In addition, some states may be more
rigorous in protecting the confidentiality of UI records through privacy
laws, which may add obstacles to collecting performance data. For example,
Oregon law prohibits the release of WIA participants' records without
informed consent. Consequently, program providers had to enter into an
agreement that established a protocol for collecting and sharing the
data-one that developed safeguards to protect confidentiality. In addition,
the state had to develop a process to ensure that WIA participants consented
to the use of their protected records in this way.
States Believe Performance Levels Are Set Too High
All the states we visited believed that some of the established performance
levels for their measures were set too high for them to meet-either because
they were set in absence of historical or baseline data or because
negotiations did not sufficiently account for variations in economic
conditions or population served. States reported that limitations in
available baseline data made it difficult to set fair, realistic performance
levels. The new measures on credentials and customer satisfaction, for
instance, had no prior data available on which to set performance levels.
Where baseline data were available, such as for the wage-related measures,
the data were collected under JTPA, a program whose goals were different
from those of WIA.20 In addition, some states believe that the performance
levels did not account for variations in economic conditions, such as the
slow growth in new or existing businesses that some areas have experienced.
Performance levels also did not account for the many economically
disadvantaged or hard-to-serve individuals seeking services in some local
areas.
States Report that Performance Levels May Determine Who Receives WIA-Funded
Services
Many states reported that the need to meet performance levels may be the
driving factor in deciding who receives WIA-funded services at the local
level. All the states we visited told us that local areas are not
registering many WIA participants, largely attributing the low number of WIA
participants to concerns by local staff about meeting performance levels.
20While JTPA focused on providing training to the economically
disadvantaged, including the hardest-to-serve, WIA provides a broader range
of services to all individuals, regardless of their eligibility for other
services.
Local staff are reluctant to provide WIA-funded services to job seekers who
may be less likely to get and keep a job. One state official described how
local areas were carefully screening potential participants and holding
meetings to decide whether to register them. As a result, individuals who
are eligible for and may benefit from WIA-funded services may not be
receiving services that are tracked under WIA.
Performance levels for the measures that track earnings change for adults
and earnings replacement for dislocated workers may be especially
problematic. Several state officials reported that local staff were
reluctant to register already employed adults or dislocated workers.
Officials in one state reported that some local areas had not yet registered
any dislocated workers. State and local officials explained that it would be
hard to increase the earnings of adults who are already employed or replace
the wages of dislocated workers, who are often laid off from high- paying,
low-skilled jobs or from jobs that required skills that are now obsolete. In
addition, for dislocated workers, employers may provide severance pay or
workers might work overtime prior to a plant closure, increasing these
workers' earnings before they are dislocated. As a result, many dislocated
workers who come to the one-stop center have earned high wages just prior to
being dislocated, making it hard to replace-let alone increase- their
earnings. If high wages are earned before dislocation and lower wages are
earned after job placement through WIA, the wage change will be negative,
depressing the wage replacement level. As a result, a local area may not
meet its performance level for this measure, discouraging service to those
who may need it.
A hypothetical example involving two workers dislocated at the same time
illustrates this point (see table 3). One worker is a sales clerk with
limited skills earning $25,000, the other a long-time factory worker with
obsolete skills earning $60,000. Both are laid off from work and go to their
local one-stop center seeking job placement assistance. The clerk is placed
in a new job as a receptionist paying $25,000. By calculating his wage
replacement from his salary as a clerk, the one-stop can claim a wage
replacement rate of 100 percent. The factory worker eventually gets a job as
a security guard earning $30,000, netting a wage replacement rate of 50
percent. As this example shows, a one-stop center can meet its performance
levels more easily by serving the clerk than by serving the factory worker
even though both job seekers may need the one-stop system's resources to
find a job or enhance their skills.
Table 3: Comparing Wage Replacement Rates for Two Job Seekers
Scenario 1: Sales Clerk Scenario 2: Factory Worker
Sales clerk earning $25,000 Factory worker earning $60,000 after 20 years of
employment
* Gets laid off * Gets laid off
* Cannot find a job * Cannot find a job
* The center gets
him placed as a
security guard
earning $30,000
Wage replacement
rate: $25,000 = 100%
$25,000
Wage replacement rate: $30,000 = 50% $60,000
Note: Example uses annual full-time earnings. When replacement rates are
actually calculated using UI wage records, they will be based on earnings
reported on a quarterly basis.
Some states and Labor are making efforts to address this disincentive to
serve certain job seekers. Indiana instituted a policy allowing local areas
to adjust their dislocated worker wage replacement rate in light of the
significant dislocations they are facing.21 Texas uses a regression model to
establish local performance levels that adjust for differences in factors,
such as economic conditions and the characteristics of individuals served.
Without this policy, said a Texas official, WIA programs would have
registered fewer workers. Similarly, Michigan substantially reduced the
penalties to local areas for failing to meet performance levels and found
that the number of registered participants increased as a result of
instituting less threatening sanctions. WIA requires that states be allowed
to renegotiate their performance levels based on unanticipated
circumstances. Labor is currently developing criteria that states can use to
renegotiate their performance levels based on unanticipated circumstances,
such as changes in economic conditions due to plant closings or shifts in
unemployment for the current and future years. The guidance is expected to
be released soon.
21The adjusted levels would apply only to the dislocated workers from the
company that has been affected, not to the entire local area.
Performance Measures May Not Accurately Assess Performance of the Three
WIA-Funded Programs
Even when fully implemented, WIA performance measures may still not provide
an accurate picture of performance for the three WIA-funded programs largely
because data are neither comparable across states nor timely. State and
local officials generally supported many of the performance measures as
relevant indicators of the success of an employment and training program.
However, the performance data collected and reported by states and
localities are not comparable largely because of the lack of clear guidance
on when to collect and report performance data and what constitutes a
credential-the attainment of a certified skill or degree. In addition, while
UI wage records are one of the best available sources of employment and
earnings data, limitations in the data may hinder the ability of states and
local areas to meet their performance levels and use the measures for
short-term program management.
Performance Measures Include Relevant Indicators, but Lack of Clear Guidance
Affects Accuracy and Comparability of Performance Data
State and local officials in the states we visited generally support many of
the performance measures as relevant indicators of the success of an
employment and training program. For example, several officials cited the
wage-related measures, such as job placement, retention, and earnings
change, as important indicators of a successful employment and training
program. The measures are also generally consistent with the goal of WIA to
help individuals get and keep jobs and increase their wages and skills. In
addition, the states noted that the measures provide a good basis for
long-term evaluation.
However, the performance data collected and reported by states and
localities are not comparable-a critical component in creating a level
playing field from which states' relative performance can be evaluated.
While there are various reasons that performance data are not comparable,
one of the chief reasons is the lack of clear guidance for collecting and
reporting performance data on participants. Labor has provided detailed
written guidance to states on who should be registered under WIA and when
this registration should occur, but the guidance is open to interpretation
in some areas.22 The lack of a uniform understanding of when registration
occurs and thus who should be counted toward the measures raises questions
about both the accuracy and comparability of states' performance data. For
example, the guidance tells states to register adults and dislocated workers
who need significant
22TEGL No. 7-99.
staff assistance designed to help with job seeking or acquiring occupational
skills, but the state can decide what constitutes significant staff
assistance.23 The guidance provides examples of when to register job
seekers, but it sometimes requires staff to make subtle and subjective
distinctions. For example, those who receive initial assessment of skill
levels and the need for supportive services are not to be registered; those
requiring comprehensive assessment or staff-assisted job search and
placement assistance, including career counseling, are to be registered. In
another example involving the classification of workshops, job seekers who
participate are to be registered in some cases, but not in others.
Labor has allowed states and local areas flexibility in implementing the
registration policy, and we found that local areas differed on when they
registered WIA job seekers. In one local area we visited, the one-stop
center registers most job seekers who come into the center, even if staff
assistance is minimal. At this center, a general orientation is sufficient
for the job seeker to be registered under WIA. In contrast, another center
in the same state registers only those job seekers who require significant
staff assistance and are likely to benefit from intensive services. Similar
disparities occurred in other states we visited. Labor has said there is
little consistency across states in registering participants and has
convened a work group to develop additional guidance on registration, but as
yet, the issues remain unresolved.
The lack of a definition for the credential measure is also leading to
performance data that are not accurate or comparable across states. Labor
allows the states and local areas to determine what constitutes a credential
and to develop a statewide list of approved credentials with input from
employers. Because states and, in many cases, local areas must define what
constitutes a credential, what is currently counted as a credential differs
within and across states. Some states may strictly define credentials to
include only diplomas from accredited institutions or use only formal
training completion criteria as defined by education partners. Other states
may expand their criteria to count a broad variety of credentials, such as
job readiness, on-the-job experience, and completion of workshops. Labor
officials note that states' performance levels for the credential measure
are negotiated to take state and local definitions into account, and the
measure is intended to help local employers gauge the readiness and skill
level of job seekers. Nevertheless, given the broad
23All youth who receive WIA-funded services are required to be registered.
range of definitions states and localities employ, the outcomes on the
credential measure may be of limited value, even within a single state.
Challenges in Using UI Wage Record Data Also May Affect the Accuracy and
Comparability of Performance Data
UI wage records are one of the best available data sources for tracking the
employment and earnings of individuals-a significant improvement over the
less objective self-reporting methods of JTPA-but the limitations of the
database pose challenges that need to be addressed. These challenges, if
unresolved, may hinder states' ability to meet their performance levels. As
we have reported in prior work,24 one such limitation is that UI wage
records, while covering about 94 percent of workers, exclude certain
employment categories, such as self-employed persons, most independent
contractors, military personnel, federal government workers, and postal
workers. States, therefore, must develop alternative methods to track WIA
participants who are employed in these uncovered occupations. Pennsylvania,
for example, developed a partnership with other states in its region to
share the cost of purchasing the rights to federal civil service and
military personnel data. And Florida has developed agreements with the
Department of Defense, the Office of Personnel Management, and the U.S.
Postal Service to access employment and wage information on an annual basis.
Our survey data indicate that 33 states are using additional or supplemental
data to compensate for uncovered occupations, with only 27 of those using
the supplemental data to count towards their performance levels. Thus, at
least 23 states have not used additional data to help them meet their
performance levels.
Another limitation is that state UI databases include only wage record
information on job seekers who get jobs within their state; they do not
track job seekers who find jobs in other states. States cannot readily
access UI wage records from other states to track outcomes under WIA, making
it difficult to track individuals who receive services in one state but get
a job in another. Over one-third of all of the states we surveyed reported
that an estimated 16 to 30 percent of cases are not being picked up by their
state's UI wage record system. To fill in these gaps, seven states have
agreements with other states-often those that share a common border-to
exchange UI information. Indiana, for example, established an agreement with
Illinois to trade data. If data are missing on
24See U.S. General Accounting Office, Veterans' Employment and Training
Service: Proposed Performance Measurement System Improved, But Further
Changes Needed,
GAO-01-580 (Washington, D.C.: 2001).
particular participants, Illinois sends the cases to Indiana to see if the
Indiana UI wage records have information on the job seeker. The value of
these agreements, however, may be limited because job seekers may find work
in a state that does not have an agreement with the one in which they
received services.
Another way to obtain UI data on workers who are employed out of state is
through WRIS, a clearinghouse that makes UI wage records available to states
seeking employment and wage information on their WIA participants. This
information can provide outcome data on WIA participants to help states meet
their performance levels. While WRIS was available for states to use by July
2001, only 7 out of the 50 states are currently able and ready to
participate, with 8 others in various stages of completing the requirements
for participation.25 Although many states have shown an interest in a system
such as WRIS, many are reluctant to participate because Labor, while
agreeing to cover all the costs of operating WRIS for its first year, has
not yet agreed to pay for subsequent years. The estimated total cost of
operating WRIS is $2 million annually, but states have not been given a
definitive answer about how much it would cost them to participate after
this first year if Labor does not continue funding.26 Because of this
uncertainty regarding future costs, states are hesitant to commit to
participation in WRIS. If not all states participate, the value of WRIS will
be diminished-even for participating states-because no data will be
available from nonparticipating states' UI wage records.
Lack of Timely Data Limits Use of Performance Measures for Short-Term
Program Management
The lack of timely data, due to the time lag in obtaining UI wage records,
makes it difficult for state and local officials to use the performance
measures for short-term program management-because, for the wage-related
measures, current available data on the measures will reflect performance
from the previous program year. While UI wage records are the best available
data source for documenting employment, the data collection and reporting
process is slow and time-consuming. Data are generally collected from
employers only once every quarter, and
25To participate, states are required to be technically and administratively
ready. For example, states must be able to respond to queries for UI wage
data and establish internal security measures to ensure the confidentiality
of data.
26A national association is currently working a contractor to study the
costs associated with WRIS.
employers have 30 days after the quarter ends to report the data. In many
states, employers-especially small businesses-are allowed to submit data in
paper format, which then must be converted to electronic media. After data
entry, information must be checked for errors and corrected. All of these
steps take time. As a result, WIA program administrators are unable to get a
timely picture of program performance. For example, we asked states in our
survey how quickly job placement outcome data would be available to them
from UI wage records. On the basis of our survey, we found that for 30
states, the earliest time period that job placement data would be available
is 6 months or more after an individual entered employment, with 15 states
reporting that it may take 9 months or longer. Similarly, for the employment
retention measure, over half of states report that obtaining this
information could take a year or longer. (See fig. 3.)
Figure 3: Number of Months it Takes to Receive UI Data for Job Placement
Note: Our survey asked two questions regarding UI time frames. One question
asked for the earliest date states would expect to receive UI data; the
other asked for the latest date.
Source: GAO survey.
The time delay in receiving UI wage record data makes it difficult for state
and local officials to use the performance measures to gauge the
effectiveness of their services. States report that not being able to get
performance results in the same program year is a problem: it makes it
difficult to manage programs and improve one-stop services. Labor reports
that the performance measures are not intended to be a management tool.
State and local officials, therefore, must develop alternative methods if
they want to assess the quality of their services so they can identify
problems and improve programs in a timely way. Labor has encouraged these
efforts, but, while some local areas are finding ways to collect data to
help them manage their programs, there is no cohesive effort at the federal
level to share strategies and promising approaches for the Adult and
Dislocated Worker Programs.27
Existing Performance Measures Fail to Gauge Overall One-Stop Performance
Although there are performance measures for the three WIA-funded programs
and most of the programs required to provide services in the one-stop, no
measures exist to assess how well the overall one-stop systems are working.
The success of the one-stop system as a whole is not captured by the program
measures of individual one-stop partners. Furthermore, combining the
performance measures from mandatory programs does not provide a
comprehensive picture of one-stop performance. Even when measures appear the
same, comparing them is difficult because of differences in definitions and
calculations. Beyond failing to provide a complete evaluation of one-stop
performance, state officials reported that the separate reporting
requirements of the partner programs have hampered coordination within their
one-stop systems. While WIA did not establish any comprehensive measures to
assess the overall one-stop system, it required that Labor take the lead in
developing optional measures to help states assess progress toward their
workforce investment goals. Labor has made limited progress on such
performance measures, and only a few states have developed their own overall
measures.
27Labor's Office of Youth Services has taken action to help youth programs
in this area, such as creating a set of short-term performance measures.
Existing Performance Measures Fail to Capture Important One-Stop Features
The existing performance measures for participating one-stop programs fail
to capture important one-stop features. First, it is difficult to get an
unduplicated count of job seekers using the one-stop. While an individual
may have multiple outcome measures for the services received through each of
the programs at a one-stop, there is no single outcome measure for multiple
services. In addition, separate reporting systems for each of the programs
make it difficult to disaggregate data and track an individual's outcome for
those receiving multiple services.
Second, other important aspects of one-stop performance are not included
within the existing measures. Customer satisfaction measures used in support
of WIA-funded programs and the Employment Service fail to measure how job
seekers and employers believe they are being served by the one-stop system
as a whole. Instead, these measures show satisfaction with the individual
programs. Employer satisfaction is important under WIA because WIA created a
more private-sector driven system. Capturing customer satisfaction of the
system as a whole would reflect whether job seekers are successful at
attaining the services they need to get jobs and would assess whether
employers are satisfied with job applicants sent to them from different
one-stop programs.
Finally, state and local officials expressed concern that a large portion of
one-stop participants are not included in performance measures. Many job
seekers use self-service and informational activities but they are not
tracked and counted in any program measures. While staff time and resources
are used to establish and maintain self-service resource rooms and web
sites, job seekers who use only these services will not be included in any
of the performance measures. Without any information on individuals who use
self-service, it will be difficult for Labor to show how effectively
one-stops are being used.
Multiple Measures Cannot Be Used to Measure the One-Stop System's
Performance
Performance measures for different programs often track similar outcomes, as
figure 4 shows. However, the measures cannot be combined to obtain an
overall view of one-stop performance. Although the same terms are used in
various performance measures, their definitions are not identical. For
example, while WIA older and younger youth programs define youth as being
between the ages of 14 and 21, the laws governing Job Corps and HUD's
Youthbuild define youth as being between the ages of 16 and 24. Similarly,
the definition of veterans is different for the Employment Service and
Veterans' Employment and Training Service program. The differences in
definitions mean that assessing the outcomes for youth or veterans by
combining the performance measures of individual programs within the
one-stop setting would be difficult.
Figure 4: Comparison of Performance Measures for 17 One-Stop Programs
aMay include retention in education and/or training.
bMay include the attainment of a credential or placement in education or
training.
c Two measures for customer satisfaction- one for employers, one for job
seekers-are used for all WIA-funded programs.
d This program reports by submitting individual records on program
participants.
eBased on proposed regulations.
fUnemployment Insurance does not have participant performance measures.
gSCSEP is required to develop performance measures, which are expected to be
in place for program year 2003, according to a Labor official.
h Grantees for this program are required to use the core indicators of
performance common to WIA adult and youth programs.
iIndividual grantees propose performance in their individual grantee plans,
which Labor must approve.
jMeasures must include either placement or retention and may include both.
kFor this program, states must report on at least one measure from each of
six national goals. The six national goals are (1) number employed, (2)
number maintaining 90-day retention, (3) number making progress toward
literacy/GED, (4) number gaining health care coverage through employment,
(5) increase in the availability and affordability of essential services,
and (6) increase in access to community services and resources by low-income
people. States may select from among national measures or develop their own.
Source: Based on agency regulations and documents.
Besides variations in definitions, there are also variations in how measures
are calculated in different programs. For example, while the entered
employment rate for WIA's adult program is defined as the percentage of
workers who get a job by the end of the first quarter after exit, the
entered employment rate for the Employment Service is defined as the
percentage of workers who get a job or changed employers in the first or
second quarter after registration. As a result, performance data from these
separate programs cannot be combined to yield a single overall score to
assess various performance outcomes of the one-stop system. The Office of
Management and Budget (OMB) convened a work group representing federal WIA
partners to look at common definitions and measures across programs, which
may issue guidance to states and localities on WIA performance measures and
federal requirements.
meant to establish a more coordinated workforce development system through
the use of one-stops, over one-third of states surveyed expressed concern
that individual program performance measures may impede this process. Some
states even believed that separate measures caused competition among
programs. They said that if staff did not understand that a participant
could be counted in more than one program, they might not direct them to
other one-stop programs. For example, one state reported in its written
comments to our survey that competition for participants and duplication of
services due to lack of coordination with
28This issue was also raised in a study that found that the continued
emphasis on measuring performance in categorical programs undermines the
integration expected in WIA. Evelyn Ganzglass, Martin Jensen, Neil Ridley,
Martin Simon, and Chris Thompson. Transforming State Workforce Development
Systems: Case Studies of Five Leading States (National Governors'
Association, 2001).
Separate Measures May Beyond failing to provide an overall picture of
one-stop performance, state
Impede Coordination
officials reported that separate performance measures impede the cooperation
of one-stop partners.28 As a result, even though WIA was
other programs would continue as long as each program is required to meet
its own performance and participation levels. Fourteen states volunteered in
their written comments to our survey that the federal government should work
to coordinate performance measures across programs or develop systemwide
measures. In addition, while states agree that systemwide measures are
needed, they caution against making any additional measures mandatory, since
states are still adapting to the existing measures.
Limited Progress on Developing Performance Measures for One-Stop Systems
Although WIA did not establish one-stop measures, it does require that Labor
develop additional optional measures to assist states in assessing progress
toward their workforce investment goals,29 which Labor has interpreted to
include one-stop measures. Labor began developing workforce development
performance measures to capture overall one-stop use after one-stop systems
were piloted. Since the passage of WIA, Labor has continued its efforts to
develop systemwide measures, but it has made limited progress. Labor
convened a working group in September 2001 to develop additional indicators
of one-stop performance. Partner representatives at this group included
national workforce-related organizations including the National Association
of State Workforce Agencies, National Governors' Association, National
Association of Counties, U.S. Conference of Mayors, and representatives of
states and regional boards. This group is working to develop a menu of
indicators that will help provide a comprehensive picture of WIA system
activity. Such measures may include capturing information on self-service
customers and the cost of services at the one-stop.30 These measures, while
optional, would help provide information on overall one-stop use across the
country if all states report on at least some of the measures. Labor plans
to have guidelines for these optional indicators in place for use in program
year 2002, which begins on July 1, 2002. In order for states to be able to
implement them for the coming program year, Labor will need to provide final
guidance well before July 2002.
29Section 136 (i)(1) requires Labor to work with workforce development
system partners to develop terms for a menu of additional indicators of
performance to help states assess their progress toward their workforce
investment goals. The purpose of developing these additional measures is to
ensure nationwide comparability.
30Labor is currently seeking OMB approval to collect data on the costs and
usage of WIA and Wagner-Peyser funded services that do not require
registration. Labor issued a Federal Register Notice on January 16, 2002
seeking public comment on this proposal.
In addition to these national efforts, some states, on their own initiative,
have attempted to develop additional measures for one-stop systems, but
these efforts are not coordinated and do not allow for nationwide assessment
of the one-stop system. According to our survey, eight states have created
or are developing additional systemwide measures, but of those, only three
are reporting them to Labor. Pennsylvania, for example, developed five
measures specific to its one-stop system's performance. These indicators,
intended to measure the overall effectiveness of the one-stop system,
include median cycle time to fill a job, and the percentage of employers and
individuals using services through the one-stop. Florida, on the other hand,
has developed "tiers" of measures that focus on the outcome of their
workforce development programs. In the first tier, state-generated
systemwide indicators measure many employment and training programs
together. The second tier clusters similar types of programs and captures
measures relevant to particular groups (e.g., continued education status of
youth in youth programs). The third tier captures all the federally mandated
measurements, as well as measures for the other tiers, such as caseloads for
specific programs. In this way, Florida has attempted to measure the system
overall as well as outcomes for individual programs. Measures developed by
other states include the number of people using the resource rooms at
one-stops and the increase over time in the number of unemployed people
getting a job. Despite these states' efforts, the absence of nationally
established systemwide measures means that Labor cannot ensure nationwide
comparability.
WIA represents a fundamental shift in the way federally funded employment
and training services are provided to job seekers and in the way WIA
programs measure and monitor success. Despite obstacles, in just over a year
states have made good progress in implementing the new requirements under
WIA-developing new processes and designing new systems. Labor, for its part,
has been working to find ways to allow states and localities greater
flexibility to design their programs to meet local needs and has been
actively seeking opportunities for states to have input into the process,
particularly in the area of performance measurement. But given the
challenges states have faced in implementing the new performance measurement
system, more time is needed before the measures can meaningfully gauge the
success of the programs.
This new performance measurement system under WIA is a high-stakes game-a
state's future funding and, therefore, its ability to serve its citizens may
depend upon how well it performs compared to how well it is expected to
perform. It should be no surprise that states and localities are
Conclusions
designing their systems and processes in ways that will enhance their
ability to meet their performance levels. Because states see the current
performance levels as too high for the current economy, states and
localities may choose not to serve those job seekers who may be helped by
their services, but who may not help in achieving their negotiated
performance levels. Unless the performance levels can be adjusted to truly
reflect differences in economic conditions and the population served, local
areas will continue to have a disincentive to serve some job seekers that
could be helped.
WIA's requirement to use UI data to track outcomes is a step in the right
direction-it provides federal, state, and local government entities with an
objective means to evaluate program success. But it brings challenges that
need to be addressed, and states will need help to do so. Establishing the
means to routinely share data across state lines through WRIS and developing
ways to share promising approaches in the use of supplemental data sources
and in managing the assessment of short-term program needs would go far in
moderating these challenges. Without this help and the cooperative efforts
of states and localities toward this end, developing a useful performance
measurement system will take longer and cost more.
In general, WIA's performance measurement system captures some useful
information, but it may not capture all the right information. The measure
to track credentials has limited value because it lacks a standard
definition for what's being measured. For other measures, the lack of clear
definitions for whom to track limits their usefulness in drawing conclusions
about program success at both the state and national levels. Without clear
definitions and processes, the measures will not provide the Congress with a
true picture of how well the programs are performing. Furthermore, WIA
performance measures gauge only WIA-funded services; yet there is widespread
agreement that measures are needed to gauge the effectiveness of the entire
one-stop system. The system's narrow focus on program outcomes for a limited
number of participants misses a key requirement of WIA to support the
movement toward a coordinated system. In fact, the measures may foster the
opposite-a siloed approach that encourages competition among programs and
limits their cooperation. Without global one-stop measures, the Congress
will not be able to assess how well states and localities are doing in
meeting WIA's requirement to coordinate services. The lack of such measures
may, instead, send a signal to states that service coordination is a minor
goal.
Recommendations for Executive Action
To give states and local areas more time to implement WIA performance
measures and establish baseline data needed to determine performance levels,
we recommend that the Department of Labor
*
delay the application of financial sanctions for at least 1 year or until it
is judged that states have their data systems sufficiently in place to
successfully track WIA outcomes.
To eliminate possible disincentives to serve some job seekers and ensure
that states and local areas will not be unduly penalized for economic
downturns, we recommend that the Department of Labor
*
expedite the release of guidance on revising negotiated performance levels
and allow states to immediately begin the process of re-negotiation.
To ensure uniformity in data collection and reporting so that performance
results are more accurate and comparable across states, we recommend that
the Department of Labor
*
provide clearer guidance using objective criteria on who should or should
not be registered as a WIA participant for tracking purposes and, once the
guidance is released, work proactively with states to implement it, and
*
issue guidance delineating a clear definition for what constitutes a
credential, and, once the guidance is released, ensure that states use it to
report on this indicator.
To help states address the challenges of using UI data to measure outcomes,
we recommend that the Department of Labor
*
continue to fully fund the Wage Record Interchange System in order to
facilitate the sharing of UI data across state lines; and
*
develop ways for states to share promising approaches in the use of
supplemental data sources in closing the data gaps for covered and uncovered
employment in UI; and
*
develop ways for states to share promising approaches that help states
address the UI timeliness issue, providing methods to help states monitor
and improve their programs in a timely manner.
Agency Comments
To help states measure one-stop performance, we recommend that the
Department of Labor
*
ensure that the development of optional one-stop system measures is
completed in enough time for states to implement them at the beginning of
program year 2002.
We provided a draft of this report to Labor for its review and comment.
Labor's comments are in appendix II. We incorporated comments and
clarifications where appropriate.
Labor generally agreed with our findings and recommendations, noting that
they are consistent with information they have gathered from state and local
partners. In its comments, Labor expressed concern that negotiated
performance levels may be determining who receives WIA-funded services,
indicating that it will work with states and local areas to address this
issue. Labor also commented on our finding regarding the lack of clear
guidance on certain policies, stressing the importance of state and local
flexibility in determining specific policies and practices to fit local
needs. While state and local flexibility is important, we continue to be
concerned that the lack of a uniform understanding of when registration
occurs and what constitutes a credential raises questions about both the
accuracy and the comparability of states' performance data. We are pleased
to note that Labor is in the process of reviewing this issue. Finally, Labor
cites its efforts to collaborate with states and local areas in developing a
performance accountability system and increasing partnerships. We commend
Labor for obtaining states' input and participation in developing such a
system.
We are sending copies of this report to the Secretary of Labor, relevant
congressional committees, and others who are interested. Copies will also be
made available to others upon request. The report is also available on GAO's
home page at http://www.gao.gov.
Please contact me on (202) 512-7215 if you or your staff have any questions
about this report. Other major contributors to this report are listed in
appendix III.
Sigurd R. Nilsen Director, Education, Workforce, and Income Security Issues
Appendix I: WIA Performance Measures for Adults, Dislocated Workers, and
Older Youth
The following table includes descriptions of the performance measures
reviewed in this report. It does not include the calculations required to
attain the final value of the performance measures.
Adults Definition
Entered employment rate Of those who did not have a job when they registered
for WIA, the percentage of adults who got a job by the end of the 1st
quarter after exit. This measure excludes participants who are employed at
the time of registration.
Employment retention rate at 6 months Of those who had a job in the
1the 3rd quarter after exit.
Average earnings change in 6 months Of those who had a job in the 1quarter after exit, the post-program earnings increases as with pre-program earnings.
Employment and credential rate Of those adults who received WIA training
services, the percentage who were employed in the 1st quarter after exit and
received a credential by the end of the 3rd quarter after exit.
Dislocated workers
Entered employment rate The percentage of dislocated workers who got a job
by the end of the 1st quarter after exit. This measure includes dislocated
workers who are employed at the time of registration.
Employment retention rate at 6 months Of those who had a job in the 1st
quarter after exit, the percentage of dislocated workers who have a job in
the 3rd quarter after exit.
Earnings replacement rate in 6 months Of those who had a job in the
1earned post-program. Since it may be difficult to find dislocated workers
jobs with equivalent or better wages, this measure captures the percentage
of earnings of the new job in relation to the old.
Employment and credential rate Of those dislocated workers who received WIA
training services, the percentage who were employed in the 1st quarter after
exit and received a credential by the end of the 3rd quarter after exit.
Older youth (age 19-21)
Entered employment rate Of those who are not employed at registration and
who are not enrolled in post-secondary education or advanced training in the
1st quarter after exit, the percentage of older youth who have gotten a job
by the end of the 1st quarter after exit. This measure also excludes youth
that move on to post-secondary education or advanced training and not
employment.
Employment retention rate at 6 months Of those who are employed in the 1st
quarter after exit and who are not enrolled in post- secondary education or
advanced training in the 3rd quarter after exit, the percentage of older
youth that are employed in the 3rd quarter after exit.
Average earnings change in 6 months Of those who had a job in the 1 st
quarter after exit and who are not enrolled in post- secondary education or
advanced training, the post-program earnings increases as compared with
pre-program earnings.
Employment/education/training and The percentage of older youth who are in
employment, post-secondary education, or
credential rate advanced training in the 1st quarter after exit and received
a credential by the end of the 3rd quarter after exit.
Customer satisfactiona
Employer customer satisfaction The average of three statewide survey
questions rated 1-10: (1 being " very dissatisfied" and 10 being "very
satisfied"):
* was the employer satisfied with services
Appendix I: WIA Performance Measures for Adults, Dislocated Workers, and
Older Youth
Adults Definition
* did the service meet the expectations of the customer
* how well did the service compare to the ideal set of services
Participant customer satisfaction The average of three statewide survey
questions rated 1-10:
* was the participant satisfied with services
* did the service meet the expectations of the customer
* how well did the service compare to the ideal set of services
aA statewide telephone survey of a sample of 500 is conducted for all the
WIA-funded programs. Source: U.S. Department of Labor TEGL 7-99.
Appendix II: Comments From the Department of Labor
Appendix II: Comments from the Department of Labor
Appendix II: Comments from the Department of Labor
Appendix III: GAO Contacts and Staff Acknowledgments
GAO Contacts Dianne Blank (202) 512-5654 Ronni Schwartz (202) 512-7033
Staff Abbey Frank, Mikki Holmes, and Amanda Ahlstrand made significant
contributions to this report. In addition, James Wright assisted in the
study
Acknowledgments design and the national survey; Jessica Botsford and Richard
Burkard provided legal support; and Patrick DiBattista and Barbara Alsip
assisted in the message and report development.
Related GAO Products
U.S. General Accounting Office. Workforce Investment Act: Better Guidance
Needed to Address Concerns Over New Requirements. GAO-02-
72. Washington, D.C.: 2001.
U.S. General Accounting Office. Veterans' Employment and Training Service:
Proposed Performance Measurement System Improved, But Further Changes
Needed. GAO-01-580. Washington, D.C.: 2001.
U.S. General Accounting Office. Multiple Employment and Training Programs:
Overlapping Programs Indicate Need for Closer Examination of Structure.
GAO-01-71. Washington, D.C.: 2000.
U.S. General Accounting Office. Workforce Investment Act: Implementation
Status and the Integration of TANF Services. GAO/T-HEHS-00-145. Washington,
D.C.: 2000.
(130009)
GAO's Mission
Obtaining Copies of GAO Reports and Testimony
The General Accounting Office, the investigative arm of Congress, exists to
support Congress in meeting its constitutional responsibilities and to help
improve the performance and accountability of the federal government for the
American people. GAO examines the use of public funds; evaluates federal
programs and policies; and provides analyses, recommendations, and other
assistance to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
The fastest and easiest way to obtain copies of GAO documents is through the
Internet. GAO's Web site (www.gao.gov) contains abstracts and full-text
files of current reports and testimony and an expanding archive of older
products. The Web site features a search engine to help you locate documents
using key words and phrases. You can print these documents in their
entirety, including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its Web
site daily. The list contains links to the full-text document files. To have
GAO e-mail this list to you every afternoon, go to www.gao.gov and select
"Subscribe to daily e-mail alert for newly released products" under the GAO
Reports heading.
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out to
the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:
U.S. General Accounting Office P.O. Box 37050 Washington, D.C. 20013
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
Visit GAO's Document GAO Building
Distribution Center Room 1100, 700 4th Street, NW (corner of 4th and G
Streets, NW) Washington, D.C. 20013
To Report Fraud, Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm,
Waste, and Abuse in E-mail: [email protected], or
Federal Programs 1-800-424-5454 or (202) 512-7470 (automated answering
system).
Jeff Nelligan, Managing Director, [email protected] (202) 512-4800
Public Affairs U.S. General Accounting Office, 441 G. Street NW, Room 7149,
Washington, D.C. 20548
*** End of document. ***