Workforce Investment Act: Labor Actions Can Help States Improve  
Quality of Performance Outcome Data and Delivery of Youth	 
Services (23-FEB-04, GAO-04-308).				 
                                                                 
The Workforce Investment Act (WIA) has been in effect for several
years and is currently undergoing reauthorization. In order to	 
provide the Congress with information on the implementation and  
effectiveness of the WIA youth program, GAO was asked to explore 
how services have been delivered at the local level, whether the 
Department of Labor's guidance has addressed challenges faced by 
local areas, and how effective the program has been.		 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-04-308 					        
    ACCNO:   A09295						        
  TITLE:     Workforce Investment Act: Labor Actions Can Help States  
Improve Quality of Performance Outcome Data and Delivery of Youth
Services							 
     DATE:   02/23/2004 
  SUBJECT:   Federal aid programs				 
	     Locally administered programs			 
	     Program evaluation 				 
	     Strategic planning 				 
	     Youth employment programs				 
	     Workforce Investment Act Standardized		 
	     Record Data					 
                                                                 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-04-308

United States General Accounting Office

GAO 	Report to the Ranking Minority Member, Committee on Health,
Education, Labor, and Pensions, U.S. Senate

February 2004

WORKFORCE INVESTMENT ACT

 Labor Actions Can Help States Improve Quality of Performance Outcome Data and
                           Delivery of Youth Services

GAO-04-308

Highlights of GAO-04-308, a report to the Ranking Minority Member,
Committee on Health, Education, Labor, and Pensions, U.S. Senate

The Workforce Investment Act (WIA) has been in effect for several years
and is currently undergoing reauthorization. In order to provide the
Congress with information on the implementation and effectiveness of the
WIA youth program, GAO was asked to explore how services have been
delivered at the local level, whether the Department of Labor's guidance
has addressed challenges faced by local areas, and how effective the
program has been.

GAO is recommending that the Departments of Labor and Education coordinate
efforts to clarify how schools can work with workforce officials to help
connect school dropouts to local WIA youth programs. GAO is also
recommending that the Department of Labor provide states and local areas
with technical assistance necessary to address ongoing implementation
challenges and establish standard monitoring procedures to improve the
quality of data reported by states.

In formal comments on a draft of this report, Education concurred with our
recommendation to work with Labor to connect out-ofschool youth to local
WIA youth programs. In its informal comments, Labor said that many of the
findings corroborated its own observations and that the recommendations
are consistent with Labor's current program direction.

www.gao.gov/cgi-bin/getrpt?GAO-04-308.

To view the full product, including the scope and methodology, click on
the link above. For more information, contact David Bellis (415) 904-2272
or [email protected].

February 2004

WORKFORCE INVESTMENT ACT

Labor Actions Can Help States Improve Quality of Performance Outcome Data and
Delivery of Youth Services

Local areas primarily used the WIA program for dropout prevention and
other efforts to improve academic achievement for in-school youth.
Nationally, about 70 percent of youth served were in school, but
percentages ranged from 38 to 86 percent by state. Officials in the five
states GAO visited said that they focused on in-school youth because
serving out-of-school youth was much more difficult and expensive, and
less effective. Local areas emphasized learning-related summer employment
for in-school youth and occupational skills training and supportive
services for out-of-school youth. Over half of local boards nationwide
used providers that had subcontracting arrangements with others to deliver
youth services. The majority of youth were served primarily from
educational institutions and community organizations.

Despite Labor's guidance, local areas continue to face implementation
challenges in identifying and retaining out-of-school youth, providing
youth with mentoring and follow-up services, and using interim measures
for ongoing program assessment. While Labor supports information exchange
forums, a promising practices Web site, and technical assistance, some
local areas may have difficulties gaining access to and using these
resources.

Little is known about the effectiveness of the WIA youth program because
Labor has not yet conducted an impact evaluation. In addition, while the
youth program exceeded most of its performance goals, these data were
questionable because of problems with state information systems and
inadequate oversight of data quality. While states will be required to
verify data, concerns remain about their ability to fully implement the
requirement and Labor's ability to monitor implementation consistently.

Source: GAO.

WIA youth gain summer work experience by landscaping a local high school's
grounds.

Contents

Letter

Results in Brief
Background
WIA Program Used Primarily for Dropout Prevention and

Emphasized Summer Services and Skills Training Coordinated among Multiple
Service Providers Labor's Guidance Has Not Fully Addressed Implementation
Challenges Faced by Local Areas Lack of Program Evaluations and
Questionable Performance Data

Prevent Assessments of WIA Youth Program Conclusions Recommendations for
Executive Action Agency Comments and Our Evaluation

                                       1

                                      3 4

                                       9

21

26 30 31 31

Appendix I Comments from the Department of Education

Appendix II GAO Contacts and Staff Acknowledgments 35

GAO Contacts 35 Staff Acknowledgments 35

Related GAO Products

    Tables                                                                
                       Table 1: WIA Youth Performance Measures              8 
             Table 2: Percentage of Local Boards Using Different Types of 
                        Service Delivery Arrangements, by Number of Youth 
                                       Served                              17 

Figures

Figure 1: Percentage of All WIA Youth Served Who Were in School, by State
(including the District of Columbia) 10 Figure 2: WIA Services Provided to
In-School and Out-of-School Youth 13 Figure 3: Number of WIA Youth Service
Providers Used by Local Boards 16 Figure 4: Primary Providers of Services
to WIA Youth 18

Abbreviations

OIG Office of Inspector General
JTPA Job Training Partnership Act
WIA Workforce Investment Act
WIASRD Workforce Investment Act Standardized Record Data

This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.

United States General Accounting Office Washington, DC 20548

February 23, 2004

The Honorable Edward M. Kennedy

Ranking Minority Member

Committee on Health, Education, Labor, and Pensions United States Senate

More than 5 million young people-15 percent of the nation's youth
population between ages 16 and 24-are out of school and out of work,
according to a recent study. Further, many teenagers and young adults who
are in school are at risk of dropping out. According to some experts, this
indicates that a considerable portion of the country's emerging workforce
may face significant difficulty making the transition to productive
adulthood.1 At the same time, the Department of Labor projects that some
labor demands will go unmet because there will be too few workers in the
labor market with the necessary skills. Enacted in 1998, the Workforce
Investment Act (WIA) aims to address these issues by assisting the
nation's emerging workforce in realizing its full potential. As the
administering agency, Labor has budgeted about $1 billion annually on WIA
Title I-B youth employment and training programs to serve an estimated
721,000 of the nation's most at-risk young people.

WIA services are based on promising practices in the fields of youth
development and employment. Research suggests, for example, that youth can
achieve positive outcomes when they interact with caring adults, engage in
hands-on education and training activities, and receive support for
personal growth. The WIA program has sought to make these sorts of
experiences available to both in-school and out-of-school youth
participants. Under WIA, local areas can tailor their approach to the
types of youth served, the services provided, and how they are delivered.
To ensure that youth programs are tailored to local areas, WIA requires
the participation of a wide variety of people-youth policy experts,
representatives from youth-serving agencies, parents, and others with a
vested interest in the local youth programs. These participants serve on
local workforce investment boards created by WIA to establish workforce

1Center for Law and Social Policy, Leave No Youth Behind: Opportunities
for Congress to Reach Disconnected Youth, (Washington, D.C.: July 2003).

development policies and oversee implementation at the local level, or on
youth councils, subgroups of the local board that plan and coordinate the
local youth program.

We previously reported on the implementation challenges local areas faced
during the first few years of implementation.2 Now that the program has
been ongoing for several years and WIA is undergoing reauthorization, you
asked us to review (1) what approaches local areas have taken to serve
at-risk youth, (2) whether Labor's youth program guidance has addressed
ongoing implementation challenges, and (3) what is known about the
effectiveness of the WIA youth program.

To obtain information on what approaches local areas have taken to deliver
youth services, we administered a survey to the directors of all 604 local
workforce investment boards across the nation, including those in the 50
states, the District of Columbia, Puerto Rico, and U.S. territories. We
received responses from 496 local workforce investment boards (82 percent)
and relied on self-reported data. To further understand local area
approaches to service delivery, we visited nine local workforce investment
areas in five states: California, Louisiana, New Hampshire, Ohio, and
Virginia. We selected these states to obtain a mix based on their
differences in geographic location, number of workforce investment areas,
amount of youth funding, and presence of a state youth council. In
addition, our selection was informed by recommendations from Labor, youth
policy experts, and state workforce officials. We interviewed officials
representing state and local boards, youth councils, one-stop centers,
youth-service providers, business representatives, and state and local
education agencies. We reviewed Labor's program evaluation agenda and
published guidance letters from program year 2000 to the present. We also
assessed the reliability of the performance data submitted by states in
their annual reports and compiled in Labor's WIA database known as the
Workforce Investment Act Standardized Record Data (WIASRD) by performing
checks for internal consistency, reviewing Office of Inspector General
reports and other relevant documents, and speaking with the contractor for
WIASRD. We determined that the data were not sufficiently reliable to use
for the purposes of this report. We also interviewed officials from Labor,
including each of its six regional offices, as well as from the

2U.S. General Accounting Office, Workforce Investment Act: Youth
Provisions Promote New Service Strategies, but Additional Guidance Would
Enhance Program Development, GAO-02-413 (Washington, D.C.: April 5, 2002).

  Results in Brief

Department of Education and national associations. Our work was conducted
between January and December 2003 in accordance with generally accepted
government auditing standards.

Local WIA youth programs primarily focused on preventive strategies to
help in-school youth avoid academic failure and dropping out of school,
emphasizing summer enrichment services that were coordinated through
individual case managers and multiple service providers. Our survey of
local boards showed that about 70 percent of WIA youth were in school,
although this percentage varied widely, from 38 percent in South Dakota to
86 percent in Nebraska. State and local education officials said that the
WIA program allowed communities to provide complementary services needed
to support at-risk in-school youth. For example, local officials from one
urban area in Virginia said that while a local school provided 1 counselor
for as many as 300 students, the WIA program funded 1 counselor for every
50 WIA participants. In addition, local officials in four states we
visited said they focused on serving in-school youth because it was easier
and less costly than recruiting and retaining youth once they had dropped
out of school. Local boards reported that more than half of in-school
youth received summer employment services that were linked to classroom
learning. For example, a service provider in a rural area of California
enrolled in-school youth in a 6-week summer enrichment program where
students worked part-time while learning reading skills. In contrast, WIA
youth who were out of school were more likely to receive occupational
skills training and supportive services, such as assistance with child
care, transportation, and housing. While in-school and out-ofschool youth
usually participated in programs separately, both groups worked with case
managers who helped develop individual service strategies and coordinate
delivery of services. The majority of WIA youth were served primarily by
community organizations or by educational institutions such as high
schools, colleges, and universities.

Despite guidance issued by Labor, local areas continued to face challenges
in serving out-of-school youth, providing mentoring and follow-up
services, and establishing and using optional interim performance
measures. One reason serving out-of-school youth continued to be
challenging was that such youth can be difficult to locate within the
community. Some local workforce and regional Labor officials said that
identifying youth was problematic, in part because schools did not always
ensure that dropouts were linked with the WIA program. Another reason
cited was the difficulty in retaining youth who were primarily interested
in immediate employment rather than in participating in WIA's long-term

youth development activities. Local officials added that mentoring and
follow-up services were challenging to provide, in part because of the
difficulty in finding enough qualified mentors to work with at-risk youth
and sustaining a connection with youth once they had exited the program.
Some state workforce officials said that to more effectively serve
out-ofschool youth, they needed better-targeted guidance from Labor that
addressed their local areas' particular service delivery issues. Local
areas also faced challenges establishing interim measures, such as
enrollment and service participation, to assess program performance and
improve service delivery in a timely manner. Despite Labor's
encouragement, some regional Labor officials said that interim performance
measures were not widely used by local areas. Labor has established WIA
youth learning exchanges, a Web site, and a technical assistance program
to help address WIA youth program implementation challenges, but some
states and local areas may not have access to or be aware of these
resources.

Little is known about the effectiveness of the WIA youth program because
program impact evaluations have not been performed and performance outcome
data that have been collected may not be reliable. Impact evaluations
provide information on program effectiveness by differentiating between
outcomes that result from the program itself and those that result from
other factors. Because of possible legislative changes to the WIA youth
program, Labor does not plan to undertake an impact evaluation until 2006,
with initial results expected by 2009. While states annually report
performance data to Labor, these data are questionable because of
insufficient state monitoring of data quality and the inadequacies of some
state management information systems. Labor officials said that they will
require states to verify local area data beginning with program year 2002
data, but concerns remain about states' ability to fully implement
validation requirements and Labor's ability to monitor implementation
consistently.

We are recommending that Labor and Education coordinate efforts to clarify
how schools can work with workforce officials to connect dropouts to WIA
youth services. We are also recommending that Labor provide guidance to
address specific concerns identified by local implementers and establish
standards to monitor data quality.

Background 	Enacted in 1998, WIA replaced the fragmented and overlapping
programs under the Job Training Partnership Act (JTPA) with a system that
sought to connect employment, education, and training services to better
match job seekers to labor market needs. The youth program falls under
Title I

of WIA along with programs for adults and dislocated workers; Title II
deals with adult and family literacy; Title III pertains to employment
services under the Wagner-Peyser Act; and Title IV addresses vocational
rehabilitation. To better prepare low-income youth who face barriers to
employment and education, WIA requires youth programs to focus on
long-term, comprehensive services delivered year-round through a
coordinated network of service providers. In addition to meeting the
lowincome requirement, to be eligible for WIA services youth must be
between the ages of 14 and 21 and faced with one or more of six barriers
to employment.3 WIA serves both in-school and out-of-school youth. A
significant portion of out-of-school youth are high school dropouts and
unemployed youth who are struggling to succeed in the public education
system and lack financial, family, or social support. Thus, at least 30
percent of local WIA youth funds must be spent on out-of-school youth.4

                               WIA Youth Funding

Since WIA's full implementation in 2000, funding for the youth program has
ranged between approximately $1 billion and $1.4 billion a year. Labor
follows a formula to allocate WIA funds to states, which in turn
distribute money to their local workforce investment areas.5 At the state
level, the governor can reserve up to 15 percent of the annual WIA
allotment for such statewide activities as developing the WIA strategic
plan for the state. States allocate the remaining 85 percent of funds to
local boards that use the funds to develop the local strategic plan,
establish a youth council, and award competitive contracts to youth
service providers that have been recommended by the youth council. In
addition to recommending eligible providers, the youth council coordinates
youth activities in the local area, oversees providers, and carries out
other duties authorized by the local board such as forging linkages with
educational agencies. Youth councils

3A youth is considered to face employment barriers if he or she is (1)
deficient in basic literacy skills; (2) a school dropout; (3) homeless, a
runaway, or a foster child; (4) pregnant or a parent; (5) an offender; or
(6) an individual who requires additional assistance to complete an
educational program or to secure and hold employment. Up to 5 percent of
WIA youth participants are not required to meet the income eligibility
requirements, but all youth served by WIA must meet barrier requirements.

4Out-of-school youth do not include youth enrolled in alternative schools
at the time of WIA registration.

5For a discussion of WIA formula funding, see U.S. General Accounting
Office, Workforce Investment Act: Issues Related to Allocation Formulas
for Youth, Adults, and Dislocated Workers, GAO-03-636 (Washington, D.C.:
April 25, 2003).

may also leverage additional public and private funds to supplement their
WIA funding in order to provide comprehensive youth services.

                                 Youth Services

Once they are determined to be WIA eligible, youth receive an objective
assessment of their academic level, skills, and service needs. Local youth
programs then use the assessment to create each participant's individual
service strategy, which lays out employment goals, educational objectives,
and necessary services. Every local youth program must offer the following
10 services, known as program elements, to eligible youth, though
participants may receive different combinations of these elements
depending on their service strategy. Labor groups the 10 required program
elements around four major themes:

Improving educational achievement

1. 	tutoring, study skills training, and instruction leading to completion
of secondary school, including dropout prevention,

2. alternative school services,
Preparing for and succeeding in employment

3. summer employment linked to academic and occupational learning,

4. 	paid and unpaid work experiences, including internships and job
shadowing,

5. occupational skills training,
Developing the potential of young people as citizens and leaders

6. 	leadership development, which may include community service and
peer-centered activities encouraging responsibility,

Supporting youth

7. supportive services (such as child care and housing assistance),

8. 	adult mentoring for at least 12 months that may occur both during and
after program participation,

9. follow-up upon program completion for at least 12 months, and

10. comprehensive guidance and counseling.

Eligible youth may obtain these services directly from approved youth
service providers or through WIA's one-stop system, which serves as a
gateway to a variety of employment and training services.6 In addition to
helping WIA youth gain access to the 10 program elements, the one-stop
system also provides all youth with basic services, whether or not they
are eligible for WIA. Any young person may walk into a one-stop center and
make use of the center's career resources and obtain information on and
referrals to other providers. While one-stop centers are designed
primarily to serve those 18 and older, some states have established
one-stops that serve only youth, as we reported previously.7

                             Youth Program Guidance

Labor provides guidance to help states and local areas implement WIA.
Labor's guidance includes issuing annual Training and Employment Guidance
Letters on the youth program in general as well as on specific topics,
sponsoring WIA Learning Exchanges8 in every region, and maintaining a Web
site for promising and effective practices. Labor responds to requests for
clarification or additional information by phone or e-mail or at
conferences. Labor's guidance often includes promising and

6To create a more comprehensive workforce investment system, WIA requires
states and localities to bring together 17 federally funded employment and
training services into a single system, called the one-stop center system.
Locally, the one-stop system must include at least one physical site
offering a comprehensive array of WIA services as well as those of the
other partners, and may be supplemented by satellite sites that provide
one or more WIA services. For more information on the one-stop system, see
U.S. General Accounting Office, Workforce Investment Act: One-Stop Centers
Implemented Strategies to Strengthen Services and Partnerships, but More
Research and Information Sharing Is Needed,

GAO-03-725 (Washington, D.C.: June 18, 2003).

7Local boards in Alaska, Arizona, Arkansas, California, Colorado, the
Federated States of Micronesia, Florida, Idaho, Louisiana, Missouri, New
Jersey, New York, Puerto Rico, Tennessee, Texas, and Wisconsin reported
having one or more youth-only one-stop centers. See GAO-02-413.

8WIA Learning Exchanges for Youth Systems are two-day regional meetings
designed to highlight promising practices at exemplary WIA sites, foster
peer-to-peer information sharing, and develop team-driven action plans in
three areas: (1) recruiting and retaining out-of-school youth; (2)
building and sustaining partnerships, especially with the education
community; and (3) defining and aligning assessments, skill achievement,
and credentials. Along with the National Youth Employment Coalition and
the American Youth Policy Forum, Labor conducted seven learning exchanges
around the country, one of which was specifically for rural local areas.

effective practices to reinforce state and local flexibility under WIA to
tailor programs to best meet youths' needs.

Performance Measures 	The law requires that states and local areas collect
performance information on seven youth measures, which are separated for
younger and older youth (see table 1). All seven youth measures apply to
both statewide and local performance.

                    Table 1: WIA Youth Performance Measures

Younger youth measures (ages 14-18) Older youth measures (ages 19-21)

1. Skill attainment rate 1. Entered into employment rate

2. Diploma or equivalent attainment rate 2. The employment retention rate
at six

3. Placement and retention in months postsecondary education, advanced 3.
Increase in average earnings

training, or employment 4. The credential rate

Source: GAO analysis of the Workforce Investment Act.

WIA performance information is collected from service providers and flows
upward to the local boards, then to state boards or agencies, and finally
to Labor. Local performance data are aggregated and entered into a state's
own automated data system that tracks the activities of individual WIA
participants. States use two mechanisms to report performance to Labor:
WIASRD, a database of individual records containing activity and outcome
information for each registered participant that has exited WIA, and state
annual reports. Labor uses the state annual reports to track states'
progress in meeting negotiated performance goals. Labor then awards
monetary incentives to states that meet or exceed their performance goals
and sanctions states that fail to meet at least 80 percent of each goal in
two consecutive years.

  WIA Program Used Primarily for Dropout Prevention and Emphasized Summer
  Services and Skills Training Coordinated among Multiple Service Providers

Most local areas used WIA youth program services for a range of approaches
to prevent academic failure and school dropouts, emphasizing
learning-related summer services that were coordinated through case
managers and multiple service providers. Nationally, about 70 percent of
youth served were in school, according to local boards responding to our
survey. For in-school youth, local boards provided summer employment
services linked to classroom learning more often than for out-of-school
youth, while services for this latter population more often included
occupational skills training and supportive services. Local boards
reported that most youth received services primarily from community
organizations and educational institutions. These providers were most
likely to subcontract or make informal arrangements with other
organizations to deliver the full range of WIA services to youth
participants.

Local Areas Focused on Dropout Prevention for In-School Youth

Overall, local areas' approach to serving youth was to supplement schools'
dropout prevention efforts to keep youth connected to an educational
system, according to state and local workforce and education officials.
WIA allows states and local areas to determine the proportion of in-school
and out-of school youth to be served, requiring only that they spend at
least 30 percent of funds for out-of-school youth. Labor reported that
with some exceptions, states chose to focus the majority of resources on
inschool youth,9 and our survey showed that about 81 percent of local
areas served more in-school youth than out-of-school youth. Nationally,
about 70 percent of youth served were in school, according to local boards
we surveyed. However, this percentage varied across the nation, as shown
in figure 1. For example, 38 percent of youth served in South Dakota were
in school, compared with 86 percent in Nebraska.

9Excluding the territories, Labor's data show that only eight states had
spent 50 percent or more of their program year 2001 WIA youth allotment on
out-of-school youth as of September 30, 2003: Iowa, Idaho, Illinois,
Maryland, Maine, New Hampshire, North Carolina, and South Dakota.

Figure 1: Percentage of All WIA Youth Served Who Were in School, by State
(including the District of Columbia)

Sources: GAO survey of local board directors (data); copyright (c) Corel
Corp. All rights reserved (map).

aIncludes states where less than 65 percent of boards responded.

According to officials in four states that we visited, WIA youth programs
primarily targeted in-school youth because recruiting and retaining
out-ofschool youth for the WIA program was much more difficult and
expensive. For example, officials from a rural area in Ohio had difficulty
identifying and retaining out-of-school youth whose contact information
changes frequently, and Labor reports that the average cost of serving an
out-ofschool youth under the WIA program is about $4,000 a year, twice as
much as for an in-school youth. Officials from other local areas we
visited considered other factors. A local official in Louisiana said that
in-school youth are interested in the WIA program, unlike out-of-school
youth, who

are difficult to engage. In contrast, a rural area in Virginia chose to
focus primarily on out-of-school youth because so few services were
available for this population.

The Administration has proposed amending the WIA youth program to focus
more resources on out-of-school youth. Department of Education officials
said that WIA's services to in-school youth were not unique, since schools
already offer various services to their students to deter them from
dropping out. While Education officials said that the department's grants
geared exclusively toward dropout prevention would be insufficient to
address the national dropout problem, they also stated that funding is
available under many Education programs that could be used for dropout
prevention activities.10 However, local workforce and education officials
we spoke with in three states we visited said that they were either
unaware of or unable to gain access to other available federal resources
that could be used to provide intensive services to at-risk youth. In New
Hampshire, for example, officials told us that WIA provides the only
dropout prevention program, and that they were unaware of other available
education dropout prevention resources.

State and local workforce and education officials we spoke with in the
five states we visited said that WIA funding complemented rather than
duplicated education services and was critical in preventing in-school
youth from dropping out of school. For example, in a rural area in Ohio,
workforce officials stated that the WIA program was the only dropout
prevention program and that WIA provided students with their only chance
at academic and career success. In addition, they said that WIA's services
were more intensive and comprehensive, and were delivered in a one-on-one
setting where each student received individualized attention. In one urban
school in Virginia, there was 1 WIA counselor for 50 eligible in-school
youth, compared to 1 school counselor for as many as 300 students.

10For a discussion of Education and other federal programs to assist youth
at risk of dropping out of school, see U.S. General Accounting Office,
School Dropouts: Education Could Play a Stronger Role in Identifying and
Disseminating Promising Prevention Strategies, GAO-02-240 (Washington,
D.C.: February 1, 2002).

In-School Youth Received More Summer Employment; Out-of-School Youth
Received More Occupational Skills Training

Local areas tailored services provided to youth based on their needs,
emphasizing different services, depending on whether youth were in or out
of school. To meet youth's many developmental needs, the WIA youth program
requires that local areas offer the same menu of 10 academic, employment,
and support services to all eligible youth, which WIA providers choose
from when tailoring services to an individual's service strategy based on
an assessment of needs.11 As shown in figure 2, services such as work
experience and leadership development were provided fairly equally to both
youth populations, but there were differences in other areas. For example,
in-school youth were more likely to receive tutoring services and summer
employment linked to classroom learning, while outof-school youth were
more likely to receive occupational skills training and supportive
services to help prepare them for employment.

11One of the 10 required services, follow-up, is required for all youth
for at least 12 months after they exit the WIA program.

Figure 2: WIA Services Provided to In-School and Out-of-School Youth

Services received

Summer employment linked to learning

Occupational skills training

Alternative secondary school

Paid or unpaid work experience

Leadership development

Supportive services

Mentoring

Guidance and counseling

Tutoring

0 1020304050

Percentage of youth

In-school youth

Out-of-school youth

Source: GAO survey of local board directors.

Note: Follow-up services are not depicted in the figure because they are
mandatory for all participants when they exit the WIA youth program.

WIA requires that summer employment programs be linked to academic
learning, and Labor guidance promotes meaningful summer enrichment

experiences to help youth attain skills throughout the school year.12
Local youth programs in the five states we visited were providing youth
with various types of summer experiences. For example, a service provider
in a rural area of California enrolled in-school youth in a 6-week summer
enrichment program where students worked part-time while learning reading
skills. Another service provider in a rural area in Louisiana offered
in-school youth summer services that included academic enrichment,
community service, and exposure to different career options.

Local youth programs provided out-of-school youth training in occupational
skills and job readiness, as well as offered them supportive services.
Labor's guidance states that enrolling youth in occupational skills
training and retaining them until program completion leads to better
outcomes.13 Local areas provided several different types of occupational
skills training, depending on local labor market needs. For example, one
urban local area in Ohio trained out-of-school youth in nursing, welding,
and computer repair, while another urban local area in Virginia offered
technology certification training. Local areas also provided job readiness
skills cited by employers as lacking in many youth seeking employment. For
example, a program for out-of-school youth in an urban area of Ohio taught
participants such job readiness skills as customer service, conflict
management, and other interpersonal skills. Supportive services were also
important for out-of-school youth who needed additional assistance to help
them overcome their multiple barriers to employment. A service provider in
New Hampshire provided out-of-school participants with support services
such as child care, transportation, and housing assistance.

12Research has shown that engaging youth during the summer provides them
with an opportunity to learn new skills that they can apply in school and
also learn the value of work. See Westat, The 1993 Summer Youth Employment
and Training Program, (Rockville, MD, April 1994). In addition, Labor's
guidance highlights work-based learning during summer employment as a
strategy to help out-of-school youth acquire work experience while making
the connection between staying in school and pursuing a career. See U.S.
Department of Labor, Training and Employment Guidance Letter No. 3-99,
(Washington, D.C., 2000).

13See U.S. Department of Labor, Training and Employment Guidance Letter
No. 18-00, (Washington, D.C., 2001).

Case Managers Developed Service Strategies and Providers Coordinated to
Deliver Services

Almost all local boards we surveyed reported using case managers to
coordinate services for youth in and out of school. Because at-risk youth
often require services from a host of departments, including Education,
Health and Human Services, Housing and Urban Development, Justice, and
others, Labor's guidance promotes the use of a case manager to coordinate
services among all these youth-serving partners to provide seamless access
to and delivery of services.14 Nationwide, all but 3 percent of local
boards responding to our survey reported that their youth program uses
case managers. Local boards reported that, in addition to performing other
duties, case managers assessed youth upon enrollment in the WIA program.
These assessments typically included a review of educational attainment,
work readiness, work experience, and career interests. The assessments are
then used to develop an individual service strategy for each youth
participant. Labor's guidance requires that each strategy identify
employment goals and educational objectives, and prescribe appropriate
services for each participant.15 About three-fourths of local boards
reported that youth were greatly or very greatly involved in the
development of their individual service strategies.

Most local areas used multiple service providers to deliver the youth
services spelled out in each participant's individual service strategy.
WIA does not specify how services must be provided to youth, allowing
local areas to determine how many providers they will fund and hold
responsible for delivering the services outlined in each youth's
individual service strategy. Over half of local boards responding to our
survey reported using 4 or fewer service providers, but 15 percent used
more than 10 providers, as shown in figure 3.

14See Training and Employment Guidance Letter No. 18-00.

15See U.S. Department of Labor, Training and Employment Guidance Letter
No. 9-00 (Washington, D.C., 2001).

Figure 3: Number of WIA Youth Service Providers Used by Local Boards

Percentage of local boards

40

30

20

10

0 0 to 1 2 to 4 5 to 10 >10 Number of service providers

Source: GAO survey of local board directors.

Note: Some local board directors reported using zero providers because the
board itself delivered youth services. WIA permits local boards to provide
training services if there is an insufficient number of eligible providers
to meet local demand and if the board has demonstrated that it meets the
requirements for an eligible service provider.

Over half of local boards nationwide used providers that did not deliver
all services themselves, using formal or informal subcontracting
arrangements to provide the range of services needed. The extent that
providers coordinated with others to deliver services was related to the
number of youth served in the local area. In local areas with 250 or fewer
youth, providers delivered all services themselves more often than in
areas with more than 250 youth, as indicated in table 2.

Table 2: Percentage of Local Boards Using Different Types of Service
Delivery Arrangements, by Number of Youth Served

                                                       0-250 251-750 Over 750 
                                                       youth   youth    youth 
          Providers deliver all services without using               
                                        subcontractors    44      34 
          Providers are responsible for delivering all               
                       services but use subcontractors    23      31 

Providers do not deliver all services themselves,
but make noncontractual arrangements with
other organizations to make sure all services are
delivered 22 25

Other arrangement 10 10

Source: GAO survey of local board directors.

Note: Columns may not add to 100 because of rounding.

Regardless of the level of coordination used to provide youth services,
workforce officials told us that providers often informally collaborated
with one another through regular meetings to discuss problems, train one
another, share best practices, or share other resources.

Local boards responding to our survey reported using a range of public and
private entities to deliver youth services. Over half of all youth
received services primarily from educational institutions or community
organizations, while less than 1 percent of youth received services
primarily from faith-based organizations, as shown in figure 4.

Figure 4: Primary Providers of Services to WIA Youth Types of providers
Adult one-stop

Youth one-stop

Youth opportunity centersa

Secondary schools

Colleges or universities

Community organizations

Faith-based organizations

Programs sponsored by federal agencies besides Labor

Private employer

Local or state government

Other

0 5 1015202530 Percentage of youth

                                   In-school

                                 Out-of-school

Source: GAO survey of local board directors.

a Youth opportunity centers were established under the Youth Opportunity
Grant program to enable youth living in high-poverty areas to gain access
to a wide range of services.

Local boards reported that about one-fourth of youth received services
primarily from WIA one-stops-whether the one-stops focused on serving
adults or youth. However, among the one-stops we visited, these entities

were frequently used for supplemental services such as information and
referrals. Officials we spoke with in New Hampshire and Ohio said that
service providers or schools usually took youth on a field trip to the
local one-stop for basic orientation to expose them to employment services
they could use in the future. Although most youth were served by the WIA
program through other providers, one-stop usage varied considerably by
local area. For example, the majority of WIA youth living in an urban area
of California were served by youth-exclusive one-stops, while youth in a
rural area of the state rarely interacted with the local adult one-stop.

In-school and out-of-school youth usually did not participate in WIA
programs together. Nationwide, 43 percent of local boards reported that
these two types of youth received services from different providers. In
another 27 percent of cases, the two groups received services from the
same providers but participated separately. Another 24 percent of local
boards reported that all youth were served by the same service providers
and participated together.16

Local Areas Developed Partnerships with Business and Education to Deliver
Youth Services

Local areas developed partnerships with the local business and education
communities to identify employer needs and provide comprehensive youth
services related to academic and employment preparation. In many cases,
youth councils helped facilitate these partnerships.

Local areas partnered with businesses to identify and provide employment
and training services for WIA youth needed to fill high-demand, highgrowth
occupations.17 Over 85 percent of local boards reported using each of the
following methods to develop and maintain relationships with local
businesses: consulting with businesses about their job needs, training
participants in skills needed by local businesses, training program
participants to understand the values and attitudes local businesses look
for in employees, and providing employment experience to participants that
suits the jobs available in local businesses. Officials we spoke with at
the local youth programs in all five states we visited provided such work
readiness training as punctuality, teamwork, respect for others, and

16The remaining 6 percent of local boards reported their arrangement for
service delivery to in-school and out-of-school youth as "other."

17Over 70 percent of local boards reported that their programs encouraged
youth to seek employment in each of the following sectors: retail,
hospitality and food service, health care, and information technology.

appropriate dress. In addition to identifying needed skills, businesses
also provided employment opportunities. Nationally, 34 percent of local
boards reported that businesses subsidized work experience for WIA youth.
In New Hampshire, for example, a financial services firm employed
out-ofschool youth and taught them about personal financial management. In
addition, an appliance-store owner, once an at-risk youth himself, hired
WIA youth and hoped to groom one of them to take over the business.
Finally, officials in three states we visited noted that businesses
donated in-kind assistance such as building materials, work clothes, work
readiness training workshops, and a financial management curriculum. For
example, in an urban area in Virginia, businesses donated and remodeled
the space for the one-stop center.

Local areas also partnered with schools to provide academic preparation
services to WIA youth. In a local area of Louisiana, for example, a
service provider helped eighth graders explore classes leading to high
school industry-based certifications. In all five states we visited,
representatives from local postsecondary institutions made presentations
to WIA youth, informing them about higher education opportunities. In some
instances, WIA youth were allowed to take advanced level courses at their
institutions and even earn college credit. In addition, a community
college in California provided a pathway to facilitate the transition from
high school to higher education by hosting youth for a one-day college
experience of classes and conducting workshops on financial aid. Officials
in four states we visited also said that schools provided in-kind
assistance such as office space and tutoring.

Local boards reported that youth councils performed a number of important
functions that facilitated partnerships between local boards and the
community. Over 70 percent of local boards reported that the youth council
served as a forum to bring together key community partners who may never
have collaborated with each other. Almost two-thirds of local boards
reported that they would keep their youth council even if it became
optional.18 In addition, over half of local boards reported that their
youth councils elevated the importance and visibility of local youth
issues and programs and added value to the youth program.

18Of the local boards that reported that they would not keep their youth
council, 73 percent reported that they would use a youth committee of the
local board to perform the functions of a youth council.

  Labor's Guidance Has Not Fully Addressed Implementation Challenges Faced by
  Local Areas

Local areas continue to face challenges in implementing aspects of the WIA
youth program despite guidance issued by Labor. Some of these ongoing
challenges include recruiting and retaining out-of-school youth, providing
mentoring and follow-up services, and designing interim performance
measures that can be used to continually improve aspects of program
performance before youth exit the program. Labor supplements guidance on
these issues through information forums, a Web site, and a technical
assistance program. However, regional Labor officials said that state and
local areas' access to these resources had been limited.

              Out-of-School Youth Were Hard to Recruit and Retain

Labor has issued guidance on recruiting and retaining out-of-school youth
for the WIA program during the last two years, but local areas continue to
face challenges in serving this population. Labor issued a guidance letter
to states in April 2001 outlining strategies for recruiting youth to the
program, suggesting such methods as engaging youth to recruit their peers,
collaborating with community organizations that already work with
disadvantaged youth, and offering incentives for recruiting new
participants.19 In subsequent guidance issued in May 2002, Labor described
ways to keep out-of-school youth engaged in the program, such as by
helping youth make the link between career potential and education and
skills training as well as by providing support services like child
care.20 Despite this guidance, local officials from most of the states we
visited said that recruiting out-of-school youth continues to be a
problem. According to data from Labor, the District of Columbia, Delaware,
and New Mexico had not met WIA's requirement to spend at least 30 percent
of WIA funds on out-of-school youth for program year 2001 as of September

21

30, 2003.

Some local workforce officials indicated that closer coordination with
schools to immediately connect dropouts to the WIA program would help them
identify and recruit more out-of-school youth. Some local workforce and
regional Labor officials said that one reason schools may not share
information on dropouts directly with the WIA program was because of
concerns about student privacy restrictions. However, schools we visited

19See Training and Employment Guidance Letter No. 18-00.

20See U.S. Department of Labor, Training and Employment Guidance Letter
No. 28-01, (Washington, D.C., 2002).

21Under WIA, states have three years to spend their annual allotment,
therefore program year 2001 funds must be expended by June 30, 2004.

in four states shared dropout information by referring students who had
been expelled or had dropped out directly to the WIA service provider,
notified or provided contact information to the WIA provider when a
student had dropped out, or worked with the WIA program to develop a list
of dropouts. None of the schools, however, had procedures in place to
routinely connect all dropouts with local WIA programs. Labor and
Education officials agreed that schools could do more to work closely with
local workforce officials to help connect dropouts to local WIA youth
programs.

Local officials from most of the states we visited said that retaining
out-ofschool youth in the WIA program was also difficult. According to
some local officials, one reason was that out-of-school youth tend to
prefer immediate employment instead of training and academic learning.
Other officials said that both in-school and out-of-school youth face
difficulties with transportation-particularly in rural areas--that limit
their involvement in WIA programs. For example, local officials in a rural
area in Ohio said that transportation was their biggest issue. To mitigate
this challenge, the local area offered services through a mobile one-stop
unit that traveled to WIA clients throughout the county. Some state
workforce officials said that to be more effective in serving this
population, they needed guidance and technical assistance that was more
focused on the specific service delivery issues within their local areas.

Mentoring and Follow-up Were among the Most Difficult Services to Provide

Nearly all local areas we visited indicated that providing mentoring
services continued to be a problem for the WIA youth program, but Labor
has not addressed mentoring in its annual youth program guidance or shared
best practices on the provision of high-quality mentoring services. Local
areas identified several reasons why mentoring has been difficult. In one
instance, a Virginia official in a rural area said that it was difficult
to identify sufficient numbers of qualified mentors to work with eligible
youth. In another instance, a local area official in California noted that
geographical distances within the county resulted in long commutes and
discouraged adults from mentoring youth. A service provider in New
Hampshire said that finding mentors was especially difficult for
out-ofschool youth, as some adults are uncomfortable working with school
dropouts. Labor youth program officials said that mentoring is an
important way of connecting youth to caring adults and said they would
take action to provide guidance on this topic.

Another service element that remains challenging for local areas is
providing complete and thorough follow-up services to help youth succeed

after they have exited the program. WIA regulations require that follow-up
services last for at least 12 months, and Labor's guidance states that
follow-up may include regular contact with a youth's employer to track
progress made, assistance in addressing work-related problems, and help in
securing better-paying jobs and further education. Labor's policy guidance
for program year 2001 provided some principles from best practices in the
field of youth development to help local areas develop strategies for
follow-up. The principles included developing a systematic approach for
maintaining contact and interaction with the young person; evening and
weekend social activities for informal support; helping youth access
services to fulfill physical, emotional, and vocational needs; and
visiting the job site soon after the youth has started employment.
Nevertheless, several local officials cited continued difficulties in
sustaining a connection with youth and identifying outcomes for them once
they exited the WIA program.

Few Local Areas Used Interim Measures to Gauge Program Success

Labor's guidance underscored the importance of establishing and using
optional interim performance measures to monitor the success of delivering
WIA youth services, but some regional Labor officials said that such
measures were not widely used. Six of the seven required WIA youth
measures are collected only after youth exit the program. However, because
youth may remain in the program for many years, local areas with long-term
youth retention strategies may have limited means of gauging progress
without interim measures. Labor issued guidance in May 2002, encouraging
the use of interim measures, which may be tracked and reported separately
from the required annual performance measures, to provide ongoing feedback
on which aspects of the youth program were working well or needed
modification.22 The guidance identified the following interim measures
that local boards could use to monitor progress as youth move through the
WIA program:

o  	intake, including development of a comprehensive individual service
strategy with short-term and long-term goals;

o  	participation in program elements to help ensure youth meet short-term
skill attainment goals;

22See Training and Employment Guidance Letter No. 28-01.

o  	skill attainment to help ensure successful completion of short- and
longterm goals;

o  exit data, including number of participants exiting the program; and

o  follow-up services received to help measure youth performance outcomes.

Some local areas we visited were using interim measures to hold providers
accountable for delivering services. For example, New Hampshire workforce
officials said that they tracked enrollment and expenditure levels to
monitor activity levels across their contracted service providers. In a
local area in California, workforce officials monitored the progress of
youth by tracking enrollment, participation, work readiness skills, and
the rate at which youth made a successful transition to other activities
once they exited the WIA program.

Despite the potential usefulness of these data, interim measures were not
widely used, according to Labor officials in two regions. These officials
said that states lacked the resources to properly track them.
Additionally, the guidance did not explain how states and local areas
could collect, analyze, and use the data to assess progress and make
needed adjustments. For example, officials in one state we visited said
that they were unclear about how to apply the interim measures.

Access to and Awareness of Labor's Assistance Has Been Limited

Labor has disseminated information related to these and other issues
through forums that allow local areas to exchange information with one
another, but access to these forums has been limited. From December 2002
to April 2003, for example, Labor sponsored peer-to-peer WIA Learning
Exchanges with two national youth organizations that provided venues
around the country for local areas to share information and observe an
exemplary program firsthand. However, state budgetary cuts prevented many
local areas from attending some of these forums in person, according to an
official from the sponsoring organization. Although information and ideas
shared at the Learning Exchanges were later summarized and made available
to all local areas, those that were unable to attend missed the
opportunity to network with their colleagues and develop an action plan to
take back and apply to their program.23

23Labor officials told us that their regional offices have also sponsored
youth conferences and other forums where guidance and technical assistance
have been provided.

Labor has also supported the development of an online resource to
facilitate information sharing among local areas. The agency's Promising
Practices Web site was intended to provide a mechanism for local areas to
post and share promising workforce development practices, including those
pertaining to WIA youth programs. However, some state workforce and
regional Labor officials said that states and local areas may not even
have been aware of the site, that it has been difficult to enter practices
for inclusion, and that some users found it difficult to access the
documents described in the narrative.24 We also had difficulty accessing
and using the Web site to find information. For example, when we conducted
a search on the key word "mentoring," the five results did not
satisfactorily address the topic.

Labor guidance encourages local areas to tap into other resources that can
help them develop quality youth programs.25 The Promising and Effective
Practices Network, for example, provides a useful listing of promising
practices categorized by the specific strategy and the 10 required WIA
youth services that local areas can use for improving their WIA youth
programs.26

Labor has provided technical assistance to states through its Performance
Enhancement Project, initiated in program year 2002. Under this
initiative, Labor grouped into three tiers states that failed or were at
risk of failing to meet their performance goals, based on their reported
outcomes, according to Labor officials.27 These officials also said that
states in the first tier received priority for targeted technical
assistance to improve their youth programs, with the goal of improving
performance outcomes. Labor officials said that states could use this
technical assistance to address any of the challenges they faced in
implementing their WIA youth program. However, unless the state falls into
one of these three tiers, most local areas may not receive such assistance
to help them increase the

24We previously reported on problems with the Promising Practices Web
site. See GAO-03-725. This Web site can be found at
http://www.promising-practices.org.

25See Training and Employment Guidance Letter No. 18-00.

26The National Youth Employment Coalition's Promising and Effective
Practices Network Web site can be found at http://www.nyec.org/pepnet.

27According to Labor, the first tier includes those states that were
financially sanctioned. The second tier includes those states that failed
one or more performance measures but were not sanctioned. The third tier
includes those states that did not fail any measures but were deemed at
risk of failing them.

  Lack of Program Evaluations and Questionable Performance Data Prevent
  Assessments of WIA Youth Program

proportion of out-of-school youth served, improve mentoring and followup,
and use interim measures.

Little is known about the effectiveness of the WIA youth program because
Labor has not yet initiated an impact evaluation, and results from a
planned evaluation will likely not be available until 2009, according to
Labor officials. While Labor's performance data for program year 2001
indicate that five of seven youth measures were exceeded, these results
cannot be used to infer program outcomes because of insufficient
monitoring of state data quality and inadequacies of some state management
information systems. Labor officials said that they will require states to
validate local area data beginning with program year 2002 data, but some
implementation concerns remain.

Labor Plans to Initiate First Impact Evaluation in 2006, and Results Are
At Least Five Years Away

According to department officials, Labor intends to initiate an impact
evaluation28 of the WIA youth program in 2006, and preliminary results
will not be available until about 2009. While Labor officials said that
the youth program's 3 years of full implementation was sufficient time to
initiate a comprehensive evaluation, they were anticipating significant
changes to the WIA youth program as a result of reauthorization. They did
not plan to begin an impact evaluation of the program until after these
changes had been implemented.

If reauthorization is completed by spring 2004, Labor officials said they
anticipate that the process of awarding the contract for the study will be
completed by 2006 when the impact evaluation is scheduled to begin. The
evaluation will proceed with 5 or 6 years of data collection with an
additional 3 or 4 years of follow-up activity. Labor officials said they
expect to issue a series of interim reports before the final product. If
the project begins in 2006, Labor expects to issue the first report on
short-term impacts in 2009, with a final report on long-term impacts
available in 2011. According to officials, this evaluation is part of
Labor's proposed

28By isolating a program's effects, impact evaluations provide
policymakers with key information for determining program effectiveness.
To isolate a program's effect, impact evaluations divide participants into
two groups: one that receives program services and a similar group that
does not. Some impact evaluations assign participants randomly to one or
the other group; the group that does not receive services is called the
control group. The use of random assignment allows researchers to compare
outcomes for the two groups and attribute any differences to the program
services rather than other factors.

research plan to study all of its major employment and training programs,
including the WIA youth program, on a regular cycle.29

State Performance Outcome Data Were Questionable, Partly because of
Inconsistent Data Monitoring

Performance data submitted by states to Labor in quarterly and annual
reports were not sufficiently reliable to determine outcomes for the WIA
youth program. Labor's national aggregation of WIA performance data for
program year 2001 indicated that the program exceeded its goals for five
of the seven youth measures.30 According to Labor's Office of Inspector
General (OIG), however, there is little assurance that the states'
performance data for all WIA programs, including the youth program, are
either accurate or complete because of inadequate oversight of data
collection and management at the local, state, and federal levels. The OIG
also found that just 2 of 12 local areas and none of four states it
reviewed had formal policies on documentation requirements for participant
activities and outcomes.31 At the local level, for example, another OIG
report found that WIA youth program outcomes were adequately documented
only 37 percent of the time for a sample of 420 participants across 14
local areas.32 An official in one of Labor's regional offices added that
documentation requirements are inconsistent among states and local areas.

At the state level, the OIG reported that two of four states it reviewed
had not monitored local areas' performance data at the case file level,
and that none of the four states had adequate procedures in place to
ensure the accuracy of their performance data.33 According to regional
Labor officials, some states had insufficient procedures in place for
verifying the accuracy of the data collected by their local areas. For
example, officials in one region said that state monitoring rarely
addressed data accuracy or

29Labor anticipates that the WIA youth program will be evaluated on a
10-year cycle.

30States did not meet goals for the earnings change measure and credential
rate for older youth. States reported actual earnings change in 6 months
as $3,109, compared with their negotiated goal of $3,396. For the
credential rate, about 40 percent of older youth earned credentials,
compared with the goal of 44 percent.

31U.S. Department of Labor, Office of Inspector General, Workforce
Investment Act Performance Outcomes Reporting Oversight, 06-02-006-03-390
(Washington, D.C., Sept. 30, 2002).

32U.S. Department of Labor, Office of Inspector General, Workforce
Investment Act: Evaluation of Youth Program Enrollments, Services, and
Recorded Outcomes, 06-03-006-03-390 (Washington, D.C., September 30,
2003).

33U.S. Department of Labor, Office of Inspector General, 06-02-006-03-390.

included the verification of a sample of data items against original
records. In addition, some state information systems had significant flaws
that caused them to produce incorrect information, casting further doubt
on the quality of WIA performance data. For example, Ohio state officials
said they were in the process of replacing their old information system
that had caused such errors as data changing or disappearing entirely
after it had been entered in the system.

At the federal level, Labor did not have a standard data-monitoring guide
in place, and officials in some regional offices-who, according to agency
officials, are responsible for overseeing the quality of states' reported
data--said they followed their own oversight procedures. These procedures
did not usually include verifying the accuracy of a sample of the data
submitted by states. For example, an OIG report stated that while regional
offices conducted some data accuracy reviews, such as computer edit
checks, they did not verify the data's accuracy with such tests as
comparing the data with participant case files. Consistent with the OIG's
findings, officials in all six of Labor's regional offices said that they
examined state data submissions through desk reviews, which included
checking for errors such as incorrect calculation of performance measures,
extreme outliers, and miscoding of data. However, only the Atlanta
regional office checked a sample of data records against source
documentation. In its review of data records from six of the states it
oversees, the Atlanta office examined a sample of participant records
across all WIA programs from each state and found errors that affected the
calculation of one or more performance measures. In one state, for
example, 17 percent of participant records had at least one error,
compared with 83 percent of participant records in another state. The
regional office also found that two of six states it reviewed computed the
younger youth skill attainment measure incorrectly.

Labor recognizes these problems with data reliability, but in accordance
with WIA regulations, uses states' annual performance reports--the only
reports that depict states' progress in achieving or exceeding negotiated
performance levels--to make incentive grant and sanction decisions. States
that exceed the WIA performance goals negotiated between Labor and the
states are eligible to receive incentive grants that generally range from
$750,000 to $3 million.34 States that fail to meet 80 percent of their

34To be eligible for incentive grants, states must also meet performance
goals for the Adult Education and Literacy programs under Title II of WIA
and programs authorized under the Carl D. Perkins Vocational and Applied
Technology Education Act.

WIA performance goals for 2 years in a row are subject to sanction of up
to a 5 percent reduction in their annual WIA formula grant. However,
Labor's use of questionable performance data reported by the states may
hinder its ability to negotiate realistic performance goals and make
appropriate incentive grant and sanction decisions. Ohio state officials
questioned the appropriateness of being sanctioned 2 percent of its
program year 2002 WIA youth allotment. They believed that poor performance
levels were due to problems with its information system, not its workforce
development system.

Data Validation Initiative May Improve Reliability but Concerns Remain

To address the data issues described above, Labor is implementing a new
data validation policy requiring states to ensure the accuracy of their
annual reports and verify a sample of the data they submit. According to
Labor's policy, these requirements will be phased in over a 3-year period.
In the first year, states will be required to validate their annual
reports and data submissions for the program year 2002 period.35 According
to agency officials, Labor does not plan to publicly release these initial
reports but will use their findings to work with the states to correct
their data accuracy problems. In the second year, Labor will use
validation reports covering program year 2003 data to establish acceptable
error rates. In the third year, Labor will require states to meet the
acceptable error rates for their program year 2004 data submissions. Labor
will consider failure to meet the standard a violation equivalent to
failing to submit a report, for which states may be subject to corrective
action or financial sanction, as appropriate. In addition, Labor's data
validation guidance indicates that states that do not meet data accuracy
standards will receive technical assistance.

While the data validation initiative may improve the reliability of WIA
performance data, several implementation concerns remain. First, officials
in some of Labor's regional offices said that the states they oversee will
have difficulty implementing the data element verification requirement
because of limited staff resources. Further, these officials said that in
states where local providers keep the original documentation on-site,
retrieving the documentation to check it against records in their
information system will be time-consuming and expensive. Second, a Labor
official said that the agency does not plan to issue a program

35Program year 2002 started on July 1, 2002, and ended on June 30, 2003.

Conclusions

monitoring guide to standardize procedures across regional offices for
several more years.

Many youth are struggling to be successful in the public education system
and often face substantial obstacles to obtaining a high school diploma
and going on to college or getting jobs with career advancement
possibilities. Many of these youth may lack meaningful social and family
supports and may require comprehensive, intensive services to remain
engaged in society and avoid risky behaviors that can lead to chronic
unemployment, criminal activity, and other adverse outcomes. WIA currently
provides both the education and workforce systems with strategies and
resources to engage youth in academic and job training. The opinion held
by some federal education officials that WIA in-school services overlap
with existing education programs is not necessarily seconded by officials
in local areas. State and local workforce and education officials believe
WIA's educational, occupational, and support services provide critical
support and services to at-risk youth and that without WIA's comprehensive
services, schools may face an increasing burden to keep these youth in
school and ensure their academic success. The connection between WIA youth
services and schools could be made more effective if Labor worked with
Education to find ways to connect school dropouts with local WIA youth
programs.

Since WIA's passage, Labor has provided general guidance and promising
practices in addressing implementation issues, but increased availability
of technical assistance may be necessary to overcome some of the more
difficult challenges some states and local areas face in providing youth
services. This will be especially critical for states and local areas that
are reportedly meeting performance goals but still need assistance in
improving delivery of youth services and shifting program resources to
target different populations such as out-of-school youth. In addition,
local areas will need guidance, including specific strategies, to help
them provide effective mentoring and follow-up services and use interim
measures to track program performance.

Robust research and reliable performance data are needed to obtain a
complete picture of the WIA youth program's effectiveness and outcomes and
to make quality decisions about managing the program. However, none
currently exist. Labor has not yet initiated an impact evaluation and does
not expect to report on program impacts from a planned evaluation until
2009. In addition, states and local areas continue to struggle to collect
and document accurate and complete participant and performance data and
maintain data systems that can yield reasonably reliable outcome

information. For its part, Labor is taking action to improve data
integrity by requiring states to validate WIA performance data, beginning
with data from program year 2002. However, Labor's inconsistent monitoring
processes across regions will continue to challenge Labor's capability to
ensure that all states validate and report data consistently and
effectively. In the short term, the lack of accurate outcome data will
continue to hinder Labor's ability to negotiate realistic state
performance goals and use data to make sound decisions about financial
incentives or sanctions. In the long term, lack of accurate and complete
information will keep Labor from obtaining a true picture of how
effectively the youth program is working.

To promote information sharing that improves local WIA youth programs'
ability to identify and serve out-of-school youth, we recommend that the
Secretaries of Labor and Education coordinate efforts to clarify how
schools can work with workforce officials to connect school dropouts with
local WIA youth programs.

To assist state and local WIA youth programs address ongoing
implementation challenges, we recommend that the Secretary of Labor

                              Recommendations for

  Executive Action

o  	increase availability of guidance and technical assistance to local
areas that continue to face challenges in serving out-of-school youth;

o  	disseminate guidance, including specific strategies, to help local
areas provide effective mentoring services; and

o  	develop additional guidance on providing follow-up services and using
interim measures to track program performance.

To obtain an accurate and complete gauge of WIA outcomes and determine
whether local programs are operating successfully, we recommend that the
Secretary of Labor establish standard monitoring procedures that Labor's
regional offices could use to oversee state data validation efforts.

Agency Comments 	We provided a draft of this report to Labor and Education
officials for their review and comment. Education's comments are reprinted
in

and Our Evaluation 	appendix I. In its formal comments, Education
concurred with our recommendation to work with Labor to connect
out-of-school youth to local WIA youth programs. Labor responded
informally, and said that

many of the findings corroborated its own observations and that the
recommendations are consistent with Labor's current program direction.
Both agencies also provided technical comments, which we incorporated as
appropriate.

We will send copies of this report to the Secretaries of Labor and
Education, relevant congressional committees, and other interested
parties. Copies will be made available to others upon request. In
addition, the report will be available at no charge on GAO's Web site at
http://www.gao.gov.

Please contact me at (415) 904-2272 if you or your staff have any
questions about this report. Other major contributors to this report are
listed in appendix II.

Sincerely yours,

David D. Bellis Director, Education, Workforce, and Income Security Issues

Appendix I: Comments from the Department of Education

Appendix II: GAO Contacts and Staff Acknowledgments

GAO Contacts 	Lacinda Ayers, Assistant Director (206) 654-5591 Meeta
Sharma, Analyst-in-Charge (206) 287-4806

Staff In addition to the individuals mentioned above, the following staff
made key contributions to this report: Karyn Angulo, Susan Baker,
AndrewAcknowledgments Bauck, Paula Bonin, Maya Chakko, Keira Dembowski,
Joel Grossman, Corinna Nicolaou, Rebecca Woiwode, and Monica Wolford.

Related GAO Products

Transportation-Disadvantaged Populations: Some Coordination Efforts Among
Programs Providing Transportation Services, but Obstacles Persist.
GAO-03-697. Washington, D.C.: June 30, 2003.

Workforce Investment Act: One-Stop Centers Implemented Strategies to
Strengthen Services and Partnerships, but More Research and Information
Sharing is Needed. GAO-03-725. Washington D.C.: June 18, 2003.

Workforce Investment Act: Issues Related to Allocation Formulas for Youth,
Adults, and Dislocated Workers. GAO-03-636. Washington, D.C.: April 25,
2003.

Workforce Investment Act: States' Spending Is on Track, but Better
Guidance Would Improve Financial Reporting. GAO-03-239. Washington, D.C.:
November 22, 2002.

Workforce Investment Act: Youth Provisions Promote New Service Strategies,
but Additional Guidance Would Enhance Program Development. GAO-02-413.
Washington, D.C.: April 5, 2002.

Workforce Investment Act: Improvements Needed in Performance Measures to
Provide a More Accurate Picture of WIA's Effectiveness. GAO-02-275.
Washington, D.C.: February 1, 2002.

School Dropouts: Education Could Play a Stronger Role in Identifying and
Disseminating Promising Prevention Strategies. GAO-02-240. Washington,
D.C.: February 1, 2002.

  GAO's Mission

Obtaining Copies of GAO Reports and Testimony

The General Accounting Office, the audit, evaluation and investigative arm
of Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability of
the federal government for the American people. GAO examines the use of
public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no cost
is through the Internet. GAO's Web site (www.gao.gov) contains abstracts
and fulltext files of current reports and testimony and an expanding
archive of older products. The Web site features a search engine to help
you locate documents using key words and phrases. You can print these
documents in their entirety, including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document files.
To have GAO e-mail this list to you every afternoon, go to www.gao.gov and
select "Subscribe to e-mail alerts" under the "Order GAO Products"
heading.

Order by Mail or Phone 	The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:

U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C.
20548

To order by Phone:	Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061

Contact:

To Report Fraud, Web site: www.gao.gov/fraudnet/fraudnet.htm

  Waste, and Abuse in E-mail: [email protected]

Federal Programs Automated answering system: (800) 424-5454 or (202)
512-7470

Jeff Nelligan, Managing Director, [email protected] (202) 512-4800

Public Affairs 	U.S. General Accounting Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
*** End of document. ***