Job Corps: Participant Selection and Performance Measurement Need to Be
Improved (Testimony, 10/23/97, GAO/T-HEHS-98-37).
Pursuant to a congressional request, GAO discussed Job Corps, a
$1-billion program administered by the Department of Labor, focusing on
how Labor can improve the selection of Job Corps participants and how it
can improve the information it has available on program placements in
order to adequately manage the program and assess placement contractor
performance.
GAO noted that: (1) Job Corps needs to improve the selection of program
applicants in order to decrease the early dropout rate for program
participants; (2) Job Corps needs to identify participants, from among
its eligible population of about 6 million, who have the commitment,
attitude, and motivation to complete the training and benefit from Job
Corps' comprehensive and intensive services; (3) the procedures that the
more successful outreach and admissions contractors use include
commitment checks as well as preenrollment tours and briefings, which
give applicants a more realistic basis for deciding whether to enroll in
the program; (4) Job Corps' policy guidance for 2 of the 11 eligibility
criteria is ambiguous and incomplete, leading to an eligibility
determination process that fails to follow the requirements of the
legislation and program regulations; (5) although Labor uses performance
measures to make decisions about renewing placement contractors, GAO
found that two of the four measures Labor uses do not provide
information meaningful for assessing the performance of placement
contractors; (6) related measures regarding overall program performance
are flawed, thus inaccurately assessing program performance; (7)
although Job Corps reports that about 65 percent of its participants are
placed in jobs and about 46 percent of these placements are related to
the training participants receive, GAO's work has raised questions about
the accuracy and relevancy of both of these figures; and (8) these
flawed measures jeopardize the ability of Job Corps, the Employment and
Training Administration, which administers Job Corps, and Labor in
general to determine whether the goals stated in Labor's strategic plan
are being achieved.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: T-HEHS-98-37
TITLE: Job Corps: Participant Selection and Performance
Measurement Need to Be Improved
DATE: 10/23/97
SUBJECT: Eligibility criteria
Eligibility determinations
Contractor performance
Youth employment programs
Attrition rates
Vocational education
Disadvantaged persons
Career planning
IDENTIFIER: DOL Job Corps Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Before the Subcommittee on Human Resources, Committee on Government
Reform and Oversight, House of Representatives
For Release on Delivery
Expected at 10 a.m.
Thursday, October 23, 1997
JOB CORPS - PARTICIPANT SELECTION
AND PERFORMANCE MEASUREMENT NEED
TO BE IMPROVED
Statement of Cornelia M. Blanchette, Associate Director
Education and Employment Issues
Health Education, and Human Services Division
GAO/T-HEHS-98-37
GAO/HEHS-98-37T
(205357)
Abbreviations
=============================================================== ABBREV
AWOL - absent without leave
ETA - Employment and Training Administration
JTPA - Job Training Partnership Act
JOB CORPS: PARTICIPANT SELECTION
AND PERFORMANCE MEASUREMENT NEED
TO BE IMPROVED
============================================================ Chapter 0
Mr. Chairman and Members of the Subcommittee:
We are pleased to be here today to discuss Job Corps, a $1 billion
program administered by the Department of Labor that serves youths
aged 16 to 24 who are economically disadvantaged, in need of
additional education or training, and living under disorienting
conditions, such as in a disruptive homelife. The 109 Job Corps
centers are operated under contracts with Labor, which also issues
contracts for outreach and admissions and placement services. Job
Corps is the nation's most expensive job training program with an
average cost of more than $15,000 per participant. Reasons for this
high cost include that the program serves a severely disadvantaged
clientele with multiple barriers to employment and that it provides a
comprehensive array of services in a residential setting. In spite
of this considerable investment, about 25 percent of the participants
in the program drop out within 60 days, and 57 percent of those who
leave the program do so without completing their vocational training.
In contrast, those who remain in the program and complete their
vocational training obtain higher-skill jobs at better wages.
My testimony today focuses on how Labor can improve the selection of
Job Corps participants and how it can improve the information it has
available on program placements in order to adequately manage the
program and assess placement contractor performance. Having the
right information is particularly important if Labor is to improve
program performance as envisioned by the Government Performance and
Results Act of 1993 (the Results Act). Much of my testimony is drawn
from a study being released today that we conducted at the request of
the Chairman as well as from several other Job Corps reports we have
issued over the past 2 years.\1
For the report being released today, we met with Labor officials and
reviewed Labor's eligibility policy guidance in relation to
applicable legislation and regulations. We analyzed national data on
the characteristics of program participants and early dropouts
enrolled during program year 1995.\2 We also analyzed program
retention data and placement results for each outreach and admissions
and placement contractor during program years 1994 and 1995 to
identify contractors that had higher and lower retention or placement
performance. From among these, we selected 14 contractors to
visit--2 that do only outreach and admissions, 1 that provides only
placement services, and 11 that perform outreach and admissions
functions as well as placement functions. We selected them in order
to obtain detailed information on the processes the contractors use
to admit applicants into Job Corps and place them upon their leaving
the program.
In summary, Job Corps needs to improve the selection of program
applicants in order to decrease the early dropout rate for program
participants. It needs to identify participants, from among its
eligible population of about 6 million, who have the commitment,
attitude, and motivation to complete the training and benefit from
Job Corps' comprehensive and intensive services. We found that the
procedures that the more successful outreach and admissions
contractors use include commitment checks as well as preenrollment
tours and briefings, which give applicants a more realistic basis for
deciding whether to enroll in the program. We also found that Job
Corps' policy guidance for 2 of the 11 eligibility criteria is
ambiguous and incomplete, leading to an eligibility determination
process that fails to follow the requirements of the legislation and
program regulations.
Further, although Labor uses performance measures to make decisions
about renewing placement contractors, we found that two of the four
measures Labor uses do not provide information meaningful for
assessing the performance of placement contractors. In addition,
related measures regarding overall program performance are flawed,
thus inaccurately assessing program performance. Although Job Corps
reports that about 65 percent of its participants are placed in jobs
and about 46 percent of these placements are related to the training
participants receive, our work has raised questions about the
accuracy and relevancy of both of these figures. These flawed
measures jeopardize the ability of Job Corps, the Employment and
Training Administration (ETA) (which administers Job Corps), and
Labor in general to determine whether the goals stated in Labor's
strategic plan are being achieved.
--------------------
\1 Job Corps: Need for Better Enrollment Guidance and Improved
Placement Measures (GAO/HEHS-98-1, Oct. 21, 1997).
\2 A program year begins on July 1 of a year and ends on June 30 of
the following year. A program year is designated by the year in
which it begins. Thus, program year 1995 began on July 1, 1995, and
ended on June 30, 1996.
BACKGROUND
---------------------------------------------------------- Chapter 0:1
Job Corps currently operates 109 centers throughout the United
States, including centers in Alaska and Hawaii, the District of
Columbia, and Puerto Rico. Most states have at least one center, and
several states have four or more centers. Private corporations and
nonprofit organizations, selected through competitive procurement,
operate the majority of the centers. However, the departments of
Agriculture and the Interior operate 28 centers, called civilian
conservation centers, under interagency agreements.
To address the needs of students with multiple employment barriers,
Job Corps provides a comprehensive range of services, primarily in a
residential setting.\3 It provides services 24 hours a day, 7 days a
week, including basic education, vocational skill training, social
skill instruction, counseling, health care, room and board, and
recreation. Job Corps also offers training in several vocational
areas, such as business occupations, automotive repair, construction
trades, and health occupations. Because of the comprehensive
services Job Corps provides, it is a relatively expensive program.
According to Labor's program year 1995 figures, the average cost per
Job Corps student was more than $15,000.\4 In contrast, the cost per
participant in the Job Training Partnership Act title II-C year-round
program for youths is $1,673.\5 Cost varies according to how long Job
Corps participants remain in the program--we estimated that, at the
six centers included in a previous study, those who completed
vocational training stayed an average of 13 months with an associated
cost of about $26,200.\6
Considerable Job Corps' resources are spent on participants who drop
out early and others who fail to complete their training. Although
the length of time students stay in Job Corps can vary
substantially--from 1 day up to 2 years--a large number of Job Corps
participants leave within a short time after enrollment.\7 In program
year 1995, about 15 percent of the enrollees left within 30 days of
entering the program and more than one-fourth left within 60 days.
Still others who stayed longer failed to complete their training. In
total, 57 percent of program year 1995 terminees did not complete
their vocational training. We estimated in our 1995 report that
about 40 percent of program funds at the six centers we visited was
spent on participants who did not complete vocational training.
Outreach and admissions services and placement services are provided
by private contractors, the centers, or state employment service
agencies under contract. During program year 1995, Job Corps spent
about $60 million on outreach and admissions and placement
contracts.\8
Job Corps is a performance-driven program and outreach and admissions
contractors and placement contractors must meet certain levels of
achievement in order to continue to participate and to receive
program funding. Two performance standards have been established for
outreach and admissions contractors for the enrollment of certain
numbers of male and female youths and for the proportion of enrollees
who remain in the program for more than 30 days. A third standard
relates to the percentage of participants who are eventually placed
in jobs, the military, schools, or other training programs following
program termination. Similarly, placement contractors are required
to meet standards related to the percentage of participants placed.
Additional placement contractor standards are applied to participants
who are placed in jobs, such as the percentage of participants
obtaining full-time jobs, jobs directly related to the vocational
training they receive, and the average wage they receive at
placement.
Individuals enroll in Job Corps by submitting applications through
outreach and admissions contractors. Enrollment in the program is
open entry and its training courses are self-paced, allowing students
to enroll throughout the year and to progress at their own pace.
Students leave Job Corps for a variety of reasons, including
successful completion of the program objectives, voluntary
resignation, disciplinary termination, and being absent without leave
(AWOL) for 10 consecutive training days. With a few exceptions,
participants terminating from Job Corps are assigned to a placement
contractor for assistance in finding a job or enrolling in other
education or training programs. Placement contractors are to give
priority to finding full-time, training-related jobs for
participants.
Participation in Job Corps can lead to placement in a job or
enrollment in further training or education. It can also lead to
educational achievements such as attaining a high school diploma and
gains in reading or mathematics skills. According to Labor data, 75
percent of the more than 60,000 program terminees in program year
1995 were placed--65 percent in jobs and 10 percent in education or
other training--and 46 percent of the placements were in
training-related jobs. The average wage for all placements was
$5.98; for training-related placements, $6.44.
Labor has long recognized that participants who complete their
vocational training courses tend to do better after program
termination--that is, they have significantly higher placement rates.
Information we developed during our 1995 study of Job Corps verified
this conclusion. We found that students who completed vocational
training at the six centers we visited were 50-percent more likely to
obtain a job than students who did not complete it (76 percent versus
49 percent, respectively). Furthermore, those who completed such
training were five times more likely to obtain a training-related job
at wages 25-percent higher than students who did not complete their
training.\9 In contrast, about two-thirds of the jobs obtained by
students who did not complete their training were in low-skill
positions such as fast-food worker, cashier, laborer, assembler, and
janitor.
--------------------
\3 In an earlier study, we compared the characteristics of Job Corps
terminees with comparable youths in the Job Training Partnership Act
(JTPA) youth training program and found that a greater proportion of
Job Corps youths (about 68 percent) than JTPA youths (39 percent) had
multiple barriers to employment. See Job Corps: High Costs and
Mixed Results Raise Questions About Program's Effectiveness
(GAO/HEHS-95-180, June 30, 1995).
\4 On average, students spend about 7 months in the program.
\5 The average length of stay for JTPA title II-C is about 10 months.
However, JTPA does not provide the comprehensive array of services
provided by Job Corps nor is it a residential program.
\6 Job Corps: High Costs and Mixed Results Raise Questions About
Program's Effectiveness (GAO/HEHS-95-180, June 30, 1995).
\7 Job Corps participants may be enrolled in the program for up to 2
years and may enroll for an additional year to attend advanced career
training.
\8 About $8 million of this amount was for media support contracts.
According to Labor, this high level of media expenditure should be
regarded as a one-time but necessary cost to counteract a decline in
Job Corps enrollments in program year 1994 and early program year
1995.
\9 Because of the wide latitude Labor guidance permits for
determining whether jobs participants obtained were training-related,
we analyzed each of the more than 400 placements at the six centers
and, using information contained in the Dictionary of Occupational
Titles, compared the job with the vocational training provided to
determine whether the job was related to the training received. This
resulted in a stricter interpretation of a job-training match for
both those who completed and those who did not complete vocational
training.
BETTER ELIGIBILITY GUIDANCE AND
SCREENING ARE NEEDED TO SELECT
PARTICIPANTS
---------------------------------------------------------- Chapter 0:2
The eligibility guidance that Labor provides to its outreach and
admissions contractors and the screening procedures these contractors
follow need to be improved. Labor has not provided adequate guidance
on 2 of the 11 eligibility criteria--living in an environment
characterized by disorienting conditions and having the capability
and aspiration to complete and secure the full benefits of Job Corps.
Without complete and unambiguous guidance, outreach and admissions
contractors may not be enrolling the applicants who are the most
appropriate for the program and, thus, may potentially be
contributing to the dropout rate. We found that contractors with
lower dropout rates follow procedures aimed at identifying applicants
with the commitment and motivation to remain in and benefit from the
program. However, others not following such procedures have higher
dropout rates.
GUIDANCE FOR TWO ELIGIBILITY
CRITERIA IS INADEQUATE
-------------------------------------------------------- Chapter 0:2.1
Job Corps' policy guidance for 2 of the 11 eligibility criteria is
ambiguous and incomplete, which has led to an eligibility
determination process that fails to follow the requirements of the
legislation and regulations. One of these requirements is that, to
be eligible for Job Corps, program participants must be from an
environment so characterized by cultural deprivation, disruptive
homelife, or other disorienting conditions as to impair their ability
to successfully participate in other education and training programs.
However, regarding this environmental requirement, Job Corps' Policy
and Requirements Handbook (1) does not define key terms used to
describe "other disorienting conditions," such as "limited job
opportunities," and (2) limits eligibility to a set of factors that
does not include "cultural deprivation," an environmental factor
specified in the law. Further, Labor has not provided adequate
guidance regarding another eligibility requirement--that participants
have the capability and aspiration to complete and secure the full
benefits of Job Corps. Without complete and unambiguous guidance,
outreach and admissions contractors may not be enrolling the most
appropriate applicants for the program under the law.
CONTRACTORS WITH LOWER
DROPOUT RATES HAVE BETTER
SCREENING PROCEDURES
-------------------------------------------------------- Chapter 0:2.2
In our most recent study, we found that placement contractors with
lower dropout rates differ discernibly in outreach and assessment
approaches and practices when compared with contractors having higher
dropout rates.\10 We noted that admissions contractors with lower
dropout rates--10 percent or less--tend to have better procedures for
identifying applicants with the commitment and motivation to remain
in and benefit from the program. These contractors emphasize making
sure that applicants have the capability and aspiration to complete
and secure the full benefit of the program, which is one of the
program's statutory eligibility criteria. These more-successful
contractors' procedures include "commitment checks" and preenrollment
tours and briefings that give applicants a more realistic basis for
deciding whether they want to enroll. This emphasis by these
contractors is consistent with the finding we reported in a May 1996
report on successful training programs--namely, that a key
job-training strategy shared by successful programs is a focus on
ensuring that participants are committed to their training and to
getting a job.\11
The "commitment checks" that contractors use to test Job Corps
applicants' initiative take a variety of forms. For example, several
admissions counselors require individuals interested in Job Corps to
set up application appointments. Four admissions counselors also
mentioned that they require applicants to arrive for their meetings
dressed in proper attire; otherwise, they have to schedule another
appointment. In addition, three admissions counselors require
applicants to submit written statements explaining why they want to
participate in the program and what they hope to accomplish. Several
admissions counselors require applicants to call weekly between the
date of application and the enrollment date to determine the status
of their application and to demonstrate their continued interest.
Finally, one contractor uses a nine-point checklist of documents that
all interested persons have to acquire before they set up their
application appointment.
Some outreach and admissions contractors consider preenrollment tours
of Job Corps centers and briefings to be extremely useful, although
they are not practical in every situation. They provide applicants
with a firsthand opportunity to obtain a thorough understanding of
Job Corps' rules and requirements, observe the living conditions,
erase false expectations, and determine whether they are suited for
the regimented life of Job Corps. Some preenrollment briefings occur
before application; others take place afterward. For example, one
contractor requires all interested individuals to participate in a
prearranged tour, briefing, and question-and-answer session, after
which those still interested must set up an appointment to complete
an application. Another contractor requires potential enrollees to
take a tour after the application process. After the tour,
applicants attend a briefing and a question-and-answer session,
followed by one-on-one interviews with center staff. The value of
preenrollment tours and briefings was also confirmed at two of the
centers we visited by Job Corps participants who thought the tours
and briefings were definitely worthwhile and by two regional
directors who said that preenrollment tours and briefings are very
effective in preparing applicants for Job Corps and in improving the
prospect of retention.
Several regional directors commented on the importance of identifying
applicants who are ready for Job Corps and can benefit from its
training. For example, one regional director stated that because the
program cannot afford to squander its resources on applicants who do
not really want to be in the program, admissions counselors should
ensure that applicants are ready and can benefit from the investment.
Another regional director noted that because so many people are
eligible (more than 6 million), it was important to provide the
opportunity of Job Corps to those most likely to benefit and that
commitment should be "first and foremost" when assessing applicants.
To identify other factors that might be related to program retention,
we analyzed the relationship between participant characteristics and
the likelihood of remaining in the program for at least 60 days. We
found that the participants who are more likely to leave the program
within the first 60 days included those who are younger (15 to 17
years old), have less than 12 years of education, have a dependent
child, or reside more than 50 miles from the assigned center. The
most clear-cut use of this information on participant characteristics
may be for designing efforts to improve the retention of participants
whose characteristics are associated with leaving the program early.
--------------------
\10 Job Corps: Need for Better Enrollment Guidance and Improved
Placement Measures (GAO/HEHS-98-1, Oct. 21, 1997).
\11 Employment Training: Successful Projects Share Common Strategy
(GAO/HEHS-96-108, May 7, 1996).
IMPROVED MEASURES ARE NEEDED TO
EVALUATE PLACEMENT CONTRACTOR
PERFORMANCE
---------------------------------------------------------- Chapter 0:3
While Job Corps is a performance-driven program, and Labor uses
performance measures in evaluating program performance and in making
contract renewal decisions, we found that Labor does not have the
information it needs to accurately assess either the program or
placement contractor performance. Two of the four measures Labor
uses in assessing placement contractor performance are not
meaningful. One of the measures --placement in jobs--holds
contractors accountable for placing participants who are
realistically unemployable and, therefore, could lead to an
understatement of actual placement performance. At the same time,
this measure could overstate placement performance because, as our
previous work has shown, many reported placements cannot be
confirmed. A second measure--placement in training-related
occupations--probably overstates performance for two reasons. First,
it includes participants who received little vocational training.
Second, it gives placement contractors too much latitude in deciding
whether placements are training-related. Problems in these measures
also result in flawed assessments of overall program performance.
Thus, we are uncertain about how well the program is performing.
JOB PLACEMENT MEASURES ARE
FLAWED AND MANY REPORTED
PLACEMENTS ARE UNVERIFIABLE
-------------------------------------------------------- Chapter 0:3.1
The job placement measure Labor uses for assessing contractor
performance has flaws that could lead to both understatement and
overstatement of actual performance. Labor's current methodology for
calculating a placement contractor's performance may have resulted in
an understatement of the placement rate at the contractors we visited
by an average of 8 percentage points. Labor's calculation includes
participants who remained in the program for as little as 1 day,
those who were AWOL, and those who were expelled after 30 days for
using drugs or committing violent acts--all individuals a placement
contractor would have difficulty recommending for employment. During
program year 1995, about one-third of the participants leaving Job
Corps were in these categories. We recognize that determining what
happens to every program participant is an important indicator of how
well Job Corps is performing but not necessarily an appropriate
measure of placement contractor performance. If Labor's methodology
were modified to include only participants who were in the program
long enough to obtain at least minimal benefits (that is, stayed for
at least 30 days) and were employable (that is, were not terminated
for drug and violence violations and were not AWOL), the average
placement rate for the 12 placement contractors we visited would have
been from 2.6 percentage points to 13.6 percentage points higher.
Job placement data may also be overstated. Although Labor reports
that 65 percent of Job Corps participants leaving the program are
placed in jobs, our work has raised questions about the validity of
this figure. In our June 1995 report, we questioned the validity of
about 15 percent of the reported placements at six locations that we
visited. We attempted to contact the employers of more than 400
randomly selected placements and found that, in more than 7 percent
of the cases, employers reported either that they had never hired the
participant or that the individual had never shown up for work, and
we were unable to locate the employer of record for about 8 percent
of the placements.
Although Job Corps has procedures for verifying contractors'
placements, Labor may need to take steps to ensure that it fully
implements such procedures and that it provides adequate oversight.
In a recent report, Labor's Inspector General pointed out that Job
Corps had not adequately managed or controlled the resolution of
questionable placements identified in a sample of Job Corps terminees
by the Job Corps' placement verification contractor.\12 Discrepancies
in data reported by placement contractors are referred to appropriate
Job Corps regional offices for review. The Inspector General's
report found a backlog of questionable placements stemming from Job
Corps' not adequately monitoring the resolution of such placements.
In addition, the report said that the verification process had
limited benefit because Job Corps did not take timely corrective
actions.
In a memorandum to Job Corps regional directors, the Director of the
Office of Job Corps pointed out that program guidance on what
constitutes a valid placement is clear. In addition, placement
contractors are required to verify and document 100 percent of their
placements. She stated further that "if this is being done, there is
really no excuse for a reported placement to be found invalid at a
later date." The Director pointed out that the federal responsibility
in this area is oversight. We agree and strongly encourage that
oversight be adequate to ensure that placement data are accurately
reported.
--------------------
\12 Office of Inspector General, Job Corps Needs to Improve Its
Followup on Questionable Placements (Washington, D.C.: U.S.
Department of Labor, Sept. 22, 1997).
TRAINING-RELATED PLACEMENT
MEASURE IS FLAWED
-------------------------------------------------------- Chapter 0:3.2
The value of the current job-training match data is questionable.
The job-training match measure is used to evaluate the effectiveness
of vocational training programs and placement contractors by
determining the percentage of jobs participants obtain that match the
training they receive while in Job Corps. Labor allows placement
contractors wide discretion in deciding whether a job placement they
obtain for a participant is related to the training he or she
receives. At the same time, Labor requires that participants who
receive little vocational training be included in the calculation of
this measure. Labor is developing a new system to determine
job-training matches that, it believes, will be more accurate.
Placement contractors are responsible for recording whether or not
participants are placed in jobs requiring skills similar to those
included in their training. Labor's guidance for such decisions
consists of 16 broad categories of training programs, and within each
category are a number of detailed occupations in which Job Corps
participants could have received training. In addition, each of the
16 broad categories contains a list of jobs that would be considered
a match with the training a participant receives. To illustrate, the
broad training category of construction trades includes 47 detailed
training occupations and 357 placement occupations. An individual
who was trained in any one of the 47 training occupations and was
then placed into any one of the 357 placement occupations would be
counted as a job-training match. For example, an individual trained
as a carpenter (1 of the 47 training occupations) who was placed as a
plumber, janitor, or cable television installer (3 of the 357
placement occupations) would be considered to have obtained a
training-related placement.
Among the wide range of jobs that are considered to be training
matches under each of the broad training categories, Labor's guidance
includes jobs that appear to bear little, if any, relationship to Job
Corps training. For example, a position as a key cutter would be
considered a training match for any of the 51 training categories
under the broad category of mechanics and repairers, which includes
automobile mechanic, electronics assembler, and parts clerk. A
position as a general laborer would be considered a job-training
match for any of the 30 training occupations under the precision
production category, which includes mechanical drafter, sheet metal
worker, and welder. Table 1 lists examples of some possible matches
under Labor's guidance.
Table 1
Some Occupations Considered Job-
Training Matches for Selected Vocational
Training Programs
Instructional category Occupation
---------------------------------- ----------------------------------
Automobile mechanic Band attacher (attaches wrist
bands to watches)
Feeder (stacks paper in offset
presses)
Key cutter
Washer (clock parts)
Cook Bar attendant
Carhop
Housecleaner (hotels)
Fast-food worker
Cosmetologist Hot-room attendant (gives patrons
towels)
Sales person for weed eradication
services
Shaver (brushes suede garments
after they have been cleaned)
Shaver (shaves hog carcasses)
Heavy-equipment operator Baggage checker
Freight elevator operator
Porter
Ticket seller
Medical secretary Coin counter and wrapper
General cashier
Hand packager
Linen-room attendant
Welder Antisqueak filler (shoes)
Casket liner
General laborer
Hacker (lifts bricks of clay tiles
from conveyor belt and stacks
them)
----------------------------------------------------------------------
Many of the positions that are considered to be related to Job Corps
training require relatively little training to perform. The job
placement occupational categories contained in Labor's guidance for
job-training match come from its Dictionary of Occupational Titles.
The dictionary includes, for each occupation, the average time
required to learn the techniques, acquire information, and develop a
facility for average performance in a specific job situation. For
more than 700 of the jobs in Labor's guidance, the average training
time is indicated as requiring either only a short demonstration or
training up to and including 1 month. Thus, Labor is allowing
job-training match credit for occupations such as fast-food worker,
cashier, and laborer that require relatively short training time,
even though participants spend, on average, about 7 months in the
program. While we recognize that some of these positions provide
entry into an occupational area that may lead to a better job, it is
questionable in our view to consider such positions to be a
job-training match until a participant advances into a job
commensurate with the training he or she has received.
Further, Labor guidance encourages placement contractors to search
among the allowable jobs for a job-training match. Its policy
handbook states that, if a job-training match is not generated when a
job-placement code is entered in its automated system, the placement
contractor is allowed to enter a different code that may generate a
job-training match, "so long as integrity of data is maintained." We
found that the placement contractors' practice of recording
job-training matches does indeed raise questions about the integrity
of the data. One contractor told us that if a placement specialist
obtains a job for a participant that is not a job-training match
under Labor's guidance, then the manager and placement specialist
meet to determine how to make it a match. This same contractor
claimed that it is possible to get a job-training match for
participants who were trained as bank tellers, secretaries, and
welders and were subsequently placed in fast-food restaurants. For
the most part, the placement contractors we visited similarly
indicated that they use creativity when entering the code for the
placement job in order to obtain a job-training match and raised
concerns about the validity of reported job-training match
statistics.
The job-training match performance measure may also unfairly hold
placement contractors accountable for placing certain participants in
training-related jobs. All individuals placed in a job or the
military are included in the calculation of job-training match,
regardless of how long they received vocational training. Thus,
individuals who were in the program for a few days or weeks and had
little chance to participate in vocational skill-training would be
included in the calculation of the job-training match measure. Most
of the placement contractors and regional staff we spoke with agreed
that it would be more meaningful to include only participants who
entirely or substantially completed their vocational skills training
when this measure was calculated.
According to Labor officials, they are revising the methodology for
determining job-training matches, which is currently based on the
Dictionary of Occupational Titles. The proposed methodology will use
a system that the Bureau of Labor Statistics uses to collect
occupational employment data by various industry classifications,
with about 830 five-digit codes rather than the 5,700 nine-digit
codes taken from the dictionary. According to Labor, the proposed
system will be more accurate and easier to maintain and monitor in
terms of egregious job-training matches. Labor hopes to have the new
methodology in place by July 1, 1998. In addition, Labor stated that
the job-training match issue is primary on the agenda of a committee
established by Job Corps to improve the quality of vocational
outcomes.
JOB CORPS MEASURES UNDER THE
RESULTS ACT COULD BE IMPROVED
---------------------------------------------------------- Chapter 0:4
Labor's strategic plan with regard to Job Corps would be more useful
if the measures for the two performance goals it articulated were
based on valid information and included existing performance
indicators. Further, relying on Job Corps' invalid placement
performance data compromises one of the basic purposes of the Results
Act--measuring the extent to which goals are achieved--and
jeopardizes Labor's ability to effectively manage the program.
The Results Act requires virtually every executive agency to develop
a strategic plan, covering a period of at least 5 years from the
fiscal year in which it is submitted. The act is aimed at improving
program performance. It requires that agencies, in consultation with
the Congress and other stakeholders, clearly define their missions
and articulate comprehensive mission statements that define their
basic purpose. It also requires that they establish long-term
strategic goals, as well as annual goals linked to them. Agencies
must then measure their performance against the goals they have set
and report publicly on how well they are doing. In addition to
monitoring ongoing performance, agencies are expected to evaluate
their programs and to use the results from these evaluations to
improve the programs.
The strategic plan Labor submitted under the Results Act consists of
a department-level strategic plan overview supplemented by strategic
plans for 15 of its offices or units, including ETA. The ETA
strategic plan includes six goals, one of which is to increase the
number of youths, particularly at-risk youths, who successfully make
the transition into the workforce resulting in self-sufficiency. Job
Corps is one of several programs aimed at addressing this goal. In
that regard, ETA's plan includes two performance goals for Job Corps:
(1) increase the number of youths retaining jobs and increase their
earnings and (2) enhance their employability and increase their
educational attainment levels.
While ETA's strategic plan structure appropriately defines the role
of Job Corps within its mission and strategic goal of increasing the
participation of at-risk youths in the workforce, resulting in
self-sufficiency, we have concerns about the validity of one of the
measures articulated for one of its performance goals. As previously
noted, our past work and that of Labor's Inspector General has
questioned the validity of placement information. Thus, one of the
basic measures with which Labor proposes to assess the performance of
Job Corps under the Results Act does not provide accurate and
meaningful information.
ETA's articulation of performance measures could be improved. As
shown in table 2, its plan identifies proposed performance measures
for one of its goals but not the other. The plan states that
performance indicators for the goal of increasing job retention and
earnings will eventually include postprogram job retention and
postprogram earnings gains. Job Corps will be developing these
indicators by collecting data over the next 2 years to develop
baseline measures. In the interim, its proposed measures for program
years 1998 and 1999 include placement rates and placement wages. As
we noted previously, we question the validity of these data. In
addition, while wages provide some measure of program success, an
additional indicator of program quality that would be useful is an
improved measure of job-training match. ETA's strategic plan has no
related performance indicators for the second goal--enhancing
employability and increasing educational attainment. Nonetheless,
Job Corps' current performance measurement system contains measures
directly related to this goal, including measures of functional
literacy, functional numeracy, the attainment of a general
equivalency diploma, and completing vocational training. In our
opinion, measures such as these would be useful as performance
indicators for this second goal.
Table 2
Job Corps' Performance Goals and
Measures
ETA strategic goal Performance goal Performance measures
------------------------- ------------------------- --------------------------
Increase the number of Increase the number of Existing:
America's youths, youths retaining jobs and
particularly at-risk increase earnings, --number of terminees
youths, who make a resulting in greater entering employment or
successful transition self-sufficiency further education
into the labor force
resulting in self- --at an average placement
sufficiency wage
To be developed:
--postprogram job
retention
--postprogram earnings
gains
Enhance employability and No measure identified
raise educational
attainment of program
terminees
--------------------------------------------------------------------------------
Labor also needs accurate information to effectively manage Job
Corps. This is particularly important given the program's complex
structure, involving three independent functions--recruiting,
training, and placement--that are often contracted for separately.
Without accurate information on contractor performance, Labor does
not have the data for making proper decisions on contractor renewal.
CONCLUSIONS AND RECOMMENDATIONS
---------------------------------------------------------- Chapter 0:5
Job Corps is the nation's most expensive job-training program, with
an average cost of more than $15,000 per participant. And, although
there are reasons for this high cost, a considerable amount of this
investment is being spent on participants who drop out early and who
fail to complete their vocational training. It has been recognized
that those who complete their vocational training do better--that is,
they get higher-skill jobs at better wages. To ensure that Job Corps
operates the most effectively and that benefits accrue to the
greatest number of eligible youths, it is imperative that the program
identify, from among its eligible population, the applicants who both
need Job Corps' intensive services and have the commitment, attitude,
and motivation to complete the training and benefit from the program.
Furthermore, without meaningful and accurate program performance
information, Labor's ability to effectively manage the program is
jeopardized.
In the report we are releasing today, we make several recommendations
to the Secretary of Labor to help ensure that Job Corps uses its
resources to serve the most appropriate participants. We recommend
that the Secretary provide clear and complete guidance on program
eligibility criteria and provide better guidance to ensure that
outreach and admissions contractors assess each applicant's
capability and aspirations to complete training and obtain a positive
outcome. We also recommend improvements in the measures Labor uses
to assess placement contractor performance, to make them more
meaningful as tools for improving the selection and retention of
contractors.
In commenting on a draft of our report, Labor disagreed with our
recommendation that it clarify and expand its program eligibility
guidance in order to ensure that it is consistent with the law and
gave no indication of any formal action it planned to take on this
recommendation. Labor expressed concern with our characterization of
program eligibility guidance as inadequate. It commented that
guidance on one eligibility factor--limited job opportunity--was
provided to all admissions counselors during training conducted in
program year 1995. Labor also stated that another eligibility
factor--cultural deprivation--was not included in its policy handbook
because other specific factors were more useful. We disagree that
sufficient policy guidance was provided on both factors. Providing
guidance on the term "limited job opportunity" during a training
program was not adequate because, even if all admissions counselors
at that time attended this training, contractors and staff have since
turned over. And, as mentioned in our report, the admissions
counselors we interviewed interpreted this term in different ways, as
they did "cultural deprivation," thus indicating that contractors
need clearer guidance for interpreting these terms consistently. In
addition, Labor fails to explain how its guidance on either term
satisfies other specific provisions contained in the legislation and
program regulations.
Although Labor expressed some concern with our remaining
recommendations, it acknowledged that they had merit and warranted
consideration, and the agency identified actions that it would take.
-------------------------------------------------------- Chapter 0:5.1
Mr. Chairman, this concludes my prepared statement. We would be
happy to answer any questions that you or Members of the Subcommittee
may have.
*** End of document. ***