Job Corps: Need for Better Enrollment Guidance and Improved Placement
Measures (Letter Report, 10/21/97, GAO/HEHS-98-1).
Pursuant to a congressional request, GAO provided information on the Job
Corps program's recruitment and placement contractors, focusing on: (1)
whether Job Corps' policy guidance regarding eligibility criteria is
consistent with the legislation and regulations; (2) how the use of
recruiting contractors could be improved to increase participant
retention in the program; and (3) how the use of placement contractors
could be improved to enhance positive outcomes.
GAO noted that: (1) Job Corps' policy guidance for 2 of the 11
eligibility criteria was ambiguous and incomplete, which has led to an
eligibility determination process that fails to follow the requirements
of the law and program regulations; (2) in GAO's visits to several
outreach and admissions contractors, GAO found that those with higher
retention rates follow procedures aimed at identifying applicants with
the committment and motivation to remain in and benefit from the
program; (3) in GAO's analysis of participant characteristics, GAO
identified certain characteristics significantly related to the
likelihood of remaining in the program for at least 60 days; (4) Labor
could use some of these characteristics to design outreach efforts or to
establish priorities among establish priorities among eligible
applicants; (5) although Job Corps is a performance-driven program and
Labor uses performance measures to make decisions on placement
contractor renewal, two of the measures Labor used were not meaningful
and, thus, Labor did not have the information it needed to accurately
assess the performance of placement contractors; (6) placement measures
held contractors responsible for placing individuals who may have
received little or no benefit from the program or who demonstrated
behavior that normally would be unacceptable to most employers; (7) the
job-training match measure did not accurately portray the extent to
which participants obtained jobs related to their vocational training
because of the wide latitude placement contractors have in deciding
whether a job is related to the training received and the creativity
contractors used in recording the occupational titles of the jobs
obtained; (8) one aspect of placement contractors' operations associated
with better performance was having staff solely responsible for placing
Job Corps participants; (9) seven contractors visited by GAO with high
placement rates had staff solely responsible for placing Job Corps
participants; (10) in contrast, four of the five contractors having
lower placement rates had the same staff responsible for performing
outreach and assessment as well as placement; and (11) as a result of
its concern about performance, Labor has not renewed 12 of the 18
contracts with state agencies.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: HEHS-98-1
TITLE: Job Corps: Need for Better Enrollment Guidance and Improved
Placement Measures
DATE: 10/21/97
SUBJECT: Attrition rates
Vocational education
Disadvantaged persons
Youth employment programs
Contractor performance
Training utilization
State-administered programs
Eligibility criteria
IDENTIFIER: DOL Job Corps Program
DOL Student Pay and Allowance Management Information System
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to the Chairman, Subcommittee on Human Resources, Committee on
Government Reform and Oversight, House of Representatives
October 1997
JOB CORPS - NEED FOR BETTER
ENROLLMENT GUIDANCE AND IMPROVED
PLACEMENT MEASURES
GAO/HEHS-98-1
Job Corps Recruitment and Placement Process
(205329)
Abbreviations
=============================================================== ABBREV
AWOL - absent without leave
SPAMIS - Student Pay, Allotment, and Management Information System
Letter
=============================================================== LETTER
B-272492
October 21, 1997
The Honorable Christopher Shays
Chairman, Subcommittee on Human Resources
Committee on Government Reform
and Oversight
House of Representatives
Dear Mr. Chairman:
Job Corps is an employment and training program that is aimed at
providing severely disadvantaged youths with a comprehensive array of
services, generally in a residential setting. Job Corps is one of
the few remaining federally administered training programs. The
Department of Labor contracts with private and nonprofit
organizations to (1) recruit and enroll individuals in the program,
(2) operate its 109 centers throughout the nation, and (3) place
program participants in jobs or additional training upon termination
from the program.\1 About $1 billion a year is appropriated for Job
Corps, and it serves about 68,000 youths. However, about one-quarter
of the participants leave the program after a short time--many of
them expelled for disciplinary reasons.
In your continued interest in the Job Corps program, you asked us to
provide you with information on Job Corps recruitment and placement
contractors. Specifically, the objectives of our study were to
determine (1) whether Job Corps' policy guidance regarding
eligibility criteria is consistent with the legislation and
regulations, (2) how the use of recruiting contractors could be
improved to increase participant retention in the program, and (3)
how the use of placement contractors could be improved to enhance
positive outcomes.
In carrying out our work, we met with Labor officials and reviewed
Labor's eligibility policy guidance in relation to applicable
statutes and regulations. We analyzed national data on the
characteristics of program participants and early dropouts enrolled
during program year 1995.\2 We also analyzed program retention data
and placement results for each outreach, admission, and placement
contractor during program years 1994 and 1995 to identify contractors
that had higher and lower retention or placement performance. From
among these, we selected 14 contractors to visit--2 that did only
outreach and admissions, 1 that provided only placement services, and
11 that performed outreach and admissions functions and placement
functions--to obtain detailed information on the processes used to
admit applicants into Job Corps and place them upon their leaving the
program. We also interviewed Job Corps participants at three centers
to learn about their experiences when they were recruited for the
program and to obtain their views about the enrollment process.
(App. I contains a more detailed discussion of our scope and
methodology.)
--------------------
\1 Placement is defined as getting a job, entering the military,
returning to school, or entering another training program.
\2 A program year begins on July 1 of a year and ends on June 30 of
the following year. A program year is designated by the year in
which it begins. Thus, program year 1995 began on July 1, 1995, and
ended on June 30, 1996.
RESULTS IN BRIEF
------------------------------------------------------------ Letter :1
Job Corps' policy guidance for 2 of the 11 eligibility criteria was
ambiguous and incomplete, which has led to an eligibility
determination process that fails to follow the requirements of the
law and program regulations. Under Job Corps' enabling act and its
regulations, program participants must be from an environment so
characterized by cultural deprivation, a disruptive homelife, or
other disorienting conditions as to impair the applicant's ability to
successfully participate in other education and training programs.
However, regarding this environmental criterion, Job Corps' Policy
and Requirements Handbook (1) did not provide definitions of key
terms to describe "other disorienting conditions," such as "limited
job opportunities," and (2) limited eligibility to factors that do
not include "cultural deprivation," an environmental factor specified
in the law. Further, Labor has not provided adequate guidance
regarding another eligibility requirement--that participants have the
capability and aspirations to complete and secure the full benefits
of Job Corps. Without complete and unambiguous guidance, outreach
and admissions contractors may not be enrolling those who are most
appropriate, under the act and regulations, for the program.
We used two ways to identify how outreach and admissions contractors
could target the recruitment and selection of participants to those
more likely to stay in and benefit from Job Corps. In our visits to
several outreach and admissions contractors, we found that those with
higher retention rates followed procedures aimed at identifying
applicants with the commitment and motivation to remain in and
benefit from the program. And in our analysis of participant
characteristics, we identified certain characteristics significantly
related to the likelihood of remaining in the program for at least 60
days. Labor could use some of these characteristics to design
outreach efforts or to establish priorities among eligible
applicants. In addition, this information may be useful to Labor
should it decide to undertake an effort to improve the retention rate
for participants with characteristics associated with leaving the
program within 60 days of enrollment.
Although Job Corps is a performance-driven program and Labor uses
performance measures to make decisions on placement contractor
renewal, we found that two of the measures Labor used were not
meaningful and, thus, Labor did not have the information it needed to
accurately assess the performance of placement contractors. We found
that the placement measure held contractors responsible for placing
individuals who may have received little or no benefit from the
program or who demonstrated behavior that normally would be
unacceptable to most employers. In addition, the job-training match
measure did not accurately portray the extent to which participants
obtained jobs related to their vocational training because of the
wide latitude placement contractors have in deciding whether a job is
related to the training received and the creativity contractors used
in recording the occupational titles of the jobs obtained.
One aspect of placement contractors' operations associated with
better performance was having staff solely responsible for placing
Job Corps participants. The seven contractors we visited that had
higher placement rates (over 73 percent) had staff solely responsible
for placing Job Corps participants. Most of these contractors were
also responsible for managing Job Corps centers or had placement
staff located at Job Corps centers. In contrast, four of the five
contractors having lower placement rates had the same staff
responsible for performing outreach and assessment as well as
placement; none had placement staff located at the Job Corps center.
In addition, three of the contractors we visited were state
employment service agencies that provided services to Job Corps
participants similar to those provided to regular employment service
clients. As a result of their concern about performance, in the past
2 years Labor has not renewed 12 of the 18 contracts with state
employment service agencies.
BACKGROUND
------------------------------------------------------------ Letter :2
Job Corps was established as a national employment and training
program in 1964 to mitigate employment barriers faced by severely
disadvantaged youths. Job Corps enrolls youths aged 16 to 24 who are
economically disadvantaged, in need of additional education or
training, and living in disorienting conditions such as a disruptive
homelife.\3 Students may enroll in training programs throughout the
year and progress at their own pace.
Job Corps provides participants with a wide range of services,
including basic education, vocational skills training, social skill
instruction, counseling, health care, room and board, and recreation.
The program offers vocational skills training in areas such as
business occupations, automotive repair, construction trades, and
health occupations. Participation in Job Corps can lead to placement
in a job or enrollment in further training or education. It can also
lead to educational achievements such as attaining a high school
diploma and skills in reading or mathematics.
Job Corps is unique in that, for the most part, it is residential.
About 90 percent of the youths enrolled each year live at Job Corps
centers and are provided services 24 hours a day, 7 days a week. The
premise for boarding participants is that most come from a disruptive
environment and, therefore, can benefit from receiving education and
training in a different setting in which a variety of support
services are available around the clock.
Job Corps operates in a very structured and disciplined environment.
For example, established daily routines must be followed, as must
specific rules and regulations governing such areas as acceptable
dress and behavior. Furthermore, Job Corps participants must have
permission to leave the Job Corps center grounds, and participants
"earn" home leave, which must be approved before being taken and can
be denied for a number of reasons such as failure to follow a
center's rules of conduct. Job Corps typically employs residential
staff to oversee dormitory living and security staff for the safety
and well-being of its participants. The program recently implemented
a "zero tolerance" policy for violence and drugs. This policy
includes a "one-strike-and-you're-out" provision for the most serious
violent or criminal offenses as well as for drug violations.
Job Corps currently operates 109 centers throughout mainland United
States, Alaska and Hawaii, the District of Columbia, and Puerto Rico.
Most states have at least one center, and several states have four or
more centers.\4 Job Corps' nine regional directors are responsible
for the day-to-day administration of the Job Corps program at the
centers within their geographic boundaries. Private corporations and
nonprofit organizations, selected through competitive procurement,
operate the majority of the centers. However, the departments of
Agriculture and Interior directly operate 28 centers, called civilian
conservation centers, under interagency agreements.
The regional directors are also responsible for overseeing the
recruitment of youths for program participation and the placement of
participants after they leave Job Corps. Recruitment, referred to as
outreach and admissions by program managers, and placement services
are provided by private contractors, the centers, or state employment
service agencies under contract with the regional offices. During
program year 1995, Job Corps spent about $60 million on outreach and
admissions as well as placement contracts.\5 This included amounts
paid contractors solely for outreach and admissions and placement
services. In addition, a portion of the funding for some Job Corps
center operation contracts was specifically designated for outreach
and admissions and placement services.
Job Corps contractors are expected to meet certain levels of
achievement in order to continue to participate in the program and
receive program funding. A performance standard has been established
for outreach and admissions contractors with respect to "quotas" of
male and female youths to be enrolled (as specified in the contract),
and a second standard relates to the proportion of participants who
are to remain in the program for more than 30 days (90 percent). A
third standard relates to the percentage of participants who are
eventually placed following termination from the program (70
percent). Similarly, placement contractors are required to meet
established standards related to the percentage of participants
placed in jobs, the military, schools, or other training programs (70
percent). Additional standards are applied to participants who are
placed in jobs. These standards relate to the percentage obtaining
full-time jobs (70 percent) and jobs directly related to the
vocational training received (42 percent). A fourth placement
standard relates to the average wage received at placement.\6
Individuals enroll in Job Corps by submitting applications through
outreach and admissions contractors. The length of time students
stay in Job Corps can vary substantially--from 1 day to 2 years.\7 In
program year 1995, about 15 percent of the enrollees left Job Corps
within 30 days of entering the program and more than one-fourth left
within 60 days. On the average, however, students spend about 7
months in the program. Students leave Job Corps for a variety of
reasons, including successful completion of the program objectives,
voluntary resignation, disciplinary termination, and being absent
without leave (AWOL) for 10 consecutive training days. With a few
exceptions, participants terminating from Job Corps are assigned to a
placement contractor for assistance in finding a job or enrolling in
other education or training programs. Placement contractors are to
give priority to finding full-time, training-related jobs for
participants.
--------------------
\3 Although the act includes 14- and 15-year-old youths in the age
criteria, Job Corps regulations provide that youths 14 and 15 years
of age may be eligible "upon a specific determination by the program
director to enroll them."
\4 Delaware, New Hampshire, Rhode Island, and Wyoming have no
centers. California, Kentucky, New York, North Carolina, Oklahoma,
Oregon, Pennsylvania, Texas, and Washington have four or more Job
Corps centers. See Job Corps: Where Participants Are Recruited,
Trained, and Placed in Jobs (GAO/HEHS-96-140, July 17, 1996).
\5 About $8 million of this amount was for media support contracts.
According to Labor, this high level of media expenditures should be
regarded as a one-time but necessary cost to counteract a decline in
Job Corps enrollments in program year 1994 and early program year
1995.
\6 Because economic conditions vary by location, the standard for
this measure is adjusted by a model that adjusts for local
conditions.
\7 Job Corps participants may be enrolled in the program for an
additional year to attend advanced career training.
JOB CORPS ELIGIBILITY GUIDANCE
IS INADEQUATE
------------------------------------------------------------ Letter :3
We found that Job Corps' policy guidance on two of its eligibility
criteria was ambiguous and incomplete. As a result, the program's
eligibility process was not following all the requirements of the law
or program regulations. The law specifies program eligibility
requirements, including age, economic status, educational needs,
medical condition, and behavioral condition--all defined in the
legislation, implementing regulations, or Labor policy guidance.\8
Another legislative requirement--living in an environment
characterized by disorienting conditions--has not been clearly
defined in the statute, regulations, or Labor's guidance. Further,
Labor has not provided adequate guidance regarding the requirement
that participants have the capability and aspirations to complete and
secure the full benefits of Job Corps.
Contractors are required to follow Labor's Policy and Requirements
Handbook, which sets out 11 eligibility criteria for the program that
all participants must satisfy: age, economically disadvantaged,
requires additional education or training, environment, health
history, behavioral adjustment history, capability and aspirations to
participate, legal U.S. resident, child care, parental consent, and
Selective Service registration (see app. II).\9
The first seven are specified in the law. The policy handbook
generally provides guidance on what is needed to meet most of these
criteria. For example, to be eligible under the education or
training criterion, an applicant must be a dropout or in need of
additional education, training, or related support services in order
to hold meaningful employment, participate in regular school work,
qualify for other training, or satisfy armed forces requirements.
However, guidance on two of the criteria (environment and capability
and aspirations) is vague.
--------------------
\8 29 U.S.C. 1501. The law also allows the Secretary of Labor to
prescribe other eligibility requirements for enrollment.
\9 Three of these criteria do not apply to all applicants. For
example, child care applies only to those with a dependent child;
parental consent, only to those who are minors; and Selective Service
registration, only to male applicants.
ENVIRONMENTAL CRITERION IS
OPEN TO INTERPRETATION
---------------------------------------------------------- Letter :3.1
One of Job Corps' eligibility criteria specified in the law for
participation in the program relates to environment: A participant
must come from "an environment so characterized by cultural
deprivation, a disruptive homelife, or other disorienting conditions
as to substantially impair prospects for successful participation in
other programs providing needed training, education, or assistance."
Program regulations go on to explain that the disorienting condition
must be one that would impair the applicant's chance of success in a
nonresidential program rather than a residential Job Corps program.
Job Corps legislation, Labor's program regulations, and Job Corps'
policy handbook list environmental factors to be considered when
assessing eligibility, but these sources of program guidance are not
entirely consistent nor do they contain adequate definitions (see
table 1). With the exception of the regulatory definition of
disruptive homelife, program guidance does not define the factors
that make up the environmental criterion. In the absence of specific
definitions of the environmental criterion, admissions counselors
applied their own interpretations.
Table 1
Comparison of Elements of the
Environmental Criterion in Job Corps
Program Guidance
Labor's policy
Criterion Statute Regulation handbook
---------------- ---------------- ---------------- ----------------
Currently living
in an
environment
characterized by
Cultural Cultural
deprivation deprivation
Disruptive home Disruptive home Disruptive home
life life\a life, unsafe,
overcrowded
dwelling
Other Other Limited job
disorienting disorienting opportunities;
conditions conditions disruptive
community; high
crime rates
----------------------------------------------------------------------
\a Defined in the regulations as a homelife characterized by
conditions such as (1) living in an orphanage or other institution,
(2) suffering from parental or familial neglect or abuse, and (3)
having parents or guardians who are chronic invalids, alcoholics, or
drug addicts or have other serious health conditions.
As shown in table 1, Labor includes "limited job opportunities" in
its policy handbook as a disorienting condition that fulfills the
environmental eligibility requirement. However, none of the sources
of program guidance specifically defines this factor or gives any
direction to assessment counselors to help them interpret it, nor do
they explain how limited job opportunities affect the chance of
success in a residential program compared to a nonresidential one.
In prior Job Corps regulations, Labor included among "disruptive
conditions" that could impair an applicant's prospect to participate
fully in nonresidential training "a neighborhood or community
characterized by high crime rates, high unemployment rates, high
school dropout rates, and similar handicaps." Unlike the present
regulations, the prior version made clear that applicants might be
subject to more than one disruptive factor and that several factors
in combination might satisfy this impairment criterion. Labor's
present guidance does not explain how "limited job opportunities" by
themselves can satisfy this criterion. Nonetheless, limited job
opportunities was the factor cited as fulfilling the environmental
eligibility requirement for 92 percent of the 68,000 Job Corps
enrollees in program year 1995. Because admissions counselors
generally indicate only one environmental factor on the Job Corps
application form, we have no way of knowing how many of these
participants would have met the environmental criterion had limited
job opportunities not been used to fulfill the requirement.
Further, the admissions counselors we interviewed had varying
interpretations of limited job opportunity. Some thought that it
referred to the applicants' lack of job skills or lack of education,
whereas others thought that it referred to the economic condition of
the geographic areas in which applicants resided or their being too
young or lacking transportation.
Cultural deprivation, another eligibility factor that could fulfill
the environmental criterion, was not clearly defined--in fact, it is
not even listed in Labor's policy handbook--and was also interpreted
differently by various admissions counselors. One contractor
referred to persons who had never gone to a museum or the beach;
another thought it applied to a situation such as raising a minority
child in a nonminority family; a third referred to living in a
housing project. Most admissions counselors we interviewed admitted
that they had no idea what this term meant.
Finally, Labor's policy handbook restricts what can be considered
under the environmental criterion, stating that to be eligible an
applicant must be living in an environment characterized by
-- disruptive homelife; unsafe, overcrowded dwelling;
-- limited job opportunities; or
-- disruptive community; high crime rates.
However, the handbook excludes cultural deprivation--specified in the
statute and Labor's own regulations--from permitted environmental
factors.
INADEQUATE GUIDANCE ON
CAPABILITY AND ASPIRATIONS
CRITERION
---------------------------------------------------------- Letter :3.2
The Job Corps law states that to enroll in Job Corps, an applicant
must, after careful screening, have the present capability and
aspirations to complete and secure the full benefit of the program.
However, in determining whether applicants meet this requirement,
Labor relied primarily on an evaluation form that assesses behavior
that would be expected of any and all applicants. Without more
detailed guidance on the use of this criterion, the program may not
always be serving those who are most likely to benefit from it. In
previous work, we found that ensuring that project participants are
committed to training and getting a job is a key feature of
successful employment training projects.\10
The law does not define "capabilities and aspirations" but leaves to
Labor the tasks of defining this term and providing guidance on how
it is to be implemented. Labor has developed the "Capability and
Aspirations Assessment Tool," which admissions counselors must
complete for each applicant (see app. III). This "tool" formulates
four categories of factors--commitment, attitude, capability, and
compatibility of applicant and program goals--that are used to assess
capability and aspirations and to demonstrate suitability for the
program. Factors under commitment include meeting scheduled
appointments on time, providing requested documents such as birth
certificates, and reacting favorably to program requirements such as
following center rules and living away from home. Attitude includes
willingly responding to questions and behaving respectfully during
the interview. Capability involves obtaining documentation that
supports an applicant's ability to benefit from the program such as
school, court, or medical records or a letter from a former employer.
Compatibility of applicant and program goals relates to the
admissions counselor's opinion that an applicant's expressed
goals--for example, for job placement or vocational training--can be
realistically achieved through Job Corps.
The factors specified in Labor's assessment tool include
characteristics that if not displayed would be an appropriate basis
for rejecting an application. However, the possession of these
characteristics does not necessarily demonstrate that an applicant
has the ability and motivation to benefit from Job Corps. Job Corps
outreach and admissions contractors and regional staff whom we spoke
with pointed out shortcomings in the current approach to assessing
applicants' capability and aspirations. Staff in one of Labor's
regional offices stated that admissions counselors have asked for
additional guidance in making better decisions on capability and
aspirations. An admissions contractor with statewide recruiting
responsibility in one state said that there is a need for a valid
assessment tool for this criterion because the current tool is
inadequate. Another contractor stated that it filled out Labor's
assessment tool because it is a program requirement but did not use
it in assessing the suitability of applicants. One of Labor's
regional offices has started to develop a more meaningful tool.
--------------------
\10 Employment Training: Successful Projects Share Common Strategy
(GAO/HEHS-96-108, May 7, 1996).
RECRUITMENT AND SELECTION OF
JOB CORPS PARTICIPANTS COULD BE
IMPROVED
------------------------------------------------------------ Letter :4
A substantial number of Job Corps participants leave the program
within a short time after enrollment--about one-fourth of program
year 1995 participants left within 2 months. Therefore, we believed
that it would be useful to identify ways contractors could target
recruitment efforts and the selection of applicants to the eligible
youths who are more likely to stay in the program and, thus, more
likely to benefit from it. To determine the factors that might be
related to program retention, we visited a number of outreach and
admissions contractors to examine their practices in assessing and
screening applicants for the program. We also analyzed the
characteristics of the more than 68,000 program year 1995
participants to determine the characteristics that were associated
with remaining in Job Corps for at least 60 days.\11 In our visits,
we identified several procedures that distinguished outreach and
admissions contractors with higher retention rates from other
outreach and admissions contractors. In general, these procedures
were aimed at identifying applicants with the commitment and
motivation to remain in and benefit from the program. Our
statistical analysis provides some information about characteristics
significantly related to the likelihood of remaining in the program
for at least 60 days that Labor could use to design outreach efforts,
establish priorities among applicants, or improve the retention rate
for those who might otherwise leave the program early.
--------------------
\11 We performed a multivariate logistic regression analysis to
identify characteristics associated with individuals staying in Job
Corps longer (at least 60 days).
CONTRACTORS WITH HIGHER
RETENTION RATES HAVE BETTER
ASSESSMENT PROCEDURES
---------------------------------------------------------- Letter :4.1
Of the 11 outreach and admissions contractors that we visited, those
with higher retention rates (10 percent or fewer of their enrollees
dropping out within the first 30 days) tended to have better
procedures for identifying applicants with the commitment and
motivation to remain in and benefit from the program. That is, these
contractors emphasized making sure that applicants met the programs'
statutory eligibility criterion of having the capability and
aspirations to complete and secure the full benefit of the program.
These more-successful contractors' procedures included "commitment
checks" and preenrollment tours and briefings, which gave applicants
a more realistic basis for deciding whether they wanted to enroll.
The emphasis in these programs was consistent with the finding we
reported in a May 1996 report on successful training programs--that a
key job-training strategy shared by successful programs was a focus
on ensuring that participants are committed to training and getting a
job.\12 It was also consistent with the opinions expressed by several
regional directors we interviewed.
The "commitment checks" contractors' used were designed to test Job
Corps applicants' initiative. For example, several contractors
required individuals interested in Job Corps to set up application
appointments. Admissions counselors at four contractors also
mentioned that they required applicants to arrive for their meetings
dressed in proper attire; otherwise, they had to schedule another
appointment. In addition, three admissions counselors required
applicants to submit written statements of why they wanted to
participate in the program and what they hoped to accomplish.
Several admissions counselors required applicants to call weekly
between the date of application and the enrollment date to determine
the status of their application and to demonstrate their continued
interest in the program. Finally, one contractor also used a
nine-point checklist of documents that all interested persons had to
acquire before they set up their application appointment.
Some outreach and admissions contractors considered preenrollment
tours and briefings to be extremely useful, although they were not
practical in every situation. They provided applicants with a
firsthand opportunity to obtain a thorough understanding of Job Corps
rules and requirements, observe the living conditions, erase false
expectations, and determine whether they were suited for regimented
life. In some instances, these preenrollment briefings were given
prior to application while others took place afterward. For example,
one contractor required that all interested individuals attend a
prearranged tour and briefing. After taking the tour, attending the
briefing, and participating in a question and answer session, those
still interested had to set up an appointment to complete an
application. Another contractor required potential enrollees to take
a tour after the application process. Following the tour, applicants
attended a briefing and question and answer session, followed by
one-on-one interviews with center staff. The value of preenrollment
tours and briefings was also confirmed by Job Corps participants at
two of the centers we visited who thought the tours and briefings
were definitely worthwhile and by two regional directors who agreed
that the preenrollment tours and briefings were very effective in
preparing applicants for Job Corps and in improving program
retention. These tours and briefings would help meet the law's
requirements that applicants be given a full understanding of Job
Corps as well as what is expected of them after enrollment.
Several regional directors commented on the importance of identifying
applicants who are ready for Job Corps and can benefit from its
training. For example, one regional director stated that because the
program cannot afford to squander its resources on applicants who do
not really want to be in the program, admissions counselors should
ensure that applicants are ready and can benefit from the investment.
Another regional director noted that because so many people are
eligible for Job Corps (over 6 million) it was important to provide
this opportunity to those most likely to benefit and that commitment
should be "first and foremost" when assessing applicants. Another
regional director agreed that commitment was important but considered
the program's Capability and Aspirations Assessment Tool to be
ineffective in measuring it.
--------------------
\12 Employment Training: Successful Projects Share Common Strategy
(GAO/HEHS-96-108, May 7, 1996).
CHARACTERISTICS ASSOCIATED
WITH PROGRAM RETENTION
---------------------------------------------------------- Letter :4.2
In our analysis, we identified several characteristics associated
with program retention that Labor might consider in designing
outreach efforts, establishing priorities among applicants, or
improving participant retention rates. Some of these characteristics
would be of limited value nationwide, however, because so few
participants nationwide had those characteristics. In addition, when
considering how to use the results from our analysis, Labor also
needs to consider other factors.
Two of the characteristics most strongly related to the likelihood of
remaining in the program were need for bilingual education and years
of education. Of the characteristics we examined, the need for
bilingual education had the strongest relationship with the
likelihood of remaining in the program. Participants needing
bilingual training--Spanish as well as other languages--were much
more likely than others to remain in the program for at least 60
days. Education was also an important factor--participants with 12
or more years of education were more likely to remain than
participants with 8 or fewer years of schooling.
Another characteristic with a strong relationship to retention was
age. Our analysis indicated that older participants had a greater
likelihood than younger participants of remaining in the program.
Specifically, when compared to 15-17-year-old participants, those
aged 18 to 20 and 21 to 25 were more likely to remain in the program
for at least 60 days.\13 This analysis supported the concern
expressed by many of the admissions counselors we interviewed
regarding enrollment, retention, and placement of 16- and 17-year-old
youths, who make up nearly 40 percent of the program year 1995
enrollees. The concerns they expressed were that these younger
youths are often victimized by older participants at the center, have
a harder time adjusting to center life, are more likely to drop out,
and are difficult to place. Labor program year 1995 outcome data
showed that 16- and 17-year-old terminees were less likely to be
placed once they left the program (see fig. 1). Because of the
difficulty in placing 16- and 17-year-old participants, one regional
Labor official believed that the minimum age for enrollment should be
increased, while another thought that there should be separate
standards for these participants. In contrast, a third regional
Labor official thought that maturity, and not age, should be the
deciding factor for enrollment. He acknowledged, however, that the
program should probably have different expectations and performance
standards for 16-year-old participants. Another Labor official told
us that a work group has been established to look into the problem of
serving 16- and 17-year-old participants.
Figure 1: Percentage of
Program Year 1995 Terminees Not
Placed by Age
(See figure in printed
edition.)
Appendix IV discusses our statistical analysis of characteristics
related to remaining in the program at least 60 days, including
limitations associated with the analysis. Table IV.3 in that
appendix contains the final model and significance levels. For
example, it shows that other factors that had a significant
relationship to the likelihood of remaining in the program for at
least 60 days included residing less than 50 miles from the assigned
Job Corps center, being a nonresidential student, having no
dependents, and having served in the military. Additionally, some of
the factors that proved to be useful predictors of remaining in the
program were characteristics of only small subsets of participants.
For example, because relatively few participants had a need for
bilingual education (less than 3 percent of the Job Corps
population), that characteristic was limited in its value as a
feature for nationwide use in screening. Because we found no large
subgroups with great differences, the ability of the model we used in
our analysis to predict 60-day retention for the program's full
population is limited.
In deciding how to use the results of this analysis, Labor would need
to consider more than the statistical results. For example, it would
clearly be inappropriate to use these findings to exclude applicants
who met the statutory eligibility requirements because they had
characteristics associated with a low likelihood of completing the
program. If Labor chose to consider these characteristics in
designing outreach efforts or establishing priorities for eligible
applicants, it would be faced with the complexity of integrating
these results with existing eligibility requirements and program
policy. For example, our results showed that participants with at
least 12 years of education were more likely to remain for 60 days
than those with less education. Many youths with that many years in
school, however, might not meet the eligibility requirement of
needing additional education or training to secure and hold
meaningful employment, participate successfully in regular school
work, qualify for other suitable training programs, or satisfy armed
forces requirements. The most clear-cut use of this information on
participant characteristics may be in designing efforts to improve
the retention rate of participants with characteristics associated
with leaving the program early.
--------------------
\13 We obtained data for this analysis from Labor's national database
and they showed that less than 1 percent of program year 1995
enrollees were either 15 or 25 years old.
PERFORMANCE MEASURES ARE
INADEQUATE FOR ASSESSING
PLACEMENT CONTRACTOR
PERFORMANCE
------------------------------------------------------------ Letter :5
Labor uses performance measures in deciding whether contractors are
to continue to participate in the program. However, Labor does not
have the information it needs to accurately assess the performance of
its placement contractors. We found that two of the four measures
Labor used in assessing placement contractor performance were not
meaningful. One of the measures held contractors accountable for
placing participants who were realistically unemployable. A second
measure, relating to the placement of terminees in training-related
occupations, included terminees who received little vocational
training and also gave placement contractors wide latitude in
deciding whether placements were related to training.
Job Corps requires placement contractors to assist all terminees with
placement regardless of how long they were in the program or the
reason they left, and it has established the following standards to
measure contractor performance:\14
-- 70 percent of all terminees assigned to a contractor are to be
placed,
-- 70 percent of all placements are to be in full-time jobs,
-- the average wage paid to participants placed in jobs is to be
equal to or greater than a specified level, and
-- 42 percent of all job placements are to be in occupations
related to the training received.
--------------------
\14 Job Corps contractors provide placement services to all program
participants once they leave the program, except those who are
terminated within the first 30 days for violating the program's zero
tolerance policy for drugs and violence and those found to be
ineligible after enrollment.
MEASUREMENT OF JOB
PLACEMENTS INCLUDES
UNEMPLOYABLE TERMINEES
---------------------------------------------------------- Letter :5.1
In calculating a contractor's placement performance, Labor includes
participants who remained in the program for as little as 1 day,
those who were AWOL, and those who were expelled from Job Corps after
30 days for using drugs or committing violent acts--all individuals a
placement contractor would have difficulty recommending for
employment. During program year 1995, about one-third of the
participants leaving Job Corps were in these categories. If Labor's
methodology were modified to include only participants who were in
the program for sufficient time to obtain at least minimal benefits
(that is, stayed for at least 30 days) and were employable (that is,
were not terminated for drug violations and violence and were not
AWOL), the average placement rate for the 12 placement contractors we
visited would be about 8 points higher--ranging from an increase of
2.6 points for one contractor to 13.6 points for another
contractor--and the rank order among the 12 contractors would change
somewhat. (See fig. 2.)
Figure 2: Program Year 1995
Placement Rates for Selected
Contractors Using Existing and
Modified Methodology
(See figure in printed
edition.)
About half of the placement contractors we visited suggested that
Labor should exclude certain individuals when calculating placement
rates. For example, one contractor noted that it is unreasonable to
expect contractors to recommend to an employer someone who was
expelled for taking drugs or committing a violent act. Another
contractor believed that it was a waste of resources to try to place
participants who were AWOL because they were not only difficult to
locate but also undependable to an employer. A third contractor
suggested that Labor's methodology include only participants who are
truly employable. Similarly, a regional director stated that it is
ridiculous to require placement specialists to be responsible for
placing participants who stayed in the program a very short time,
were expelled for drug use or violence, or were AWOL. He said that
this responsibility asks the placement specialist to lie to employers
by recommending they hire these people. Another regional director
agreed that placement contractors should not be responsible for
participants who received no benefit from the program or who were
kicked out for violating the program's drug and violence policies.
TRAINING-RELATED PLACEMENT
MEASURE IS FLAWED
---------------------------------------------------------- Letter :5.2
The job-training match measure is used to evaluate the effectiveness
of vocational training programs and placement contractors by
determining the percentage of jobs terminees obtain that matches the
training they received while in Job Corps. Labor allows placement
contractors wide discretion in deciding whether a job placement they
obtain for a terminee is related to the training received--another
measure of performance. At the same time, Labor requires that
terminees who receive little vocational training be included in the
calculation of this measure. As a result, the value of the current
job-training match performance measure is questionable. Labor is
developing a new system to determine job-training matches that, it
believes, will be more accurate.
Labor's guidance gives placement contractors wide latitude in
deciding whether a job placement was a job-training match. According
to Labor guidance, a job-training match results when a participant is
placed in a job requiring skills similar to those included in the
participant's training. Placement contractors are responsible for
recording this information. Labor's guidance for these decisions
consists of 16 broad categories of training programs, and within each
category are a varying number of detailed occupations in which Job
Corps participants may be trained. In addition, each of the 16 broad
categories contains a list of jobs that would be considered a match
with the training received. To illustrate, the broad training
category of construction trades includes 47 detailed training
occupations and 357 placement occupations. An individual who was
trained in any one of the 47 training occupations and then was placed
into any one of the 357 placement occupations would be counted as
having made a job-training match. Overall, Labor's system includes
nearly 300 detailed training occupations and more than 5,700 job
placement occupations.
In addition to the wide range of jobs that are considered to be
training matches under each of the broad training categories, Labor's
guidance includes jobs that appear to bear little, if any,
relationship to the training received. For example, a position as a
key cutter would be considered to be a training match for any of the
51 training categories under the broad category of mechanics and
repairers, which includes auto mechanic, electronics assembler, and
parts clerk. A position as a general laborer would be considered to
be a job-training match for any of the 30 training occupations under
the precision production category, which includes mechanical drafter,
sheet metal worker, and welder. Table 2 lists examples of some
possible matches under Labor's guidance.
Table 2
Examples of Occupations Considered to Be
Job-Training Matches for Selected
Vocational Training Programs
Instructional category Occupation
---------------------------------- ----------------------------------
Automobile mechanic Band attacher (attaches wristbands
to watches)
Feeder (stacks paper in offset
press)
Key cutter
Washer (clock parts)
Cook Bar attendant
Car hop
Housecleaner (hotel)
Fast-food worker
Cosmetologist Hot-room attendant (gives patrons
towels)
Sales person for weed eradication
services
Shaver (brushes suede garment
after it has been cleaned)
Shaver (shaves hog carcasses)
Heavy equipment operator Baggage checker
Freight elevator operator
Porter
Ticket seller
Medical secretary Coin counter-and-wrapper
General cashier
Hand packager
Linen-room attendant
Welder Antisqueak filler (shoes)
Casket liner
General laborer
Hacker (lifts bricks and clay
tiles from conveyor belt and
stacks them)
----------------------------------------------------------------------
Many of the positions that are considered to be related to Job Corps
training require relatively little training to perform. The job
placement occupational categories contained in Labor's guidance for
job-training match come from its Dictionary of Occupational Titles.
The dictionary includes, for each occupation, the average time
required to learn the techniques, acquire information, and develop
the facility for average performance in a specific job situation.
For more than 700 of the jobs included in Labor's guidance, the
average training time is indicated as either only a short
demonstration or training up to and including 1 month. Thus, Labor
is allowing job-training match credit for occupations requiring
relatively short training time even though participants spend an
average of about 7 months in the program at an average cost of about
$15,300 each.\15 While we recognize that some of these positions
provide entry into an occupational area that may lead to a better
job, in our view it is questionable to consider such positions to be
a job-training match until the participant advances into a job
commensurate with the training received.
Further, Labor guidance encourages placement contractors to search
among the allowable jobs for a job-training match. Its policy
handbook states that, if a job-training match is not generated when a
job placement code is entered in its automated system, the placement
contractor is allowed to enter a different code that may generate a
job-training match, "so long as integrity of data is maintained." We
found that placement contractors' practice of recording job-training
matches does indeed raise questions about the integrity of the data.
One contractor told us that if a placement specialist obtained a job
for a terminee that was not a job-training match under Labor's
guidance, then the manager and placement specialist would meet to
determine how to make it a match. This same contractor claimed that
it is possible to get a job-training match in fast-food restaurants
for participants trained as bank tellers, secretaries, and welders.
For the most part, the placement contractors we visited similarly
indicated that creativity is used when entering the code for the
placement job in order to obtain a job-training match and raised
concerns about the validity of reported job-training match
statistics.
The job-training match performance measure may also unfairly hold
placement contractors accountable for placing certain terminees in
training-related jobs. All participants placed in a job or the
military are included in the calculation of job-training match,
regardless of how long they received vocational training. Thus,
participants for a few days or weeks who had little chance to
participate to any extent in vocational skill training would be
included in the calculation of the job-training match measure. Most
of the placement contractors and regional staff we spoke with agreed
that when calculating this measure it would be more meaningful to
include only participants who completed their vocational skills
training.
Labor officials told us that they are revising the methodology for
determining job-training matches. The proposed methodology will use
an existing system used by the Bureau of Labor Statistics to collect
occupational employment data by various industry classifications.
This system uses about 830 five-digit codes rather than the 5,700
nine-digit codes used in the current methodology based on the
Dictionary of Occupational Titles. In its comments on a draft of our
report, Labor acknowledged that we made a legitimate point about the
need to strengthen the job-training match process. According to
Labor, the proposed system will be more accurate and easier to
maintain and monitor in terms of egregious job-training matches.
Labor hopes to have implemented the new methodology by July 1, 1998.
In addition, Labor stated that the job-training match issue is one of
the primary projects being addressed by a Job Corps committee to
improve the quality of vocational outcomes.
--------------------
\15 Job Corps: High Costs and Mixed Results Raise Questions About
Program's Effectiveness (GAO/HEHS-95-180, June 30, 1995).
CHARACTERISTICS OF CONTRACTORS
WITH HIGHER PLACEMENT RATES
------------------------------------------------------------ Letter :6
We found that a characteristic common to the contractors we visited
that had higher placement rates was having staff solely responsible
for providing placement services to Job Corps participants. In
addition, most of these placement contractors were either Job Corps
centers or had staff located at the centers they served. In
contrast, Labor regional officials have been concerned with the
performance of state employment service agencies and have not renewed
many of their contracts during the past 2 years. We also noted that
Labor and several of the Job Corps centers we visited were starting
to improve links to the business community in an effort to increase
placements.
The placement contractors we visited had had varying success in
placing Job Corps participants in program year 1995. Placement
included getting a job, entering the military, or returning full-time
to school. The seven contractors that had relatively high placement
rates (over 73 percent) included four Job Corps centers and three
private organizations. A common characteristic among these
contractors was having staff who had only one responsibility--placing
Job Corps participants. Other contractors that were not as
successful used the same staff to perform outreach and admissions as
well as placement. One contractor whose staff performed these
functions noted that with the program's emphasis on maintaining
centers at full capacity, placement is often secondary to admissions.
We also noted that most of the contractors with higher placement
rates were either Job Corps centers or had staff at the center.
Placement specialists at the Job Corps center contended that being at
the center allowed them easy access to instructors, counselors, and
participants. One Labor regional director also mentioned the
importance of having a continuity of services from the time enrollees
arrive at the center until they are placed, noting that it was no
accident that every center in his region also has a placement
contract.
In contrast, the placement contractor we visited with the highest
placement rate was not a Job Corps center and did not have staff at a
center. The program manager of this private company viewed Job Corps
placement as a business and ran the organization accordingly--either
placement specialists produced jobs for Job Corps participants or
else the program manager found someone who could. Thus, having a
focus on the ultimate goal--placement in a job--is a strategy
associated with a high placement rate.
One type of contractor that generally has not had high placement
rates is state employment service agencies. Between program years
1994 and 1996, Labor regional offices did not renew two-thirds (12 of
18) of the placement contracts they had with state employment service
agencies (see table 3). Labor officials in three regional offices
informed us that they cancelled the placement contracts with state
employment service agencies because of poor performance. A Labor
official in a fourth region stated that the agency had sent a letter
of concern to the state employment service agency because it was the
worst-performing placement contractor in the region. Five of the six
remaining state employment service placement contractors had
placement rates in program year 1995 below the national Job Corps
standard of 70 percent.
Table 3
State Employment Service Agencies Having
Placement Contracts With Job Corps,
Program Years 1994-96
Program year
----------------------------------------------------
1994 1995 1996
---------------- ---------------- ---------------- ----------------
State agency Missouri Missouri Missouri
Nevada Nevada Nevada
North Dakota North Dakota North Dakota
Oklahoma Oklahoma Oklahoma
South Dakota South Dakota South Dakota
Texas Texas Texas
Alabama Alabama
Arkansas Arkansas
Florida Florida
Georgia Georgia
Louisiana Louisiana
Mississippi Mississippi
Tennessee Tennessee
Virgin Islands Virgin Islands
Kansas
Kentucky
Washington
Wyoming
----------------------------------------------------------------------
Officials from two of the three state employment service agencies we
visited expressed reservations about continuing to contract with Job
Corps for placement services. For example, one employment service
official said that the agency might not seek contract renewal because
of its strained relations with Labor's regional office. An official
from another employment service commented that its Job Corps contract
was really "small potatoes" and insufficient to provide for adequate
staffing and that the only reason it was still involved was that the
employment service commissioner believed that Job Corps was
worthwhile and wanted to assist disadvantaged youths. An official
from the third employment service agency we visited noted that the
Labor regional office threatened to cancel its placement contract 2
years ago for poor performance and gave the agency another 6 months
to improve. The official noted that, under new management,
performance did improve and Labor renewed the agency's contract for
another 2 years.
Placement specialists at the three employment service offices we
visited stated that they have no contact with Job Corps participants
before their termination. It also appeared that the major placement
emphasis was to register Job Corps participants in the employment
service databank. While this did provide access to a major source of
potential jobs, it was the same service provided to regular job
seekers using the employment service and was not any kind of
specialized assistance.
As pointed out by the Chairman of the Senate Subcommittee on
Employment and Training, Committee on Labor and Human Resources,
during hearings on Job Corps in April 1997, a key to program success
is the development of links to the business community. However,
concerns were raised about whether Job Corps had developed such
links. We noted that several of the centers we visited that had
higher placement rates also had good relationships with local
businesses. For example, one center had established a physical
therapy program to meet the needs of local health facilities, and
another center used temporary agencies as a springboard for their
computer services trainees to gain access to the area's computer
industry. A third center was working on improving its work
experience component to better match participants' skills and
abilities to the needs of local businesses so that more permanent
hires would result.
Labor regional offices are also exploring ways to improve links to
the business sector. For example, one office has recently started a
business roundtable of 18 employers in the region who discuss
placement issues. Another regional office has begun a project to get
local employers involved with training and placement. The idea is to
have employers identify what they need in terms of training
curriculum, equipment, and skills and then determine how the program
can meet these needs. Recognizing the importance of employer links,
Labor has launched a new school-to-work initiative within Job Corps
to involve more employers in placing program terminees and to
establish the basic framework for a school-to-work program. It
started as a pilot program at three Job Corps centers and will be
expanded to 30 centers this year. Further expansion will depend on
the availability of funding.
CONCLUSIONS
------------------------------------------------------------ Letter :7
Labor's program guidance to admissions counselors on two eligibility
requirements was ambiguous and incomplete. One of the program's
eligibility criteria--living in an environment characterized by
disorienting conditions--has not been clearly defined in the statute,
regulations, or Labor's guidance. In addition, Labor has not
provided adequate guidance regarding the requirement that
participants have the capability and aspirations needed to complete
and secure the full benefits of Job Corps. As a result, outreach and
admissions contractors may not be enrolling the applicants who are
most appropriate for the program.
In the absence of specific Labor guidance, we noted that outreach and
admissions practices varied among contractors. Those with higher
participant retention rates tended to have better procedures to
identify applicants who have the capability and aspirations to remain
in and benefit from the program. A particularly effective tool in
preparing applicants for Job Corps appeared to be preenrollment tours
and briefings. Most admissions counselors expressed concern about
the enrollment of 16- and 17-year-old applicants. Labor data confirm
that these youths are more likely to drop out early for disciplinary
reasons and less likely to be placed once they leave the program.
Although Job Corps is a performance-driven program, the measures used
to assess placement performance may not be meaningful and thus may
not provide Labor with the information it needs to accurately assess
placement contractor performance. Labor's system for calculating a
contractor's placement performance included program terminees who
were realistically unemployable. Determining what happens to every
program participant is an important indicator of how well Job Corps
is performing but not necessarily an appropriate measure of a
contractor's placement performance. Guidance related to another
placement measure--the extent to which terminees were placed in
training-related occupations--gave contractors such wide latitude
when deciding whether a job was related to the training received that
the validity of the measurement was questionable. In addition, the
performance measure included terminees who received little vocational
skills training and, therefore, were unlikely to be placed in jobs
requiring an acquired skill. Labor is redesigning the methodology
for determining job-training matches, which may help address some of
these problems. However, any system would still be susceptible to
manipulation by placement contractors without proper oversight and
monitoring.
We noted similarities in the procedures the placement contractors
with higher placement rates used. One common characteristic was that
they all had staff whose sole responsibility was placing program
participants, whereas other contractors had the same staff performing
outreach and admissions functions and providing placement services.
In contrast, five of the six state employment service agencies were
performing below Labor's placement performance standard in program
year 1995. We noted that between program years 1994 and 1996, Labor
did not renew the contracts with 12 of the 18 state employment
service agencies that had Job Corps placement contracts. None of the
placement specialists we interviewed at the three employment service
offices we visited had contact with students before termination, and
it appeared that their primary effort was to register participants in
the employment service databank. At these agencies, it appeared that
Job Corps participants received similar services as regular
employment service clients, raising questions as to why Job Corps is
paying for services that could be obtained free of charge
RECOMMENDATIONS TO THE
SECRETARY OF LABOR
------------------------------------------------------------ Letter :8
To help ensure that Job Corps' resources serve the most appropriate
participants, we recommend that the Secretary of Labor provide clear
and complete guidance on program eligibility criteria, ensuring that
the guidance is consistent with the law, and provide better guidance
to ensure that outreach and admissions contractors assess each
applicant's capability and aspirations to complete training and
attain a positive outcome.
Improvements are also needed to make the measures used to assess
placement contractor performance more meaningful. Therefore, we
recommend that the Secretary of Labor modify certain measures for
placement contractors, including
-- eliminate from the placement pool participants whom contractors
realistically could not or should not be expected to place, such
as participants who were expelled for criminal or violent
behavior;
-- replace the current job-training match system with one that
captures realistic information and provide guidance to regional
offices to ensure that the data are accurately recorded;
-- establish separate placement performance standards for
participants with different levels of program
accomplishment--for example, those who completed program
requirements and those who dropped out early.
AGENCY COMMENTS
------------------------------------------------------------ Letter :9
In Labor's comments on a draft of this report, the agency disagreed
with our recommendation that it clarify and expand its program
eligibility criteria in order to ensure that they are consistent with
the law. Labor stated that our report lacked acknowledgment of the
detailed specifications for eligibility requirements developed over
the years in conjunction with the Office of Inspector General and
that the eligibility, verification, and documentation requirements
contained in its policy handbook are detailed and specifically
related to guidance for Job Corps admissions counselors. Labor gave
no indication of any formal action it planned to take on this
recommendation. Although Labor expressed some concern with our
remaining recommendations, it acknowledged that they have merit,
warrant consideration, and identify actions that the agency would
take in response to them.
Labor's specific concerns with our report are in three broad
areas--adequacy of program eligibility guidance, potential effect of
additional assessment procedures, and recommended changes to
placement performance measures, including training-related
placements. Labor also pointed out a number of items in the draft
report that it believed should be modified or clarified, and we acted
on these, where appropriate. For example, we clarified that our
discussion of the ambiguity of program eligibility guidance related
to only 2 of the 11 criteria. We also made a number of other
technical changes to our report to respond to Labor's comments.
Following is a summary of Labor's concerns and our responses.
Labor's full comments are printed in appendix VI.
ELIGIBILITY GUIDANCE
---------------------------------------------------------- Letter :9.1
Labor stated that our report lacked acknowledgment of the detailed
specifications for eligibility requirements developed over the years
in conjunction with the Office of Inspector General and that the
eligibility, verification, and documentation requirements contained
in its policy handbook are detailed and specifically related to
guidance for Job Corps admissions counselors. Labor expressed
concern with our characterization of the program eligibility guidance
as inadequate. For example, regarding the lack of definition in
Labor's policy handbook for "limited job opportunities," Labor
commented that training conducted in program year 1995 for all
admissions counselors included technical assistance material that
defined this term as follows: "scarcity of jobs, commensurate with
the skill levels of Job Corps-eligible youth and which has been
designated as an area of substantial unemployment." Labor added that
"In essence, any applicant who lacks the specific skills required by
the local labor market to obtain meaningful employment is a
legitimate candidate for Job Corps."
Labor acknowledged that another eligibility factor--cultural
deprivation--is not included in the policy handbook because
more-specific factors--including (1) disruptive homelife, (2) unsafe
or overcrowded dwelling, (3) disruptive community with high crime
rates, and (4) limited job opportunities--were more useful to
admissions counselors than the general term itself. Finally, Labor
expressed concern with our discussion of the tool used in assessing
another eligibility requirement--capability and aspirations.
According to Labor, this assessment by its very nature must rely on
the judgment of admissions counselors and determining aspirations is
very difficult and challenging; Labor stated that the current
assessment tool will be revisited and modified according to
suggestions from regional offices and admissions counselors.
We disagree that sufficient policy guidance defining "limited job
opportunities" was provided to admissions counselors at a training
seminar. Even if all admissions counselors at that time received
such guidance, contractors and staff have since turned over. And, as
mentioned in our report, admissions counselors we interviewed had
different interpretations of "limited job opportunities," indicating
that something more is needed to ensure the consistent interpretation
of limited job opportunities. Because Labor's policy handbook was
created to be "the single document containing all policy and
requirements which would be: clear and concise, and up-to-date, and
consistent with legislative provisions," any definition of "limited
job opportunities" that Labor develops should be incorporated into
this policy handbook. In addition, the law states that environmental
factors substantially impair an individual's ability to succeed in
training, not his or her ability to find employment. But Labor fails
to explain the connection between its definition and the impairment
of ability to succeed in training. And there is a separate
eligibility requirement in the law that the applicant must "require
additional education, training, or intensive counseling and related
assistance in order to secure and hold meaningful employment . . .
." Labor's interpretation of limited job opportunities appears to
duplicate or at least overlap that separate requirement. Finally,
Labor fails to explain how its definition satisfies the program
regulations that stipulate that the environmental criteria are to be
used in the context of residential versus nonresidential programs.
Nowhere in its guidance does Labor mention this distinction.
We also disagree that Labor provided adequate guidance regarding the
term "cultural deprivation." On the Job Corps application form, Labor
not only lists each of the four factors it says define "cultural
deprivation" as separate and distinct eligibility factors (any one of
which would satisfy the eligibility requirement) but also adds the
term "cultural deprivation" as a fifth factor that can be used to
meet program eligibility. Guidance for completing the application
form does not define this term and, as noted in our report, most of
the admissions counselors we spoke with admitted that they did not
know what the term meant. Furthermore, cultural deprivation cannot
include disruptive homelife, as Labor says it does, because the law
lists these as two separate environmental conditions.
Regarding the eligibility requirement that participants have the
capability and aspirations to complete and benefit from Job Corps, we
agree with Labor that making such a determination is very difficult
and challenging and, therefore, we believe that it is important that
admissions counselors have guidance adequate to assist them in making
these judgments. Furthermore, we agree with one regional official's
portrayal of the current assessment tool as a beginning step in
providing guidance on this criterion. Accordingly, we support
Labor's decision to revisit this assessment tool and to obtain
regional office and admissions contractors' suggestions for improving
it.
ASSESSMENT PROCEDURES
---------------------------------------------------------- Letter :9.2
With respect to assessment procedures, Labor agreed that Job Corps
should not enroll youths who obviously have no desire to be in the
program or capability to succeed and that assessing the interest and
ability to benefit are important parts of the intake procedure.
Labor also noted that participants' leaving the program within the
first 2 months is a cost that Job Corps must do whatever it can to
minimize. However, Labor points out the need for a balance between
this goal and the goal of serving youths who truly need the program,
noting that overly strict assessment procedures could be a barrier to
many severely disadvantaged youths. Furthermore, Labor states that
the Congress clearly intended that Job Corps serve a severely at-risk
population. Labor acknowledged that our report contained a number of
positive suggestions (that is, "best practices") that will be made
available to outreach and admissions as well as placement
contractors.
Labor cautioned that the results of our analysis of characteristics
associated with program retention could be misinterpreted because the
report lacks the proper context. Labor further suggested that the
detailed appendix related to this discussion be removed. Finally,
Labor stated that the age data relating to participants who were 15
and 25 years old was inaccurate because Job Corps serves individuals
aged 16 to 24.
While we do not disagree that the program is to target persons most
in need, the law states that the purpose is to assist youths who both
need and can benefit from an intensive program. And the law requires
that enrollees have the capabilities and aspirations to complete and
secure the full benefits of the program. Several Labor regional
directors commented on the importance of identifying applicants who
are ready for Job Corps and can benefit from its training. For
example, one director stated that with more than 6 million people
eligible for Job Corps, admissions counselors have to identify those
most likely to benefit from the program and that commitment should be
first and foremost when they assess applicants. We also note that,
in a previous report, we found that a key element of successful
job-training projects was ensuring that participants are committed to
training and to getting a job.\16 Accordingly, we endorse Labor's
decision to make available to admissions contractors the procedures
noted in our report that help identify the applicants who have the
commitment and motivation to remain in and benefit from the program.
We modified the report to provide our reasons for performing our
analysis of characteristics associated with program retention and to
highlight the limitations associated with our approach as well as the
results. However, we do not believe the detailed appendix should be
eliminated. In addition to describing our analysis and results in
detail, it describes the related limitations. Regarding our mention
of 15- and 25-year-old program participants being inaccurate, we
obtained our data from Labor's national database, which showed that
less than 1 percent of program year 1995 enrollees were either 15 or
25 years old. We have added a relevant footnote.
--------------------
\16 Employment and Training: Successful Projects Share Common
Strategy (GAO/HEHS-96-108, May 7, 1996).
PLACEMENT PERFORMANCE
MEASURES
---------------------------------------------------------- Letter :9.3
Labor expressed concern with our recommendation with respect to
placement performance measures that Job Corps eliminate from the
contractors' placement pool individuals who realistically could not
or should not be expected to be placed, such as those expelled from
the program for using drugs or engaging in violent behavior. Labor
believes that the program has the responsibility to provide placement
services to all participants and that it is not asking placement
contractors to mislead or lie to employers during placement. Labor
further commented that the current placement measure resulted from a
recommendation by its Office of Inspector General that all
participants who leave the program should be included in the
placement pool, thus creating incentives to keep students as long as
possible. Labor acknowledged that the points we made in this portion
of the report merit serious consideration and, therefore, it will
convene a workgroup to discuss our recommendations and examine the
incentives and disincentives resulting from any proposed changes to
the performance management system.
Labor also acknowledged that our report contained "some good points"
with respect to training-related placements but expressed concern
about our use of hypothetical examples of questionable job-training
matches and the lack of data to indicate the degree to which these
occur. Labor also commented that the claim by a contractor about
obtaining a job-training match for participants trained as bank
tellers, secretaries, and welders and placed in fast-food restaurants
is inaccurate, noting that the system does not permit such matches.
Although Labor may not be asking its placement contractors to lie to
or mislead employers when attempting to place individuals who
realistically could not be placed, by holding contractors responsible
for placing individuals expelled for criminal or violent behavior,
the program may be encouraging such practices. We agree with Labor
that determining what happens to every participant is an important
indicator of program performance, but we do not believe that it is
necessarily an appropriate measure of a contractor's placement
performance. We also acknowledge that establishing an effective
performance management system is complex and agree with Labor that,
before any changes are made to this system, the incentives and
disincentives should be thoroughly examined, and we commend Labor for
its proposed action.
We used "hypothetical" examples of job-training matches to illustrate
the wide latitude Job Corps permits. Labor data were not available
to identify the extent of abuse, but as we mentioned in the report,
most placement contractors we interviewed indicated that creativity
is used when entering codes for placement jobs, and they expressed
their concern about the validity of reported job-training match
statistics. In response to Labor's contention that the system does
not permit job-training matches for participants trained as bank
tellers, secretaries, and welders who obtain jobs in fast-food
restaurants, we agree that if such jobs were reported as "fast-food
workers," the system would not permit a job-training match. But, as
a contractor we spoke with pointed out, reporting such jobs in
fast-food restaurants as "cashier" would be an allowable match for
participants trained as bank tellers and secretaries, and reporting
such placements as "machine cleaners" would be an allowable match for
participants trained as welders.
---------------------------------------------------------- Letter :9.4
As arranged with your office, unless you publicly announce the
contents of this report earlier, we plan no further distribution of
it until 15 days after its issue date. We will then send copies to
the Secretary of Labor, the Director of the Office of Management and
Budget, relevant congressional committees, and others who are
interested. Copies will be made available to others on request.
If you or your staff have any questions concerning this report,
please call me at (202) 512-7014 or Sigurd R. Nilsen at (202)
512-7003. GAO contacts and staff acknowledgments are listed in
appendix VII.
Sincerely yours,
Carlotta C. Joyner
Director, Education and
Employment Issues
SCOPE AND METHODOLOGY
=========================================================== Appendix I
We designed our study to identify whether Labor's policy guidance on
eligibility was consistent with legislation and regulations and to
collect information on contractors' practices in enrolling
individuals for the program and in placing them in jobs after they
leave Job Corps. We reviewed Job Corps legislation as well as
Labor's program regulations and policy guidance on program
eligibility, outreach and assessment of individuals for participation
in the program, and placement of participants after termination.
We also interviewed national and regional Job Corps officials and
conducted site visits to 14 outreach, admissions, and placement
contractors. We augmented the information we collected during the
site visits with data from Labor's Student Pay, Allotment, and
Management Information System (SPAMIS), a database containing
nationwide Job Corps data on all Job Corps participants as well as
information on the outreach, admissions, and placement contractors
for each participant. We analyzed program year 1995 enrollee data,
the most recent full program year for which SPAMIS data were
available. While we did not verify the accuracy of Labor's SPAMIS
data, we performed internal validity checks to ensure the consistency
of the database. We performed our work between October 1996 and July
1997 in accordance with generally accepted government auditing
standards.
SITE VISITS
We visited 14 Job Corps outreach and assessment and placement
contractors. We selected the sites judgmentally to provide a mixture
of contractors that were private contractors, Job Corps centers, and
state agencies. We also selected contractors that provided both
outreach and assessment services and placement services or that
provided only one of these services. In addition, we considered past
contractor performance in making our selections. We selected
contractors located in 5 of Labor's 10 regions to provide some
regional management diversity and geographic dispersion and to allow
us to visit multiple contractors during individual trips.
In making our site selections, we identified contractors that had
outreach and admissions or placement contracts with Labor during
program years 1994 and 1995 and that were still under contract in
program year 1996. This provided us with contractors that had
multiyear program experience and were currently under contract with
Job Corps. In order to select among the larger contractors, we
included only contractors who enrolled or were responsible for
placing at least 150 participants each year. We then ranked outreach
and admissions contractors according to the percentage of program
year 1995 enrollees who stayed in the program for more than 30 days
and placement contractors according to the percentage of program year
1995 terminees placed in jobs, school, the military, or other
training.\17 We then selected contractors from among the top, middle,
and bottom third of these rankings. Table I.1 lists the contractors
we visited and their characteristics.
Table I.1
Outreach, Admissions, and Placement
Contractors We Visited
Outreach
and
Labor Contractor admissions Placement
region Location Contractor type ranking\a ranking\b
----------- ----------- ----------- ----------- ----------- ---------------
IV Kittrell, Kittrell Center Top third Top third
N.C. Job Corps
Center
Raleigh, North State Bottom Not a placement
N.C. Carolina third contractor
Department
of Human
Resources
V St. Paul, Dynamic Private Top third Middle third
Minn. Educational
Systems,
Inc./
Hubert H.
Humphrey
Job Corps
Center\c
VI Dallas, Dynamic Private Did not Bottom third
Tex. Educational meet
Systems, selection
Inc. criteria
El Paso, Education Center Top third Top third
Tex. Foundation/
David L.
Carrasco
Job Corps
Center
New New Orleans Center Middle Bottom third
Orleans, Job Corps third
La. Center
Oklahoma Oklahoma State Bottom Bottom third
City, Okla. Employment third
Security
Commission
Austin, Texas State Middle Bottom third
Tex. Workforce third
Commission
IX Sacramento, Nero Private Not an Top third
Calif. Support outreach
Services and
admissions
contractor
Sacramento, Sacramento Center Middle Top third
Calif. Job Corps third
Center
San Jose, San Jose Center Did not Top third
Calif. Job Corps meet
Center selection
criteria
Carson State of State Middle Bottom third
City, Nev. Nevada third
Department
of
Employment,
Training,
and
Rehabilitat
ion
San Women In Private Middle Not a placement
Francisco, Community third contractor
Calif. Service
X Seattle, Del Jen, Private Top third Top third
Wash. Inc.
--------------------------------------------------------------------------------
\a Ranking based on percentage of program year 1995 enrollees staying
in the program for at least 30 days.
\b Ranking based on percentage of program year 1995 assigned
terminees placed in job, school, military, or other training.
\c Hubert H. Humphrey Job Corps Center subcontracts with Dynamic
Educational Systems, Inc., to perform outreach, admissions, and
placement.
We visited 11 outreach and admissions contractors from which varying
percentages of program year 1995 enrollees left the program within
the first 30 days. As shown in figure I.1, the percentages ranged
from about 1 percent for one contractor's enrollees to nearly 20
percent for another's.
Figure I.1: Percentage of
Program Year 1995 Enrollees
Leaving Within 30 Days for
Selected Contractors
(See figure in printed
edition.)
We also selected 12 placement contractors to visit that had varying
success in placing Job Corps participants in program year 1995. As
shown in figure I.2, placement rates ranged from about 54 percent to
about 85 percent.
Figure I.2: Program Year 1995
Placement Rates for Selected
Contractors
(See figure in printed
edition.)
INTERVIEWS WITH CONTRACTORS
To obtain information on how contractors enroll individuals in Job
Corps and place them after their termination, we interviewed
contractor personnel using a semistructured interview protocol. We
asked outreach and admissions contractors questions related to their
practices and procedures in attracting youths to Job Corps and in
screening applicants. We also asked about their understanding and
implementation of program eligibility criteria as specified by Labor
and about their views on what affects program retention. We
questioned placement contractors on their procedures in placing
terminees in jobs, the military, or other training; the types of
services they provided to terminees; and their practices when
deciding whether a placement is a job-training match. We asked both
groups of contractors about their views on current Labor performance
standards related to recruitment and placements and their opinions on
improvements needed in the Job Corps program. At three centers
(David L. Carrasco, Kittrell, and Sacramento), we also interviewed
Job Corps participants (approximately six from each center) to learn
about their experiences when they were recruited for Job Corps and to
obtain their views about the enrollment process.
NATIONAL AND REGIONAL JOB CORPS
OFFICES
We interviewed Labor officials at national and selected regional
offices to obtain an overview of Job Corps enrollment, placement, and
contracting. We also obtained information on Labor's policy guidance
on eligibility and how it relates to the Job Corps legislation;
outreach, admissions, and placement contractors' performance; and the
program's performance management system. In addition, we reviewed
Labor's Policy and Requirements Handbook, which was designed to
include all program policy and requirements concerning eligibility
criteria and policies and standards related to program enrollment and
participant placement.
DATA ANALYSIS
We analyzed Job Corps participant retention data, reasons for
termination, and placement information for program year 1995. We
used 30-day retention data, part of Labor's standard for evaluating
outreach and admissions contractor performance, as a basis for
selecting outreach and admissions contractors to visit. We expanded
our analysis of retention beyond the 30-day standard and determined
how many terminees left Job Corps within 60 days of enrollment in
order to look at retention beyond the realm of outreach and
admissions contractor performance. We also used one of Labor's
placement standards--the extent to which terminees are placed in
jobs, the military, school, or other training--as a basis for
selecting placement contractors to visit. Furthermore, we used the
data from our analysis to supplement information obtained in
discussions with admissions counselors and placement specialists.
--------------------
\17 One of Labor's standards for measuring the performance of
outreach and admissions contractors is the extent to which enrollees
remain in Job Corps more than 30 days. The current standard is that
90 percent of enrollees arriving at the center will remain more than
30 days.
LABOR'S JOB CORPS ELIGIBILITY
CRITERIA
========================================================== Appendix II
Labor's Policy and Requirements Handbook includes 11 eligibility
requirements for enrollment in Job Corps. As noted below, three of
these requirements--child care, parental consent, and Selective
Service registration--do not apply to all applicants.
1. Age
-- at least 16 but not yet 25 years old at enrollment
-- no upper age limit for those who are physically or mentally
disabled
2. Selective Service registration
-- all male applicants, who must sign a consent form authorizing
the Selective Service System to register them automatically at
age 18
3. Legal U.S. residency
-- a U.S. citizen or national, including naturalized citizens, or
-- a lawfully admitted permanent resident alien, refugee, parolee,
or other alien permitted to accept permanent employment in the
United States or
-- a resident of a U.S. territory or
-- a Canadian-born American Indian ("Jay Treaty Indian")
4. Economic disadvantage
-- an individual receiving or member of a family receiving cash
welfare payments, government-provided medical assistance, or
food stamps or
-- a foster child for whom state or local government payments are
made or a ward of the state or court or
-- an individual with physical or mental disabilities that present
barriers to obtaining employment and whose own income meets the
income criteria or
-- an individual or member of a family receiving total family
income not in excess of the higher of the poverty level or 70
percent of the lower living standard income level
5. Requirement for additional education or training
-- a school dropout or
-- an individual in need of additional education, vocational
training, or related support services in order to hold
meaningful employment, participate successfully in regular
school work, qualify for other suitable training programs, or
satisfy armed forces requirements
6. Environment
-- an individual living in an environment characterized by
disruptive home life, unsafe, overcrowded dwelling; limited job
opportunities; or disruptive community, high crime rates
7. Health history
-- a drug-free individual also free of any health condition
(medical, mental, emotional, or dental) that represents a
potentially serious hazard to youths or others, precludes
participation in Job Corps with a reasonable expectation of
successful completion followed by employment, or requires
intensive or costly treatment
-- an individual with a number of other health conditions that
Labor's policy handbook specifies for consideration
-- all applicants with disabilities, who must be referred to the
regional office for evaluation and determination of eligibility
and assessed for appropriate assignment to centers equipped to
handle each particular disability
8. Behavioral adjustment history
-- an individual free of behavioral problems so serious that the
applicant cannot adjust to the standards of conduct, discipline,
work, and training required or would prevent others from
benefiting from the program or requires face-to-face court
supervision or court-imposed financial obligations
-- a youth on probation or parole or under other supervision as a
result of court action, who may be eligible only if the agency
with jurisdiction states that the youth has responded positively
to supervision, will permit the applicant to leave the local
area or state, and will not require personal, face-to-face
supervision during participation in the program
-- all applicants, who must sign the zero tolerance for violence
certification
9. Child care
-- all applicants with dependent children who provide primary or
custodial care, who must have established suitable child care
arrangements
10. Parental consent
-- youths who have not reached the age of majority as defined by
state law, who must have parental or legal guardian consent to
participate
11. Capability and aspirations to participate
-- all applicants, who must have the capability and aspirations to
complete and secure the maximum benefits of Job Corps
(See figure in printed edition.)Appendix III
LABOR'S CAPABILITY AND ASPIRATIONS
ASSESSMENT TOOL
========================================================== Appendix II
(See figure in printed edition.)
(See figure in printed edition.)
ANALYSIS OF THE RELATIONSHIP
BETWEEN PARTICIPANT
CHARACTERISTICS AND THE LIKELIHOOD
OF REMAINING IN JOB CORPS FOR AT
LEAST 60 DAYS
========================================================== Appendix IV
In our analysis, we examined the relationship between the
characteristics of Job Corps participants and the likelihood of their
remaining in the program for at least 60 days. We used the data that
were available from Labor's Student Pay, Allotment, and Management
Information System (SPAMIS) on the characteristics of the more than
68,000 participants enrolled in Job Corps during program year 1995.
We performed a three-stage analysis resulting in a logistic
regression model that used these characteristics to predict the odds
of a participant's remaining in the program for at least 60 days.
While the information from our analysis provides some indication of
whether participants with specific characteristics will remain in Job
Corps for at least 60 days, we do not intend to imply that only
individuals with these characteristics should be enrolled in the
program or that outreach and assessment efforts should be focused on
them. Rather, this information is a source of insight into early
program attrition for Labor's use in Job Corps management. We also
recognize that being in the program for at least 60 days indicates
only longevity, not necessarily success.
For our initial exploration of the data, we selected the participant
characteristics from SPAMIS that appeared to be conceptually relevant
to the likelihood of remaining in the program for at least 60 days.
These included age at enrollment, distance between a participant's
home and the assigned Job Corps center, and educational status. We
first used crosstabulations to examine the relationship of these
variables to whether the participant remained in the program for 60
days. The chi-square statistics from these analyses showed the
variables that seemed to exhibit no relationship to 60-day retention
and helped us eliminate certain characteristics and select a set of
variables for further analysis.\18 The set of variables we selected
is shown in table IV.1.
Table IV.1
Percentage of Participants Remaining in
Job Corps for at Least 60 Days by
Selected Characteristics
Did not remain
Remained at at least 60
least 60 days days
-------------- --------------
Signif
icance
of
chi-
Percen Percen square
Characteristic Number tage Number tage \a
------------------------------ ------ ------ ------ ------ ------
Age at enrollment .00
15-17 19,148 72% 7,539 28%
18-20 21,650 75 7,109 25
21-25 9,815 78 2,829 22
Need for bilingual education .00
Spanish-English 1,067 87 163 13
Other-English 621 93 48 7
No need for bilingual 48,941 74 17,272 26
education
Distance from home to center .00
Less than 50 miles 16,180 78 4,710 22
50-149 miles 10,979 73 4,108 27
150-299 miles 10,556 72 4,032 28
300 miles or more 10,301 72 4,029 28
High school diploma .00
No 38,663 73 14,591 27
Yes 11,966 80 2,892 20
Last school grade completed .00
0-8 7,324 68 3,368 32
9-11 31,148 74 11,177 26
12-15 12,157 80 2,938 20
Participant has dependents .92
No 44,380 74 15,325 26
Yes 6,210 74 2,139 26
Months out of school .00
0-2 13,026 75 4,245 25
3-6 9,425 74 3,383 26
7-12 9,659 74 3,436 26
Over 12 18,519 74 6,419 26
Prior conviction .00
No 48,671 74 16,665 26
Yes 1,958 70 818 30
Prior military service .00
No 50,217 74 17,385 26
Yes 412 81 98 19
Size of participant's home .00
city or town
Fewer than 2,500 5,361 74 1,937 26
2,500-9,999 5,627 72 2,238 28
10,000-49,000 10,301 74 3,667 26
50,000-249,000 8,781 74 3,160 26
250,000 or more 20,559 76 6,481 24
Resident at the center .00
No 7,384 80 11,814 20
Yes 43,245 73 15,669 27
----------------------------------------------------------------------
\a We used the chi-square test of independence to test for a
relationship between remaining in the program at least 60 days and
the student characteristic examined. The chi-square significance
represents the probability that no relationship exists. For
instance, the probability that no relationship exists between age and
remaining at least 60 days is less than 1 in 100.
With these variables, we then performed a bivariate logistic
regression to estimate the effects of each individual factor on
remaining in Job Corps for at least 60 days. The results from the
regression models are stated as odds ratios, which tell us how much
more likely participants with certain characteristics are to remain
in Job Corps for at least 60 days than participants without those
characteristics. We give a chi-square test of significance for each
of these odds ratios.
To calculate the odds of a specific group remaining in Job Corps for
at least 60 days, the percentage remaining and not remaining must be
determined. For example, 26,687 participants aged 15-17 enrolled in
Job Corps during program year 1995. As shown in table IV.1, 19,148
of these participants remained in the program for at least 60 days,
while 7,539 did not. The odds of 15-17-year-old participants
remaining in the program for at least 60 days were calculated by
dividing the number remaining (19,148) by the number not remaining
(7,539). Therefore, the odds for this group's remaining were 2.54,
meaning that 2.54 individuals remained for every 1 who did not.
Similar calculations for participants aged 18 to 20 and 21 to 25
yield higher odds of 3.04 and 3.47, respectively.
The logistic regression model provides us with odds ratios that tell
us how different the odds are for each group and whether the
differences are statistically significant. For example, when the
15-17-year-old group is used as a benchmark for comparing the two
other groups, the resultant odds ratios are 3.04/2.54 = 1.20 and
3.47/2.54 = 1.37 for participants aged 18 to 20 and 21 to 25,
respectively. Thus, the odds of 18-20-year-old participants
remaining in Job Corps at least 60 days are 1.20 times the odds of
15-17-year-old participants, and the odds of 21-25-year-old
participants remaining are 1.37 times the odds of 15-17-year-old
participants. Odds ratios that deviate from 1.0 the most, in either
direction, represent the most sizable effects (for example, odds
ratios of 0.5 and 2.0 represent effects that are similar in size,
since 0.5 indicates that one group is half as likely as the other to
remain in the program for at least 60 days, while 2.0 indicates that
one group is twice as likely as the other to remain). We performed
this type of bivariate analysis for each characteristic we selected.
The resulting odds ratios are shown under the "bivariate results"
column of table IV.2.
Table IV.2
Bivariate and Multivariate Effects of
Various Factors on the Odds of Remaining
in Job Corps for at Least 60 Days
Odds ratio
--------------
Bivari Multiv
ate ariate
Independent variable result result
------------------------------------------------------ ------ ------
Age at enrollment
18-20 vs. 15-17 1.20* 1.16*
21-25 vs. 15-17 1.37* 1.27*
Need for bilingual education
Spanish-English vs. no need 2.31* 1.90*
Other-English vs. no need 4.48* 3.13*
Distance from home to center
Less than 50 miles vs. 300 miles or more 1.34* 1.15*
50-149 miles vs. 300 miles or more 1.05 1.03
150-299 miles vs. 300 miles or more 1.02 1.00
High school diploma
Yes vs. no 1.56* 1.02
Last school grade completed
0-8 vs. 12-15 0.53* 0.57*
9-11 vs. 12-15 0.67* 0.72*
Participant has dependents
No vs. yes 1.00 1.27*
Months out of school
0-2 vs. over 12 1.06* 1.39*
3-6 vs. over 12 0.97 1.17*
7-12 vs. over 12 0.97 1.06*
Prior conviction
Yes vs. no 0.82* 0.94
Prior military service
Yes vs. no 1.46* 1.28*
Size of participant's home city or town
2,500-9,999 vs. under 2,500 0.91* 0.93
10,000-49,000 vs. under 2,500 1.02 0.95
50,000-249,000 vs. under 2,500 1.00 0.97
250,000 or over vs. under 2,500 1.15* 1.07*
Resident at the center
No vs. yes 1.47* 1.20*
----------------------------------------------------------------------
* Statistical significance = .05.
Note: We also included two additional characteristics
(race-ethnicity and gender) in our analysis to ensure that we had
used all available and relevant data. However, for Labor to use
these characteristics to distinguish between applicants would raise
serious legal concerns because, in various rulings, the Supreme Court
has made clear that using race or gender as a basis on which to treat
people differently is unconstitutional unless stringent conditions
are met. We have, therefore, not reported the coefficients for these
characteristics. Although federal law generally prohibits
discrimination in federally funded programs based on a third
characteristic--age--this characteristic may be considered by Job
Corps because the program legislation itself makes age a factor.
After performing the bivariate analysis, we used the same set of
variables in a multivariate logistic regression analysis, identical
to the bivariate analysis except that it provides estimates of the
effects of each characteristic on the likelihood of remaining in the
program for at least 60 days while holding constant, or controlling
for, the effects of the other characteristics. We included all
factors (and levels of factors), even if their effects were not
statistically significant in the bivariate analysis because, in some
cases, effects that are suppressed in bivariate analysis emerge as
significant in multivariate analysis. Similarly, effects that were
significant in the bivariate analysis may be insignificant in the
multivariate analysis. The results of the multivariate logistic
regression are shown in column 2 of table IV.2 ("multivariate
result").
As this column shows, when we entered all variables into the model,
some variables and levels of variables had odds ratios that were not
significantly different from the reference category.\19 We dropped
these variables or, in cases in which levels of variables were not
significantly different from other levels within the same variable,
we combined levels. For instance, in the multivariate model, the
odds of remaining in Job Corps for at least 60 days for participants
having a prior conviction were not significantly different from the
odds of remaining for participants not having had a conviction. As
shown in table IV.2, the odds ratio of .94 is not statistically
significant. Therefore, we dropped this variable from subsequent
analysis. Similarly, the odds of remaining for two levels of the
variable "distance from home to center" (50-149 miles and 150-300
miles) were not significantly different from the odds of the
reference category (over 300 miles). Therefore, we combined these
two levels with the reference category to create a two-level variable
for subsequent analysis. Thus, we included in the final model only
the variables, and levels of variables, that were shown to be
significant in the previous multivariate analysis.
The results of this final model, as well as statistics related to how
well the model performs, are shown in table IV.3. Model performance
can be measured by the likelihood ratio method, which evaluates the
probability of the observed results, given the parameter estimates.
These results are shown under the -2 Log Likelihood (-2LL) entries in
the note to table IV.3. As shown, the model containing the predictor
variables shows an improved (smaller) -2LL compared with the model
containing only the constant (that is, the model that assumes no
differential effects resulting from individual variables). The model
chi-square, which tests that the coefficients for all the terms in
the model (except the constant) are 0 (that is, the null hypothesis),
was significant at the .0000 level.
Table IV.3
Final Multivariate Model of Effects of
Various Factors on the Odds of Remaining
in Job Corps for at Least 60 Days
Odds
Independent variable ratio
-------------------------------------------------------------- ------
Age at enrollment
18-20 vs. 15-17 1.15*
21-25 vs. 15-17 1.27*
Need for bilingual education
Spanish-English vs. no need 1.90*
Other-English vs. no need 3.15*
Distance from home to center
Less than 50 miles vs. 50 miles or more 1.14*
Last school grade completed
12-15 vs. 0-8 1.82*
12-15 vs. 9-11 1.41*
Participant has dependents
No vs. yes 1.27*
Months out of school
0-2 vs. over 12 1.39*
3-6 vs. over 12 1.17*
7-12 vs. over 12 1.06*
Prior military service
Yes vs. no 1.28*
Size of participant's home city or town
Over 250,000 vs. under 250,000 1.11*
Resident at the center
No vs. yes 1.20*
----------------------------------------------------------------------
* Statistical significance = .05.
Note: We also included two additional characteristics
(race-ethnicity and gender) in our analysis to ensure that we had
used all available and relevant data. However, for Labor to use
these characteristics to distinguish between applicants would raise
serious legal concerns because, in various rulings, the Supreme Court
has made clear that using race or gender as a basis on which to treat
people differently is unconstitutional unless stringent conditions
are met. We have, therefore, not reported the coefficients for these
characteristics. Although federal law generally prohibits
discrimination in federally funded programs based on a third
characteristic--age--this characteristic may be considered by Job
Corps because the program legislation itself makes age a factor.
Model chi-square: chi-square, 1651.166; degree of freedom, 18;
significance, .0000
Improvement: chi-square, 1651.166; degree of freedom,18;
significance, .0000
Goodness-of-fit statistics
-2 Log Likelihood initial model (constant only), 74294.86
-2 Log Likelihood final model, 72643.70
Goodness of fit, 64973.69
--------------------
\18 In some cases, we suspected that variables that showed no
relationship in bivariate analysis might be important in multivariate
analysis. In these cases, we retained the variable for subsequent
analysis.
\19 The reference category is the one category against which other
categories are compared. For example, the reference category for age
in table IV.2 is 15-17 years old. In this instance, 18-20-year-old
participants are compared to those 15-17 years old, as are
21-25-year-old participants.
RESULTS OF MULTIVARIATE
ANALYSIS
------------------------------------------------------ Appendix IV:0.1
The results of our multivariate analysis revealed that older
participants have greater odds of remaining in the program 60 days or
more. When compared to 15-17-year-old participants, those aged 18 to
20 and 21 to 25 had odds of remaining that were 15-percent and
27-percent greater, respectively. In addition, we found that
participants with 12 or more years of school had about 80-percent
greater odds of remaining in Job Corps for at least 60 days than
participants with 8 years or less of school. (See table IV.3.)
We also found a relationship between the need for bilingual education
and the likelihood of remaining in the program for at least 60 days.
Of the variables we examined, the need for bilingual education
yielded the highest odds ratio. Spanish-speaking participants
needing bilingual training had odds of remaining that were almost
twice the odds of those not needing bilingual education. Other
non-English-speaking participants needing bilingual assistance had
odds that were more than 3 times the odds of those not needing
bilingual education.
LIMITATIONS OF THE ANALYSIS
------------------------------------------------------ Appendix IV:0.2
Our attempt to construct a model for predicting the characteristics
of participants who are more likely to remain in the program for at
least 60 days was limited by the variables available to us in Labor's
SPAMIS extracts. Most of these variables were demographic
characteristics. We were unable to include in the analysis measures
of such things as student ability, attitude, and motivation, as well
as other characteristics that could potentially affect the likelihood
of participants remaining in the program for at least 60 days.\20
Additionally, the factors that proved to be the most useful
predictors of remaining in the program for at least 60 days were
characteristics of small subsets of participants. For example, there
is evidence that participants in need of bilingual education are more
likely to remain, but this group made up less than 3 percent of the
Job Corps population. Similarly, participants who had completed 12
years or more of school had odds of remaining that were more than
80-percent greater than those of participants who completed 8 or
fewer grades, but almost two-thirds of the participants were in
neither of these groups. Consequently, while the model is very
useful in predicting whether participants with specific
characteristics will remain in Job Corps for at least 60 days, the
model's ability to predict 60-day retention for the program's full
population is limited because we found no large subgroups with great
differences. Finally, in this analysis, we examined only main
effects for the variables we investigated. An examination of the
interactions among the variables might produce useful information and
improve the predictive ability of the model.
--------------------
\20 Although SPAMIS files include tests of adult basic educational
skills, we were unable to include these scores in our analysis
because of problems with the data in the files we received.
DATA SUPPORTING REPORT FIGURES
=========================================================== Appendix V
Table V.1
Percentage of Program Year 1995
Terminees Not Placed by Age (Data for
Fig. 1)
Not
Age at termination placed
-------------------------------------------------------------- ------
16 44.9%
17 36.1
18 30.5
19 28.4
20 24.8
21 24.5
22 23.5
23 23.3
24 22.7
25+ 14.9
----------------------------------------------------------------------
Table V.2
Comparison of Program Year 1995
Placement Rates for Selected Contractors
(Data for Fig. 2)
Placed Placed
using using
existing modified
methodolo methodolo
Placement contractor gy gy
------------------------------------------------ --------- ---------
A 83.7% 87.3%
B 65.1 74.3
C 64.3 76.3
D 54.2 65.6
E 76.5 86.6
F 63.4 69.2
G 73.5 87.1
H 78.3 82.9
I 73.2 82.3
J 80.1 84.5
K 59.6 62.2
L 84.7 89.5
----------------------------------------------------------------------
Table V.3
Percentage of Program Year 1995
Enrollees Staying in Program for Less
Than 30 Days (Data for Fig. I.1)
Enroll
Outreach and admissions contractor ees
-------------------------------------------------------------- ------
A 1.2%
B 12.9
C 18.6
D 10.6
E 12.0
F 12.1
G 10.1
H 7.5
I 10.0
J 12.3
K 17.9
----------------------------------------------------------------------
Table V.4
Program Year 1995 Placement Rates (Data
for Fig. I.2)
Enroll
ees
Placement contractor placed
-------------------------------------------------------------- ------
A 83.7%
B 65.1
C 64.3
D 54.2
E 76.5
F 63.4
G 73.5
H 78.3
I 73.2
J 80.1
K 59.6
L 84.7
----------------------------------------------------------------------
(See figure in printed edition.)Appendix VI
COMMENTS FROM THE DEPARTMENT OF
LABOR
=========================================================== Appendix V
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
GAO CONTACTS AND STAFF
ACKNOWLEDGMENTS
========================================================= Appendix VII
GAO CONTACTS
Sigurd R. Nilsen, Assistant Director, (202) 512-7003
Wayne Sylvia, Evaluator-in-Charge, (617) 565-7492
ACKNOWLEDGMENTS
In addition to the contacts named above, the following persons made
important contributions to this report: Thomas N. Medvetz, Wayne
Dow, Deborah Edwards, Jeremiah Donoghue, Robert Crystal, and Sylvia
Shanks.
RELATED GAO PRODUCTS
=========================================================== Appendix 0
Job Corps: Where Participants Are Recruited, Trained, and Placed in
Jobs (GAO/HEHS-96-140, July 17, 1996).
Employment Training: Successful Projects Share Common Strategy
(GAO/HEHS-96-108, May 7, 1996).
Job Corps: Comparison of Federal Program With State Youth Training
Initiatives (GAO/HEHS-96-92, Mar. 28, 1996).
Job Corps: High Costs and Mixed Results Raise Questions About
Program's Effectiveness (GAO/HEHS-95-180, June 30, 1995).
*** End of document. ***