Job Training Partnership Act: Long-Term Earnings and Employment Outcomes
(Letter Report, 03/04/96, GAO/HEHS-96-40).
The federal government spends billions of dollars annually to support
employment training programs, but little is known about their long-term
effect on participants' earnings and employment rates. GAO analysis
found some positive effects of the Job Training Partnership Act--the
cornerstone of the federal employment training effort--in the years
immediately following training. However, neither employment rates nor
earnings were significantly higher for participants than for
nonparticipants five years after training. In some earlier years, adults
(but not youth) who received training had earnings or employment rates
significantly higher than those of the control group. By the fifth year,
each of the four treatment groups had earnings and employment rates that
were nominally higher than those of the control group. Because none of
the fifth-year differences were statistically significant, however, GAO
could not attribute the higher earnings to training provided under the
act rather than to chance alone.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: HEHS-96-40
TITLE: Job Training Partnership Act: Long-Term Earnings and
Employment Outcomes
DATE: 03/04/96
SUBJECT: Youth employment programs
Labor statistics
Employment or training programs
Disadvantaged persons
Proposed legislation
Vocational rehabilitation
Education or training
Income statistics
Unemployment rates
IDENTIFIER: CAREERS Act
JTPA
Job Training Partnership Act Program
National JTPA Study
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to Congressional Requesters
March 1996
JOB TRAINING PARTNERSHIP ACT -
LONG-TERM EARNINGS AND EMPLOYMENT
OUTCOMES
GAO/HEHS-96-40
JTPA Outcomes
(104778)
Abbreviations
=============================================================== ABBREV
BLS - Bureau of Labor Statistics
CPS - Current Population Survey
JTPA - Job Training Partnership Act
NJS - National JTPA Study
SDA - service delivery area
SSA - Social Security Administration
Letter
=============================================================== LETTER
B-256749
March 4, 1996
The Honorable Paul Simon
United States Senate
Dear Senator Simon:
Although the federal government spends billions of dollars annually
to support employment training programs, little is known about their
long-term effects on participants' earnings and employment rates.\1
Few training programs have been rigorously evaluated to assess their
net impact, and, for those that have, the research results have often
been inconclusive. Furthermore, most research on training programs
has focused on short-term effects--the year or two immediately
following completion of training.
Enacted in 1982, title II of the Job Training Partnership Act (JTPA)
has been the cornerstone of federal employment training programs,
providing block grants to state and local governments to administer
these federally funded programs. JTPA supports job training for
individuals facing barriers to employment and needing special
training to obtain productive employment. Under recent legislative
proposals\2 to consolidate multiple federally funded training
programs, states would have the flexibility to design and implement a
statewide approach to job training based on the concept of one-stop
career centers. As states design and implement their approaches to
job training, lessons learned from JTPA can help in reallocating
training dollars and in setting performance standards.
Because of concerns about the long-term impact of job training
programs, you asked us to (1) ascertain the long-term earnings of
participants in JTPA-sponsored programs and to compare their earnings
with those of nonparticipants and (2) calculate the long-term
employment rates for these JTPA participants and to compare their
employment rates with those of nonparticipants. For this study, we
defined long-term earnings and employment rates as the annual
earnings and employment rates achieved in the fifth year after
applying for JTPA training.
To develop this information, we merged data from the National JTPA
Study\3 (NJS) with annual earnings records from the Social Security
Administration (SSA). Participants in the NJS were randomly assigned
to either the treatment group (allowed to enroll in JTPA training) or
the control group (not allowed to enroll in JTPA training for 18
months). We calculated the average earnings and employment rates\4
of four target groups (adult men, adult women, male youths, and
female youths\5 ) for 5 years after their acceptance into the study.
We considered a 5-year period sufficient to provide meaningful data
on the long-term effects of JTPA training.
Because of the NJS' inherent design problems, we cannot unequivocally
use our findings to draw conclusions about JTPA's effectiveness. For
example, the participating areas were neither randomly selected nor
necessarily representative of JTPA training nationally. Moreover,
not everyone assigned to the treatment group enrolled in or completed
JTPA training. Also, many control group members received some
training services from other sources. Nevertheless, the NJS had some
design advantages, such as random assignment of applicants for JTPA
training, that made the data appropriate for our study.
Appendix I has further information on the NJS and SSA data sets and
our statistical results. We conducted our work between April 1994
and January 1996 in accordance with generally accepted government
accounting standards.
--------------------
\1 Multiple Employment Training Programs: Most Federal Agencies Do
Not Know If Their Programs Are Working Effectively (GAO/HEHS-94-88,
Mar. 2, 1994).
\2 Both the Senate and the House have passed bills that would
consolidate over 90 federal education, employment, and job training
programs. The Senate's Workforce Development Act of 1995 (S. 143)
would replace the programs with one block grant to each state, while
the House's Consolidated and Reformed Education, Employment, and
Rehabilitation Systems Act (CAREERS Act, H.R. 1617) would authorize
three consolidation grants to each state.
\3 The Department of Labor commissioned Abt Associates to conduct the
National JTPA Study in 1986 to evaluate the impact of JTPA on adults
and youths.
\4 We defined employment on the basis of SSA earnings records. If a
person's SSA earnings record showed positive earnings in a given
calendar year, we considered that person to be employed. If 78 out
of 100 people in a group showed positive earnings in their SSA
earnings records, the employment rate for that group would be 78
percent.
\5 Adults were defined as those 22 years old and older. Youths were
defined as those aged 16 to 21 who no longer attended school when
applying for JTPA training. This included both high school dropouts
and graduates.
RESULTS IN BRIEF
------------------------------------------------------------ Letter :1
Although our statistical analysis showed some positive effects of
JTPA in the years immediately following training, we found no
significant\6 effect of JTPA on earnings or employment rates after 5
years. In some earlier years, adult men and women who received
training--but not male or female youths--had earnings or employment
rates significantly higher than those of the control group. By the
fifth year, each of the four treatment groups had earnings and
employment rates that were nominally higher than those of the control
group. Because none of the fifth-year differences were statistically
significant, however, we could not attribute the higher earnings to
JTPA training rather than to chance alone.
--------------------
\6 "Significance" refers to statistical significance at the 5-percent
level. This significance means that we can be 95-percent confident
that the observed difference between groups is not due to chance or
random variation. Our convention is to calculate significance at the
5-percent level. We use this convention both in reference to other
reports and in presenting our own findings.
BACKGROUND
------------------------------------------------------------ Letter :2
Enacted in 1982, JTPA is the largest federal employment training
program, with titles II-A and II-C intended to prepare economically
disadvantaged adults and youths, respectively, for entry into the
labor force.\7 JTPA emphasizes state and local government
responsibility for administering federally funded job training
programs. In fiscal year 1995, JTPA title II-A and II-C programs
received approximately $1.6 billion in funding.
JTPA training programs annually provide employment training for
specific occupations and services, such as job search assistance and
remedial education, to roughly one million economically disadvantaged
individuals. Training is provided in local service delivery areas
(SDA) through service providers, such as vocational-technical high
schools, community colleges, proprietary schools, and community-based
organizations. The program objectives are to increase earnings and
employment and to reduce welfare dependence for participants of all
ages. During the NJS, participation in JTPA involved roughly 3 to 4
months of training at an average cost of about $2,400 per
participant.
In 1986, Labor commissioned the NJS to evaluate the impact of JTPA on
adults and youths because previous findings on the effects of job
training programs had been hampered by poor data and statistical
problems. The NJS randomly assigned persons who sought JTPA
services, and were eligible for them, to a treatment group or a
control group. The treatment group was offered JTPA training, and
the control group was not. The study was intended to ensure that the
two groups would not differ systematically in any way except access
to the program, so any subsequent differences in outcomes could be
attributed solely to JTPA.
The study included over 20,000 eligible participants who applied for
JTPA services between November 1987 and September 1989 in 16 local
SDAs. The study followed up on a sample of people in the treatment
and control groups 18 months after assignment\8 and then again at 30
months.\9
The NJS showed mixed results on the impact of JTPA programs.\10
Adult women assigned to JTPA training had significantly higher
earnings than the control group of adult women after 18 and 30
months, but the treatment group of adult men, as well as of both male
and female youths, did not have significantly higher earnings than
its respective control groups.
--------------------
\7 JTPA title II-A was originally targeted to both adults and youths.
The 1992 JTPA Amendments split title II-A into two components: title
II-A for adults and title II-C for youths.
\8 See Howard S. Bloom and others, The National JTPA Study: Title
II-A Impacts on Earnings and Employment at 18 Months (Bethesda, Md.:
Abt Associates, Inc., Jan. 1993).
\9 See Larry L. Orr and others, The National JTPA Study: Impacts,
Benefits, and Costs of Title II-A (Bethesda, Md.: Abt Associates,
Inc., Mar. 1994) for the results at 30 months.
\10 Bloom and others, The National JTPA Study: Title II-A Impacts on
Earnings and Employment at 18 Months and Orr and others, The National
JTPA Study: Impacts, Benefits, and Costs of Title II-A.
PARTICIPANT EARNINGS NOT
SIGNIFICANTLY GREATER THAN
CONTROL GROUP EARNINGS AFTER 5
YEARS
------------------------------------------------------------ Letter :3
Participants assigned to receive JTPA training did not have
significantly greater earnings than control group members 5 years
after their assignment. For some of the four targeted worker
categories--adult men, adult women, male youths, and female
youths--treatment group earnings exceeded those of the control group
in some of the intervening years, but any statistically significant
effects disappeared by the fifth year.\11
--------------------
\11 Generally, our results matched the findings of the NJS for the 30
months immediately following assignment.
EARNINGS OUTCOMES FOR ADULT
MEN
---------------------------------------------------------- Letter :3.1
Annual earnings of adult men increased in each year following
assignment for both the treatment and control groups. As shown in
figure 1, in the first year after assignment, the average annual
earnings of adult men in the treatment group grew from about $4,400
to about $6,900. This group's earnings continued to rise in the
subsequent years, reaching approximately $8,700 in the fifth year
after assignment to receive JTPA training. The earnings of adult men
in the control group, which did not receive JTPA training, also rose
following assignment, but this group's earnings were less than those
of the treatment group for each of the 5 years.
Figure 1: Earnings of Adult
Men Before and After Assignment
(See figure in printed
edition.)
After 5 years, the difference between earnings of the treatment and
control groups was not statistically significant. Five years after
assignment, the treatment group's earnings had exceeded those of the
control group by approximately $300 to $500 annually, but only in the
first 3 years were these differences statistically significant.
EARNINGS OUTCOMES FOR ADULT
WOMEN
---------------------------------------------------------- Letter :3.2
Earnings of adult women showed a pattern similar to those of adult
men, increasing in each year after assignment. Figure 2 shows that
the annual earnings of adult women assigned to the treatment group
increased from approximately $2,800 in the year of assignment to
approximately $4,700 in the first year following assignment. This
group's earnings continued to climb, reaching approximately $6,600 in
the fifth year. Earnings of adult women in the control group
followed a similar pattern, but this group's earnings were lower than
those of the treatment group in each year, reaching approximately
$6,200 during the fifth year.
Figure 2: Earnings of Adult
Women Before and After
Assignment
(See figure in printed
edition.)
As with the earnings of adult men, 5 years after assignment the
difference between the treatment and control groups' annual earnings
was not statistically significant. However, during the first 4 years
after assignment, the differences between the treatment and the
control groups' earnings were statistically significant in each year,
with treatment group earnings approximately $300 to $600 higher than
control group earnings annually.
EARNINGS OUTCOMES FOR MALE
YOUTHS
---------------------------------------------------------- Letter :3.3
The earnings of male youths in the control group, like those of adult
men and adult women, increased in each year following assignment.
Figure 3 shows that the earnings of male youths in the treatment
group increased from approximately $2,900 in the year of assignment
to approximately $4,600 in the first year after assignment. This
group's earnings continued to grow during the 5-year period, reaching
a high of approximately $7,600 in the fifth year. The earnings of
male youths in the control group also rose during the 5-year period
following assignment, climbing from approximately $4,800 in the first
year to approximately $6,800 in the fifth year.
Figure 3: Earnings of Male
Youths Before and After
Assignment
(See figure in printed
edition.)
We found no significant difference between the treatment and control
groups' annual earnings 5 years after assignment. Although the
control group's earnings were higher than the treatment group's
during the first 3 years following assignment, the differences, which
ranged from approximately $200 to $400 each year, were not
statistically significant. During the fourth and fifth years, the
treatment group had higher earnings than the control group, but these
differences too were not statistically significant.
EARNINGS OUTCOMES FOR FEMALE
YOUTHS
---------------------------------------------------------- Letter :3.4
Earnings of female youths showed a pattern similar to that of male
youths, growing in each year following assignment. Earnings of
female youths in the treatment group rose from approximately $2,000
during the year of assignment to approximately $3,300 in the first
year following assignment (see fig. 4). This group's earnings
continued to climb, reaching approximately $5,400 in the fifth year
following assignment. The earnings of female youths in the control
group also rose during the 5-year period, climbing from approximately
$3,400 in the first year to a high of approximately $5,200 in the
fifth year.
Figure 4: Earnings of Female
Youths Before and After
Assignment
(See figure in printed
edition.)
We found no significant differences between the treatment and control
groups' annual earnings 5 years after receiving their assignments.
During the first 2 years following assignment, the control group's
earnings were higher than the treatment group's, but the differences
of less than $100 annually were not statistically significant. In
the fourth and fifth years following assignment, the treatment group
had earnings of approximately $100 to $300 higher than the control
group, but these differences also were not statistically significant.
EMPLOYMENT RATES OF
PARTICIPANTS NOT SIGNIFICANTLY
GREATER THAN CONTROL GROUP
EMPLOYMENT RATES AFTER 5 YEARS
------------------------------------------------------------ Letter :4
As with earnings, employment rates of those assigned to receive JTPA
training were not significantly greater than employment rates of
control group members 5 years after assignment. For some of the four
targeted worker categories, treatment group employment rates were
higher than those of the control group in some years, but any
statistically significant effects disappeared by the fifth year.
EMPLOYMENT RATES OF ADULT
MEN
---------------------------------------------------------- Letter :4.1
The employment rates of both treatment and control group adult men
peaked during the calendar year of assignment and then declined in
subsequent years, eventually reaching levels lower than those of the
men before entering the study (see fig. 5). For example, the
employment rate for adult men in the treatment group was 87 percent
in the year of assignment. The percent employed declined in the
following years, reaching 72 percent by the fifth year following
assignment, which was lower than the group's employment rate of 79
percent in the year before entering the study. The adult men in the
control group showed a similar pattern--their employment rate was 87
percent in the year of assignment but dropped to 71 percent in the
fifth year after assignment.
Figure 5: Employment Rates of
Adult Men Before and After
Assignment
(See figure in printed
edition.)
After 5 years, the difference between the treatment and control
groups' employment rates was not statistically significant. The
treatment group's employment rates were higher than the control
group's in each year following assignment, although the differences
in the employment rates were statistically significant only in the
fourth year following assignment.
EMPLOYMENT RATES OF ADULT
WOMEN
---------------------------------------------------------- Letter :4.2
The pattern of employment rates of adult women was somewhat similar
to that of adult men. The employment rates of adult women were
highest during the calendar year following assignment, with 80
percent of the treatment group and 77 percent of the control group
employed (see fig. 6). After the first year, however, the
employment rates for both the treatment and control groups fell,
reaching 69 percent and 67 percent, respectively, in the fifth year
following assignment. These rates in the fifth year were also lower
than each group's employment rate in the year before assignment.
Figure 6: Employment Rates of
Adult Women Before and After
Assignment
(See figure in printed
edition.)
We found no significant differences between the treatment and control
groups' employment rates 5 years after assignment. The treatment
group's employment rates exceeded the control group's in all 5 years
following assignment, usually by about 2 to 3 percent, but only in
the first 3 years were these differences statistically significant.
EMPLOYMENT RATES OF MALE
YOUTHS
---------------------------------------------------------- Letter :4.3
The pattern of employment rates of male youths was somewhat similar
to that of adult men and women: the male youths' employment rates
peaked during the calendar year following assignment--reaching nearly
91 percent for the treatment group and over 92 percent for the
control group--but then declined (see fig. 7). However, in contrast
to the employment rates of adults, those of male youths were slightly
higher 5 years after assignment than before assignment, reaching 81
percent for the treatment group in the fifth year, compared with 80
percent in the year before assignment.
Figure 7: Employment Rates of
Male Youths Before and After
Assignment
(See figure in printed
edition.)
We found no significant differences between the treatment and control
groups' employment rates 5 years after assignment. While the
employment rates for the control group actually exceeded those for
the treatment group in the year of assignment and the first and third
years following assignment, none of the differences were
statistically significant.
EMPLOYMENT RATES OF FEMALE
YOUTHS
---------------------------------------------------------- Letter :4.4
The employment rates of female youths in both the treatment and
control groups peaked during the calendar year of assignment,
declined somewhat over the next 4 years, and then slightly increased
in the fifth year (see fig. 8). As with those of male youths, the
employment rates of female youths were slightly higher 5 years after
assignment than before assignment. The employment rates of female
youths were 74 percent for the treatment group and 73 percent for the
control group in the fifth year following assignment, compared with
71 and 73 percent, respectively, in the year before assignment.
Figure 8: Employment Rates of
Female Youths Before and After
Assignment
(See figure in printed
edition.)
We found no significant differences between the treatment and control
groups' employment rates 5 years after assignment. Employment rates
for the treatment group exceeded those for the control group in 4 of
the 5 years following assignment, but none of the differences in
employment rates were statistically significant.
CONCLUSIONS
------------------------------------------------------------ Letter :5
Though both long-term earnings and employment rates for NJS treatment
groups surpassed those for their respective control groups, the
differences did not meet our test for statistical significance. Five
years after expressing an interest in JTPA-sponsored job training,
individuals assigned to participate in the program did not have
earnings or employment rates significantly higher than individuals
not assigned to participate.
AGENCY COMMENTS
------------------------------------------------------------ Letter :6
In commenting on a draft of this report, Labor expressed several
concerns. It took exception to what it characterized as unwarranted
negative conclusions that are not consistent with the report
findings. Labor also took issue with the importance the report
places on tests of statistical significance applied to earnings of an
individual group in a given year, preferring to emphasize other
evidence of the positive effect of JTPA on participant earnings over
the 5-year period. Labor also expressed concerns that the report
findings have limited relevance to current job training programs.
We believe that our conclusions are well supported by our findings.
On several occasions where appropriate, we have noted comparisons
favorable to the JTPA treatment groups, including in the "Results in
Brief" and "Conclusions" sections. Although other evidence covering
the 5-year period might be found to better highlight the positive
effects of JTPA training, our research focused on the earnings and
employment rates of each target group in the fifth year after
applying for JTPA training. Also, we do not believe that current or
proposed job training programs sufficiently differ from JTPA training
at the time of the NJS to limit the relevance of our report findings.
In its response, Labor enclosed an attachment with specific comments
on the report and additional information. This attachment and our
evaluation of the comments appear in appendix III. Labor also
provided us with technical comments, which we have incorporated in
the report where appropriate.
---------------------------------------------------------- Letter :6.1
As agreed with your office, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30 days
from the date of this letter. At that time, we will send copies to
interested parties and make copies available to others upon request.
This report was prepared under the direction of Wayne B. Upshaw,
Assistant Director, who may be reached on (202) 512-7006 if you or
your staff have any questions. Gene Kuehneman, Senior Economist,
(202) 512-4091, Jill Schamberger, Senior Evaluator, and Thomas L.
Hungerford, Senior Economist, were major contributors to this report.
Sincerely yours,
Cornelia M. Blanchette
Associate Director, Education
and Employment Issues
DATA SOURCES AND METHODOLOGY
=========================================================== Appendix I
To select our sample of individuals assigned to receive training
under JTPA and our control group, we used participation information
from the National JTPA Study (NJS). We then obtained long-term
earnings and employment information for these individuals from SSA.
Our analysis compared earnings and employment levels of individuals
in the treatment and control groups to determine whether differences
between these groups were statistically discernable.
NATIONAL JTPA STUDY
--------------------------------------------------------- Appendix I:1
The original NJS data set contained demographic and program
information on 20,601 people who applied for JTPA services between
November 1987 and September 1989 in 16 local service delivery areas.
Program applicants were recruited, screened to determine their
eligibility, assessed to determine their service needs and wants, and
recommended for services. NJS participants were then randomly
assigned to either the treatment group, which was allowed to
participate in JTPA title II-A programs, or the control group, which
was not allowed to participate in these programs for 18 months.
Approximately two-thirds of the applicants were assigned to the
treatment group and one-third to the control group. The control and
treatment groups were closely matched in demographic variables such
as age, race, and education, which typically allows a meaningful
comparison of average outcomes between the two groups.
However, two factors intervened to make such a comparison
problematic. First, not all members of the treatment group
participated in JTPA programs. For example, about two-thirds of the
adult treatment group members enrolled in JTPA, but the other
one-third either found jobs on their own or decided not to
participate in the program. Second, a substantial minority of the
control group members chose to participate in some alternative,
non-JTPA training programs. These complications preclude attributing
earnings differences between the two groups solely to JTPA training.
Therefore, our findings refer to differences between the treatment
and control groups rather than between individuals who did or did not
receive JTPA training. Furthermore, we do not know which of the
control and treatment group members chose to receive training later
than 18 months after assignment.
The NJS was not designed to track treatment or control group members
beyond 30 months. Therefore, to calculate and compare longer term
earnings and employment outcomes for these groups, we needed
information from another source.
SOCIAL SECURITY EARNINGS
RECORDS
--------------------------------------------------------- Appendix I:2
We obtained annual earnings records from SSA for the individuals in
the NJS treatment and control groups. SSA maintains information on
annual earnings of individuals contributing to either Social Security
or Medicare. We assumed that an individual was employed if his or
her SSA records showed positive earnings for a given year. We
adjusted data for what we assumed were data entry or processing
errors, and we also rounded reported negative earnings to zero.
DATA ANALYSIS
--------------------------------------------------------- Appendix I:3
We analyzed the NJS and SSA earnings records of 13,699 NJS
participants\12 to determine their annual earnings and employment
outcomes for the 3 years before assignment to the treatment or
control group, the year of assignment, and 5 years following
assignment. The 3 years of prior earnings and employment data served
to demonstrate the prior comparability of treatment and control
groups. The 5 years of postassignment data effectively doubled the
30-month follow-up period for the NJS. The treatment group had 9,275
individuals, and the control group had 4,424 individuals.
We used individual earnings data and calculated means and variances
for each of the four target groups--adult men, adult women, male
youths, and female youths--to compare the treatment groups' earnings
and employment outcomes with those of the control groups. We tested
for differences in earnings and employment outcomes at the 5-percent
significance level.
We calculated annual earnings amounts using SSA information on Social
Security-covered earnings for nonfederal workers and on earnings
covered by Medicare for federal workers. We calculated employment
rates as the percentage of each group with positive covered earnings
in a calendar year. Individuals with unreported earnings may have
had their earnings and employment understated in our analysis.
Individuals whose earnings exceeded the Social Security withholding
ceiling may also have had their earnings understated in our
analysis.\13 These limitations applied to both the treatment and
control groups, and we do not believe they affected the two groups
differently.
--------------------
\12 We did not analyze earnings and employment information for the
NJS participants who were assigned after 1988 because we had only 4
years of postassignment data for them. We also excluded the NJS
participants whose SSA records did not adequately match information
collected for the NJS, such as name or birth month and year.
\13 Only a certain amount of workers' earnings--for example $55,500
in 1992--were subject to the Social Security tax.
DATA AND STATISTICS FOR FIGURES
========================================================== Appendix II
Table II.1
Data Points for Figure 1--Earnings of
Adult Men Before and After Assignment
Statistically
significant
Treatment Control difference at
Time period group group 5% level?
---------------- ------------ ---------- --------------
3 years before $5,883 $5,924 no
2 years before 5,680 5,894 no
1 year before 5,106 5,246 no
Assignment 4,439 4,242 no
1 year after 6,901 6,410 yes
2 years after 7,792 7,254 yes
3 years after 7,936 7,363 yes
4 years after 8,282 7,725 no
5 years after 8,651 8,326 no
----------------------------------------------------------
Table II.2
Data Points for Figure 2--Earnings of
Adult Women Before and After Assignment
Statistically
significant
Treatment Control difference at
Time period group group 5% level?
---------------- ------------ ---------- --------------
3 years before $3,262 $3,020 no
2 years before 3,377 3,215 no
1 year before 3,230 3,048 no
Assignment 2,823 2,703 no
1 year after 4,702 4,323 yes
2 years after 5,705 5,047 yes
3 years after 5,902 5,319 yes
4 years after 6,367 5,811 yes
5 years after 6,556 6,154 no
----------------------------------------------------------
Table II.3
Data Points for Figure 3--Earnings of
Male Youths Before and After Assignment
Statistically
significant
Treatment Control difference at
Time period group group 5% level?
---------------- ------------ ---------- --------------
3 years before $860 $828 no
2 years before 1,456 1,575 no
1 year before 2,179 2,303 no
Assignment 2,894 3,014 no
1 year after 4,612 4,792 no
2 years after 5,620 5,963 no
3 years after 6,130 6,497 no
4 years after 6,687 6,425 no
5 years after 7,554 6,778 no
----------------------------------------------------------
Table II.4
Data Points for Figure 4--Earnings of
Female Youths Before and After
Assignment
Statistically
significant
Treatment Control difference at
Time period group group 5% level?
---------------- ------------ ---------- --------------
3 years before $629 $663 no
2 years before 1,069 1,090 no
1 year before 1,529 1,707 no
Assignment 1,974 2,098 no
1 year after 3,339 3,389 no
2 years after 4,045 4,125 no
3 years after 4,393 4,383 no
4 years after 4,934 4,610 no
5 years after 5,433 5,209 no
----------------------------------------------------------
Table II.5
Data Points for Figure 5--Employment
Rates of Adult Men Before and After
Assignment
Statistically
significant
Treatment Control difference at
Time period group group 5% level?
---------------- ------------ ---------- --------------
3 years before 79.2 78.8 no
2 years before 77.6 79.0 no
1 year before 78.3 80.6 no
Assignment 87.0 87.4 no
1 year after 86.7 86.4 no
2 years after 81.3 80.7 no
3 years after 76.3 73.9 no
4 years after 73.7 70.0 yes
5 years after 72.3 71.1 no
----------------------------------------------------------
Table II.6
Data Points for Figure 6--Employment
Rates of Adult Women Before and After
Assignment
Statistically
significant
Treatment Control difference at
Time period group group 5% level?
---------------- ------------ ---------- --------------
3 years before 65.8 65.8 no
2 years before 67.1 66.2 no
1 year before 69.4 68.4 no
Assignment 76.7 75.7 no
1 year after 80.3 77.4 yes
2 years after 76.3 73.7 yes
3 years after 71.2 68.1 yes
4 years after 69.7 67.7 no
5 years after 68.7 67.4 no
----------------------------------------------------------
Table II.7
Data Points for Figure 7--Employment
Rates of Male Youths Before and After
Assignment
Statistically
significant
Treatment Control difference at
Time period group group 5% level?
---------------- ------------ ---------- --------------
3 years before 46.5 48.3 no
2 years before 63.2 66.4 no
1 year before 79.6 79.6 no
Assignment 89.2 91.8 no
1 year after 90.5 92.1 no
2 years after 88.4 87.8 no
3 years after 82.2 82.6 no
4 years after 80.4 79.4 no
5 years after 81.1 77.5 no
----------------------------------------------------------
Table II.8
Data Points for Figure 8--Employment
Rates of Female Youths Before and After
Assignment
Statistically
significant
Treatment Control difference at
Time period group group 5% level?
---------------- ------------ ---------- --------------
3 years before 41.2 43.6 no
2 years before 57.6 60.5 no
1 year before 70.7 72.8 no
Assignment 82.0 81.8 no
1 year after 82.0 79.6 no
2 years after 79.7 78.2 no
3 years after 73.8 75.1 no
4 years after 71.7 70.7 no
5 years after 73.9 73.0 no
----------------------------------------------------------
Table II.9
Standard Errors Associated With Earnings
of Adult Men Before and After Assignment
Treatmen Pooled
Time period t Control error T-ratio
------------------ -------- -------- -------- --------
3 years before 6,740 6,983 221 -0.19
2 years before 6,547 6,702 213 -1.00
1 year before 5,818 5,779 188 -0.74
Assignment 4,724 4,405 149 1.32
1 year after 6,825 6,406 216 2.27
2 years after 8,029 7,516 254 2.12
3 years after 8,753 8,335 279 2.06
4 years after 9,446 9,117 302 1.84
5 years after 10,134 9,611 322 1.01
Subgroup size 2,874 1,435
----------------------------------------------------------
Table II.10
Standard Errors Associated With Earnings
of Adult Women Before and After
Assignment
Treatmen Pooled
Time period t Control error T-ratio
------------------ -------- -------- -------- --------
3 years before 4,638 4,412 134 1.81
2 years before 4,714 4,664 137 1.18
1 year before 4,315 4,296 126 1.44
Assignment 3,297 3,128 95 1.27
1 year after 5,017 4,828 145 2.62
2 years after 6,242 5,707 178 3.70
3 years after 6,829 6,414 196 2.98
4 years after 7,646 7,039 218 2.55
5 years after 7,797 7,480 225 1.79
Subgroup size 3,631 1,723
----------------------------------------------------------
Table II.11
Standard Errors Associated With Earnings
of Male Youths Before and After
Assignment
Treatmen Pooled
Time period t Control error T-ratio
------------------ -------- -------- -------- --------
3 years before 1,689 1,687 87 0.36
2 years before 2,339 2,604 125 -0.96
1 year before 2,762 2,778 142 -0.88
Assignment 3,048 2,902 154 -0.78
1 year after 4,799 4,603 243 -0.74
2 years after 5,807 5,703 297 -1.16
3 years after 6,561 6,646 338 -1.08
4 years after 7,445 7,186 378 0.69
5 years after 8,150 7,807 413 1.88
Subgroup size 1,177 559
----------------------------------------------------------
Table II.12
Standard Errors Associated With Earnings
of Female Youths Before and After
Assignment
Treatmen Pooled
Time period t Control error T-ratio
------------------ -------- -------- -------- --------
3 years before 1,327 1,392 61 -0.56
2 years before 1,885 1,797 84 -0.25
1 year before 2,113 2,352 99 -1.80
Assignment 2,194 2,307 101 -1.23
1 year after 3,828 3,767 172 -0.29
2 years after 4,612 4,663 209 -0.38
3 years after 5,271 5,388 240 0.04
4 years after 6,053 5,803 270 1.20
5 years after 6,494 6,309 291 0.77
Subgroup size 1,593\a 707
----------------------------------------------------------
\a Subgroup size for the treatment group 3 years after assignment is
1,592.
(See figure in printed edition.)APPENDIX III
COMMENTS FROM THE DEPARTMENT OF
LABOR AND OUR EVALUATION
========================================================== Appendix II
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
The following are GAO's comments on the Department of Labor's letter
dated November 30, 1995.
GAO COMMENTS
-------------------------------------------------------- Appendix II:1
1. Labor comments that our report understates the gross returns to
JTPA training. Furthermore, Labor implies that these gross returns
calculations compare favorably with the returns to college education.
Our objective, as clearly stated in the report, was not to evaluate
the cost-effectiveness of JTPA training, but rather to determine and
compare the long-term effects of JTPA training. In fact, because we
did not calculate the gross return to JTPA participants the report
cannot have understated or overstated the values. Such calculations
were not within the scope of this report. While it may be true that
these returns are favorable, we have no basis to judge the
favorability of the gross returns to training.
2. Labor states that the report does not acknowledge favorable
aspects of this study. Specifically, Labor cited that (1) all four
target groups had higher earnings in the fifth year after assignment;
(2) both adult treatment groups had higher earnings than their
respective control groups in each of the 5 years following
assignment; and (3) for male youths, a positive trend exists, and the
fifth-year earnings exceed those of the control group by over 10
percent.
Contrary to Labor's comment, we did note many of these favorable
program outcomes in our report. We stated that adult male treatment
group members had higher earnings than adult male control group
members and presented similar findings for the other three target
groups. We further stated that adult male treatment group earnings
exceeded control group earnings in each of the 5 years and reported
similar information for adult women. Also, we noted the positive
trend for earnings of male youths. We did not note the percentage
difference for male youths in the fifth year because we did not
report percentage comparisons for any of the target groups.
3. Labor states that if the training impacts are accumulated over
time during the 5-year follow-up period, the net benefits outweigh
the costs.
As we stated in comment 1, our objective was not to evaluate the
cost-effectiveness of JTPA training, but rather to determine and
compare the long-term effects of JTPA training. While it may be true
that the net benefits outweigh the costs, we have no basis to judge
this because such calculations were beyond the scope of this report.
4. Labor states that the increase in standard errors is primarily
responsible for the decline in statistical significance of the
estimated impacts.
While Labor is correct in stating that the standard errors were
greater in the fifth year, it is not accurate to attribute a decline
in statistical significance to either the estimated training effects
or to the standard errors. The test statistics used for our
significance tests are determined by the ratio of the estimates to
their standard error, and attributing the lack of significance solely
to either component of these ratios is inappropriate.
5. Labor states that our conclusion requires assessing the total
impact of JTPA and the overall cost and benefits. The Department
states that we overemphasize the importance of year-by-year
significance tests in questioning the program's usefulness in
improving participants' long-term earnings prospects by stressing the
insignificant effect of JTPA in the fifth year.
We agree with Labor that year-by-year significance tests have limited
value in assessing the total impact of JTPA and the overall cost and
benefits. Furthermore, our year-by-year significance tests provide
statistical evidence that adult treatment group members achieved
higher earnings for several years following assignment to JTPA
training. While other evidence covering the 5-year period might be
found to better highlight the positive effects of JTPA training, we
chose to address the question of whether the fifth-year earnings of
those assigned to participate in JTPA differed significantly from the
fifth-year earnings of those not assigned to participate in JTPA.
6. While acknowledging that the observed earnings differences
between the four target groups were not statistically significant in
the fifth year, Labor asserts that the odds that all four differences
would be positive purely by chance is 6.25 percent.\14
This implies that an accumulation of not statistically significant
observations provides more compelling empirical evidence than the
actual significance test for any one group. While the probability
(not odds) that all four not significant fifth-year earnings
differences would be positive purely by chance might be low, our
research question is whether a significant earnings difference
occurred for each target group.
7. Labor comments that the report does not report the standard
errors. Labor states that the report should include confidence
intervals for the estimates, sample sizes, and standard errors and
specify significance levels for the estimates.
We have made several additions to tables in appendix II in response
to this comment. We have added the sample size, the size of the
treatment and control groups, the standard errors, and a reminder
that the significance level chosen is 5 percent for the tables in
this appendix. Since we have not presented point estimates of the
earnings effects, we did not calculate confidence intervals for these
estimates. Technical readers of our report can construct such
estimates and the associated intervals from the information in the
appendix II tables.
8. Labor claims that the report treats figures that are not
significant as zero.
We do not report any training effect as zero. The magnitude of the
earnings differences, whether significant or not, is discussed in the
report and is easily calculated from the tables in appendix II.
9. Labor states that statistical significance is not a knife edge of
yes or no but a continuum.
The level used for tests of statistical significance may be chosen
from a broad range (or continuum) of values. Although different
researchers may choose to use different values for the significance
level, choosing a significance level before analyzing any data is
common practice. Once this level has been chosen, statistical
hypothesis testing very much involves a yes or no decision. Either
the data reject the null hypothesis of no training effect at the set
significance level or not. We follow these commonly accepted
procedures for hypothesis testing, and our convention is to set the
significance level for such tests at 5 percent.
10. Labor also states that the report should discuss the results,
the probability values, and changes in the significance levels.
Our report does discuss the results as well as whether the earnings
and employment effects were significant and whether this significance
changed over time. Although we do not present the probability
values, technical readers of our report can calculate them using the
information in the appendix II tables.
11. Labor takes issue with our statement that complications (not all
treatment group members received training and some control group
members did receive training) precluded our attributing earnings
differences solely to JTPA training. It claims that these
complications led us to understate the effect of training, implying
that the earnings differences observed, along with perhaps some
further overlooked earnings effects, can be attributed to JTPA.
We clearly state that these complications preclude solely attributing
the earnings effects to JTPA training. However, we have no evidence
that these factors led to an understatement of the effect of JTPA
training. In the first place, a short delay can occur before an
assignee can begin a training program. In some of these cases,
individuals find and accept employment instead of reporting for
training. To the extent that these individuals are more fully
employed and may earn more than they might have if they had attended
JTPA training, our estimate may actually overstate the effect of
training. Second and more importantly, if those who attend training
are in some way more motivated than those who do not attend, it would
be difficult to separate any increase in earnings due to training
from the increase in earnings due to this motivation. At a minimum,
we would need to identify which control group members were motivated
to attend training to draw such inferences.
12. Labor recommends adjusting the comparison by effectively
removing treatment group members who did not enroll in training.
We chose to compare only those assigned to JTPA training with those
not assigned to training to take full advantage of the original
random assignment design. As we stated in comment 11, we would have
needed to identify which control group members were motivated to
attend training to justify removing the treatment group members who
did not attend training. Since we could not take all the necessary
steps to fully implement Labor's suggestion, we chose not to make
that or any other adjustments.
13. Labor comments that we will be providing the Department's
contractor with access to Social Security data for additional
analysis, including examining the results for subgroups.
We would like to clarify the details of this arrangement in light of
the sensitive nature and confidentiality of individual earnings
records. When we began our work, Labor was also planning to evaluate
the long-term impact of JTPA training on earnings. Both our and
Labor's evaluation (contracted out to Westat, Inc.) planned to use
Social Security earnings records to supplement the information
collected through the NJS. In the spirit of cooperation, Labor
requested and we agreed to provide aggregated earnings data to the
Department, which will submit computer programs to us; we will in
turn run the programs and provide the output to Labor. Only
aggregated information, such as means and standards deviations, will
be reported. No data will be released that could be traced to
individuals, nor will we provide Labor or its contractor with
individual earnings records.
14. Labor suggests that we include earnings and employment
information for the third of the sample for whom only 4 years of
follow-up data were available.
We did not include this group in our analysis since we could not
report on their earnings or employment 5 years after training. As
such, any additional information provided would not address the
question of whether JTPA had a long-term effect on the earnings or
employment outcomes of the treatment group.
15. Labor states that we should explain that our employment measure
is not the definition that is generally reported in government
statistics and is not comparable to figures reported in Current
Population Survey (CPS) and Bureau of Labor Statistics (BLS) reports.
Our employment rate differs from measures reported in CPS and BLS
reports but is appropriate for our purposes. Our employment rate is
the number employed divided by all who were in the treatment or
control group, which includes those workers who may have dropped out
of the labor force.\15 Since these workers presumably applied for
training because they intended to keep working, we believe that all
workers should be included in the denominator of the measure. Our
measure also counts as employed everyone who worked during the year,
even if they might have been unemployed for some portion of the year.
As such, our measure may overstate the instantaneous employment
outcomes of both the treatment and control groups relative to figures
reported in CPS and BLS reports.
16. Labor states that Social Security data are subject to
considerable revisions in the first year of availability. It
believes this calls into doubt fifth-year estimates for adult men,
adult women, and male youths.
While data are often subject to revision, we have no reason to
suspect that the data for those assigned to training in 1988 are
materially less reliable than for those assigned to training in 1987.
The fifth year (1993) of earnings data for those assigned to training
in 1988 was extracted from SSA records in March 1995. An SSA
official responsible for updates and revisions to SSA earnings data
said that we could expect the accuracy and completeness of our
extract to exceed 99 percent.
17. Labor states that we fail to recognize the limited relevance of
our findings to current job programs.
While we agree that our evaluation has limitations, we disagree that
it has little relevance to current job programs. We make it quite
clear that our analysis is not nationally representative of JTPA
training. Additionally, we cite many flaws associated with the
design and implementation of the original NJS that limit our
analysis. However, no evidence exists to suggest that job training
funded by JTPA and administered at the state and local level has
changed so dramatically since 1989 that our findings are not relevant
to the current program.
--------------------
\14 For example, if training truly had no effect on earnings, then
the probability that the estimated difference would be positive for
any one target group is 50 percent--that is, half of the time the
estimate would be positive and the other half it would be negative.
Thus, the chance that all four groups would show positive estimated
earnings effects, if no true effect existed for any group, would be
50 percent to the fourth power or 6.25 percent.
\15 The CPS counts people at any given time as (1) employed, (2)
unemployed, or (3) not in the labor force--that is, not employed and
not seeking employment. The unemployment rate in the CPS is the
number of unemployed divided by the sum of the employed plus
unemployed. If an employment rate were reported, it would be one
minus the unemployment rate, or, equivalently, the number employed
divided by the sum of the employed plus unemployed.
*** End of document. ***