Welfare to Work: State Programs Have Tested Some of the Proposed Reforms

Welfare to Work: State Programs Have Tested Some of the Proposed Reforms
(Letter Report, 07/14/95, GAO/PEMD-95-26).

As Congress considers various proposals to reform welfare, questions
have arisen about how best to reduce welfare dependency and help
recipients move from welfare to work. This report reviews the
evaluation of the many state welfare-to-work experiments completed since
the reforms enacted in 1988 to learn (1) How do they resemble the
welfare reforms now being discussed? and (2) What approaches have been
effective in increasing employment and earning or reducing benefit
receipt among welfare clients.

--------------------------- Indexing Terms -----------------------------

REPORTNUM: PEMD-95-26
TITLE: Welfare to Work: State Programs Have Tested Some of the
Proposed Reforms
DATE: 07/14/95
SUBJECT: Program evaluation
Disadvantaged persons
State programs
Welfare recipients
Welfare benefits
Federal/state relations
IDENTIFIER: AFDC
JOBS Program
Job Opportunities and Basic Skills Training Program
California Greater Avenues for Independence Program
Food Stamp Program
Florida
San Diego (CA)
Ohio
Michigan
Alabama Avenues to Self-Sufficiency Through Employment and
Training Services Project
New York
Washington Family Independence Plan
Aid to Families with Dependent Children Program

******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************

Cover
================================================================ COVER

Report to the Honorable
Daniel Patrick Moynihan, U.S. Senate

July 1995

WELFARE TO WORK - STATE PROGRAMS
HAVE TESTED SOME OF THE PROPOSED
REFORMS

GAO/PEMD-95-26

Welfare to Work

(973796)

Abbreviations
=============================================================== ABBREV

AFDC - Aid to Families with Dependent Children
ASSETS - Avenues to Self-Sufficiency through Employment and
Training Services
CAP - Child Assistance Program
CWEP - Community Work Experience Program
ET - Employment and Training
FIP - Family Independence Program
FPI - Florida Project Independence
GAIN - Greater Avenues for Independence
GED - General equivalency diploma
HHS - Department of Health and Human Services
JOBS - Job Opportunities and Basic Skills Training program
SWIM - Saturation Work Initiative Model
TI - Transitions to Independence
TSMF - To Strengthen Michigan Families

Letter
=============================================================== LETTER

B-261315

July 14, 1995

The Honorable Daniel Patrick Moynihan
United States Senate

Dear Senator Moynihan:

As the Congress has been considering various proposals to reform
welfare, questions have arisen about how best to reduce welfare
dependency and help recipients move from welfare to work. This
report responds to your request that we review the evaluations of the
numerous state welfare-to-work experiments completed since the
reforms enacted in 1988 to learn (1) How do they resemble the welfare
reforms currently being discussed? and (2) What approaches have been
effective in increasing employment and earnings or reducing benefit
receipt among welfare clients?

BACKGROUND
------------------------------------------------------------ Letter :1

The nation's major cash assistance program to poor families, Aid to
Families with Dependent Children (AFDC), provides cash benefits to
needy families with children who lack support from one or both of
their parents because of unemployment, incapacity, absence, or death.
Funded with federal and state dollars, the program operates as an
individual entitlement--that is, everyone who meets the eligibility
requirements is entitled to receive benefits. In fiscal year 1993,
AFDC benefits supported 5 million families and more than 9.5 million
children each month and cost over $25 billion in federal and state
funds. The Family Support Act of 1988 created the Job Opportunities
and Basic Skills Training (JOBS) program, which requires the states
to enroll an increasing proportion of their adult AFDC recipients
(primarily women) in the education, training, and employment-related
activities they need to get jobs and avoid long-term welfare
dependency. The states are permitted substantial flexibility in
designing and implementing their JOBS programs, but they are required
to provide participants with the support services deemed necessary,
such as child care and transportation. Federal funds to match state
JOBS expenditures are capped, but most states have not reached the
limit of that cap. However, as we reported last December, the share
of AFDC recipients active in JOBS is limited; only about one fourth
of those required to participate were served in an average month in
fiscal year 1993.\1

Rapid growth in the AFDC caseload since 1989 and concern about
program costs and beneficiaries' long-term dependence have led to
widespread dissatisfaction with the AFDC program and to several
congressional proposals to reform it. Some provisions of current
proposals represent continuity with previous legislative efforts to
strengthen the employment focus of the program, such as requiring
larger proportions of recipients to participate in a work program.
Other provisions propose dramatic changes in AFDC's structure, such
as imposing time limits on the receipt of benefits and replacing the
individual entitlement to benefits with a block grant for which
federal funding would be fixed.

Concern about welfare dependency has spurred policy initiatives since
the 1970s to encourage or assist welfare clients to get jobs. The
states have obtained waivers from existing federal statutes and
regulations to test a variety of welfare-to-work initiatives. One
condition of the waivers is that the states rigorously evaluate the
effects of these initiatives. Evaluations conducted under such
waivers informed the formulation of the JOBS program; others
completed since 1988 can similarly inform the current debate.

This report presents the results of our evaluation synthesis of nine
published high-quality studies, from eight states, of welfare-to-work
experiments for adult AFDC recipients.\2 We identified these studies
by conducting a systematic search and methodological review of all
evaluations published since the Family Support Act of 1988 that
focused, at least in part, on moving clients from welfare to work.
All nine studies used comparison groups, six of which were formed
through random assignment, making it possible to estimate the effects
of a program by comparing the outcomes for its participants with
those for nonparticipants.

To meet our first objective, we compared the approaches used in these
experiments with provisions of the proposed welfare reforms being
debated. Our list of provisions was derived primarily from the
pending House welfare reform bill, H.R. 4, but we also included a
few provisions from other bills introduced in the 104th Congress.\3
To meet our second objective--to identify approaches that
successfully moved AFDC recipients from welfare to work--we compared
and contrasted the statistically significant effects of similar and
dissimilar programs on participants' earnings, employment, and
welfare receipt. (See appendix I for details on our selection and
analysis of these studies.) We conducted our work in accordance with
generally accepted government auditing standards between December
1994 and April 1995. However, we did not independently verify the
information in the evaluation reports.

--------------------
\1 U.S. General Accounting Office, Welfare to Work: Current AFDC
Program Not Sufficiently Focused on Employment, GAO/HEHS-95-28
(Washington, D.C.: December 1994).

\2 The nine studies covered 10 programs; the Greater Avenues for
Independence (GAIN) program in Riverside, California, was
sufficiently different from the GAIN programs evaluated in the other
California counties to treat it separately.

\3 Our reference to H.R. 4 is to the March 27, 1995, version passed
by the House. We excluded bills introduced after April 1995, when we
completed our review.

RESULTS IN BRIEF
------------------------------------------------------------ Letter :2

The state welfare-to-work experiments provide evidence on the states'
experience with some features of current federal welfare reform
proposals but not others. The features common to both address work
programs for recipients, stricter requirements for participation in
work programs and child support enforcement, and increasing work
incentives. Some proposal features, such as limiting the length of
time a family can receive benefits, are currently being tested by the
states, but their evaluations are not yet complete. Other proposed
federal reforms have not been tested by the states. In addition,
some states have evaluated features of welfare-to-work programs--such
as providing a broad mix of employment services--that go beyond the
requirements of some of the current proposals. For example, one
proposal would not count some of these activities toward meeting its
work program participation standards, thereby creating a potential
disincentive for financing them. Thus, the states' positive
experience with the broader mix of employment activities is relevant
to considering this proposal. Although the states' experiences
provide information regarding some of these reform features, other
features--such as time-limiting benefits and turning the AFDC program
into a block grant--would so alter the environment that we cannot
confidently project the likely effects of the entire package of
reform proposals.

The most successful programs--those that consistently showed better
employment and welfare-related outcomes for participants--combined a
broad range of employment-related activities and support services
with some form of participation mandate, and they had adequate
funding to fully serve their clients as intended. Each of these
programs made a broad mix of services available to their clients,
such as intensive job search and placement assistance, basic skills
and secondary education, and vocational training. Other programs
using this approach were less successful when funding shortfalls
delayed full implementation or where budget freezes caused services
to be cut back. However, moving welfare recipients into
self-supporting employment has proved challenging. Even the most
successful program had modest effects; after 3 years, only one fourth
of its participants were self-sufficient in being both employed and
off welfare.

A quite different approach--increasing both work incentives (such as
allowing recipients to earn somewhat more income before their
benefits were reduced) and access to employment supports (such as
child care)--was also successful in one program at increasing
employment and earnings, but it did not reduce welfare receipt.
However, two other states that increased work incentives and mandated
work program participation, but did not expand their
employment-related services, have not yet seen clear improvements in
either employment or welfare-related outcomes.

PRINCIPAL FINDINGS
------------------------------------------------------------ Letter :3

THE COMPLETED STATE
EXPERIMENTS HAVE TESTED ONLY
SOME OF THE PROPOSED WELFARE
REFORMS
---------------------------------------------------------- Letter :3.1

The welfare-to-work experiments we reviewed tested many of the
provisions in welfare reform proposals (including H.R. 4), such as
conducting some form of work program that may provide support
services such as child care and requiring adult AFDC recipients to
participate in that work program and to cooperate with child support
enforcement. (See table 1.) In addition, some states experimented
with extending medical and child care benefits to families as they
leave welfare for work and with increasing the disregard of earnings
while on welfare, both of which are provisions in other current
proposals. Of course, the states may not have implemented these
features in quite the same form as they appear in the legislative
proposals.

Table 1

Relationship Between Features in Federal
Proposals to Reform AFDC and States'
Completed Welfare Evaluations

Federal welfare reform Included these Did not include these
proposals features features
====================== ---------------------- ----------------------
Included these Provide a work program Limit lengths of stays
features with support services on welfare

Require work program Bar additional
participation benefits for children
born while on welfare
Require cooperation
with child support Bar aid to unwed
enforcement teenage mothers living
on their own and to
Extend transitional noncitizens
medical and child care
benefits Cap the amount of
federal funds
Increase disregard of available
earnings while on
welfare End requirement for
states to match
federal expenditures

Did not explicitly Enhance employment and Not applicable
address these features training activities

Consolidate Food Stamp
and AFDC programs
----------------------------------------------------------------------
Proposed provisions not in the state experiments we reviewed include
limits on the length of welfare receipt, prohibition of additional
benefits for additional children born to families on welfare, and
requirements that unwed teenage mothers live with a parent or
guardian. These are the subject of ongoing or planned experiments
and have not yet been evaluated. Prohibiting aid to noncitizens,
creating block grants with fixed funding, and ending requirements
that the states match federal expenditures have not been options
available to the states. Replacing the current AFDC program with a
block grant would basically repeal current federal law prescribing
state procedures for determining individuals' eligibility for
benefits and benefit levels. This change aims to increase the
states' flexibility in managing their programs of assistance to needy
families and would provide the states with a fixed amount of funds
each year rather than matching (at federally specified rates)
whatever their expenditures had been.

The states have also tested several program features not explicitly
addressed in some of the legislative proposals, such as enhancing
employment and training activities and consolidating the AFDC and
Food Stamp programs. Some of these experiments were begun before the
JOBS program was enacted but tested features it currently requires,
such as providing a broad range of employment-related and support
services. Under H.R. 4, the states would be permitted but no longer
required to provide as broad a range of employment-related services
and supports. Indeed, the states might be discouraged from enrolling
clients in some types of education and training because these
activities would not count toward the bill's work program
participation requirements. The states would face financial
sanctions if they failed to meet minimum participation levels. Thus,
these state experiments are relevant to the question of whether the
more inclusive provisions of current law should be retained.

A RANGE OF PROGRAMS HAD
POSITIVE RESULTS
---------------------------------------------------------- Letter :3.2

All but three of the experiments had a statistically significant
positive effect on at least one of the following: participants'
employment, earnings, receipt of welfare, and welfare payment
amounts.\4 Four were successful on all four outcomes, three others on
only one or two. Effects were positive more often on employment and
earnings than AFDC receipt, but a variety of approaches and their
combinations had some success.

Program outcomes were often measured 1, 2, or sometimes 3, and in one
up to 5, years after clients had been enrolled. We scored them as
"positive" if a statistically significant effect in the intended
direction was recorded at any of these time points. The more complex
scores for Florida's Project Independence (FPI) program are discussed
below.\5

Table 2 summarizes the major features being tested and does not
include features that applied to both the experimental and comparison
groups. For example, programs that did not test an employment and
training program (the last four rows) offered similar levels and
kinds of employment services to both the program and comparison
groups, but only the program participants were offered an increase in
the earned income disregard.

Table 2

Ten Treatments That Were Tested in
Welfare-to-Work Evaluations and Their
Outcomes\a

Suppor
Income t Merge
disrega servic Employment progra Earnin Employme Receip Paymen
Program Mandate rd e\b and training ms gs nt t ts
----------------- ------- ------- ------ ------------- ------ ------ -------- ------ ------
Calif. Monthly No Child Emphasis: job No Positi Positive Positi Positi
SWIM partici care search. Also ve ve ve
pation; CWEP,
fixed education,
activit and training
y
sequenc
e

Mass. Work No Child Emphasis: No Positi Positive Positi Positi
ET registr care; voluntary ve ve ve
ation transi education and
only tional training.
care Also CWEP and
job search

Calif. Work No Child Emphasis: No Positi Positive Positi Positi
GAIN registr care; basic ve ve ve
ation; transi education and
activit tional training.
y care Also job
sequenc search and
es vocational
education

Riverside Work No Child Emphasis: job No Positi Positive Positi Positi
GAIN registr care; search. Also ve ve ve
ation; transi basic and
activit tional vocational
y care education and
sequenc job training
es

Fla. Orienta No Child Emphasis: No Positi None None Positi
FPI tion; care; independent ve early\; early\ ve
activit transi job search. early; positive ; none early;
y tional Also basic none later later positi
sequenc care and later\ ve
es vocational c later
education and
job training

Ohio Educati No None Emphasis: No None\c Positive None\c None\c
TI on and education and
trainin training.
g Also CWEP and
job search

N.Y. Child Yes Child No Yes Positi Positive None\c None\c
CAP support care ve
coopera
tion

Ala. Work Yes Reduce No Yes Not Not None\c None\c
ASSETS registr d availa availabl
(AFDC) ation; child ble e
child care
support
coopera
tion

Mich. Educati Yes None No No None\c None\c None\c None\c
TSMF on and
trainin
g

Wash. None Cash Child No. But small Yes None\c None\c Negati Negati
FIP bonus care; cash bonus ve\d ve\d
transi for
tional participating
care; in education
transi or training
tional
medica
l
----------------------------------------------------------------------------------------------------
\a Programs abbreviated are AFDC, Aid to Families with Dependent
Children; ASSETS, Avenues to Self-Sufficiency through Employment and
Training Services; CAP, Child Assistance Program; CWEP, Community
Work Experience Program; ET, Employment and Training; FIP, Family
Independence Program; FPI, Florida Project Independence; GAIN,
Greater Avenues for Independence; SWIM, Saturation Work Initiative
Model; TI, Transitions to Independence--Fair Work; TSMF, To
Strengthen Michigan Families.

\b Transitional care and transitional medical are the extension of
child care and medical care benefits to ease clients' transition to
work after leaving AFDC.

\c None statistically significant at p = .05 or less.

\d The program increased AFDC receipt or payment rather than reducing
them as intended.

Because the programs typically combined several features at once,
individually they do not provide clear tests of the effectiveness of
single program features. Therefore, we drew our conclusions about
the success of program approaches (including clusters of these
features) both by comparing the effects of programs that included and
did not include the same feature and by comparing the features of the
more and less successful programs. However, our sample of 10 studies
is not large enough to provide conclusive answers, because, of
course, there are many differences between the studies, some of which
might have influenced their outcomes.

--------------------
\4 At the .05 level or better. All the programs wanted to increase
client earnings and employment and decrease AFDC receipt and
payments.

\5 We did not base our analysis on reported effect sizes because to
do so would imply a false level of precision in the comparison of
very different, complex programs and evaluations. Few meaningful
conclusions could be drawn from comparing these effect sizes.

COMBINING A BROAD RANGE OF
EMPLOYMENT-RELATED SERVICES
AND SUPPORTS YIELDED THE
BEST, THOUGH MODEST, RESULTS
---------------------------------------------------------- Letter :3.3

The most successful welfare-to-work programs--those with the largest
and most consistent effects--offered participants an expanded mix of
education, training, and employment services; increased child care
assistance; and mandated some form of client participation. Four
programs using this same general approach--San Diego's Saturation
Work Initiative Model (SWIM), Massachusetts' Employment and Training
(ET) program, and California's Greater Avenues for Independence
(GAIN) program, both statewide and in Riverside county--were the only
ones to record statistically significant effects on all four
outcomes.

These programs provided a mix of employment-related services, of
which clients could receive one or more. Education and training
included assistance in basic education, preparation for the
high-school equivalency examination (or GED), English-language
training, and vocational classes. Intensive job search included
program staff working with employers to develop job placements,
assisting clients with their job search, or starting clients with job
searches immediately. In addition, some offered community work
experience (CWEP), which involves unpaid work in public or nonprofit
agencies aimed at increasing clients' employability. Their
evaluations compared participants' outcomes to those of AFDC clients
who received whatever the standard level of employment services was
at the time. Since some of these programs began operating before the
JOBS program was enacted, they typically offered either a lower level
of service than is currently required or nothing at all.

Child care assistance was increased to allow participation in
employment preparation activities and, during the first year of
postwelfare employment, to facilitate the transition off public
assistance. Participation mandates included requirements to register
for job search and apply for work, participate for a specified number
of hours per month, or enroll in a sequence of employment-related
activities. However, this does not mean that all clients actually
participated; some could be exempted for personal reasons, others for
lack of program resources.

There were, however, some significant differences in the four
successful programs. Massachusetts' ET allowed voluntary client
participation and selection of activities after a mandatory work
registration, while California's SWIM enforced a fixed sequence of
activities and GAIN allowed a variety of sequences. ET put more
emphasis on education and training, while GAIN in Riverside put more
emphasis on aggressive job search support. The statewide program
emphasized basic education more than the other programs.

Two other programs in Ohio and Florida that took the same general
approach had mixed results, which could in part be explained by
funding problems that delayed or cut short the full experiment.
Ohio's economy took a downward turn at the start of the Transitions
to Independence--Fair Work (TI) program evaluation period, causing an
influx of cases and lengthy backlogs. In fact, a majority of clients
did not even receive their employment and training assignments. TI
achieved effects on only one of the four outcomes. Florida's FPI
showed positive effects for first-year participants on two outcomes,
but an economic downturn combined with a budget freeze led to program
reductions in the second year. This provided the opportunity to test
the effects of the changes--increases in caseloads and the
elimination of child care assistance. However, the contribution of
these features is unclear because both the early and later groups of
participants achieved mixed results.

The effects of even the most successful program were modest. The
Riverside GAIN program is arguably the most successful of the
welfare-to-work programs. It increased the proportion of clients
ever employed in 3 years to 67 percent, or 14 percentage points over
the comparison group, but this means that 33 percent of clients in
the best program were never employed in 3 years. Of those who were
employed at the end of 3 years, only 24 percent made more than $5,000
per year. Thus, Riverside GAIN participants averaged a 49-percent
increase in earnings over 3 years compared to nonparticipants
receiving only traditional AFDC, but this amounted to only $3,113, or
about $1,000 per year. The Riverside program lowered average AFDC
payments for all participants over 3 years by 15 percent, or $1,983,
and reduced the percentage who were receiving AFDC payments after 3
years by 5 percent, compared to the nonparticipants. However, after
3 years only one fourth of its participants had achieved
self-sufficiency by being both employed and off welfare.

That the successful programs only modestly reduced welfare dependency
has, no doubt, a variety of causes. Even when participation was
mandated, not all recipients were required to enroll in activities,
some were exempt for ill health or to care for an infant, and others
had to wait for assignments. In addition, some education and
training programs had participation and attendance problems that
diminished their success. These might reflect problems that clients
had that support services like those in these programs could address,
or perhaps other interventions are needed. Researchers also point to
other barriers to moving welfare recipients into self-supporting
employment--in particular, their low skill levels and the low wages
and short tenure of low-skill jobs. In 1992, 45 percent of the
single mothers receiving AFDC lacked a high school diploma and
another 38 percent had no schooling beyond high school.\6 Yet
occupations that accept limited schooling pay fairly low wages, have
limited fringe benefits (such as health insurance), and are
characterized by high job turnover.\7 Thus, relatively short-term
training and job search interventions may have a limited effect on
recipients whose skill levels are low.

--------------------
\6 U.S. General Accounting Office, Families on Welfare: Sharp Rise
in Never-Married Women Reflects Social Trend, GAO/HEHS-94-92
(Washington, D.C.: May 1994).

\7 Rebecca M. Blank, "Outlook for the U.S. Labor Market and
Prospects for Low-Wage Entry Jobs," in Demetra Smith Nightingale and
Robert H. Haveman (eds.), The Work Alternative: Welfare Reform and
the Realities of the Job Market (Washington, D.C.: Urban Institute
Press, 1995).

INCREASING WORK INCENTIVES
ALSO SUCCEEDED WHEN
REINFORCED BY EMPLOYMENT
SUPPORTS
---------------------------------------------------------- Letter :3.4

Rather than enhancing work-related services, the New York Child
Assistance Program (CAP) took a different approach, providing an
incentive to work by increasing the amount of earned income working
recipients could keep. The program supported this incentive by
lowering barriers to reentering the job market; it provided child
care stipends in advance for clients to use during job search and
training. New York's program successfully increased employment and
earnings but did not reduce welfare receipt.

In contrast, two programs that increased work incentives and mandated
work program participation without expanding employment-related
services or child care assistance have not yet succeeded. Michigan's
"To Strengthen Michigan Families" (TSMF) program increased the amount
of the income disregard and also required participation in some form
of work program. AFDC clients were required to enter into "a social
contract" in which they had to complete 20 hours a week of broadly
defined "useful" activities of their own choice, such as education or
job search. However, no additional child care assistance was
provided to assist them in keeping this contract, and there were no
significant effects in the first year. During the second year, some
small effects were achieved for both earnings and welfare receipt for
some subgroups but typically only in the final quarter or month for
those with 2 years of data. Evaluation of the effects on the full
sample and their stability will have to await future reports.\8

Similarly, Alabama's Avenues to Self-Sufficiency through Employment
and Training Services (ASSETS) program increased work incentives and
strengthened its work registration and child support cooperation
requirements. In addition to raising the amount of the basic
earnings disregard, ASSETS raised the limits on savings and other
resources that families were allowed to have while remaining eligible
for AFDC. However, it also reduced the amount that could be
specifically deducted from earnings for child care expenses. The
implementation of their planned employment and training component was
delayed by 2 years, so available results do not fully reflect it.
This program has had no significant effects on welfare receipt or
average payment so far, although the evaluation is not yet complete.

Finally, like New York's CAP, Washington's Family Independence
Program (FIP) both provided economic incentives to encourage work and
increased child care assistance. It also aimed to increase
participation in education and training by offering small cash
bonuses to the participants. However, FIP's plans became difficult
to implement under budget restrictions, and caseloads increased
sharply without a corresponding increase in staff. Several features
were implemented minimally, such as improving a client's contact with
a case manager and increasing resources to pay for education and
training. In addition, the comparison group began getting very
similar services in 1990, about a year and a half into the program,
when JOBS was implemented in Washington state. Thus, it is difficult
to know how to attribute the significant increase in AFDC receipt and
payments experienced by this program's participants.

--------------------
\8 Table 2 includes only the first year's effects because data for
the full sample were not provided in the most recent report.

CONCLUSIONS
------------------------------------------------------------ Letter :4

Our review of state experiences suggests that the most successful
programs offered a broader package of employment-related services
than some proposed reform legislation encourages. The programs that
successfully increased employment and earnings and reduced welfare
receipt offered a broad mix of education, training, and
employment-related services and supports like those in the current
JOBS program. However, under H.R. 4, welfare recipients enrolled in
some education and training activities would not count toward meeting
the work program participation levels that are required in order to
avoid financial sanctions.

Some provisions of the proposed reforms--like the time limit on
benefit receipt--have not yet been tested and thus we cannot
confidently project the future effects of either those individual
provisions or the entire package of reforms. For example, imposing a
strict limit on the length of time a family can receive benefits
might influence participants' work behavior. This could influence
the effectiveness of both types of work programs, those offering
either a broad or narrow package of services; we simply do not have
similar past experiences to draw upon.

The modest results of even the most successful programs implies that
(1) within the current program structure, even increasing investments
in employment and support services will not quickly reduce caseloads
or welfare dependency, and (2) additional research is needed to
understand the barriers to better program performance and to develop
and test more successful approaches. However, it should be
recognized that some of these barriers may reside outside the welfare
program's control, including poor school preparation and the limited
availability and low wages of low-skill jobs.

Although federal funds for AFDC benefits have not been capped before,
the states have limited the funds available for their work programs.
Our review suggest that adequacy of funds can be a critical barrier
to the success of efforts to help clients move from welfare to work.
Three states in our review were unable to sustain or fully implement
their planned level of service because state budget constraints kept
them from increasing program capacity to match their growing
caseloads. However, by reducing federal prescriptions on the use of
these funds, the reform proposals aim to increase the states'
flexibility to manage such resource constraints.

Many of the program evaluations that we reviewed were conducted under
the requirement that waivers of federal regulations be rigorously
evaluated. The pending welfare reform legislation would reduce
federal regulation in order to foster further state experimentation,
but it would, thereby, effectively remove that evaluation requirement
and thus possibly reduce the incentive for future evaluations of
state experiments.

RECOMMENDATIONS
------------------------------------------------------------ Letter :5

We are not making recommendations in this report.

AGENCY COMMENTS AND OUR
RESPONSE
------------------------------------------------------------ Letter :6

The U.S. Department of Health and Human Services (HHS) commented on
a draft of this report and generally agreed with our conclusions but
argued that (1) the differences between the programs studied and
those that would be offered under H.R. 4 are so substantial that one
must conclude that the proposed reforms have not been tested and (2)
the report makes too strong a case for individual factors explaining
program success or failure and should instead describe the "package"
of services that may have led to certain effects. On the first
point, we agree that some features of the proposed reforms have not
been tested, but we believe that the states' experiences with the
program features that would be included under some of the current
proposals, as well as with other features that might be discouraged,
are relevant to consideration of these reforms. The text has been
altered, as necessary, to clarify this distinction. On the second
point, our general approach was to focus on packages of services.
However, where appropriate we have made changes to clarify this. In
addition, HHS provided suggestions for clarifications that we have
incorporated, as appropriate, throughout the text. HHS's comments
are reprinted in appendix II.

We will send copies of this report to the Chairman of the House
Subcommittee on Human Resources of the Committee on Ways and Means,
the Chairman of the Senate Finance Committee, the Secretary of Health
and Human Services, and others who are interested. Copies will also
be made available to others on request. If you have any questions
concerning this report or need additional information, please call me
on (202) 512-2900 or Robert L. York, Director of Program Evaluation
in Human Services Areas, on (202) 512-5885. Other major contributors
to this report are listed in appendix III.

Sincerely yours,

Joseph F. Delfico
Acting Assistant Comptroller General
for Program Evaluation
and Methodology

OUR EVALUATION SYNTHESIS
METHODOLOGY
=========================================================== Appendix I

We conducted an evaluation synthesis to identify approaches that have
successfully helped welfare clients achieve economic independence.
That is, we conducted a systematic review and analysis of the results
of previous evaluation studies of programs sharing this goal.
Whereas some evaluation syntheses examine studies of similar programs
to learn whether a treatment consistently has had the intended
effect, we examined studies of programs that used a range of
different approaches toward the same goal to learn which ones had
been successful.

Our evaluation synthesis consisted of several steps. The first step
entailed locating state welfare-to-work experiments and screening
them to identify rigorous evaluation studies with reliable results in
terms of the intended outcomes. In the second step, we identified
the commonalities and differences among the programs and assessed
whether these were related to the programs' demonstration of effects.
We then drew conclusions from the cumulative picture of existing
research about what approaches have helped AFDC clients move from
welfare to work.

SEARCH FOR AND SELECTION OF
STUDIES
--------------------------------------------------------- Appendix I:1

We identified relevant, potentially high-quality studies by searching
for as many existing evaluation studies as possible of
welfare-to-work programs for adult AFDC clients. Our criteria were

-- A program could have started before 1988 but its evaluation had
to have been reported after the passage of the Family Support
Act of 1988.

-- A study had to be testing, at least in part, the effect of
welfare-to-work initiatives on adult AFDC single parents.

-- The study measured the effects of the program on employment or
AFDC receipt.

-- The program's effects were measured through a comparison group
of nonparticipants (not necessarily a control group).

We searched for references to terms such as Family Support Act, JOBS,
and welfare reform in on-line bibliographic databases, including
CCRSP, ERIC, Sociological Abstracts, the PAIS International index of
the Public Affairs Information Service, and the NIS index of the U.S.
Department of Commerce. From the resulting abstracts, we were able
to screen the hundreds of citations down to six promising
evaluations.

In addition, we reviewed the bibliographies of research studies and
interviewed experts on welfare evaluation to identify other studies
we should consider. The experts identified an additional three
studies that had only just been published and therefore had not yet
appeared in databases or bibliographies. This gave us a total of
nine potentially high-quality evaluations of 10 different programs
from eight states. (The Riverside County GAIN evaluation included
treatments and effects sufficiently different from the rest of
California's GAIN evaluation that we considered them as separate
programs.\1 ) Finally, we confirmed this list of nine evaluations
with program and evaluation officials at HHS. They suggested several
studies that we might consider as background but no additional impact
evaluations.

We explicitly excluded programs focused exclusively on AFDC
teenagers, who may have very different needs. We also excluded
unpublished studies, implementation studies, evaluations of single
program features rather than complete programs, and many studies and
reviews that did not examine program effects. So, for example, we
excluded the Utah Unemployed Parents evaluation and the National Job
Training Partnership Act study, because they did not focus on single
parents.

--------------------
\1 Of course, Riverside GAIN also had much in common with the other
GAIN programs in California; while we considered Riverside
separately, we were unable to exclude Riverside data from the larger
evaluation of the GAIN program in six counties.

QUALITY REVIEW OF EVALUATION
STUDIES
--------------------------------------------------------- Appendix I:2

After identifying the 10 programs, we rated the quality of each study
to ensure that the research was rigorous and would produce reliable
results. We used six specific criteria, adapted from dimensions in
The Evaluation Synthesis, that together would reflect the rigor,
consistency, and reliability of an evaluation study:\2

-- similarity of the comparison group to the program's clients,

-- adequacy of the sample size for the analyses performed,

-- standardization of data collection procedures,

-- appropriateness of the measures used to represent the outcome
variables,

-- adequacy of the statistical or other methods used to control for
threats to validity, and

-- presence and appropriateness of the methods used to analyze the
statistical significance of observed differences.

We rated each study on a three-point scale from "unacceptable,"
because the report provided no information on the dimension or the
method was so flawed that the data were probably wrong, to
"acceptable," indicating that an appropriate method had been used or
attempts had been made to minimize problems.

--------------------
\2 See U.S. General Accounting Office, The Evaluation Synthesis,
GAO/PEMD-10.1.2 (Washington, D.C.: March 1992), p. 31. A similar
set of dimensions was used in U.S. General Accounting Office,
Teenage Pregnancy: 500,000 Births Year But Few Tested Programs,
GAO/PEMD-86-16BR (Washington, D.C.: July 1986), p. 34.

RESULTS OF QUALITY REVIEW
------------------------------------------------------- Appendix I:2.1

Most of the 10 programs had well-designed and rigorously structured
quasi-experimental or experimentally based evaluations. Six of the 9
evaluations had comparison groups formed by random assignment. In
Alabama and Washington, the comparison groups were drawn from AFDC
clients in demographically similar jurisdictions; in Massachusetts,
from a random sample of clients who did not start a program activity
within a specified time period. The rigor of our first screening of
programs was reflected when all 10 programs met our standards.
However, there were problems with the implementation and execution of
several of these programs, rather than with their evaluation designs,
that have to be kept in mind when interpreting them. A weakness, or
confounding factor, in 3 programs was the similarity in services
received by the program participants and the comparison group. (This
was a serious problem in Washington but only a minor problem in
California's SWIM and GAIN programs.) This type of confounding factor
means that the standard measure of a program's effect--the difference
between outcomes for the two groups--most likely underestimates the
program's potential effect.

OVERVIEW OF PROGRAMS
------------------------------------------------------- Appendix I:2.2

All 10 programs targeted single adult AFDC recipients, but 2 also
included a small number of unemployed couples in their results. The
recipients were overwhelmingly women. Some programs were statewide
while others were conducted in several counties or just one county.
A few were voluntary; most were mandatory. Some included mothers
with children younger than 6 but older than 3; others simply excluded
mothers with preschool children. Some delivered services directly;
others provided referrals or did nothing at all. Some programs
included new AFDC applicants, others included people already
enrolled, and some used both.

SYNTHESIS OF PROGRAM EVALUATION
RESULTS
--------------------------------------------------------- Appendix I:3

We focused on program effects on aspects of economic
self-sufficiency: employment, earnings, and public assistance
receipt (any effects reported on additional outcomes are not included
here). For each outcome in each study, we compared the participants
receiving program services (treatment group) with those of the
control (or comparison) group; statistically significant differences
were deemed to be program effects. The evaluation reports estimated
the likelihood that these differences stemmed from random chance by
using standard tests of statistical significance. For our
interpretation, we used a common significance level of 5 percent
(.05) or less, which was stricter than that used by some of the
evaluations.

We used a structured approach to look for program features or
characteristics that might explain why some programs had positive
effects and others did not, for each of the desired outcomes. First,
we hypothesized how each of a program's features might affect each of
its outcomes. Then we compared the results of the programs that had
each of those features and those that did not. We found mixed
results, and we found that programs tended to group in clusters of
features, which we examined for their successes.

We also examined features of the studies themselves that might have
influenced the reporting of statistically significant results, such
as whether the treatment and comparison groups received similar
services. We reviewed the comments of the evaluators about any
problems they had encountered in program or study implementation. We
considered not only what services were delivered and how but also how
services might have influenced the participants' behavior.

STRENGTHS AND LIMITATIONS OF
OUR SYNTHESIS
--------------------------------------------------------- Appendix I:4

Clearly, looking across the studies provided us with information not
readily seen by looking only at individual studies. Including
several program approaches in our review allowed us to see that while
a particular approach can be successful, this does not mean that it
is the only successful approach. Examining patterns across a group
of studies may allow inferences about which of the variety of a
program's components were probably responsible for its effects;
examining single studies ordinarily does not. However, our sample of
nine studies cannot provide conclusive answers, since there are many
potential differences between studies that might be related to why
one has significant results and another does not.

(See figure in printed edition.)Appendix II
COMMENTS FROM THE DEPARTMENT OF
HEALTH AND HUMAN SERVICES
=========================================================== Appendix I

(See figure in printed edition.)

The following are GAO's comments on the June 19, 1995, HHS letter.

GAO COMMENTS

1. The text has been changed to more clearly highlight the
differences in employment and training programs between the proposals
and the successful programs we reviewed and to indicate that the
states may not have implemented features exactly as they appear in
current bills. We have also clarified issues relating to program
design and environment differences.

2. Our general approach was to focus on the package of features
unique to the successful programs, while also noting differences
among them. Characteristics such as the age of a mother's youngest
child, noted in appendix I, did not distinguish the four successful
programs from the others. However, we have made changes to the text
to remove the impression that a single factor was claimed as
responsible for program failures.

3. The text has been changed to indicate study results that are not
yet final.

4. The text has been changed to indicate that in Massachusetts,
after registering for work, clients could choose whether to engage in
other employment-related activities.

5. The text has been clarified to indicate our belief in the
importance of the package of services provided by the successful
programs. Although some of these programs resemble the current JOBS
program, we do not believe they offer sufficient evidence from which
to draw conclusions about the JOBS program per se.

6. The names of the programs not using random assignment are now
noted in appendix I.

7. Table 2 has been changed to denote the availability of child care
in the SWIM program.

8. The text has been changed to clarify that the evaluation of the
statewide GAIN program was limited to six counties.

9. The Florida groups have been explained in the text.

MAJOR CONTRIBUTORS TO THIS REPORT
========================================================= Appendix III

PROGRAM EVALUATION AND METHODOLOGY
DIVISION

Stephanie L. Shipman, Assistant Director
Daniel G. Rodriguez, Project Manager

BIBLIOGRAPHY
=========================================================== Appendix 0

EVALUATIONS IN THIS REPORT
--------------------------------------------------------- Appendix 0:1

Fein, David J., Erik Beecroft, and John Bloomquist. The Ohio
Transitions to Independence Demonstration: Final Impacts for JOBS
and Work Choice. Cambridge, Mass.: Abt Associates, 1994.

Friedlander, Daniel, and Gayle Hamilton. SWIM: The Saturation Work
Initiative Model in San Diego: A Five-Year Follow-up Study. New
York: Manpower Demonstration Research Corporation, 1993.

Hamilton, William L., et al. The New York State Child Assistance
Program: Program Impacts, Costs, and Benefits. Cambridge, Mass.:
Abt Associates, 1993.

Hargreaves, Margaret, and Alan Werner. The Evaluation of the Alabama
Avenues to Self-Sufficiency Through Employment and Training Services
(ASSETS) Demonstration: Interim Implementation and Process Report.
Cambridge, Mass.: Abt Associates, 1993.

Kemple, James J., Daniel Friedlander, and Veronica Fellerath.
Florida's Project Independence: Benefits, Costs and Two-Year Impacts
of Florida's JOBS Program. New York: Manpower Demonstration
Research Corporation, 1995.

Kemple, James J., and Joshua Haimso. Florida's Project Independence:
Program Implementation, Participation Patterns, and First-Year
Impacts. New York: Manpower Demonstration Research Corporation,
1994.

Long, Sharon K., and Douglas A. Wissoker. The Evaluation of the
Washington State Family Independence Program: Final Impact Analysis
Report. Washington, D.C.: Urban Institute Press, 1993.

Long, Sharon K., Demetra Smith Nightingale, and Douglas A. Wissoker.
The Evaluation of the Washington State Family Independence Program.
Washington, D.C.: Urban Institute Press, 1994.

Nightingale, Demetra Smith, et al. Evaluation of the Massachusetts
Employment and Training (ET) Program. Washington, D.C.: Urban
Institute Press, 1991.

Riccio, James, Daniel Friedlander, and Stephen Freedman. GAIN:
Benefits, Costs, and Three-Year Impacts of a Welfare-to-Work Program.
New York: Manpower Demonstration Research Corporation, 1994.

Werner, Alan, and Robert Kornfeld. The Evaluation of "To Strengthen
Michigan Families." Second annual report. First-Year Impacts.
Cambridge, Mass.: Abt Associates, 1994.

Werner, Alan, and Robert Kornfeld. The Evaluation of "To Strengthen
Michigan Families." Third annual report. Second-Year Impacts.
Cambridge, Mass.: Abt Associates, 1995.

Werner, Alan, and David Rodda. Evaluation of the Alabama Avenues to
Self-Sufficiency Through Employment and Training Services, (ASSETS)
Demonstration. Interim impact report. Cambridge, Mass.: Abt
Associates, 1993.

OTHER STUDIES
--------------------------------------------------------- Appendix 0:2

Bloom, Howard S., et al. The National JTPA Study Overview: Impacts,
Benefits, and Costs of Title II-A. Cambridge, Mass.: Abt
Associates, 1994.

Brock, Thomas, David Butler, and David Long. Unpaid Work Experience
for Welfare Recipients: Findings and Lessons from MDRC Research.
New York: Manpower Demonstration Research Corporation, 1993.

Burghardt, John, and Anne Gordon. The Minority Female Single Parent
Demonstration: More Jobs and Higher Pay--How an Integrated Program
Compares with Traditional Programs. New York: Rockefeller
Foundation, 1990.

Burghardt, John, et al. The Minority Female Single Parent
Demonstration. Vol. 1. Summary Report. Princeton, N.J.:
Mathematica Policy Research, 1992.

Friedlander, Daniel. The Impacts of California's GAIN Program on
Different Ethnic Groups: Two-Year Findings on Earnings and AFDC
Payments. New York: Manpower Demonstration Research Corporation,
1994.

Greenberg, David, Robert Meyer, and Michael Wiseman. "When One
Demonstration Site Is Not Enough." Focus, 16:1 (Spring 1994), 15-20.

Gueron, Judith M., and Edward Pauly. From Welfare to Work. New
York: Manpower Demonstration Research Corporation, 1991.

Hamilton, Gayle. The JOBS Evaluation: Early Lessons from Seven
Sites. New York: Manpower Demonstration Research Corporation, 1994.

Hargreaves, Margaret, et al. Illinois Department of Public Aid:
Community Group Participation and Housing Supplementation
Demonstration. Fourth interim report. Cambridge, Mass.: Abt
Associates, 1994.

Levin-Epstein, Jodie, and Mark Greenberg. The Rush to Reform: 1992
State AFDC Legislative and Waiver Actions. Washington, D.C.: Center
for Law and Social Policy, 1992.

Manski, Charles F., and Irwin Garfinkel (eds.). Evaluating Welfare
and Training Programs. Cambridge, Mass.: Harvard University Press,
1992.

Nightingale, Demetra Smith, and Robert H. Haveman (eds.). The Work
Alternative: Welfare Reform and the Realities of the Job Market.
Washington, D.C.: Urban Institute Press, 1995.

O'Neil, June E. Congressional Budget Office Cost Estimate of H.R.
1214, The Personal Responsibility Act of 1995. Washington. D.C.:
Congressional Budget Office, 1995.

Porter, Kathryn H. Making JOBS Work: What The Research Says About
Effective Employment Programs for AFDC Recipients. Washington, D.C.:
Center on Budget and Policy Priorities, 1990.

U.S. Department of Labor. What's Working (and What's Not): A
Summary of Research on the Economic Impacts of Employment and
Training Programs. Washington, D.C.: 1995.

U.S. Department of Labor, Employment and Training Administration.
"American Poverty: The Role of Education, Training, and Employment
Strategies in the New Anti-Poverty Struggle." Evaluation Forum, 10
(Summer 1994).

Zambrowski, Amy, and Anne Gordon. Evaluation of the Minority Female
Single Parent Demonstration: Fifth Year Impacts at CET. Princeton,
N.J.: Mathematica Policy Research, 1993.

RELATED GAO PRODUCTS

Welfare to Work: Most AFDC Training Programs Not Emphasizing Job
Placement (GAO/HEHS-95-113, May 19, 1995).

Welfare to Work: Participants' Characteristics and Services Provided
in JOBS (GAO/HEHS-95-93, May 2, 1995).

Welfare to Work: Current AFDC Program Not Sufficiently Focused on
Employment (GAO/HEHS-95-28, Dec. 19, 1994).

Child Care: Current System Could Undermine Goals of Welfare Reform
(GAO/T-HEHS-94-238, Sept. 20, 1994).

Families on Welfare: Sharp Rise in Never-Married Women Reflects
Societal Trend (GAO/HEHS-94-92, May 31, 1994).

*** End of document. ***