Welfare Reform: Work-Site-Based Activities Can Play an Important Role in
TANF Programs (Letter Report, 07/28/2000, GAO/HEHS-00-122).

Pursuant to a congressional request, GAO reviewed Temporary Assistance
for Needy Families (TANF) work-site programs, focusing on: (1) the key
characteristics of work-site activities states and localities are using
in their TANF programs; (2) the key challenges to implementing and
administering work-site activities and some of the ways that states and
localities have addressed these challenges; and (3) what is known about
the effects work-site activities have had on participants' ability to
successfully make the transition to unsubsidized employment and on their
communities.

GAO noted that: (1) the work-site activities at the locations GAO
visited have in common the assignment of TANF recipients to public or
private sector employers to perform work in areas such as building
maintenance, clerical work, unskilled health care, and food service; (2)
however, these work-site activities differ in two key ways; (3) they
play varying roles in their state's or locality's TANF program, with
some TANF programs requiring almost all recipients who have not found a
job after a fixed period of time to participate in work-site activities
and others targeting work-site participation to individuals who face
multiple barriers to work; (4) the work-site activities GAO reviewed
range from those in which participants are expected to work in exchange
for a welfare check to those in which participants: (a) receive
paychecks subsidized by TANF funds or other funding sources from which
payroll taxes, including Social Security, are deducted; and (b) qualify
for the Earned Income Credit (EIC); (5) states and localities GAO
visited have overcome multiple challenges in implementing and
administering their work-site activities, including the challenge of
recruiting employers and involving participants; (6) states and
localities GAO visited generally have outcome data on key aspects of
their TANF programs, such as the number of TANF recipients who find
unsubsidized employment or leave welfare, less is known about outcomes
specifically for work-site activity participants, and information is not
available on the effectiveness of work-site activities in promoting
employment; (7) in cases where outcome the data on employment and
earnings for work-site participants are available, data are not
comparable among the different work-site activities; (8) while outcome
data are valuable in helping to understand the extent to which work-site
participants become employed, a evaluation using these data would be
needed to determine a work-site activity's effectiveness; (9) program
administrators, participants, and others suggest that work-site
activities can help participants with no prior work experience develop a
resume and at the same time provide community services; (10) however,
some critics say that some work-site activities do not provide the
necessary level or experience of skills to enable participants to make
the transition to unsubsidized employment that leads to economic
independence; and (11) some critics also say that work-site participants
are brought in to replace regular salaried employees.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  HEHS-00-122
     TITLE:  Welfare Reform: Work-Site-Based Activities Can Play an
	     Important Role in TANF Programs
      DATE:  07/28/2000
   SUBJECT:  Workfare
	     Public assistance programs
	     Welfare benefits
	     Welfare recipients
	     Employment or training programs
	     Disadvantaged persons
	     State-administered programs
IDENTIFIER:  HHS Temporary Assistance for Needy Families Program
	     San Franciso Community Jobs Initiative (CA)
	     Oregon JOBS Plus Program
	     Chicago WorkFirst Program (IL)
	     Earned Income Credit
	     EIC
	     Philadelphia Paid Work Experience Program (PA)

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Testimony.                                               **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************

GAO/HEHS-00-122

Appendix I: Comments From the Department of Health and Human Services

38

Appendix II: GAO Contacts and Staff Acknowledgments

40

42

46

Table 1: Federal Law Sets Increasing TANF Participation
Requirements for All Families for Fiscal Years 1997-2002 7

Table 2: Key Characteristics of the Work-Site Activities Reviewed 10

Table 3: Selected Data on TANF Caseload for Each Work-Site
Activity, November 1999 14

Table 4: Information on Administration and Funding for Nine
Work-Site Activities 18

Table 5: Outcomes for Five Work-Site Activities 28

Figure 1: TANF Recipients Working or Participating in Work-
Related Activities, Fiscal Year 1998 8

Figure 2: Continuum of Work-Site Activities 15

Figure 3: Characteristics of Work-Site Activities Along the
Continuum 16

AFDC Aid to Families With Dependent Children

CWEP Community Work Experience Program

EIC Earned Income Credit

FICA Federal Insurance Contributions Act

HHS Department of Health and Human Services

MDRC Manpower Demonstration Research Corporation

PRWORA Personal Responsibility and Work Opportunity Reconciliation Act of
1996

TANF Temporary Assistance for Needy Families

WEP Work Experience Program

Health, Education, and
Human Services Division

B-283158

July 28, 2000

The Honorable Daniel P. Moynihan
Ranking Minority Member
Committee on Finance
United States Senate

The Honorable Sander M. Levin
House of Representatives

The Personal Responsibility and Work Opportunity Reconciliation Act of 1996
(PRWORA) significantly changed federal welfare policy for low-income
families with children, building upon and expanding state-level reforms.
Under the Temporary Assistance for Needy Families (TANF) block grant created
by PRWORA, states have much greater flexibility than before to design and
implement programs that meet state and local needs. At the same time, states
must enforce a lifetime limit of 5 years (or less, at state option) on the
length of time adults receive federal assistance and impose federal work
requirements on most adults receiving aid. More specifically, states are to
require adults receiving TANF aid to be engaged in work or work-related
activities, as defined by the states, after 2 years of receiving assistance.
In addition, to avoid financial penalties, states must ensure that
increasing proportions of adult TANF recipients are working or engaged in
federally specified activities for increasing numbers of hours through
fiscal year 2002. Recipients' participation in PRWORA-specified activities,
including unsubsidized employment, can be counted toward this target
participation rate. These activities can be grouped into three broad
categories: those that focus on finding a job, those that take place in a
classroom or other educational setting, and those that take place at a job
or work site. We refer in this report to activities that are based at a work
site--with the exception of unsubsidized employment--as work-site
activities: subsidized employment, community service, on-the-job training,
and work experience. Many policymakers anticipate that as the federal
participation rates increase and as state welfare caseloads become
increasingly composed of those who are most difficult to employ--or if the
economy falters--states will rely more heavily on work-site activities to
meet federal work requirements or states' own welfare reform goals.

To better understand work-site activities currently in place and the role
they may play as TANF evolves, you asked us to review (1) the key
characteristics of work-site activities states and localities are using in
their TANF programs, (2) the key challenges to implementing and
administering work-site activities and some of the ways that states and
localities have addressed these challenges, and (3) what is known about the
effects work-site activities have had on participants' ability to
successfully make the transition to unsubsidized employment and on their
communities. To identify characteristics of work-site activities used across
the nation, we analyzed participation data reported by states to the
Department of Health and Human Services (HHS), which administers TANF at the
federal level, and interviewed selected state TANF administrators and
experts on TANF work programs. On the basis of this work, we selected for
review nine work-site activities representing a range of activity types:
four at the state level (two in Massachusetts and one each in Oregon and
West Virginia) and five at the local level (in Chicago; New York City;
Onondaga County, New York; Philadelphia; and San Francisco).1 To describe
work-site activities in more detail and to understand implementation
challenges and performance results, we visited these locations, collecting
data and documentation and conducting interviews with state and local TANF
administrators and others involved in the work-site activities, including
contractors and some clients and client advocacy groups. Although we
identified a broad range of work-site activities in this way, they were
judgmentally selected, and our results cannot be generalized to work-site
activities across the nation. We conducted our work from July 1999 to July
2000 in accordance with generally accepted government auditing standards.

The work-site activities at the locations we visited have in common the
assignment of TANF recipients to public or private sector employers to
perform work in areas such as building maintenance, clerical work, unskilled
health care, and food service. However, these work-site activities differ in
two key ways. First, they play varying roles in their state's or locality's
TANF program, with some TANF programs requiring almost all recipients who
have not found a job after a fixed period of time to participate in
work-site activities and others targeting work-site participation to
individuals who face multiple barriers to work, such as limited work
experience or English-language speaking skills. For example, New York City's
TANF program assigns most adults to work-site activities soon after they
begin receiving cash aid. Nationwide, New York City's program had the
largest number of TANF recipients, almost 21,000, assigned to work sites in
November 1999. In contrast, Chicago's TANF program targets participants with
multiple barriers for work-site activities, relying on other activities for
recipients with different employment-related needs. Work-site activities
also differ in how closely they resemble welfare as opposed to employment,
particularly in regard to participant payments. The work-site activities we
reviewed range from those in which participants are expected to work in
exchange for a welfare check to those in which participants (1) receive
paychecks subsidized by TANF funds or other funding sources from which
payroll taxes, including Social Security, are deducted and (2) qualify for
the Earned Income Credit (EIC), which can provide additional income.

States and localities we visited have overcome multiple challenges in
implementing and administering their work-site activities, including the
challenge of recruiting employers and involving participants. In New York
City, for example, the Mayor's decision to direct city government agencies
to develop work-site assignments for welfare recipients was key to
implementing such a large-scale program. Even in much smaller programs,
administrators needed to develop relationships with employers and emphasize
that they would save recruiting costs and benefit from access to
prescreened, prequalified job candidates. Once work sites are in place,
administrators must attend to participants' child care and transportation
needs. Most of the sites we visited address these needs by having staff help
participants secure appropriate child care services and transportation
assistance in the form of fare cards or passes for public transportation. In
addition, to deal with motivational problems that can interfere with
attendance, administrators are relying on a combination of "carrots and
sticks." For example, San Francisco offers financial incentives to
participants ranging in value from $25 to $150 for good attendance or for
staying on the job, and all states have policies requiring reductions in or
elimination of cash aid, called sanctions, if a TANF recipient fails to
comply with work requirements.

While the states and localities we visited generally have outcome data on
key aspects of their TANF programs, such as the number of TANF recipients
who find unsubsidized employment or leave welfare, less is known about
outcomes specifically for work-site activity participants, and current
information is not available on the effectiveness of work-site activities in
promoting employment. In those cases in which outcome data on employment and
earnings for work-site participants are available, the data are not
comparable among the different work-site activities. In addition, while
outcome data are valuable in helping to understand the extent to which
work-site participants become employed, a rigorous evaluation using these
data would be needed to determine a work-site activity's effectiveness.
Program administrators, participants, and others suggest that work-site
activities can help participants with no prior work experience develop a
resume and at the same time provide community services, such as preparing
food for delivery to the elderly. However, some critics say that some
work-site activities do not provide the necessary level of experience or
skills to enable participants to make the transition to unsubsidized
employment that leads to economic independence; some critics also say that
work-site participants are brought in to replace regular salaried employees.

To provide more information on work-site activities, we recommend that HHS
(1) require states to provide more information on the key characteristics of
work-site activities they are using; (2) use its regional offices,
HHS-sponsored conferences, and other means available to collect and
disseminate information on promising work-site approaches; and (3) support
evaluations to increase knowledge about the effectiveness of various types
of work-site activities. HHS generally agreed with our recommendations and
cited actions planned or under way to address them.

The welfare reform law transformed federal welfare policy for needy families
with children by replacing the individual entitlement to benefits under the
61-year-old Aid to Families With Dependent Children (AFDC) program with TANF
block grants to the states. The TANF block grant makes $16.8 billion
available to the states each year through 2002, and in June 1999, states
were providing TANF assistance to about 2.5 million families. Under TANF,
aid recipients face stronger work requirements than under its predecessor
program, and the population that can be required to work now includes all
parents, rather than only those with children 3 years of age or older. If
recipients fail to participate as required, states must at least reduce the
family's grant and may opt to terminate the grant entirely. In addition,
states must now enforce a 5-year limit (or less, at state option) on the
length of time a family may receive federal TANF assistance.

States must require adults in families receiving TANF assistance to
participate in work or work-related activities, as defined by the states,
after receiving assistance for 24 months (or sooner, at state option). In
addition, to avoid financial penalties, states must ensure that at least a
specified minimum percentage of adults are participating in work or
work-related activities each year. The required number of hours of
participation and the percentage of a state's caseload that must participate
to meet mandated rates have steadily increased since 1997, as shown in table
1.

                                  1997  1998  1999  2000 2001  2002
 Minimum weekly hours             20    20    25    30   30    30
 Minimum percentage participation 25    30    35    40   45    50

Note: PRWORA also sets higher minimum hour and participation rates for
two-parent families. The minimum hour requirement is 35 hours, and the
minimum participation rate rises from 75 percent in 1997 to 90 percent in
1999 and beyond.

The welfare reform law also allows each state to reduce its annual mandated
participation rate by an amount equal to the percentage that the state's
welfare caseload has declined since fiscal year 1995. As a result, while
states were to meet a 30-percent minimum participation rate in fiscal year
1998, the adjusted required participation rates ranged from 0 percent in
four states--Indiana, Oklahoma, Oregon, and Wisconsin--to 28 percent in
Hawaii. In fiscal year 1998, every state met its adjusted participation rate
for single-parent families. As figure 1 shows, the largest percentage of
TANF recipients meeting federal participation standards were engaged in
unsubsidized employment--that is, they combined welfare and work. About 17
percent of TANF recipients were engaged in work-site activities--subsidized
employment, community service, on-the-job training, and work experience.

Note: Percentages add up to more than 100 because participants may be
counted in more than one work activity.

Source: HHS, Administration for Children and Families.

As specified in PRWORA, approved activities include, in addition to
unsubsidized employment and work-site activities,2 those focused on finding
a job and those that take place in a classroom or other educational setting.
The activities countable for participation rate purposes under PRWORA are
more work-focused than those under AFDC, which generally allowed more
education and training activities. Moreover, although subsidized employment,
on-the-job training, and work experience were approved activities under AFDC
as well, requirements relating to a number of aspects of each of these
activities were specified, including such features as length of
participation allowed and type and amount of payments to be made to
participants and employers.3 PRWORA does not specify any of the
characteristics of these activities, and in the general explanation of the
final TANF rule, HHS stated that it would not further define the terms in
order to provide states with maximum flexibility in designing their TANF
programs. HHS does, however, require states to provide it with state TANF
plans and annual reports that include at least some information on TANF
program activities, although detailed information describing work-site
activities is not required. At the time of our review, little information
describing work-site activities nationwide was available at the federal
level.

and in How Much They Mirror Unsubsidized Employment

The TANF programs in the states and localities we visited all emphasize the
need for TANF recipients to find unsubsidized employment. To help those
recipients who have difficulty finding unsubsidized employment, states and
localities have implemented work-site activities that differ in two key
ways: the extent to which unemployed TANF recipients are required to
participate in the activity and the degree to which the work-site activity
has characteristics of an unsubsidized job. Some TANF programs require all
unemployed recipients to participate in a work-site activity either
immediately after becoming a cash assistance recipient or after a specified
period of time on assistance. Other TANF programs target their work-site
activities to recipients with specific barriers to employment, such as
limited work experience. Moreover, some states and localities have
implemented work-site activities for which participants receive a subsidized
paycheck and qualify for the EIC; participants in other work-site activities
work in exchange for their cash assistance check.

Although in all the locations we visited work-site participants generally
are performing the same type of work--such as housekeeping, janitorial,
clerical, unskilled health care, or food service--we identified differences
in the extent to which unemployed TANF recipients are required to
participate in these work-site activities. These differences primarily
concern when unemployed adult recipients are required to participate and how
broadly the work-site activity requirement is applied to the TANF caseload.
Accordingly, we categorized the work-site activities by one of three
characteristics: those that require immediate work-site participation, those
that require work-site participation after a specified delay, and those that
are targeted toward a particular group. Table 2 shows the work-site
activities we reviewed and how we classified them.

                                                  Number of  Maximum length
   Work-site        Who is            When          hours        of time
   activity      required to    participation is required at   allowed to
                participatea       required
                                                  work site   participate
 Immediate work-site requirementb
 New York City                                               As long as
                                                             receiving cash
 Work           All unemployed Once approved for 35 hours    assistance, up
 Experience     recipients     cash aid          per weekc   to the 5-year
 Program (WEP)                                               aid limit
                                                             As long as
 Onondaga                                        Varies;     receiving cash
 County, N.Y.   All unemployed 4 weeks after     generally   assistance, up
 WEP            recipients     cash aid approval 20 hours    to the 5-year
                                                 per weekc
                                                             aid limit

 Massachusetts                                               As long as
                All unemployed 60 days after     20 hours    receiving cash
 Community      recipientsd    cash aid approval per week    assistance, up
 Service                                                     to the 2-year
                                                             aid limit
 Delayed work-site requirement
 San Francisco                 18-24 months
                All unemployed after signing     32 hours
 Community Jobs recipients     welfare-to-work   per week    9 months
 Initiative                    plane
 Philadelphia
                All unemployed After 24 months   20 hours
 Paid Work      recipients     of receiving cash per weekf   6 months
 Experience                    aid
 Targeted
 work-site
 activity
                Recipients
                with no child                                As long as
 Chicago        under age 13   Any time during   Up to 40    receiving cash
                or with        receipt of cash   hours per   assistance, up
 WorkFirst      multiple       aid               week        to the 5-year
                barriers to                                  aid limit
                working

 Massachusetts  Recipients
                with multiple  At participant's  30-40 hours
                                                             39 weeksg
 Supported Work barriers to    option            per week
                working
                                                             No overall
                Recipients who                               limit
                                                             specified, but
 Oregon         need work to                     No minimum; recipients may
                develop        At participant's  generally
                                                             be at one
 JOBS Plus      specific job   option            30-40 hours particular
                skills or work                   per week
                habits                                       site for
                                                             maximum of 6
                                                             months
 West Virginia                                               As long as
                Recipients
                               Any time during   Up to 30    receiving cash
 Community Work with little or receipt of cash   hours per   assistance, up
 Experience     no work        aid               weekh       to the 5-year
 Program (CWEP) experience                                   aid limit

aAll locations we visited exempt some portion of the TANF caseload from the
requirement to work or participate in a work-related activity on the basis
of age, disability, or other reasons specific to the location. This category
refers to individuals who have not been exempted from the location's
requirement.

bIn addition to individuals applying for TANF, those already receiving TANF
may be required to participate.

cThe maximum number of hours required equals the TANF grant plus the food
stamp amount divided by the minimum wage. Most TANF recipients in New York
City receive a large enough grant to require 35 hours of work or other
concurrent activity per week.

dThis applies only to recipients whose youngest child is over age 5.
Massachusetts is still operating under a waiver granted under prereform
rules to exempt parents with younger children.

eCalifornia requires new TANF recipients to meet a community-service
requirement 18 months after signing a welfare-to-work plan, or after 24
months if they were on assistance when the community-service requirement
took effect. Counties have the option to extend the time to 24 months for
new recipients on a case-by-case basis. The welfare-to-work plan documents
how the participant will meet the requirement to participate in a work
activity for 32 hours per week during the first 18 to 24 months.

fParticipant engages in 20 hours of paid work experience and 5 hours of
other countable activity, such as budgeting or life skills classes.

gIf a participant has not been hired at the work site after 39 weeks, the
placement will be reevaluated. If the participant may be hired at that site,
the participant may receive an extension at the work site. Otherwise,
another work-site position for the participant will be pursued.

hParticipants must meet the federal standard of 30 hours per week, but the
number of hours may be less if the TANF grant plus the food stamp amount
divided by the minimum wage equals less than 30 hours per week.

Source: States and localities visited.

Three of the TANF programs whose work-site activities we reviewed have an
immediate work-site requirement for almost all unemployed recipients. In
these programs, TANF recipients who have not found jobs after an initial
search for work are required to participate in a specific work-site
activity, for a fixed amount of time per week, almost immediately or soon
after they have been approved for cash assistance. Not only do these TANF
programs provide a way for recipients to meet their participation
requirements, but they also discourage those who are already working, or who
are able to find a job, from applying for welfare.4

ï¿½ In New York City, the TANF program assigns almost all unemployed
recipients to its WEP immediately after cash aid approval. Recipients who
have been identified as having medical limitations or limited
English-language skills may be referred to other activities, such as the
recently implemented Personal Roads for Individual Development and
Employment project or the ongoing Begin Employment Gain Independence Now,
both of which combine a work-site assignment with specialized activities or
education and training as needed to meet participants' needs. According to
New York City officials, they are planning to combine work-site activities
with other activities such as job search and courses for the General
Education Diploma even if recipients do not have specific barriers.

ï¿½ In Onondaga County, which includes the city of Syracuse in New York State,
the TANF program requires unemployed recipients to participate in a 4-week
class after being approved for cash aid that is designed to help them find a
job and determine their interests and skills. Recipients who are not
employed at the end of the class are assigned to a work site.

ï¿½ In Massachusetts' TANF program, adult recipients whose youngest child is
over the age of 5 have 60 days from application approval to find a job. If a
recipient is not employed or participating in another work activity at that
point, he or she must participate in the Community Service work-site
activity for 20 hours a week.

Two of the other TANF programs whose work-site activities we reviewed also
require unemployed recipients to participate in a work-site activity, but
the requirement is delayed. In San Francisco and Philadelphia, recipients
are allowed up to 24 months to find unsubsidized employment, after which
they are assigned to a work-site activity. At the time of our visit, San
Francisco was operating its work-site activity, called Community Jobs
Initiative, on a pilot basis for recipients who had been receiving aid for
at least 12 months, had little or no work experience, and had been unable to
find a job. In anticipation of recipients reaching the 18- to 24-month mark
in the summer of 2000 and thus needing to meet the state community service
requirement, San Francisco officials are expanding the pilot to the entire
caseload. Pennsylvania requires participation in its Paid Work Experience
activity after 2 years.

In contrast to work-site activities designed for all unemployed recipients,
some of the TANF programs we visited target work-site activities to
recipients with specific characteristics. TANF programs in Chicago, Oregon,
and West Virginia and Massachusetts' Supported Work offer work-site
activities designed to address barriers that recipients face in finding
unsubsidized employment, such as lack of specific job skills, poor work
habits, or inability to exhibit appropriate workplace behaviors; limited
reading proficiency; limited English-language speaking skills; and little or
no work experience.5

The work-site activities we reviewed varied in both the absolute number of
participants and the proportion of unemployed recipients participating in
the work-site activity, ranging from 83 participants in San Francisco's
Community Jobs Initiative to approximately 21,000 participants in New York
City's WEP. Several factors influenced the scale of the work-site activity,
including the role of the work-site activity within the TANF program, the
total adult TANF caseload size, the number of employed TANF recipients, and
the number of adult recipients participating in other activities. To better
understand the role of particular work-site activities in our sites, we
calculated the percentage of unemployed TANF recipients meeting their
participation requirement through participation in the work-site activity
under review. Table 3 shows that two of the broad-coverage TANF work-site
activities had higher proportions of participants than TANF programs that
use a targeted approach. However, as in the case of Chicago, a work-site
activity with a targeted approach can involve a significant percentage of
active TANF participants if its target population is sizable. With the
exception of New York City, the scale of the work-site activities in
absolute numbers is relatively small.

                                                                   Percentage of
                                      Unemployed     Unemployed    all
              Total     Adult TANF    adult TANF     adult TANF    participants
 Work-site    adult     recipients in recipients     recipients    that
 activity     TANF      unsubsidized  participating  participating participate
              caseload  employment    in any         in this       in this
                                      activitya      work-site     work-site
                                                     activity
                                                                   activity
 Immediate work-site requirement
 New York City
              156,357   29,374        38,652         20,951        54
 WEP
 Onondaga
 County, N.Y.
              3,015     987           364            209           57
 WEP
 Massachusetts

 Community    29,769    4,716         8,036          696           9
 Service
 Delayed work-site requirement
 San Francisco

 Community    5,622     1,891         2,864          83            3
 Jobs
 Initiative
 Philadelphia

 Paid Work    37,852    7,158         5,142          1,528         30
 Experience
 Targeted
 work-site
 activity
 Chicago
              59,252    24,282        7,918          2,119         27
 WorkFirst
 Massachusetts

 Supported    29,769    4,716         8,036          564           7
 Work
 Oregonb
              11,541    2,775         10,287         403           4
 JOBS Plus
 West Virginia
              13,838    1,816         5,768          1,060         18
 CWEP

Note: Recipients in the process of being sanctioned are not included in this
chart. Recipients may be counted in more than one work activity.

aOther activities are those defined by the states and those that are listed
in the federal law.

bThese figures include both one- and two-parent families.

Source: GAO analysis of state administrative data.

Welfare-Like

Another key factor distinguishing one type of work-site activity from
another is how much these activities mirror a "real job." All work-site
activity participants were generally performing similar types of work , such
as housekeeping, janitorial service, clerical work, unskilled health care,
or food service. However, what differs among work-site activities is the
extent to which participants' financial benefits and responsibilities are
more similar to those of traditional welfare programs than to those of
unsubsidized employment. As shown in figure 2, the work-site activities we
reviewed can be arrayed along a continuum, ranging from activities that are
more welfare-like to those that are more job-like.

Work-site activities that are more welfare-like generally emphasize a TANF
recipient's obligation to work in exchange for cash assistance. As shown in
figure 3, participants in these work-site activities receive a TANF check,
and the maximum number of hours they are required to work is calculated by
dividing the TANF grant amount plus food stamp benefits by the minimum wage.
A slight difference exists in Onondaga County, where work-site positions are
supplemented with job search assistance while the participants are in the
work-site activity, thus emphasizing the need for participants to find
unsubsidized employment.

aIntermediary organizations issue TANF benefits in the form of a paycheck,
but the benefits are not considered earned income.

bBecause of policies that disregard some earnings in calculating recipients'
eligibility for and amount of aid (called earnings disregards), some
work-site activity participants may receive a reduced TANF benefit check in
addition to the paycheck. For instance, Philadelphia disregards 50 percent
of Paid Work Experience participants' earnings; thus, participants may be
eligible to receive a small supplemental TANF check.

cFederal Insurance Contributions Act.

In the middle of the continuum, Philadelphia's Paid Work Experience and San
Francisco's Community Jobs Initiative are more job-like than those discussed
above. Participants in both activities receive paychecks, qualify for the
EIC, have payroll taxes deducted, and are able to earn more than their grant
amount.6 Nonetheless, participants' paychecks are subsidized by TANF funds
and are issued from an intermediary organization, such as a nonprofit
organization, rather than the work-site employer, and participants are not
considered employees of the work site.7

Finally, work-site activities on the right side of the continuum have
additional characteristics not found in Philadelphia's and San Francisco's
work-site activities. Participants in Oregon's JOBS Plus are considered
employees of the work site. They are paid by the work-site employer and make
at least the state minimum wage. The welfare agency reimburses the employer
for 100 percent of the minimum wage for up to 40 hours a week, for workers'
compensation, and for other payroll taxes.8 In addition, while there is no
contractual obligation on the part of work-site employers to hire JOBS Plus
participants, there is a mutual expectation that work-site employers will
hire participants for an unsubsidized job if they perform well. Placements
of participants at subsidized work sites in Massachusetts' Supported Work
are also expected to lead to unsubsidized employment with that work-site
employer. The key difference is that Supported Work participants are
considered employees of an intermediary organization rather than of the work
site.

Along with differences in participants' financial benefits and
responsibilities are differences in how the work-site activities are
administered and funded, as shown in table 4. Generally, the work-site
activities that are more welfare-like are administered by the state or local
TANF agency, which places recipients at work sites and monitors them to
ensure that they are meeting their work-site obligation. However, the New
York City TANF agency uses a hybrid approach, assigning recipients to either
a city agency or a community-based organization that supervises participants
at a work site and monitors their attendance. In October 1999, almost 70
percent of New York City WEP participants were assigned in large groups to
public sector agencies, where participants performed such tasks as cleaning
city court buildings and providing administrative support, among others. New
York City WEP participants referred to a community-based agency performed
the same types of work, such as maintenance or clerical tasks, but for
nonprofit organizations.

 Work-site activity Work-site administrator    Funding source
 Massachusetts
 Community Service  TANF agency                TANF
 New York City      TANF agency, city
                    agencies, nonprofit and    TANF; federal
 WEP                for-profit contractors     welfare-to-work grant
 West Virginia
                    TANF agency                TANF
 CWEP
 Onondaga County,
 N.Y. WEP           Nonprofit contractor       TANF
 Chicago
                    Nonprofit and for- profit
                                               TANF
 WorkFirst          contractors
 Philadelphia
                    Nonprofit and for profit   TANF; federal
 Paid Work          contractors                welfare-to-work grant;
 Experience                                    private foundationsa

 San Francisco                                 TANF; federal
                                               welfare-to-work grant; city
 Community Jobs     Nonprofit contractors      general fund revenues;
 Initiative
                                               private foundations
 Massachusetts
                    Nonprofit contractors      TANF; state general fund
 Supported Work                                revenues
 Oregon
                    Nonprofit contractors      TANF; food stamps (under
 JOBS Plus                                     waiver)

aA nonprofit contractor we visited in Philadelphia, the Transitional Work
Corporation, which is headed by a board that includes the city's mayor,
state-level TANF officials, and others, had received a grant from a private
foundation to serve Paid Work Experience participants.

Locations with more job-like work-site activities typically contract with
external organizations to administer the work-site activity. For example, in
Philadelphia and San Francisco, contractors are responsible for identifying
work sites; assessing and placing participants at work sites; issuing
participants' paychecks; monitoring the quality of the work site to ensure
that the position is meeting the participant's needs; and providing support
to help participants handle any problems that may interfere with their
attendance, such as a breakdown in child care arrangements. TANF agencies
pay contractors for enrolling participants in the activity and placing
participants at a work site. Contractors can receive additional payments for
placing participants in unsubsidized jobs and if the individual retains that
job for a period of time.

Work-site activities in Philadelphia, San Francisco, Oregon, and
Massachusetts' Supported Work incur additional costs, such as payroll taxes
and workers' compensation premiums, not associated with work-site activities
in which participants work in exchange for a cash assistance check. To cover
these costs, contractors and TANF agencies in these locations have tapped
into non-TANF resources, including, in some cases, the federal
welfare-to-work grant administered by the Department of Labor.9 These
locations use contractors to administer these work-site activities and often
rely on performance-based contracts to encourage the contractors through
financial incentives to develop work-site positions that match the skills
and abilities of TANF recipients.

Work-Site Activities

States and localities must overcome a number of challenges during the
initial implementation and ongoing operation of work-site activities. Some
of these challenges include designing and implementing, within state and
federal laws, a program that reflects the vision of the local elected
leadership. Program administrators also must garner community support and
identify and involve employers. Once the work-site activity has been
established, program administrators face the additional challenge of getting
TANF recipients to report to work sites and motivating them to take
advantage of opportunities to improve their work habits and skills through
participation in these activities.

Economic Factors

Agency officials at each of the locations we visited said that they are
challenged to implement work-site activities that reflect the preferences of
local elected leaders, are in accordance with applicable laws, and take
local economic conditions into account. In particular, elected leaders
influence the choices that agencies make regarding work-site activities for
TANF recipients and can facilitate or hinder the implementation of
particular activities. For example, in New York City, the Mayor chose to
expand a work-site activity that existed prior to welfare reform when
implementing work requirements for all families receiving cash assistance.
The existing work-site activity, New York City's WEP, initially required
only childless adults receiving state-funded cash assistance to work in city
agencies. In 1996, the Mayor extended the work requirement to certain
families receiving AFDC, both to ensure that all able-bodied recipients in
the city were engaged in work and to discourage their dependency on cash
assistance. The Mayor directed the heads of municipal agencies to bring more
WEP workers into their agencies, and, in response, city commissioners had to
create the large number of assignments needed to accommodate the increase in
mandatory participants. In San Francisco, to meet state community service
requirements established after federal welfare reform, the Mayor and Board
of Supervisors, after considering unpaid work programs such as the one in
New York City, chose to implement a more job-like program in which
participants receive wages, subsidized by TANF funds, in exchange for their
participation. City leaders in San Francisco believed that this would be
more effective in moving TANF recipients to unsubsidized employment and
would have the additional benefit of making participants eligible for the
EIC.10 As a result, San Francisco's TANF agency has been charged with
creating a highly individualized program with an intensive case management
component.

Minimum wage requirements can also influence the extent to which states may
use particular work-site activities. In particular, the Department of Labor
has determined that under the Fair Labor Standards Act of 1938, as amended,
TANF recipients who are required to participate in work-site activities must
receive assistance that is at least equal in value to the minimum wage
multiplied by the number of hours worked.11 This requirement affects the
number of hours that states may require participants to engage in work-site
activities, particularly in states that provide relatively low levels of
TANF cash assistance.12 For example, while the maximum monthly amount of
TANF cash assistance for a family of three in San Francisco is $626, in West
Virginia, the maximum monthly grant amount is $353. To increase the number
of hours participants may work, states may include the value of their
federal food stamps when calculating assistance to TANF recipients, and most
of the states we visited did so. This addition brings the number of hours
close to the 30-hour-per-week federal target in most of the states that we
visited. In cases in which it does not, such as for some two-parent families
in West Virginia, states may require TANF recipients to supplement work-site
activities with other countable work activities, such as community
volunteering.

Local economic conditions, especially the availability of jobs for less
skilled workers, can also influence the types of work-site activities that
agencies develop for TANF recipients. In West Virginia, officials told us
that work-site assignments in state agencies are one of the few options
available for participants with limited skills who have been unable to
secure unsubsidized employment, particularly in areas where the coal
industry has been in decline. West Virginia has used such assignments
extensively since the 1980s as a means of promoting work among those
receiving cash assistance.13 A recent study on the history of mandatory work
programs for welfare recipients suggests that when the economy is weak, the
option of expanding public employment opportunities can become attractive
and cites West Virginia's long-standing CWEP as an example that maintains a
safety net for families while ensuring that cash assistance recipients meet
their social obligation to work in exchange for this assistance.14

West Virginia has also gone beyond its more traditional program and created
public sector jobs reserved for TANF recipients in the state's Courtesy
Patrol, which provides aid to motorists. The program is funded primarily by
the state's Department of Highways but uses supplemental funding from
private foundation grants and from federal government sources, including
welfare-to-work grant funds and, for support services, some TANF funds.
Since 1998, the Courtesy Patrol has provided temporary jobs for 133 former
TANF recipients while also providing a valuable service to motorists in West
Virginia.

Contractors and agencies that administer work-site activities face the
ongoing challenge of recruiting employers who can accommodate the
developmental needs of TANF recipients. At the same time, they must ensure
that they have adequate numbers of TANF participants to fill work sites. For
contractors, this is a particular challenge, because they must rely on case
managers from other agencies to make referrals or recruit TANF recipients
themselves.

At some of the sites that we visited, program officials have been able to
develop work-site positions that meet their program needs, primarily through
the efforts of specialized program staff or employers who have had positive
experiences with their programs. Nonetheless, program officials reported
that they face employers' negative stereotypes about welfare recipients, or
employers' unrealistically high expectations of participants. Job developers
in Massachusetts' Supported Work program address negative stereotypes by
avoiding use of the word "welfare" in discussions with potential employers,
and in Oregon, potential JOBS Plus employers are carefully screened to
ensure that placements are appropriate and mutually beneficial to employers
and participants. Also, in previous work we found that employers may be
reluctant to provide work-site positions when a subsidy is not considered
adequate to compensate them for the additional costs that are involved in
providing these positions.15 To overcome employer reluctance, one vendor
tells employers it will save the employers recruiting costs by referring
people who are prescreened and prequalified for a position.

Work-site activity administrators in San Francisco and Philadelphia cited
internal, administrative difficulties in getting state or local agencies to
refer recipients to work-site activities during the start-up of their
programs. Administrators in San Francisco have focused on improving
communications with case managers who perform individual assessments of TANF
recipients and who provide referrals for work-site activities in an effort
to obtain more referrals. Contractors offering Paid Work Experience in
Philadelphia present information about the services that they offer directly
to TANF recipients to encourage participation. At the time we visited,
several of the sites had more work-site positions than participants.

Activities Are Difficult

When recipients are referred, many fail to report to the work site. This
creates additional paperwork burdens for agencies and contractors, who must
follow up with these nonparticipating TANF recipients, either to provide
needed support services or to ensure that any penalties for nonattendance
are assessed.

Program administrators at most of the sites we visited reported that once
people are assigned to work sites, additional challenges related to
facilitating and motivating recipients' continued participation arise.
Officials in New York City, Chicago, San Francisco, and West Virginia noted
that securing child care is a key challenge for participants in work-site
activities, as is typically the case when single parents work or participate
in work-related activities. To address this issue, each of the sites that we
visited provides child care assistance to families participating in
work-site activities. At three of the sites that we visited, child care
liaisons or specialists are available to help work-site participants secure
appropriate child care services. In San Francisco, counselors work with TANF
recipients before placing them at work sites to ensure that, if their
children get sick, they have alternative child care arrangements.
Massachusetts has avoided many child care problems because it exempted
parents of non-school-age children from the work requirement under its
prereform waiver program. Also, TANF recipients who perform Community
Service in Massachusetts are required to spend only 20 hours per week in
that activity and may schedule these hours around their children's
schedules.

Most of the sites that we visited also provide transportation assistance to
participants in work-site activities in the form of fare cards, passes, or
reimbursement for public transportation. West Virginia, much of which is
inaccessible by public transportation, is piloting a car leasing program
that will enable TANF recipients to lease vehicles at reduced cost. Onondaga
County has also established a car buying program, in which donated cars are
repaired and sold at cost to those who need them.

In Chicago, Massachusetts, New York City, Philadelphia, and San Francisco,
language barriers were cited as a particular challenge. At each of these
locations, agencies have established contracts with local community-based
organizations that specialize in serving non-English-speaking populations.
In Chicago and Massachusetts, contractors serving those with limited English
skills actively recruit employers who do not require English proficiency, or
who share a TANF recipient's first language. The Massachusetts Supported
Work program also offers an "English for Employment" training component in
its basic skills training curriculum, and San Francisco is considering
exempting Community Jobs Initiative participants with limited English
proficiency from work requirements until after they have received services
to improve their language skills.

After agencies have taken steps to ensure that TANF recipients are able to
participate in work-site activities, they may face additional challenges
with recipients who are reluctant or resistant to engaging in these
activities. Officials at six of the locations that we visited reported some
difficulty with motivating continued participation in work-site activities.
Other research confirms that work-site administrators can expect levels of
nonparticipation as high as 50 percent.16 A 1998 survey conducted by the
Massachusetts TANF agency of community organizations that provided work-site
assignments for TANF recipients cited motivational and attitudinal problems,
such as unwillingness to work and poor attitude, as obstacles that these
organizations had encountered with work-site activity participants.

Some work sites have addressed the issue of motivation by contracting with
community-based organizations for liaisons to provide personal outreach to
TANF recipients. These liaisons help TANF recipients to access services that
may enable them to fulfill their obligation to perform work-site activities
or to resolve work-site issues that may limit recipients' willingness to
report for their assignments. Involving the recipients in the up-front
assessment and giving them the flexibility to set their schedules for
training and work-site activities is also seen as a key to breaking down
motivational barriers. Oregon also colocates its employment and TANF
programs to streamline the referral process and to improve access to needed
employment services.

Agencies or contractors may also offer incentives to motivate TANF
recipients' participation in work-site activities. Program administrators
from San Francisco's Community Jobs Initiative program noted that 31 percent
of the 221 clients enrolled in the program between February 1999 and
February 2000 did not complete it. The Community Jobs Initiative program has
always offered participants food or clothing gift certificates ranging in
value from $25 to $150 to reward them for good attendance or retaining an
unsubsidized job for more than 1 month.

States also use financial penalties, or sanctions, to motivate TANF family
members to meet their work obligations.17 For example, if an adult member of
a family receiving TANF in New York City refuses to participate in a
work-site activity, the city's Human Resources Administration may reduce the
family's grant. On November 15, 1999, 34,000 cases were under consideration
for financial penalties for failure to comply with work requirements, and
the cash grants for another 19,000 cases had been reduced because of
participants' failure to comply with program requirements.

and Communities

In the states we visited, TANF program administrators, recipients,
employers, and others said that participation in work-site activities helps
TANF recipients make the transition into unsubsidized employment and
contributes to the community. While states generally have data on key
employment outcomes for TANF recipients, this information is not uniformly
collected specifically for those engaged in work-site activities. Moreover,
additional information would be needed to assess program effects. One of the
sites we visited plans to evaluate its work-site activity. In addition to
positive comments about work-site activities, we also heard some concerns
that (1) work-site activities do not provide the necessary level of
experience or skills to enable participants to make the transition to
unsubsidized employment that leads to economic independence and (2)
work-site participants replace regular salaried employees.

Work-Site Activities

Proponents of work-site activities believe that they contribute to the
employability of welfare recipients, deter individuals from going on or
staying on welfare, provide useful community services, and meet the public's
preference for providing cash assistance to only those individuals who are
working or taking steps to become employed. To determine whether
participation in work-site activities increases employment and earnings, it
is necessary to have information on key outcomes, such as whether
participants become employed and how much they earn, as well as a way to
determine if these outcomes differ from what would have happened if
participants had not taken part in the work-site activity. To determine
whether these activities deter people from welfare is even more difficult
because it requires having information about individuals who do not enter
the system at all or who leave earlier than they would have if there had
been no requirement to participate in these activities.

The states and localities that we visited have collected some data on
outcomes for TANF recipients, both for federal reporting purposes and to
meet their own management information needs.18 At the federal level, states
generally report, among other things, the employment status of TANF
recipients, the number who leave welfare because of employment, their
earnings, and how long they retain their jobs. However, states do not report
these outcomes by type of activity in which TANF recipients are engaged. At
the state and local levels, program administrators collect the data they
need to manage their programs, which may include outcomes for specific
work-site activities. The states and localities we visited that have
contracted out administration of work-site activities generally collect
employment outcome data to help them monitor contractor performance;
however, these data may not be reported or summarized for all the
contractors conducting the activity in an entire locality or state. An
Oregon official told us that although program administrators could report
employment by work-site activity in their state, they generally did not do
so because their interest is in the performance of the entire TANF program
and not an individual component activity. The Pennsylvania Department of
Public Welfare, which monitors the Philadelphia work-site activity
contracts, also does not summarize employment outcomes by work-site
activity. In contrast, the Illinois Department of Human Services reports
employment and earnings outcomes for each Chicago WorkFirst contractor as
well as totals for all contractors. WorkFirst contractors report monthly
billing and participant information, including the number of participants
who become employed and their wage at placement, which the Department
summarizes and uses in its quarterly meetings with the contractors. The
Department also summarizes the information for all contractors semiannually
to assess the performance of the overall Chicago WorkFirst activity.
Illinois used these data to help redesign the WorkFirst program to more
effectively serve participants who need additional support to make the
transition to unsubsidized employment.

Some of the work-site administrators we spoke with believed that their
programs showed promise in increasing employment outcomes for TANF
recipients. As shown in table 5, Philadelphia's Transitional Work
Corporation, one of several work-site activity contractors in Philadelphia,
reported that 244, or 24 percent, of the TANF recipients enrolled in its
program from May 1999 to March 2000 made the transition to unsubsidized
employment, with average wages of $7.27 per hour. Program administrators
from San Francisco's pilot program were encouraged that 38 of the 245
recipients enrolled since the program began had moved into full-time
unsubsidized jobs with benefits and an average wage of $9.53, even though
most enrollees had not yet been at their work sites for the expected 9-month
period.19 Massachusetts reported that more than half of the 1,586
participants in its Supported Work program were placed in unsubsidized
employment during July 1998 to June 1999. While placement rates play an
important role in assessing program performance, additional information and
analysis would be needed to determine the relative effectiveness of each
work-site activity and to assess the contribution of other factors, such as
the economy or participant characteristics.

 Work-site       Data           Number      Number placed in  Average wage
 activity        collection     enrolled    unsubsidized      at placement
                 period                     jobs
 Chicago
                 April 1996 to              1,015 (9
                                10,750                        $6.89a
 WorkFirst       July 1999                  percent)
 Philadelphia
                 May 1999 to
 Paid Work       March 2000     1,029       244 (24 percent)  7.27
 Experienceb
 San Francisco
                 February 1999
 Community Jobs  to May 2000    245         38 (16 percent)   9.53
 Initiative
 Massachusetts
                 July 1998 to   1,419       470 (percentage
 Community       June 1999      (monthly                      6.51
 Service                        average)    not applicable)
 Massachusetts
                 July 1998 to
                                1,586       829 (52 percent)  7.08
 Supported Work  June 1999

aBased on the 301 placements for which there were wage data.

bThese data were reported by the Transitional Work Corporation, which is one
of several work-site activity contractors in Philadelphia.

Source: Data from state and local administrators.

In a TANF program in which a work-site activity is the primary activity for
a substantial portion of TANF recipients, such as New York City's program,
data on outcomes for the entire TANF caseload could be considered to reflect
to some extent the outcomes for the work-site activity. A New York City
administrator told us that the city's WEP and the emphasis on the importance
of work have been key factors in reducing the total number of AFDC/TANF
recipients from about 860,000 in March 1995 to about 530,000 in November
1999. However, without a rigorous evaluation, it is not possible to
determine how much of this caseload reduction is the result of required
participation in the work-site activity and how much is due to other
factors, such as the economy.

Activities

While research on work-site activities in the new welfare environment is
lacking, evaluations of similar activities implemented during the 1980s, in
which welfare recipients were expected to work in exchange for their
benefits, did not find that participation in these work-site activities
increased employment or earnings.20 However, both participants and their
supervisors reported that the work was meaningful. These evaluations were of
small programs, which in many ways were different from the programs we
reviewed. The Manpower Demonstration Research Corporation (MDRC) evaluated
nine welfare-to-work programs that included work experience as a major or
ancillary activity. Generally, in these early programs, welfare recipients
were assigned to work sites for 13 weeks and received only a benefit check.
In West Virginia, however, a CWEP placement lasted for as long as a
recipient received a welfare check, and weekly hours of participation were
limited to the amount of a recipient's cash benefit divided by the minimum
wage. The evaluators concluded that the value of the work performed by
participants generally outweighed the costs of running the programs. Given
the small scale of these programs, however, it remains unclear whether these
findings would hold up for much larger programs or for programs in which the
work-site activity lasts longer than 13 weeks.

The West Virginia CWEP program evaluated by MDRC was a statewide initiative
that continues today as a major work-site activity for West Virginia's TANF
recipients. The MDRC evaluation concluded that the state had succeeded in
providing a substantial number of welfare recipients with productive,
long-term work experience in a labor market suffering one of the highest
unemployment rates in the nation. However, the program had little effect on
moving recipients into unsubsidized employment. Although today's economy in
West Virginia is substantially improved, there remain pockets of very high
unemployment where few jobs exist for welfare recipients. CWEP may be the
only opportunity for welfare recipients in these areas who cannot relocate
to meet their work participation obligation.

MDRC also evaluated a work-site activity known as supported work in the
1970s. Supported work was a highly structured, paid work experience program
specifically designed for a disadvantaged subset of the welfare population.
Participants volunteered to be placed at work sites in closely supervised
groups and were gradually subjected to higher expectations on the job. They
could remain in the program for 12 to 18 months, earning wages that were
partially subsidized with welfare benefits, and were then assisted in
locating unsubsidized jobs. After 3 years in the program, supported work
participants were 10 percent less likely than a control group to be
receiving welfare assistance and had average earnings 23 percent higher than
the control group. Although elements of the supported work model are being
incorporated into programs targeting TANF recipients who are the most
difficult to employ, because of the ways in which the new welfare
environment is different from the old one, we cannot conclude that these
programs will achieve the same effect as the supported work demonstrations.

HHS has recently taken some initial steps to identify current supported work
programs with the goal of supporting research in this area. HHS'
Administration for Children and Families and the Office of the Assistant
Secretary for Planning and Evaluation have authority to spend funds to
support such research, demonstration, and evaluation efforts. In May 2000,
HHS contracted with Mathematica Policy Research, Inc., to identify supported
work programs serving TANF or non-TANF individuals, including those in the
area of vocational rehabilitation, that hold promise for meeting the needs
of TANF recipients and are potentially large enough to serve a significant
number of participants.

Participants and Communities

Program administrators, participants, and others reported that work-site
activities do have a positive effect on participants and communities.
Individuals with no prior work experience can develop employment skills and
a resume. Massachusetts officials reported that community work experience is
beneficial to some clients because it requires them to leave the house on a
daily basis and become involved in a work environment. They also believe
that community service instills a work ethic that some clients did not have
before. Oregon's JOBS Plus participants reported a positive change in
attitude, self-esteem, and pride.

Evidence from the sites that we visited suggests that the work performed at
these work sites also has had a positive effect on employers and their
communities. Work-site activities provide additional staff to public and
private nonprofit agencies lacking resources, which allows them to better
meet community needs. For example, in Chicago TANF recipients help prepare
food to be delivered to elderly citizens and perform janitorial work at a
low-income housing development. Further, participation in Oregon's JOBS Plus
gives private sector employers an opportunity to help the community in which
they operate by employing and training welfare recipients. JOBS Plus
employers reported a significant shift in their perception of welfare
recipients and of the challenges they face after working with them. In
Massachusetts, community service participants work in schools, churches,
hospitals, and municipal government. Participants do clerical, custodial,
housekeeping, and food preparation work as well as assist in classrooms.
Finally, in New York City, the Mayor's Management Report released in
February 2000 states that the citywide street cleanliness ratings have
improved since WEP participation in street cleaning was expanded.

While we identified many positive aspects of work-site activities for both
participants and communities, we also identified some negative aspects.
Program administrators reported that for some clients participation can be
stressful, particularly as they adjust to juggling child-care arrangements
and the demands of a job, sometimes for the first time. In New York City, we
spoke with representatives of a legal and advocacy group for low-income
families that works with the city's WEP participants. These representatives
expressed concern that the program did not do enough to assess TANF
recipients' needs and skills and to match them with work sites that would
provide them the job search assistance and training needed to find
employment at wages that would move them toward economic independence. They
were also concerned that many people in need of assistance may have left the
TANF rolls or received reduced grants because they did not want to, or could
not, perform the assigned work and that some of these families may move more
deeply into poverty without the aid they need. WEP officials we spoke with
noted that a new assessment component is planned for the program and that
new components, such as the Personal Roads for Individual Development and
Employment project for individuals with medical limitations, have been added
to the program to better meet the needs of the TANF caseload. In addition,
the New York City Council has recently voted to create a pilot transitional
jobs program for welfare recipients. Modeled after work-site activities such
as the Paid Work Experience in Philadelphia, upon implementation, the New
York pilot will provide recipients with transitional jobs and some job
training.

A long-standing concern about public work programs that put welfare
recipients into work-site positions is that these individuals may take jobs
away from regular employees or others seeking work. Consequently, PRWORA,
state TANF plans, and work-site contracts contain language designed to
protect against displacement. The locations we visited took different
approaches to addressing the concern that welfare recipients might displace
other employees. For example, in Philadelphia the Mayor's office negotiated
with the unions when the public sector transitional jobs were first created.
The agreements with the unions stipulate that the work-site positions are to
be short-term and cannot replace other jobs. In Chicago, an accounting of
what welfare recipients are doing at the work sites is sent regularly to the
unions to allay any concerns they may have about displacement.

The only site where displacement appears to be an unresolved issue is New
York City. A confederation of 56 New York City public employee unions has
filed several lawsuits against the mayor and New York City agencies,
including the Human Resources Administration; the Administration for
Children's Services; and the Departments of Sanitation, Transportation, and
Parks and Recreation. The basis for the lawsuits is the unions' claim that
paid city employees are being replaced with WEP participants or are being
denied promotional opportunities. There is also concern that the increasing
use of work-site participants to perform work equivalent to that typically
performed by entry-level personnel in city agencies prevents individuals
from obtaining city agency employment. A union representative we spoke with
said that entry-level jobs in the city's public agencies have historically
served an important role in helping low-income individuals find stable
employment and move up over time into the middle class. There is no
expectation under WEP, however, that city agencies will hire program
participants. The union believes that the disappearance of permanent
entry-level jobs in city agencies and the lack of entry-level jobs in the
private sector will inhibit individuals with lower skills and abilities from
becoming employed. The issue of whether displacement of regular workers is
occurring is currently being litigated in the courts.

Passage of the welfare reform law changed the federal role in providing
assistance to needy families with children. States and localities have been
given broader flexibility to design and implement their welfare programs,
and coupled with this flexibility is a stronger emphasis on work, with work
requirements for most adults receiving cash assistance, limits on the length
of stay on federal cash assistance, and financial penalties for states that
do not have an increasing percentage of their adult caseload working or in
work-related activities over time. As a result, states and localities are
requiring more clients to find jobs or participate in work-related
activities to prepare for jobs. For the type of work-related activities we
focused on--work-site activities, rather than job search or classroom
activities--the characteristics of activity design and structure were
strongly influenced by elected officials, economic environments, and the
support the work-site activities received from their communities. Because
work-site activities must necessarily involve employers or community
organizations--public, nonprofit, or private--these activities cannot
succeed without the support of these entities.

The work-site activities we reviewed are, like welfare reform, a work in
progress. Given the flexibility of TANF, states and localities are providing
enhanced support services and are changing and tailoring their programs to
meet clients' needs. New York City's new work-site activity designed for
TANF adults with medical limitations who had typically been exempted from
work requirements in previous programs and San Francisco's pilot Community
Jobs Initiative, which creates temporary jobs for TANF recipients who have
not yet found unsubsidized employment, both represent local responses to
welfare reform's emphasis on the importance of work for all of those
receiving aid. It is likely that changes such as these will continue as
states and localities strive to be successful in implementing welfare
reform.

To date, states and localities have implemented their welfare reform
programs in a strong national economy, which has no doubt facilitated their
reform efforts, as reflected in the fact that in fiscal year 1998, the
highest percentage of participating TANF recipients were engaged in
unsubsidized employment. States have also been aided in meeting federal
participation rates by receiving credits for the recent caseload reductions.
In the event of an economic downturn when jobs may be less readily
available, more states and localities may turn to work-site activities as an
alternative for meeting their work requirements. Yet, nationwide, most of
the work-site activities are currently being operated on a relatively small
scale. If states and localities need to expand work-site activities to serve
increasing numbers of participants, they will need to find increasing
numbers of employers and organizations willing to participate. In addition,
states and localities will need to ensure that participants and work-site
activities are monitored to determine if they meet the desired program
outcomes. Moreover, the greater the scale of work-site activities, the
greater the challenge to ensure that participants in work-site activities do
not displace regular employees from their jobs.

The work-site activities currently in operation can provide valuable
information for administrators and policymakers on what could in the future
become an increasingly important part of TANF programs nationwide, and HHS
could play a role in making such information available. While HHS requires
states to report the number of participants in work-site activities, it does
not require states to describe key characteristics of these activities. By
providing some additional guidance to states on how to describe their
work-site activities in their state TANF plans, HHS has the opportunity to
gather useful information without compromising state flexibility. In
addition, HHS could take steps, through its regional offices and at
HHS-sponsored conferences, for example, to identify and disseminate
information on promising approaches to implementing and operating work-site
activities in TANF programs. Finally, as our work has shown, little is known
about how effective these approaches are in the current welfare environment
in moving TANF recipients toward employment and economic independence. With
the broad flexibility granted to them under TANF, states and localities are
implementing alternative approaches, including work-site activities, to
achieve the goals of welfare reform, which provides an opportunity to
compare the effectiveness of various approaches. HHS' new project designed
to collect information on supported work programs that could potentially
provide models for serving TANF recipients is a positive move toward
understanding how to use work-site activities to serve TANF recipients
effectively. HHS could continue in this and other efforts to encourage
states and localities to evaluate rigorously some of the work-site
approaches used in TANF by providing financial support for such evaluations.
If information on these work-site approaches is not collected and rigorous
evaluations are not conducted, the opportunity to learn implementation
lessons and to measure the value of work-site activities in moving clients
to unsubsidized employment for the future will be lost.

To make available more information on the effectiveness of work-site
activities, which could play an increasingly important role as welfare
reform evolves, we recommend that the Secretary of Health and Human Services
do the following:

ï¿½ Require states to provide more information in the state TANF plans or
annual reports that they must file with HHS on the key characteristics of
work-site activities they are using. For example, states could include in
their plans such information as whether a work-site activity participant
receives a welfare check or paycheck, qualifies for the EIC, can earn more
than his or her grant amount, and is considered to be receiving TANF
assistance that counts toward the federal time limit.

ï¿½ Use HHS' regional offices' technical assistance activities, HHS-sponsored
conferences, and other means available to identify promising work-site
approaches used by the states and localities in their TANF programs, and
collect and disseminate information on them.

ï¿½ Encourage states and localities to rigorously evaluate their work-site
activities, particularly by providing financial support for evaluations to
increase knowledge of the effectiveness of various types of work-site
activities.

We provided HHS and the states and localities we reviewed with an
opportunity to comment on the report. HHS generally agreed with our findings
and recommendations. The states and localities also generally agreed with
the report.

With respect to the recommendation that HHS should require states to provide
more information on the key characteristics of the work-site activities they
are using, HHS said that it is statutorily limited in what information it
may require. HHS pointed out that starting with fiscal year 2000, each state
must file an annual report, which will include information on how a state
defines its various work activities. HHS expects that this report will help
provide information about the different work activities carried out in a
state. As discussed in the report, we believe that by providing some
additional guidance to states on how to describe key characteristics of
their work-site activities, HHS could gather useful information without
compromising state flexibility.

Regarding our recommendation on information dissemination, HHS noted that it
will use several methods, such as periodic conferences on critical welfare
issues and interactive work groups, to collect and disseminate promising
approaches. While agreeing with our third recommendation supporting
evaluations, HHS cautioned that there is limited potential for conducting
rigorous evaluations to determine the effects of work-site interventions on
moving TANF recipients to unsubsidized employment. It noted that differences
in the characteristics of populations served by existing work-site
activities would make it difficult to compare the relative effectiveness of
different approaches. Moreover, it noted that the small scale of these types
of programs would make it difficult to generate adequate-sized research
samples for random assignment to experimental and control groups. However,
as we note in our report, HHS has already taken initial steps to identify
current employment-focused programs using the supported work model with the
goal of supporting research in this area. We encourage HHS to identify
opportunities in which rigorous evaluations may be feasible, and, as
discussed in the report, we believe that studies of program implementation
and outcomes can also be useful to program administrators and policymakers.

HHS' comments appear in appendix I. HHS and the states and localities we
visited also provided technical comments, which we have incorporated where
appropriate.

We are sending copies of this report to the Honorable William V. Roth, Jr.,
Chairman, Senate Committee on Finance; the Honorable Don Nickles, Chairman,
Subcommittee on Social Security and Family Policy, Senate Committee on
Finance; the Honorable Nancy L. Johnson, Chairman, and the Honorable
Benjamin L. Cardin, Ranking Minority Member, Subcommittee on Human
Resources, House Committee on Ways and Means; the Honorable Donna E.
Shalala, Secretary of Health and Human Services; the Honorable Alexis M.
Herman, Secretary of Labor; appropriate congressional committees; and other
interested parties. We will also make copies available upon request.

If you have any questions about this report, please contact me on (202)
512-7215. Other GAO contacts and staff acknowledgments are listed in
appendix II.

Cynthia M. Fagnoni
Director, Education, Workforce,
and Income Security Issues

Comments From the Department of Health and Human Services

GAO Contacts and Staff Acknowledgments

Gale C. Harris, (202) 512-7235
Kevin M. Kumanga, (202) 512-4962

In addition to those named above, Margaret Boeckmann, Regina Santucci, and
Andrea Romich Sykes made key contributions to this report.

Bibliography

Abramovitz, Mimi. Workfare and the Non-Profits: Myths & Realities. New York,
N.Y.: Task Force on Welfare Reform, New York City Chapter, National
Association of Social Workers. Dec. 1997.

American Federation of State, County, and Municipal Employees. "Philadelphia
Points the Way." Public Employee, AFSCME, Vol. 62, No. 1 (Jan./Feb. 1997).

_____. Workfare or Work Scared? Washington, D.C.: AFSCME, Mar. 3, 1997.

_____. Legislative Alert: Workfare. Washington, D.C.: AFSCME, Sept. 11,
1997.

_____. Report Card on Welfare Reform: Did States Make the Grade? (A critique
of AFSCME's issues). Washington, D.C.: AFSCME, 1997.

_____. Thinking Creatively About Welfare-to-Work Job Creation. Washington,
D.C.: AFSCME, July 1998.

Bartik, Timothy J. Short-Term Employment Persistence for Welfare Recipients:
The "Effects" of Wages, Industry, Occupation, and Firm Size. Kalamazoo,
Mich.: W.E. Upjohn Institute for Employment Research, June 1997.

_____. Displacement and Wage Effects of Welfare Reform. Kalamazoo, Mich.:
W.E. Upjohn Institute for Employment Research, Jan. 1999.

Brock, Thomas, and others. Unpaid Work Experience for Welfare Recipients:
Findings and Lessons from MDRC Research. New York, N.Y.: Manpower
Demonstration Research Corporation, Sept. 1993.

Brown, Amy, and others. Business Partnerships: How to Involve Employers in
Welfare Reform. New York, N.Y.: Manpower Demonstration Research Corporation,
May 1998.

Burke, Vee. Welfare Recipients and Workforce Laws (97-1038), CRS Report for
Congress. Washington, D.C.: Congressional Research Service, Apr. 1, 1999.

_____. Welfare Reform: An Issue Overview, CRS Issue Brief 93034. Washington,
D.C.: Congressional Research Service, May 24, 1999.

California Legislative Analyst's Office. CalWORKs Community Service: What
Does It Mean for California? Sacramento, Calif.: California Legislative
Analyst's Office, Feb. 4, 1999.

Cohen, Marie. "Work Experience and Publicly-Funded Jobs for TANF
Recipients." Issue Notes, Welfare Information Network, Vol. 2, No. 12 (Sept.
1998).

Cohen, Steven. Managing Workfare: The Case of the Work Experience Program in
the New York City Parks Department. New York, N.Y.: Columbia University,
Mar. 1999.

Ellwood, David T., and Elisabeth D. Welty. Public Service Employment and
Mandatory Work: A Policy Whose Time Has Come and Gone and Come Again?
Cambridge, Mass.: Kennedy School of Government, Harvard University, Mar. 6,
1999.

Friedman, Pamela. "Community Work Experience and Publicly Funded
Jobs−Helping the Hard-to-Serve Meet Work Requirements." Issue Notes,
Welfare Information Network, Vol. 3, No. 5 (July 1999).

Gallagher, L. Jerome, and others. One Year After Federal Welfare Reform: A
Description of State Temporary Assistance for Needy Families (TANF)
Decisions as of October 1997. Washington, D.C.: The Urban Institute, May
1998.

Holcomb, Pamela A., and others. Building an Employment Focused Welfare
System: Work First and Other Work-Oriented Strategies in Five States.
Prepared by the Urban Institute for the Office of the Assistant Secretary
for Planning and Evaluation, HHS. Washington, D.C.: HHS, June 1998.

Institute for Women's Policy Research. "Workfare." Welfare Reform Network
News, Issue No. 5 (Apr. 30, 1997).

Johnson, Clifford M. Frequently-Asked Questions About Public Job Creation.
Washington, D.C.: Center on Budget and Policy Priorities, Dec. 2, 1998.

_____. Provisions of Final TANF Regulations Pertaining to Wage Subsidies and
Publicly-Funded Jobs. Washington, D.C.: Center on Budget and Policy
Priorities, Apr. 21, 1999.

_____. Publicly-Funded Jobs for Hard-to-Employ Welfare Recipients.
Washington, D.C.: Center on Budget and Policy Priorities, May 3, 1999.

McEntee, Gerald W. Statement by AFSCME President Gerald W. McEntee on Why
Workfare Participants Should Receive Basic Workplace Protections. Washington
D.C.: AFSCME, June 4, 1997.

National Employment Law Project. Survey of State & Local Workfare, Jobs &
Low-Wage Worker Legislation. New York, N.Y.: National Employment Law
Project, Sept. 1997.

Nightingale, Demetra Smith, and others. Early Implementation of the
Welfare-to-Work Grants Program: Findings From Exploratory Site Visits and
Review of Program Plans. Prepared by the Urban Institute for the Office of
the Assistant Secretary for Planning and Evaluation, HHS. Washington, D.C.:
HHS, Feb. 2000.

New York City Independent Budget Office. Welfare Reform Revisited:
Implementation in New York City. New York, N.Y.: New York City Independent
Budget Office, Sept. 24, 1998.

Pavetti, LaDonna. What Will the States Do When Jobs Are Not Plentiful?
Policy and Implementation Challenges. Prepared for the conference "Welfare
Reform and the Macro-Economy" organized by the University of
Chicago/Northwestern University, Joint Center for Poverty Research, and
funded by the Office of the Assistant Secretary for Planning and Evaluation,
HHS, Nov. 19-20, 1998.

_____ and others. The Role of Intermediaries in Linking TANF Recipients with
Jobs. Prepared by Mathematica Policy Research, Inc., for the Office of the
Assistant Secretary for Planning and Evaluation, HHS. Washington, D.C.: HHS,
Feb. 10, 2000.

Pepper, John V. What Do Welfare-to-Work Demonstrations Reveal to Welfare
Reformers? Charlottesville, Va.: University of Virginia, Aug. 25, 1999.

Perez-Johnson, Irma, and Alan Hershey. Early Implementation of the
Welfare-to-Work Grants Program: Report to Congress. Prepared by Mathematica
Policy Research, Inc., for the Office of the Assistant Secretary for
Planning and Evaluation, HHS. Washington, D.C.: HHS, Mar. 1999.

Savner, Steve, and Mark Greenberg. Community Service Employment: A New
Opportunity under TANF. Washington, D.C.: Center for Law and Social Policy,
Nov. 1997.

_____ and Maurice Emsellem. The Fiscal and Legal Framework for Creating a
Community Service Employment Program. Washington, D.C.: Center for Law and
Social Policy, Nov. 1997.

Sherwood, Kay. Designing and Administering a Wage-Paying Community-Service
Employment Program under TANF. New York, N.Y.: Manpower Demonstration
Research Corporation, Apr. 1999.

Stettner, Andrew. Welfare to Work: Is It Working? The Failure of Current
Welfare-to-Work Strategies to Move the Hardest to Employ Into Jobs: A Case
for Public Job Creation. New York, N.Y.: Community Voices Heard, 1999.

Strawn, Julie. Beyond Job Search or Basic Education: Rethinking the Role of
Skills in Welfare Reform. Washington, D.C.: Center for Law and Social
Policy, Apr. 1998.

_____ and Robert Echols. Welfare-to-Work Programs: The Critical Role of
Skills. Washington, D.C.: Center for Law and Social Policy, Apr. 1999.

U.S. Department of Labor. How Workplace Laws Apply to Welfare Recipients.
Washington, D.C.: U.S. Department of Labor, May 1997 (Rev. 2/99).

U.S. Internal Revenue Service. Notice 99-3: Treatment of Certain Payments
Received as Temporary Assistance for Needy Families (TANF). Washington,
D.C.: IRS, Jan. 11, 1999.

Wood, Robert G., and Diane Paulsell. Helping TANF Recipients Stay Employed:
Early Evidence from the GAPS Initiative. (MPR Reference No.: 84654-520).
Princeton, N.J.: Mathematica Policy Research, Inc., Apr. 7, 1999.

Zedlewski, Sheila R. Work-Related Activities and Limitations of Current
Welfare Recipients. Discussion Paper 99-06, Assessing the New Federalism.
Washington, D.C.: The Urban Institute, July 1999.

Related GAO Products

Welfare Reform: Improving State Automated Systems Requires Coordinated
Federal Effort ( GAO/HEHS-00-48, Apr. 27, 2000).

Welfare Reform: State Sanction Policies and Number of Families Affected (
GAO/HEHS-00-44, Mar. 31, 2000).

Medicaid Enrollment: Amid Declines, State Efforts to Ensure Coverage After
Welfare Reform Vary ( GAO/HEHS-99-163, Sept.10, 1999).

Welfare Reform: Assessing the Effectiveness of Various Welfare-to-Work
Approaches ( GAO/HEHS-99-179, Sept. 7, 1999).

Food Stamp Program: Various Factors Have Led to Declining Participation
(GAO/RCED-99-185 , July 2, 1999).

Welfare Reform: Information on Former Recipients' Status ( GAO/HEHS-99-48,
Apr. 28, 1999).

Welfare Reform: States' Experiences in Providing Employment Assistance to
TANF Clients ( GAO/HEHS-99-22, Feb. 26, 1999).

Welfare Reform: Status of Awards and Selected States' Use of Welfare-to-Work
Grants ( GAO/HEHS-99-40, Feb. 5, 1999).

Welfare Reform: Few States Are Likely to Use the Simplified Food Stamp
Program ( GAO/RCED-99-43, Jan. 29, 1999).

Welfare Reform: Child Support an Uncertain Income Supplement for Families
Leaving Welfare ( GAO/HEHS-98-168, Aug. 3, 1998).

Welfare Reform: States Are Restructuring Programs to Reduce Welfare
Dependence ( GAO/HEHS-98-109, June 18, 1998).

Welfare Reform: States' Efforts to Expand Child Care Programs (
GAO/HEHS-98-27, Jan. 13, 1998).

Welfare Reform: Three States' Approaches Show Promise of Increasing Work
Participation ( GAO/HEHS-97-80, May 30, 1997).

(116034)

Table 1: Federal Law Sets Increasing TANF Participation
Requirements for All Families for Fiscal Years 1997-2002 7

Table 2: Key Characteristics of the Work-Site Activities Reviewed 10

Table 3: Selected Data on TANF Caseload for Each Work-Site
Activity, November 1999 14

Table 4: Information on Administration and Funding for Nine
Work-Site Activities 18

Table 5: Outcomes for Five Work-Site Activities 28

Figure 1: TANF Recipients Working or Participating in Work-
Related Activities, Fiscal Year 1998 8

Figure 2: Continuum of Work-Site Activities 15

Figure 3: Characteristics of Work-Site Activities Along the
Continuum 16
  

1. While we have highlighted these types of work-site activities, states and
localities may use variations and combinations of them to serve TANF
clients.

2. For the purposes of meeting federal participation rates, PRWORA
specifically defines approved work activities as unsubsidized employment,
subsidized private or public sector employment, work experience, on-the-job
training, job search and job readiness assistance, community service
programs, vocational educational training, job skills training directly
related to employment, education directly related to employment,
satisfactory attendance at a secondary school or a course of study leading
to a certificate of general equivalence, or the provision of child care
services to an individual who is participating in a community service
program.

3. For more information on work-site activities allowed under the AFDC
welfare-to-work program, the Job Opportunities and Basic Skills Training
Program, see Welfare to Work: Most AFDC Training Programs Not Emphasizing
Job Placement (GAO/HEHS-95-113 , May 19, 1995).

4. See Moffitt, Robert A., "The Effect of Employment and Training Programs
on Entry and Exit from the Welfare Caseload," Journal of Policy Analysis and
Management, Vol. 15, No. 1, pp. 32-50 (1996) for a discussion of the effects
various work-related activities can have on an individual's decision to
apply for and enter the rolls or to leave the rolls.

5. Chicago's WorkFirst originally targeted TANF recipients who had no
children under age 13, but as of January 2000, it also targets recipients
who face multiple barriers to employment.

6. Under TANF regulations, when TANF funds are used to subsidize wages, the
resulting wage payments to recipient-employees do not constitute assistance
and do not count against the 60-month federal time limit. This does not
prevent states from counting months in subsidized jobs against their own
state time limits, and most states in our review did so.

7. California allows counties to adopt one of two approaches to community
service: either a work-for-cash-assistance approach or an approach in which
recipients work and receive a paycheck subsidized by their cash assistance
grant. San Francisco elected to adopt the second approach.

8. This approach, in which a welfare recipient's grant is "diverted" to an
employer who agrees to employ the person, has been known as grant diversion
under past welfare programs.

9. The Balanced Budget Act of 1997, amended by the Consolidated
Appropriations Act of 2000, appropriated nearly $3 billion in
welfare-to-work grants to be awarded by the U.S. Department of Labor to
states and local communities to move welfare recipients into jobs. For a
review of the early implementation of programs funded with welfare-to-work
grants, see Demetra Smith Nightingale and others, Early Implementation of
the Welfare to Work Grants Program: Findings From Exploratory Site Visits
and Review of Program Plans (Washington, D.C.: The Urban Institute, Feb.
2000).

10. San Francisco officials said recipients may qualify for an EIC benefit
that averages $115 per month.

11. U.S. Department of Labor, How Workplace Laws Apply to Welfare Recipients
(Washington, D.C.: Department of Labor, May 1997).

12. Welfare Reform: Few States Are Likely to Use the Simplified Food Stamp
Program ( GAO/RCED-99-43, Jan. 29, 1999).

13. GAO/HEHS-95-113, May 19, 1995.

14. David T. Ellwood and Elisabeth D. Welty, Public Service Employment and
Mandatory Work: A Policy Whose Time Has Come and Gone and Come Again?
(Cambridge, Mass.: The Kennedy School of Government, Harvard University,
Mar. 6, 1999), pp. 4-5.

15. GAO/HEHS-95-113, May 19, 1995.

16. LaDonna Pavetti and others, The Role of Intermediaries in Linking TANF
Recipients with Jobs, report by Mathematica Policy Research, Inc., for the
Office of the Assistant Secretary for Planning and Evaluation, HHS
(Washington, D.C.: HHS, 2000), p. xv.

17. For a discussion of the sanction policies of all states, see Welfare
Reform: State Sanction Policies and Number of Families Affected (
GAO/HEHS-00-44, Mar. 31, 2000).

18. For a discussion of the information needs of states under welfare
reform, see Welfare Reform: Improving State Automated Systems Requires
Coordinated Federal Attention ( GAO/HEHS-00-48, Apr. 27, 2000).

19. San Francisco officials stated that 92 percent of recipients who had
completed their work-site assignments had been placed in an unsubsidized
job.

20. See Thomas Brock, David Butler, and David Long, Unpaid Work Experience
for Welfare Recipients: Findings and Lessons from MDRC Research (New York,
N.Y.: MDRC, Sept. 1993).
*** End of document. ***