Welfare to Work: Measuring Outcomes for JOBS Participants (Letter Report,
04/17/95, GAO/HEHS-95-86).
The Department of Health and Human Services (HHS) does not know whether
the Job Opportunities and Basic Skills Training (JOBS) program is
reducing welfare dependency because it does not gather enough
information on critical program outcomes, such as the number of
participants entering employment and leaving welfare each year. Also,
states are held accountable for the number and type of participants
enrolled in education and training but not for outcomes, such as the
number of participants finding work. Although little progress has been
made in monitoring JOBS outcomes at the federal level, the picture is
better at the state level. The current national interest in making
welfare more employment focused, as well as provisions in the Government
Performance and Results Act requiring performance monitoring to become
more outcome oriented governmentwide, indicate that HHS needs to move
decisively to ensure that it meets its current schedule for developing
outcome measures and goals for JOBS. A critical first step in developing
performance goals will be to work the states and others to resolve
differences over whether the main objective of JOBS is to help
participants find work quickly or receive the education and training
needed for better-paying jobs.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: HEHS-95-86
TITLE: Welfare to Work: Measuring Outcomes for JOBS Participants
DATE: 04/17/95
SUBJECT: Education or training
Aid to families with dependent children
Disadvantaged persons
Work measurement standards
State-administered programs
Aid for training or employment
Employment or training programs
Federal/state relations
Welfare recipients
IDENTIFIER: Job Opportunities and Basic Skills Training Program
Job Training Partnership Act Program
JOBS Program
JTPA
AFDC
AFDC Quality Control System
**************************************************************************
* This file contains an ASCII representation of the text of a GAO *
* report. Delineations within the text indicating chapter titles, *
* headings, and bullets are preserved. Major divisions and subdivisions *
* of the text, such as Chapters, Sections, and Appendixes, are *
* identified by double and single lines. The numbers on the right end *
* of these lines indicate the position of each of the subsections in the *
* document outline. These numbers do NOT correspond with the page *
* numbers of the printed product. *
* *
* No attempt has been made to display graphic images, although figure *
* captions are reproduced. Tables are included, but may not resemble *
* those in the printed version. *
* *
* A printed copy of this report may be obtained from the GAO Document *
* Distribution Facility by calling (202) 512-6000, by faxing your *
* request to (301) 258-4066, or by writing to P.O. Box 6015, *
* Gaithersburg, MD 20884-6015. We are unable to accept electronic orders *
* for printed documents at this time. *
**************************************************************************
Cover
================================================================ COVER
Report to the Ranking Minority Member, Committee on Finance, U.S.
Senate
April 1995
WELFARE TO WORK - MEASURING
OUTCOMES FOR JOBS PARTICIPANTS
GAO/HEHS-95-86
JOBS Outcomes
Abbreviations
=============================================================== ABBREV
ACF - Administration for Children and Families
AFDC - Aid to Families With Dependent Children
APWA - American Public Welfare Association
FSA - Family Support Act
FY - fiscal year
GPRA - Government Performance and Results Act
HHS - Department of Health and Human Services
JOBS - Job Opportunities and Basic Skills Training
JTPA - Job Training Partnership Act
NAPA - National Academy of Public Administration
NPR - National Performance Review
NPRM - notice of proposed rule making
WIN - Work Incentive program
Letter
=============================================================== LETTER
B-256125
April 17, 1995
The Honorable Daniel P. Moynihan
Ranking Minority Member
Committee on Finance
United States Senate
Dear Senator Moynihan:
Between 1989 and 1994, federal and state governments spent about 8
billion dollars on the Job Opportunities and Basic Skills Training
(JOBS) program. The program helps recipients of Aid to Families with
Dependent Children (AFDC) obtain the education, training, and
services necessary for employment. For program managers and
policymakers to determine whether this investment has helped achieve
the objective of reducing welfare dependency requires information on
JOBS participants' outcomes, such as whether they are becoming
employed and leaving AFDC. In working toward welfare reform,
information on the extent to which JOBS is achieving its objectives
is more important than ever for the Congress.
This report responds to your request that we study the use of outcome
measurement by the Department of Health and Human Services (HHS) and
states in determining whether JOBS participants are finding
employment and leaving AFDC. Specifically, we addressed the
following questions: (1) What progress has HHS made in measuring the
employment and AFDC status of JOBS participants at the national level
and setting national goals against which program performance will be
measured? (2) To what extent are states measuring participant
outcomes and setting performance goals? (3) What major issues should
be considered in establishing a national approach to measuring JOBS
participant outcomes and setting performance goals?
To assess the progress that HHS has made in establishing outcome
indicators and goals, we interviewed officials from HHS and various
welfare research and interest groups. We also reviewed JOBS
regulations and reports. Using a mail questionnaire, we surveyed
JOBS administrators in the 50 states and the District of Columbia to
determine state performance monitoring practices. To determine the
issues that should be considered in developing indicators and goals,
we reviewed HHS' proposed approach to developing outcome indicators,
examined the literature on the development of performance monitoring
systems, and interviewed experts in the field. See appendix I for
further details on our scope and methodology.
RESULTS IN BRIEF
------------------------------------------------------------ Letter :1
HHS does not know whether JOBS is reducing welfare dependency because
it does not gather enough information on critical program outcomes,
such as the number of participants entering employment and leaving
AFDC annually. In addition, states are held accountable for the
number and type of participants enrolled in education and training
but not for outcomes, such as the number of participants finding
employment. While the current approach to monitoring performance
provides important information on the activities of JOBS
participants, state JOBS directors are concerned that the approach
provides little incentive for states to focus on moving participants
off AFDC and into jobs.
While little progress has been made in monitoring JOBS outcomes at
the federal level, the picture is better at the state level. Nearly
all states use some information on participant outcomes to manage
their individual programs, although the extent to which states
monitor outcomes varies widely. At least in part to demonstrate to
their state legislatures that program objectives are being achieved,
a majority of states monitor the number of JOBS participants entering
employment and hourly wages at hire. In addition, over one-half of
the states have established annual outcome goals. Although many
states gather some JOBS outcome data, without a standard federal
approach, few states could provide us with comparable data. However,
our survey of JOBS directors found that 27 states could provide
annual data on individuals entering employment. In these states,
about 21 percent of JOBS participants found jobs in 1993.
The current national interest in making welfare more employment
focused, as well as requirements in the Government Performance and
Results Act (GPRA) that performance monitoring become more outcome
oriented governmentwide, indicate a need for HHS to move decisively
to ensure that it meets its current schedule for developing outcome
measures and goals for JOBS. HHS has reported to the Congress that
it plans to finalize JOBS outcome measures by October 1996 and
outcome goals by October 1998. A critical first step in developing
performance goals will be working with the states and other concerned
parties to resolve differences regarding whether the primary
objective of JOBS is to help participants (1) obtain employment
quickly or (2) get the education and training needed for
better-paying jobs. Congress is considering whether AFDC and JOBS
should be replaced with a welfare-to-work block grant program that
includes some JOBS' objectives and activities. However this issue is
resolved, the need for federal accountability would be well served by
clearly defined program objectives and outcome goals.
BACKGROUND
------------------------------------------------------------ Letter :2
JOBS, created by the Family Support Act (FSA) of 1988, is designed to
help families avoid long-term welfare dependency. The act requires
all states to establish JOBS programs that make available to AFDC
recipients the education, training, and support services they need to
prepare for, accept, and retain employment. States can provide these
services either directly or through local service providers. Both
the states and the federal government share in the costs of the
program. States were required to begin their JOBS programs by
October 1, 1990, and be in full statewide operation by October 1,
1992. HHS' Administration for Children and Families (ACF) is
responsible for managing JOBS at the federal level.
To understand whether a human services program, such as JOBS, is
achieving its objectives or in need of improvement requires a system
for gathering information about program performance. Evaluating a
program by regularly collecting and analyzing performance information
is known as performance monitoring. Performance monitoring systems
include two key elements: (1) indicators--which define what
performance information will be gathered and (2) goals--a target
level of performance against which actual program performance will be
gauged.
There are two basic types of indicators: process and outcome.
Process indicators for JOBS would provide information about program
activities, such as the number of AFDC recipients participating in
JOBS, the number of JOBS participants receiving training, and the
amount of money being spent on teenage participants. Outcome
indicators for JOBS, on the other hand, would capture what happens to
people after participating in program activities, such as the number
of people who begin working, the number who leave AFDC, and the
number still employed after 6 months.
Goals establish the levels of performance that programs are expected
to achieve. For example, the goal for participants starting work
could be "25 percent of those participating in JOBS each year will
enter full-time employment." Goals can be established for outcomes,
such as the percentage of participants finding employment, and
processes, such as the percentage of participants involved in jobs
skills training, for example. Goals are often accompanied by
financial incentives for meeting or penalties for not meeting goals
whether they are related to outcomes or processes.
Because the overall goal of JOBS was defined very broadly, states
have had the flexibility to focus on a variety of different
objectives in an effort to achieve the broader purpose. These
varying objectives can result in different approaches for providing
JOBS services and different program results. For example, programs
with the objective of quickly increasing welfare recipients' earnings
may emphasize helping participants find any job; whereas, pursuing
the objective of long-term self-sufficiency may lead to more of an
emphasis on education and training activities with the hope of
placing participants in employment that allows them to move off and
stay off AFDC. If there is no agreement on program objectives,
reaching agreement on the outcome indicators and goals needed to
monitor achievement of the objectives will be very difficult.
FSA mandated that HHS develop outcome goals (known as standards in
the act) for JOBS outcomes over time and established goals for
certain processes. It initially required HHS by October 1993 to
recommend JOBS goals based on specific outcome indicators, such as
the number of participants who obtained jobs and moved off welfare.
This requirement was later amended to allow HHS until October 1994 to
develop criteria for outcome goals for JOBS. Through its funding
formula, FSA, in effect, set minimum goals for two JOBS process
indicators: rate of program participation and target group\1
expenditures. FSA specified that (1) at a minimum, 20 percent of
nonexempt\2
adult AFDC recipients participate in JOBS in fiscal year (FY) 1995
and (2) 55 percent of JOBS program funds be spent in each FY on
specified target groups. States are held accountable for meeting
both of these process goals and can lose a portion of their federal
funding if they fail.
Recent legislation reinforces the expectation, originally articulated
in FSA, that HHS develop outcome indicators and goals for JOBS. GPRA
seeks, among other objectives, to transform the focus of federal
agencies from what they are doing to what results they are
accomplishing. To accomplish this purpose, the act requires agencies
to develop 5 year strategic plans beginning in FY 1998 and annual
performance plans beginning in FY 1999. The strategic plans need to
include comprehensive mission statements and general goals and
objectives for the agencies' major functions. The annual performance
plans, which are based on the strategic plans, should set specific
performance goals for the year. Performance indicators will then be
used to monitor progress toward meeting the goals. By adopting a
focus on outcomes, agency effectiveness and congressional
decision-making are expected to improve.
Outcome indicators are useful to program managers and policymakers in
assessing the status of program operations, identifying areas needing
improvement, and ensuring accountability for end results. Indicators
alone, however, do not show the extent to which the program accounts
for an observed outcome. For example, suppose 25 percent of JOBS
participants become employed in a certain time period. JOBS
activities as well as events outside the program, such as
participants' independent efforts to find work or an upsurge in the
economy, could account for participants finding employment.
Determining the extent to which the program contributed to the
observed outcome involves studies that use experimental designs to
estimate what would have happened without the program. In this
example, to estimate the program's impact, such studies might compare
the percentage of JOBS participants becoming employed with the
percentage of comparable AFDC recipients becoming employed without
the program. To measure the impact of JOBS, FSA authorized studies
using experimental designs to isolate the actual impact of the
program. Because such evaluations are usually costly, they are done
infrequently and often involve only select locations.
--------------------
\1 JOBS target group members include AFDC recipients or applicants
who have received AFDC for at least 36 months out of the past 5
years; are under 24 years old and have not completed nor are enrolled
in high school or have little or no work experience for the preceding
year; or are a member of a family in which the youngest child is
within 2 years of being ineligible for AFDC because of age.
\2 Subject to the availability of state resources, AFDC recipients 16
through 59 years old must participate in JOBS unless they are exempt.
Reasons for exemption include illness or incapacity, working 30 hours
or more per week, attending high school, or caring for children under
3 years old. However, teenage parents who have not completed high
school and have children under 3 years old are not exempt.
HHS HAS MADE LITTLE PROGRESS IN
ESTABLISHING AN OUTCOME-FOCUSED
JOBS PERFORMANCE MONITORING
SYSTEM
------------------------------------------------------------ Letter :3
Six years after passage of FSA, HHS only holds state JOBS programs
accountable for participation, not employment. As a result, very
limited national data are available regarding the outcomes of JOBS
participants. In addition, the current approach to performance
monitoring does not assist states in determining whether they are
meeting program goals related to employment and independence from
welfare. According to HHS, a combination of technical and
environmental factors has impeded the development of outcome
indicators for JOBS.
HHS DATA GATHERING FOR JOBS
FOCUSES ON PARTICIPATION,
NOT OUTCOMES
---------------------------------------------------------- Letter :3.1
HHS focuses its JOBS data collection primarily on indicators of
participation. It collects information from all states on the
numbers of program participants, expenditures on target group
members, and the activities individuals are participating in on a
monthly basis. States are accountable for meeting process goals; if
they fail to meet these goals the rates at which state expenditures
are matched by federal dollars may be reduced.\3
Although HHS has established some outcome indicators, data on these
indicators present a very incomplete picture of JOBS outcomes. HHS
data on job entry show that on an average monthly basis in FY 1993, 8
percent of JOBS participants entered employment.\4 In addition to job
entry, HHS gathers data on hourly wages and whether an individual
stopped receiving AFDC due to increased income from working. An ACF
official told us that because states are not held accountable for
outcomes, neither the states nor HHS pays much attention to the
monthly outcome data submitted.
The usefulness of HHS' outcome indicators as tools to help manage the
program is limited for a number of reasons. Because of this approach
to gathering information, HHS cannot answer important questions
regarding whether participants are becoming self-sufficient. To
measure participants' activities, including education, training, job
search, and employment, HHS gathers data each month on a sample of
JOBS participants who (1) took part in any JOBS-sponsored activity in
that month or (2) became employed in the sample or preceding month.
The sample, therefore, excludes anyone who has been employed for more
than 2 months or did not participate in a JOBS activity in the sample
month. This approach to sampling is designed to measure
participants' current employment-related activities, not outcomes
related to whether participants remain employed and move off AFDC as
a result of their earnings. To measure such outcomes, HHS would need
to track individual participants across time. In addition, current
measurement approaches do not yield annual statistics--a common
indicator of program performance--on the percentage of JOBS
participants who became employed. HHS also believes that the quality
of some of the data is poor. For example, an ACF official told us
that the data on hourly wages are unreliable because they are missing
in many cases and often entered incorrectly.
--------------------
\3 For example, if a state failed to meet the JOBS participation goal
of 15 percent of adult nonexempt AFDC recipients in FY 1994, the
federal matching rate in FY 1995 could be reduced from 90 percent to
50 percent (for expenditures up to an amount equal to the state's
Work Incentive (WIN) program allotment for FY 1987).
\4 HHS began collecting data on job entry in FY 1992. The most
recent year for which data are available is FY 1993.
CURRENT MONITORING APPROACH
FOR JOBS DOES NOT SUPPORT AN
EMPHASIS ON EMPLOYMENT AND
REDUCING WELFARE DEPENDENCE
---------------------------------------------------------- Letter :3.2
Our 1994 survey found that state JOBS directors believe that little
progress has been made in establishing a performance monitoring
system that supports achieving program goals related to employment
and independence from welfare. Eighty-two percent of JOBS directors
we surveyed believe that HHS has not sufficiently moved to focus JOBS
on outcome measurement. Our survey also found that over one-half of
state JOBS directors believe that the data gathered on participation
rates and target group expenditures are of little or no use in
helping states achieve their employment and training program goals.
Over one-half of the states believed that the nature of federal
reporting requirements actually hindered their abilities to collect
data on outcome indicators. In addition, ACF reported in May 1994
that while it spends a significant amount of time and resources on
monitoring for JOBS and other programs, performance monitoring at ACF
is in a "state of crisis," in part because the system does not
provide a means for determining if programs are producing the desired
outcomes.\5
These beliefs echo concerns about the absence of outcome data voiced
in 1989 in response to HHS' notice of proposed rule making (NPRM) for
JOBS. In its NPRM for JOBS, HHS originally included no outcome
indicators. Numerous commenters on the NPRM, however, favored the
addition of outcome data. Some of them believed that an excessive
emphasis on participation would drive program operations toward
meeting goals not necessarily related to achieving independence from
welfare through employment. In response to the NPRM comments, HHS
amended its proposed regulations to include some outcome data related
to job entry stating that these data should be included
"since employment as a means to self-sufficiency and
independence from welfare is the objective of the JOBS program."
However, HHS chose not to include additional outcome measures at that
time, in part, because it agreed with one commenter's concern that
outcome data do not show the extent to which the program accounts for
the observed outcomes.
HHS' lack of an outcome-focused performance monitoring system also
limits the possibilities for information sharing and coordination
with other employment and training programs, such as the Job Training
Partnership Act (JTPA). At the local level, JOBS' and JTPA's
services are often combined to meet the education, training, and
support service needs of AFDC recipients. Close coordination is
necessary between the programs to facilitate effective service
delivery. In 1992, the National Governors' Association reported that
a majority of the JTPA administrators they surveyed believed that one
barrier to effective coordination was the lack of consistency between
JTPA's outcome goals and JOBS' process reporting requirements.\6
--------------------
\5 Report of the Administration for Children and Families' Monitoring
Team, Administration for Children and Families, Department of Health
and Human Services (May 1994).
\6 JTPA and JOBS: Coordination and Other Issues, National Governors'
Association (Washington, D.C.: October 1992)
HHS HAS ENCOUNTERED BARRIERS
TO DEVELOPING AN
OUTCOME-FOCUSED PERFORMANCE
MONITORING SYSTEM
---------------------------------------------------------- Letter :3.3
While FSA set process goals, it also required HHS to develop and
submit recommendations for outcome goals to the Congress by October
1993. These recommendations were to include goals for increased
earnings and reduced welfare dependency. HHS missed the October 1993
deadline, but submitted its report on September 30, 1994.\7 This
report identified problems in developing an outcome-focused
performance monitoring system and provided a detailed plan and
schedule for developing outcome indicators and goals.
In its report to the Congress, HHS identified several technical and
environmental factors that contributed to delays in the issuance of
recommendations for outcome goals by the October 1993 deadline. HHS
reported that appropriate outcome indicators had proven difficult to
define in part due to disagreements among key stakeholders, such as
researchers; congressional staff; and federal, state, and local
officials, regarding the primary objectives of the JOBS program. In
addition, setting goals was complicated by possible unintended
program effects, such as programs focusing on the most
employment-ready individuals in order to meet goals. HHS also
reported that turbulence in the welfare system--for example, funding
shortfalls and caseload growth--made it difficult to focus the
necessary attention and resources on developing outcome goals. HHS
also wanted to ensure that proposed goals were compatible with
welfare reform plans being developed by the new administration.
--------------------
\7 In legislation passed in late October 1994, the reporting
requirement was amended to allow HHS to submit a report to the
Congress by October 1, 1994, on criteria for the development of
outcome goals for JOBS.
MANY STATES ACTIVE IN
MONITORING JOBS PROGRAM
OUTCOMES, BUT HAVE MIXED VIEWS
ABOUT SETTING NATIONWIDE GOALS
------------------------------------------------------------ Letter :4
In contrast to the relatively slow progress at the federal level,
many states have been active in developing outcome indicators to
monitor JOBS participant outcomes. To a large extent, this activity
has grown out of each state program's efforts to demonstrate its
effectiveness and garner support for additional state funding.
According to our survey results, states use a variety of outcome
indicators, relying most often on the number of JOBS participants
entering employment and less frequently on job retention rate and
reductions in AFDC payments. Over one-half of the states have also
established goals for their outcome indicators. Appendix II contains
a copy of our survey questionnaire and results for selected
questions.
Although many states use their own outcome goals and support
establishing national goals, they have concerns about how these goals
will be set and used. They maintain that HHS may not be able to
adequately control for differences across states in local economic
conditions and client characteristics that can affect how successful
programs are in placing participants in jobs. They are also
concerned that certain outcome indicators will automatically favor
certain state programs and unduly influence program design decisions,
which they believe should be maintained at the state level.
STATES USE VARIOUS
INDICATORS TO MONITOR JOBS
OUTCOMES
---------------------------------------------------------- Letter :4.1
Despite the absence of a federal approach to collecting JOBS outcome
data, many state programs have been active in tracking JOBS outcomes
at the state level. Our survey of state JOBS directors showed that
states use a variety of indicators to measure outcomes. The two
indicators that states use most often are the number of JOBS
participants entering employment and their hourly wages at hire (see
table 1).
Table 1
Outcome Indicators Used by States
Number of states
using
Indicator indicator\a
------------------------------------------ ----------------
Participants entering employment 49
Hourly wages at hire 42
Participants no longer receiving AFDC due 33
to employment
Job retention rate 26
Participants with reductions in AFDC due 24
to employment
Educational/training achievement 24
------------------------------------------------------------
\a Base is 50 and includes 49 states who responded to our survey and
the District of Columbia.
As shown in figure 1, almost all states reported that they collected
data on the number of participants entering employment during FY 1993
and responded that this is one of the most important indicators to
use to monitor JOBS outcomes. A relatively large number of states
also collected data on hourly wages at hire. However, slightly fewer
states favored using hourly wages at hire as an outcome indicator.
Several states expressed concerns that hourly wages at hire were more
a reflection of local economic conditions than an outcome of the JOBS
program. One state official also noted that measuring hourly wages
at hire could discourage programs from placing participants in
low-wage entry-level positions, which often serve as stepping stones
to higher paying positions.
Figure 1: Percent of States
Using and Favoring Various
Outcome Indicators
(See figure in printed
edition.)
In comparison to employment entry and wages, only two-thirds of
states collect data on the number of participants no longer receiving
AFDC due to employment. However, as shown in figure 1, a relatively
large number of states reported that they believe this indicator
should be used to monitor program outcomes but do not collect the
information because of the difficulty and resources required to
obtain it.
Three other outcome indicators--job retention rate, the number of
participants with reductions in AFDC due to employment, and
educational achievement--are used by almost one-half of the states.
Many more states, however, favored using these three indicators.
While the survey results show that state officials believe that job
retention rate and changes in AFDC benefits are particularly useful
outcomes to monitor, tracking AFDC recipients after case closure is
difficult, time-consuming, and expensive, according to several state
officials. Staffing limitations and the inability to locate ex-AFDC
clients were two reasons they cited for not pursuing these indicators
more aggressively. Several states also explained that resources
allocated to data collection are limited and that they are bound to
first comply with federal reporting requirements. One state official
doubted the program would be given the authority or resources to
gather additional data without a federal mandate to do so.
States also have relied on different approaches to measure outcomes.
For example, to measure the employment rate of JOBS participants,
some states tracked JOBS participants over time and maintained a
count of individual JOBS participants obtaining or retaining
employment over a year. Other states performed periodic studies to
determine how many JOBS participants were working and for how long.
Other states did no more than collect the monthly caseload data
required by HHS.
Given the variation in how states measure employment rates,
determining the rate at which JOBS participants are finding
employment on a national basis is difficult. However, as part of our
survey of state JOBS program directors, we asked states to provide
the number of JOBS participants who had obtained employment during FY
1993. Based on the responses from the 27 states able to provide data
in the format requested, approximately 21 percent of JOBS
participants entered employment during FY 1993.\8 We also asked
states to provide data on the number of JOBS participants retaining
their jobs for 3 months. For the nine states reporting this
information, 33 percent of JOBS participants who entered employment
retained their jobs for at least 3 months.\9
While states have taken different approaches to measuring outcomes,
their interest in outcome measurement appears high. As mentioned
earlier, according to our survey, 82 percent of states indicated HHS
has not done enough to establish outcome indicators for JOBS.
Officials in several states emphasized that they need to establish
outcome indicators to provide their state legislatures with
information about JOBS participants outcomes. Some states were also
disappointed that HHS had not introduced outcome indicators earlier
when states were implementing their JOBS data collection systems so
that they would not have to modify their systems later to meet
federal reporting requirements.
--------------------
\8 Of the 50 states, 27 provided us with total number of JOBS
participants and the number who had entered employment during the
year. These 27 states represent 54 percent of the average monthly
participants in the JOBS program. Because we do not have a national
count of the total number of JOBS participants during the year, we
could not determine the percent of the total number of participants
in a year represented by these states.
\9 These nine states represent 16 percent of the average monthly
number of JOBS participants.
MAJORITY OF STATES SUPPORT
ESTABLISHING OUTCOME GOALS,
BUT WANT SUBSTANTIAL CONTROL
OVER THEIR DEVELOPMENT AND
USE
---------------------------------------------------------- Letter :4.2
While states recognize the need and value of outcome indicators in
managing the JOBS program, their views on establishing outcome goals
are mixed. According to our survey, 29 states had established at
least one goal at the state level for fiscal year 1993 (see fig. 2).
Of these, five states reported that they formally adjusted the
performance goals to account for local differences in client
characteristics or the availability of employment.
Figure 2: States With Outcome
Goals
(See figure in printed
edition.)
As shown in table 2, of the 29 states with established goals, 27
reported that they did so for the number of participants entering
employment. A substantial number of these states also had
established goals for hourly wages earned at the time of hire, job
retention rate, and educational/training achievement.
Table 2
Outcome Goals Used by States
Number of states
Outcome indicator with goals\a
------------------------------------------ ----------------
Participants entering employment 27
Hourly wages at hire 16
Educational/training achievement 14
Job retention rate 12
Participants no longer receiving AFDC due 9
to employment
------------------------------------------------------------
\a Base is 29 states that reported establishing at least one goal.
Several states have been particularly active in developing and using
state-level outcome goals. For example, North Carolina has
established goals for several outcome indicators, including (1) the
percentage of JOBS participants obtaining employment, (2) AFDC
closures or reductions due to earnings, and (3) the percentage of
JOBS participants returning to the AFDC rolls. New Mexico has
recently started funding programs based on performance; its JOBS
program will receive state funding based on how well it does in
meeting established goals. Similarly, California has recently
undertaken an initiative to allocate to counties a portion of state
funding based on performance against designated outcome goals. A
recent study by the American Public Welfare Association's (APWA)
Institute for Family Self-Sufficiency also confirmed that more states
are establishing outcome goals as mechanisms for managing and
improving their programs.\10
A majority of states also support the establishment of nationwide
outcome goals, although many states are concerned about how goals
will be set and used at the federal level. Over 90 percent of the
states responding to our survey indicated that they would like the
flexibility to establish their own goals or to choose their goals
from a menu established by the federal government. During follow-up
interviews, state officials emphasized that they believed certain
outcome indicators would favor particular state approaches to
implementing the JOBS program. For example, programs that invest
more in education and training would benefit from being judged on
education and training achievement or hourly wages at hire, while
programs focusing on early initial job search would probably fare
better being judged on job entry rate. In either case, states agreed
that they wanted to retain the flexibility to design their own JOBS
programs.
State views were mixed on whether federal funds should be linked to
meeting national outcome goals. Over 40 percent of the states were
against linking outcome goals to federal funding, while nearly 30
percent were in favor of doing so. Several state officials did not
believe that federal funding should be tied to national goals because
they doubted that HHS could sufficiently control for differences in
economic conditions and client characteristics across various
geographical regions.
--------------------
\10 APWA's report, Measuring Client Success: Six States Report on
Efforts to Assess What Happens to Clients After They Receive JOBS
Services is based on case studies of Kansas, Maryland, North
Carolina, Oregon, Pennsylvania, and Wisconsin. APWA's results were
similar to GAO's. APWA found that each of the six states used job
placement as well as a variety of other performance goals to manage
its programs. In all six states, performance goals were used to
publicize the achievements of the JOBS program, hold contractors
accountable to specific goals, and facilitate improvements in service
delivery. Three states also used goals to determine funding for
local programs.
ESTABLISHING OUTCOME INDICATORS
AND GOALS IS CRITICAL TO MAKING
JOBS MORE EMPLOYMENT-FOCUSED
------------------------------------------------------------ Letter :5
Establishing effective outcome indicators and goals is critical to
sharpening JOBS' focus on the ultimate goals of employment and
independence from welfare, whether JOBS remains the same or is
replaced with a welfare-to-work block grant program that includes
some JOBS' objectives and activities, as has been proposed. Current
congressional and public interest in welfare reform as well as GPRA
requirements indicate a need for HHS to move decisively to establish
national leadership regarding outcome measurement for JOBS. Before
effective outcome indicators and goals can be established, important
differences among stakeholders regarding the objectives of JOBS will
have to be resolved.
HHS PLANS TO ADD OUTCOME
INDICATORS TO THE JOBS
PERFORMANCE MONITORING
SYSTEM AND MODIFY THE
SYSTEM'S PROCESS INDICATORS
---------------------------------------------------------- Letter :5.1
In its September 1994 report to the Congress, HHS acknowledged the
value of and affirmed its commitment to using outcome measurement in
its performance monitoring system for JOBS. Specifically, HHS plans
to develop outcome indicators and goals and refine existing process
indicators and goals. In addition, HHS plans to modify the AFDC
Quality Control system\11 by adding key process indicators, such as
participation rates.
In developing outcome indicators and goals, HHS faces a complex and
difficult task. HHS recognizes that it must ensure that indicators
and goals help the program achieve its objectives and meet the needs
of numerous stakeholders, including local service providers, state
and federal managers, and policymakers. In its 1994 report to the
Congress, HHS identified and laid the groundwork for addressing a
number of critical design and implementation issues that must be
addressed to ensure that indicators and goals support program
objectives, are fair to all states, and avoid unintended program
consequences. These issues include (1) developing a process to
ensure states are given an equal opportunity to meet standards by
adjusting for differences among states (that is, levelling the
playing field), (2) designing strategies to discourage states from
serving only the easiest-to-serve clients, and (3) selecting data
collection approaches that are feasible and cost-effective.
To develop outcome indicators, HHS plans to convene a working group
composed of representatives from the Congress, HHS, the Department of
Labor, the states, AFDC recipients, community-based organizations,
and others. This group will convene by April 1995 and make
recommendations to HHS on specific outcome indicators and methods for
data collection by January 1, 1996. Proposed indicators are to be
published in the Federal Register no later than April 1996 and
finalized by October 1, 1996. Using a similar process, current plans
call for outcome goals to be developed and finalized by October 1,
1998.
HHS supports continued use of existing JOBS process indicators but
believes that changes are needed to process goals to make them more
effective. In addition, in its September 1994 report to the
Congress, HHS proposed changing the process goal from the current 20
percent rate to a rate between 45 and 55 percent. HHS also suggested
changing target-group goals to achieve higher levels of participation
among the youngest AFDC parents.
--------------------
\11 The states and HHS use the Quality Control system to evaluate
whether AFDC payments are made accurately and to determine how well
states comply with regulations to prevent waste, fraud, and abuse.
CONGRESSIONAL AND NATIONAL
INTEREST AND GPRA ADD
URGENCY TO THE NEED FOR HHS
TO ESTABLISH OUTCOME
INDICATORS
---------------------------------------------------------- Letter :5.2
Recently, there has been strong congressional and national interest
in AFDC becoming more focused on helping recipients become employed
and leave AFDC in a limited time period. Numerous bills to reform
AFDC and JOBS were introduced in the 103rd Congress and more are
likely to be introduced in the 104th. Several recent welfare reform
bills would replace AFDC and JOBS with a welfare-to-work block grant
program. To the extent that JOBS objectives and activities are
retained in the block grant, outcome indicators and goals for JOBS
would be useful in ensuring accountability and improving
congressional oversight.
While FSA, which originally required HHS to develop outcome goals
only for JOBS, was limited to one program, GPRA requires all federal
agencies to develop strategic plans, annual performance plans, and
annual performance reports. Strategic plans articulate the agency's
essential mission, long-term general goals and objectives, and a plan
of action for achieving the objectives. The annual performance
plans, by establishing a set of performance indicators and goals,
provide a link between the agency's longer-term objectives and what
managers and staff must accomplish on a daily basis to achieve those
objectives. GPRA requires that agencies must submit to Congress
5-year strategic plans beginning in FY 1998 and annual performance
plans beginning in FY 1999. The performance plans are expected to
cover each program activity set forth in the agency's budget.
Currently, over 70 agencies and programs are involved in pilot
projects for GPRA. The National Academy of Public Administration
(NAPA) recently reviewed these pilots and found that some agencies
had made limited progress in developing their plans due to
insufficient preparation time. NAPA recommended that all programs in
an agency, not only those currently operating as pilots, begin as
soon as possible to develop strategic and performance plans so that
the FY 1998 and FY 1999 deadlines could be met.
If JOBS activities are retained within an AFDC block grant, a system
of accountability for the end results of these activities would still
be needed. GPRA and the administration's National Performance Review
(NPR) promote the use of outcome indicators for all programs,
including those funded through block grants. To date, however, data
collected by states under most block grants have focused on process
indicators such as the number of clients served. In addition, past
block grant programs have not often gathered consistent information
on program activities and outcomes to support congressional
oversight. In reviewing data collection under block grants, we found
that, among other things, national leadership in directing the
development of model data-gathering criteria could increase data
comparability and, as a result, oversight.
CLARIFYING PROGRAM
OBJECTIVES IS FIRST STEP IN
ESTABLISHING INDICATORS AND
GOALS
---------------------------------------------------------- Letter :5.3
HHS and performance monitoring system experts agree that the first
critical step in developing outcome indicators and performance goals
is to reach agreement among stakeholders, such as the Congress;
researchers; and federal, state, and local officials, regarding the
objectives of the program. According to HHS, disagreements among
stakeholders about the objectives of the JOBS program have been a
major obstacle to developing JOBS outcome indicators. Difficulty
clarifying JOBS objectives may again prove to be one of the biggest
obstacles in the effort to establish outcome indicators and goals.
In our survey of state JOBS directors, we found some disagreement
regarding the programs' overriding objectives. Eighty percent of the
directors responded that the overriding objective was to prepare and
place participants in employment that allows them to move off and
stay off AFDC. The other 20 percent stated that their objective was
to get participants employed in any job, part- or full-time, even if
the job might not allow them to move off AFDC. These two objectives,
although consistent with the overall objectives of FSA, would likely
produce differently designed JOBS programs and different short-term
results, thus making the establishment of appropriate outcome goals
difficult.
CONCLUSION
------------------------------------------------------------ Letter :6
Program managers, policymakers and other stakeholders need to know
whether JOBS participants are finding employment and leaving AFDC.
Very little is known nationally about the outcomes of JOBS
participants because HHS has not moved aggressively on developing an
outcome monitoring system. Many believe that establishing effective
outcome indicators and goals is critical to refocusing JOBS on the
ultimate goals of employment and independence from welfare.
Effective indicators and goals would also help ensure accountability
for achieving these critical outcomes should AFDC and JOBS be
replaced with a welfare-to-work block grant program that includes
some JOBS' objectives and activities. The states have a strong
interest in outcome measurement and are aggressively pursuing a
variety of approaches in this area. State efforts provide a rich
well of experience that can be drawn on in developing a national
approach to measuring JOBS outcomes. To meet the requirements of FSA
and GPRA, HHS must move decisively to establish national leadership
regarding outcome measurement for JOBS.
AGENCY COMMENTS
------------------------------------------------------------ Letter :7
In its March 28, 1995, comments on our draft report, HHS generally
agreed with our conclusion that its data are incomplete and focused
on process-oriented goals. However, HHS believes that we did not (1)
adequately portray the difficulty of developing an outcome-focused
performance measurement system or (2) give adequate weight to the
importance of certain technical issues or the progress that HHS has
made in addressing those issues (see app. III). We added language
in the report to more explicitly recognize the difficulty of the task
and HHS' progress in identifying important technical issues (see p.
15). HHS also suggested minor technical revisions to the draft,
which we incorporated as appropriate.
Our work was performed between January 1994 and February 1995, in
accordance with generally accepted government auditing standards,
with the exception that we did not check the accuracy of outcome data
reported by HHS and the states.
We are providing copies of this report to the Secretary of Health and
Human Services, other HHS officials, and state JOBS program
administrators. We will also make copies available to other
interested parties upon request. Major contributors to this report
are listed in appendix IV. If you have questions about this report,
please call Robert MacLafferty on (415) 904-2000.
Sincerely yours,
Jane L. Ross
Director, Income Security Issues
SCOPE AND METHODOLOGY
=========================================================== Appendix I
To determine the progress HHS had made in establishing outcome
indicators and goals and the issues relevant to their development, we
interviewed officials from HHS' Administration for Children and
Families, which is responsible for the JOBS program at the federal
level. We also reviewed (1) the data-reporting procedures for the
JOBS program, (2) HHS reports that summarize the outcome data
collected at the federal level, and (3) various reports on the status
within ACF of monitoring and developing outcome measures. We did not
verify the accuracy of federal data, but were told by HHS officials
that the data were not complete or accurate. In addition, we
reviewed the welfare-to-work and performance measurement literature
and spoke with officials from various welfare research and interest
groups, including APWA, the National Governors' Association, and the
Manpower Demonstration Research Corporation.
To determine state performance measurement practices, in May 1994, we
surveyed JOBS program administrators in the 50 states and the
District of Columbia. We also examined 1993 and 1994 reviews of
state JOBS programs conducted by ACF's regional offices and obtained
available reports on outcome data produced by the states.
In our survey, we used a mail questionnaire to collect information on
FY 1993 general program characteristics, the use of performance
indicators and goals at the state level, and state preferences
regarding the development of nationwide indicators and goals. We
also requested information for selected outcome data elements for FY
1993. We received survey responses from the District of Columbia and
all 50 states except Iowa. However, no respondents could provide
complete responses to our request for annual, unduplicated outcome
data for FY 1993, even though many states reported that they monitor
some outcome measures.
To obtain additional information and determine why states did not
provide requested outcome data, we conducted follow-up telephone
interviews with officials in 10 states: Alaska, Arkansas,
California, Idaho, Louisiana, New Jersey, New Mexico, Nevada, New
York, and Washington. We first identified states that reported that
they monitor participants entering employment but were unable to
provide an annual, unduplicated count of JOBS participants entering
employment for FY 1993 (as we had requested in our survey). Among
these states, we then selected 10 to contact, which included states
with large, medium, and small caseloads. Based on our follow-up
work, we determined that six states could not provide annual,
unduplicated data because the data were not available. In the other
four states, the data were available but reporting them would have
required significant time or resources.
GAO QUESTIONNAIRE REGARDING JOBS
PROGRAM CHARACTERISTICS
========================================================== Appendix II
Our survey, sent to state JOBS administrators, contained questions on
general program characteristics, outcome indicators, outcome goals,
and participant outcome data. This appendix includes the full text
of the survey and the aggregate responses for the first three
sections of the survey. The number cited next to each response
category is the number of states that responsed.
The appendix does not include the responses regarding participant
outcome data. Many states were not able to provide the total number
of individuals served by their programs in fiscal year 1993 and even
fewer states could answer questions about participant outcomes in the
format that we requested. As a result, we have not annotated the
participant outcome data section of the survey with the incomplete
data reported to us.
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)Appendix III
COMMENTS FROM THE DEPARTMENT OF
HEALTH AND HUMAN SERVICES
========================================================== Appendix II
(See figure in printed edition.)
(See figure in printed edition.)
MAJOR CONTRIBUTORS TO THIS REPORT
========================================================== Appendix IV
Robert L. MacLafferty, Assistant Director, (415) 904-2000
Stephen D. Secrist, Evaluator-in-Charge, (415) 904-2000
Susan E. Arnold
Kay E. Brown
LuAnn M. Moy
Ann T. Walker
Christina L. Warren
Karen D. Wright