Social Security Disability: Improved Processes for Planning and
Conducting Demonstrations May Help SSA More Effectively Use Its
Demonstration Authority (04-NOV-04, GAO-05-19).
Since 1980, the Congress has required the Social Security
Administration (SSA) to conduct demonstration projects to test
the effectiveness of possible program changes that could
encourage individuals to return to work and decrease their
dependence on Disability Insurance (DI) benefits. To conduct
these demonstrations, the Congress authorized SSA, on a temporary
basis, to waive certain DI and Medicare program rules and to use
Social Security Trust Funds. The Congress required GAO to review
SSA's use of its DI demonstration authority and to make a
recommendation as to whether this authority should be made
permanent.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-05-19
ACCNO: A13314
TITLE: Social Security Disability: Improved Processes for
Planning and Conducting Demonstrations May Help SSA More
Effectively Use Its Demonstration Authority
DATE: 11/04/2004
SUBJECT: Disability benefits
Disability insurance
Evaluation methods
Federal social security programs
Policy evaluation
Program evaluation
Workfare
Program graduation
Budget authority
Demonstration projects
Social Security Disability Insurance
Trust Fund
Social Security Disability Insurance
Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-05-19
United States Government Accountability Office
GAO Report to the Committee on Finance, U.S. Senate, and the Committee on Ways
and Means, House of Represetatives
November 2004
SOCIAL SECURITY DISABILITY
Improved Processes for Planning and Conducting Demonstrations May Help SSA More
Effectively Use Its Demonstration Authority
On November 9, 2004, this report was reposted to the Web because the Highlights
page was missing from the electronic copy.
a
GAO-05-19
[IMG]
November 2004
SOCIAL SECURITY DISABILITY
Improved Processes for Planning and Conducting Demonstrations May Help SSA More
Effectively Use Its Demonstration Authority
What GAO Found
SSA has not used its demonstration authority to extensively evaluate a
wide range of DI policy areas dealing with return to work. Despite being
given the authority to assess a broad range of policy alternatives, SSA
has, until very recently, focused its demonstration efforts mostly on a
relatively narrow set of policy issues-those dealing with the provision of
vocational rehabilitation and employment services. SSA's recently proposed
or initiated demonstrations have begun to address a broader range of
policy issues, such as provisions to reduce, rather than terminate,
benefits based on earnings above a certain level. However, the agency has
no systematic processes or mechanisms for ensuring that it is adequately
identifying and prioritizing those issues that could best be addressed
through use of its demonstration authority. For example, the agency has
not developed a formal demonstration research agenda explicitly
identifying its broad vision for using its DI demonstration authority and
explaining how ongoing or proposed demonstration projects support
achievement of the agency's goals and objectives.
SSA's demonstration projects have had little impact on the agency's and
the Congress' consideration of DI policy issues. This is due, in part, to
methodological limitations that have prevented SSA from producing project
results that are useful for reliably assessing DI policy alternatives. In
addition, SSA has not established a formal process for ensuring that its
demonstration results are fully considered for potential policy
implications. For example, SSA does not maintain a comprehensive record of
its demonstration results that could be used to build a body of knowledge
for informing policy decisions and planning future research. Furthermore,
SSA's reporting of demonstration project results has been insufficient in
ensuring that the Congress is fully apprised of these results and their
policy implications.
United States Government Accountability Office
Contents
Letter
Results in Brief
Background
SSA Has Not Used Its Demonstration Authority to Evaluate a Wide
Range of DI Return-to-Work Policy Issues SSA's Demonstration Projects Have
Had Little Influence on
Consideration of DI Policy Changes Conclusions Recommendations Matters for
Congressional Consideration Agency Comments and Our Evaluation
1
3 4
10
18 23 24 25 27
Appendix I Scope and Methodology
Appendix II Comments from the Social Security Administration
Appendix III GAO Contacts and Staff Acknowledgments 34
GAO Contacts 34 Staff Acknowledgments 34
Tables
Table 1: Completed or Nearly Completed Projects Conducted under SSA's DI
Demonstration Authority 12 Table 2: Recently Proposed or Initiated
Projects Conducted under SSA's DI Demonstration Authority 15
Abbreviations
DI Disability Insurance
HHS Department of Health and Human Services
OASDI Old-Age and Survivors Insurance and Disability Insurance
OPDR Office of Program Development and Research
RDP Research Demonstration Program
RFP request for proposal
RSVP Referral System for Vocational Rehabilitation Providers
SGA substantial gainful activity
SPI State Partnership Initiative
SSI Supplemental Security Income
SSA Social Security Administration
SSAB Social Security Advisory Board
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
United States Government Accountability Office Washington, DC 20548
November 4, 2004
The Honorable Charles E. Grassley
Chairman
The Honorable Max Baucus
Ranking Minority Member
Committee on Finance
U. S. Senate
The Honorable William M. Thomas
Chairman
The Honorable Charles B. Rangel
Ranking Minority Member
Committee on Ways and Means
House of Representatives
From 1982 to 2002, the number of disabled workers receiving benefits
under the Disability Insurance (DI) program doubled from 2.6 million to
5.5 million, while payments quadrupled from about $14.8 billion to
$60 billion. Although technological and medical advances, along with
social changes, have increased the potential for some people with
disabilities to participate in the labor force, less than 1 percent of DI
beneficiaries leave the rolls each year because they have reentered the
workforce. The enactment of various DI work incentives that are intended
to encourage beneficiaries to work-and, potentially, to leave the rolls-
has also had little discernible impact on beneficiaries' success in
returning
to the workforce. This low rate of return to work, coupled with escalating
growth in DI program enrollment and benefit costs over the past few
decades, threatens to exhaust the DI Trust Fund by 2029. Yet relatively
little is known about the factors that help beneficiaries overcome
employment challenges and disincentives and that inhibit them from
achieving an earnings level that leads to self-sufficiency. GAO has
designated federal disability programs as a high-risk area, in part
because
of the significant growth in the DI program and the long-standing
challenges in devising effective return-to-work policies.1
In response to these challenges, the Congress has, since 1980, required
the Social Security Administration (SSA) to conduct demonstration projects
to test the effectiveness of possible program changes that could encourage
individuals to return to work and decrease their dependence on DI
benefits. In fostering return to work, these demonstrations and the
program changes they test are intended to produce savings in the Trust
Funds or improve DI program administration. To achieve these objectives,
SSA's DI demonstration authority contains several key features that
provide SSA with a potentially valuable tool for assessing the
effectiveness of policy alternatives. One of these features is SSA's
authority to waive certain DI and Medicare program rules. For example,
when conducting demonstrations, SSA is permitted to exempt certain
beneficiaries from requirements that workers with disabilities earn below
a certain amount to remain eligible for DI benefits or that they wait 24
months to become eligible for Medicare benefits. Another key aspect of
SSA's demonstration authority is the requirement that DI demonstration
projects be of sufficient scope and conducted on a wide enough scale to
ensure a thorough evaluation and results that are applicable to the DI
program as a whole. In addition, the legislation authorized SSA to use DI
Trust Fund and Old-Age and Survivors Insurance Trust Fund monies to pay
for the demonstrations and required SSA to periodically report to the
Congress on its demonstration activities, providing, when appropriate,
recommendations for legislative or administrative changes. The Congress
initially granted SSA this demonstration authority for a 5-year period but
subsequently renewed the authority several times, most recently extending
it through 2005.
The Ticket to Work and Work Incentives Improvement Act (P.L. 106-170)
mandated that GAO review SSA's use of its DI demonstration authority and
provide a recommendation as to whether SSA should be given this authority
permanently. To develop a basis on which to make this
1GAO identifies federal programs and operations that are high-risk because
of their greater vulnerability to fraud, waste, abuse, and mismanagement.
GAO has also increasingly used its high-risk designation to draw attention
to the challenges faced by government programs and operations in need of
broad-based transformation. See GAO, High-Risk Series: An Update,
GAO-03-119 (Washington, D.C.: January 2003) and GAO, Major Management
Challenges and Program Risks: Social Security Administration, GAO-03-117
(Washington, D.C.: January 2003).
Results in Brief
recommendation, we examined the extent to which (1) SSA has used the
demonstration authority to test a range of program changes and (2) SSA's
demonstration authority has been used to inform policy decisions. In
conducting our examination, we reviewed relevant federal statutes and
regulations governing DI demonstration activities and interviewed SSA
officials with current and prior responsibility for demonstration
projects. We also reviewed documents related to SSA's planning,
implementation, and evaluation of demonstration projects, including agency
reports to the Congress, public notices, and reports analyzing project
results. In addition, we examined other reviews of SSA's demonstrations
(including prior GAO and Inspector General reports) and interviewed
officials from disability research, advisory, and advocacy organizations.
Furthermore, we obtained information on research and demonstrations
conducted by other federal agencies. We performed our work in accordance
with generally accepted government auditing standards between October 2003
and August 2004. For more details about our scope and methodology, see
appendix I.
SSA has not used its demonstration authority to extensively evaluate a
wide range of DI policy areas dealing with return to work. Despite being
given the authority to assess a broad range of policy alternatives, SSA
has, until very recently, focused its demonstration efforts mostly on a
relatively narrow set of policy issues-those dealing with the provision of
vocational rehabilitation and employment services. But even in that policy
area, SSA's use of DI demonstration authority has not been comprehensive
and, therefore, did not extensively address key policy issues that the
agency is currently grappling with in its efforts to implement a new
approach for providing return-to-work assistance. SSA's recently proposed
or initiated demonstrations have begun to address a broader range of
policy issues, such as provisions to reduce, rather than terminate,
benefits based on earnings above a certain level. However, the agency has
no systematic processes or mechanisms for ensuring that it is adequately
identifying and prioritizing those issues that could best be addressed
through use of its demonstration authority. For example, the agency has
not developed a formal demonstration research agenda explicitly
identifying its broad vision for using its DI demonstration authority and
explaining how ongoing or proposed demonstration projects support
achievement of the agency's goals and objectives.
SSA's demonstration projects have had little impact on the agency's and
the Congress' consideration of DI policy issues. This is due, in part, to
methodological limitations that have prevented SSA from producing project
results that are useful for reliably assessing DI policy alternatives.
Background
In addition, SSA has not established a formal process for ensuring that
its demonstration results are fully considered for potential policy
implications. For example, SSA does not maintain a comprehensive record of
its demonstration results that could be used to build a body of knowledge
for informing policy decisions and planning future research. Furthermore,
SSA's reporting of demonstration project results has been insufficient in
ensuring that the Congress is fully apprised of these results and their
policy implications.
This report contains recommendations to help ensure the effectiveness of
SSA's DI demonstration efforts. It also identifies several matters that
the Congress may wish to consider, including continuation of SSA's
demonstration authority on a temporary basis. In its comments on a draft
of this report, SSA agreed with our recommendations. SSA also agreed that
it has not, in the past, used its demonstration authority to extensively
evaluate DI policy, but it noted that its recently initiated or proposed
demonstrations will play a vital role in testing program and policy
changes. SSA's comments appear in appendix II.
Established in 1956, DI is an insurance program that provides monthly cash
benefits to workers who are unable to work because of severe longterm
disability. Workers who have worked long enough and recently enough are
insured for coverage under the DI program. To meet the definition of
disability under the DI program, an individual must have a medically
determinable physical or mental impairment that (1) has lasted or is
expected to last at least 1 year or to result in death and (2) prevents
the individual from engaging in substantial gainful activity (SGA).
Individuals are considered to be engaged in SGA if they have countable
earnings above a certain dollar level.2 Once a person is on the disability
rolls, benefits continue until (1) the beneficiary dies, (2) the
beneficiary becomes eligible for Social Security retirement benefits at
full retirement age,3 (3) SSA determines that the beneficiary is no longer
eligible for benefits because his or her earned income exceeds the SGA
level, or (4) SSA decides that the beneficiary's medical condition has
improved to
2For 2004, SSA considers countable earnings above $810 a month to be
substantial gainful activity for persons who are not blind and $1,350 a
month for persons who are blind.
3The full retirement age is the earliest age at which an unreduced
retirement benefit is payable. The age for full retirement benefits is
scheduled to rise gradually from age 65 to
67.
the point that he or she is no longer considered disabled. In 2002, SSA
paid about $60 billion in DI cash benefits to 5.5 million disabled
workers,4 with average monthly benefits amounting to $834 per person.5 In
addition to receiving cash assistance, beneficiaries automatically qualify
for Medicare after 24 months of DI entitlement.
During the 1970s, as the number of disability awards increased
significantly and resulted in substantial cost increases for the DI
program, the Congress became concerned about the growth of the DI program
and program rules that provided disincentives to returning to work. To
encourage DI beneficiaries to return to work-and, potentially, to leave
the benefit rolls-the Congress has, over the years, enacted legislation
providing various work incentives. Such incentives include a trial work
period during which beneficiaries may earn any amount for 9 months within
a 60-month period and still receive full cash and medical benefits and
continued Medicare coverage that allows beneficiaries to maintain
eligibility for Medicare for at least 39 months following a trial work
period as long as medical disability continues.
In an effort to further address these issues, the Congress, in 1980,
required SSA to conduct demonstration projects6 to evaluate the
effectiveness of policy alternatives that could encourage DI beneficiaries
to reenter the
4In 2002, the DI program also paid about $6 billion in cash benefits to
about 1.7 million spouses and children of disabled workers. DI cash
benefits are paid from the Federal Disability Insurance Trust Fund. This
trust fund is funded through payroll deductions paid by employers and
workers.
5For disabled and blind individuals who have low income and limited
resources, SSA provides cash benefits under the Supplemental Security
Income (SSI) program. Although both the DI and the SSI programs use the
same definition of disability, SSI has no prior work requirement. In 2002,
SSA paid about $26 billion in SSI benefits to about 5.5 million people
with disabilities, of whom 3.9 million were working age adults aged 18 to
64. The average monthly federal SSI cash benefit amounted to about $398
per person. In 2002, about 1.2 million DI beneficiaries were dually
eligible for SSI benefits because of the low level of their income and
resources.
6This DI demonstration authority was provided under Section 505(a) of the
Social Security Disability Amendments of 1980 (Pub. L. No. 96-265).
Section 505(b) of this act amended Section 1110 of the Social Security Act
to provide SSA with similar demonstration authority for the SSI program,
including authority to waive SSI program rules. However, unlike the DI
demonstrations, SSI demonstration projects were to be funded from
congressional appropriations. Although SSA's DI and SSI demonstration
authorities are separate and somewhat distinct, the agency's disability
demonstration projects often involve both DI and SSI beneficiaries and
applicants. When a demonstration is conducted under both the DI and SSI
demonstration authorities, funding for the project is also split between
trust fund (i.e., DI) and appropriated (i.e., SSI) sources.
workforce.7 A key aspect of this demonstration authority is SSA's ability
to waive DI and Medicare program rules to the extent needed in conducting
these projects. The legislation granting DI demonstration authority also
identified a variety of policy alternatives for SSA to consider testing,
including (1) alternative ways of treating DI beneficiaries' work-related
activity such as methods allowing for a reduction in benefits based on
earnings and (2) modifications in other rules, such as the trial work
period and Medicare eligibility waiting period, that may serve as
obstacles to DI beneficiaries returning to work. In addition, this
legislation identified several requirements pertaining to the design and
evaluation of DI demonstration projects. In particular, these projects
were required to be of sufficient scope and carried out on a wide enough
scale to permit a thorough evaluation of the policy alternatives studied
such that the results would be generally applicable to the operation of
the DI program. The law additionally required SSA to submit reports to the
Congress announcing the initiation of DI demonstration projects as well as
periodic reports describing the status of these projects and a final
report on all projects carried out under the demonstration authority. SSA
was directed to make recommendations, when appropriate, for legislative or
administrative changes in its reports to the Congress.
Another important aspect of SSA's DI demonstration authority is that
unlike other SSA research activities, which are funded through
congressional appropriations, these projects can be paid for with DI Trust
Fund and Old-Age and Survivors Insurance Trust Fund monies. Therefore, SSA
is not required to obtain congressional approval for DI demonstration
expenditures, although it is required to receive approval from the Office
of Management and Budget for an annual apportionment of Trust Funds for
these demonstrations.
SSA's DI demonstration authority has always been granted on a temporary
basis and therefore has been subject to periodic review and renewal by the
Congress.8 After initially granting this authority for a 5-year period,
the
7In a report discussing federal evaluation activities, GAO defined
demonstration projects as "those that aim to provide evidence of the
feasibility or effectiveness of a new approach or practice." See GAO,
Program Evaluation: Improving the Flow of Information to Congress,
GAO-PEMD-95-1 (Washington, D.C.: January 30, 1995). This definition is
consistent with the purpose of SSA's DI demonstration projects, as spelled
out in legislation.
8Although SSA has temporary authority for DI demonstrations, it has
permanent authority to conduct SSI demonstrations.
Congress subsequently renewed it several times, in 1986, 1989, 1994, 1999,
and 2004. The renewal of SSA's authority has sometimes been delayed so
that SSA has, on several occasions, gone without DI demonstration
authority. For example, after its demonstration authority expired in June
1996, SSA was not again granted DI demonstration authority until December
1999. Most recently, the Congress extended this demonstration authority
through December 2005.9
In addition to granting this general DI demonstration authority, the
Congress may enact legislative mandates for SSA to conduct specific DI
demonstration projects. For example, the Ticket to Work and Work
Incentives Improvement Act of 1999 required SSA to conduct a demonstration
to assess the effectiveness of a benefit offset program under which DI
benefits are reduced by $1 for every $2 in earnings (above a certain
level) by a beneficiary. SSA's authority to conduct this demonstration is
similar in some respects to the authority it has under its general DI
demonstration statute. For instance, the statute allows waiver of DI and
Medicare program provisions to carry out this benefit offset
demonstration. However, some differences exist between the two
authorities. In particular, the benefit offset demonstration authority
provides a more detailed and comprehensive list of demonstration
objectives for SSA to fulfill than does SSA's general authority. For
example, the benefit offset demonstration authority lists six "matters to
be determined," which include assessments of project costs; savings to the
Trust Funds; and project effects on employment outcomes such as wages,
occupations, benefits, and hours worked.
Regardless of the authority under which they are carried out,
demonstration projects examining the impact of social programs are
inherently complex and difficult to conduct. Measuring outcomes, ensuring
the consistency and quality of data collected at various sites,
establishing a causal connection between outcomes and program activities,
and separating out the influence of extraneous factors raise formidable
technical and logistical problems. Thus, these projects generally require
a planned study and considerable time and expense. Adding to these
complexities are other administrative or statutory requirements affecting
SSA's DI demonstrations. For example, SSA's
9In addition to extending the authority to initiate DI demonstration
projects through 2005, Pub. L. No. 108-203 would also allow DI
demonstrations that SSA has initiated on or before December 17, 2005, to
be completed thereafter.
policy is that its demonstration projects must not make those who
participate in the project worse off, which could limit the specific types
of policy alternatives the agency can study or the methods used to study
such alternatives.
Although the legislation granting DI demonstration authority does not
prescribe the use of particular methodological approaches, SSA has
repeatedly recognized that the law's general requirements for
demonstration evaluations require SSA to conduct these projects in a
rigorous manner that provides the agency with a reliable basis for making
policy recommendations.10 Rigorous methods are required to estimate the
net impact of a tested disability policy option because many other
factors, such as the economy, can influence whether a beneficiary returns
to work. In an August 2002 report to the SSA Commissioner, an SSA advisory
panel stated that it is widely agreed that experimental designs,11 "when
feasible from operational and budgetary perspectives and when they can be
conducted without serious threats to their validity, are the best
methodology for determining the effects of changes in government
programs."12 In addition, SSA officials and other researchers have noted
the advantages of experimental designs in providing policymakers with more
clear-cut results that are less subject to debate than results derived
from other methods.13 However, when experimental designs are not
10The results of a rigorous demonstration may lead SSA to recommend
adoption of a particular policy option or, alternatively, to recommend
that such an option not be pursued. Also, to the extent that the results
of a demonstration are not definitive enough to make such policy
recommendations, SSA may instead propose that additional research be
conducted to further assess policy alternatives.
11Experimental designs involve random assignment of study participants to
either a treatment group or a control group. The treatment group is
subjected to the new program or policy, and the control group is not. The
strength of the experimental design is in its assurance that those who
experience the treatment are like, in all important ways, those who do not
experience the treatment except for the difference in receiving the
treatment itself.
12Ticket to Work and Work Incentives Advisory Panel, Advice Report to the
Commissioner of Social Security: Statutory Requirements and Design Issues
Related to SSDI $1 for $2 Benefit Offset Research, p. 12 (Washington,
D.C.: August 2002).
13For example, in a 1997 publication examining welfare reform evaluations,
Besharov and others state that with experimental designs "Policymakers can
then focus on the implications of findings, rather than `become entangled
in a protracted and often inconclusive scientific debate about whether the
findings of a particular study are statistically valid.'" See Douglas J.
Besharov, Peter Germanis, and Peter H. Rossi, Evaluating Welfare Reform, A
Guide for Scholars and Practitioners, the University of Maryland, School
of Public Affairs, 1997, p. 42.
feasible or desirable, the use of quasi-experimental designs14 offers a
reasonably rigorous evaluation alternative that may, under certain
circumstances, offer advantages over experimental designs.15
Other factors may also limit the rigor of DI demonstrations, including
insufficient sample sizes, inconsistency in demonstration design or
implementation across multiple project sites, and deficiencies in data
collection. Such design, implementation, and evaluation weaknesses may
hamper SSA's use of project results as a basis for making policy
recommendations because they limit the agency's ability to (1) control for
factors external to the demonstration, (2) generalize demonstration
results to a wider population of DI beneficiaries, and (3) isolate the
effects of specific policy interventions from the overall effects produced
by a demonstration.
The Office of Program Development and Research (OPDR) is the entity within
SSA that develops and implements demonstration projects for the DI and
Supplemental Security Income (SSI) Programs.16 OPDR program and research
staff-sometimes with the assistance of outside research
organizations-identifies the broad outlines and requirements of disability
program demonstration projects, including the basic objectives, scope, and
methodological standards for these projects. SSA then issues formal
notices requesting public or private sector organizations to submit offers
to conduct the demonstration projects, which may include development of a
detailed design plan, provision of technical support, collection of
project
14Alternative methods involving quasi-experimental designs do not involve
the use of randomized control groups. Instead, such designs rely on the
nonrandom selection of a comparison group of nonparticipants with
characteristics similar to those of study participants.
15Although experimental methods are generally preferred, they may not
always be feasible or may not necessarily provide the optimal approach for
conducting a demonstration because of various implementation or ethical
considerations. Quasi-experimental methods may instead be preferable or
may need to be used in conjunction with experimental methods to produce
the most rigorous results. For instance, quasi-experimental methods may be
preferred when there is a chance that a randomized approach will not
remain intact, such as when participants in the control group are likely
to receive elements of the intervention.
16OPDR is one of several organizational components under SSA's Office of
Disability and Income Security Programs (ODISP), which has responsibility
for carrying out DI and SSI program functions. Prior to OPDR's creation in
November 2002, responsibility for disability demonstration projects was
shared by components within ODISP and SSA's Office of Policy.
data, or evaluation of project results. On the basis of SSA's review of
submitted proposals and bids, the agency may enter into grants,
cooperative agreements, or contractual arrangements with one or more
organizations to carry out demonstration projects.17 For example, a single
demonstration may involve cooperative agreements with states to design and
implement projects as well as contracts with one or more research
institutions to provide technical assistance to the states and evaluate
demonstration results. Project managers in OPDR have the primary
responsibility for overseeing demonstration projects to ensure that they
meet SSA's technical and programmatic requirements. OPDR collaborates with
SSA's Office of Acquisition and Grants in issuing formal project notices
and solicitations and, subsequently, in overseeing grant or contract
performance.
SSA Has Not Used Its Demonstration Authority to Evaluate a Wide Range of DI
Return-to-Work Policy Issues
SSA has not used its demonstration authority to extensively evaluate a
wide range of DI policy areas dealing with return to work. Until very
recently, SSA has focused its demonstration efforts primarily on a
relatively narrow set of policy issues dealing with the provision of
vocational rehabilitation and employment services, despite being given the
authority to assess a much broader range of policy alternatives. Even in
the area of vocational rehabilitation and employment issues, SSA's use of
DI demonstration authority has not been comprehensive and, therefore, did
not extensively address key policy issues that the agency is currently
grappling with under its Ticket to Work program. SSA's recently initiated
or proposed demonstrations have begun to address a broader range of policy
issues. However, the agency has no systematic processes or mechanisms for
ensuring that it is adequately identifying and prioritizing those issues
that could best be addressed through use of its demonstration authority.
SSA Has Not Used Its Demonstration Authority Extensively
The DI demonstration projects that SSA has conducted since 1980 have not
extensively addressed a wide range of return-to-work policy issues. Since
first being granted DI demonstration authority 24 years ago, SSA has used
this authority to complete four projects, with another project nearing
17While grants and cooperative agreements are closely related to each
other, cooperative agreements generally allow for greater federal agency
involvement in the funded activities. Contracts generally provide federal
agencies with more control over funded activities than either grants or
cooperative agreements.
completion.18 Total costs for these projects amount to at least $107
million, of which about $42 million was paid for from the Old-Age and
Survivors Insurance and Disability Insurance (OASDI) Trust Funds.19
The legislation granting DI demonstration authority to SSA provided the
agency with an opportunity to examine a broad set of return-to-work policy
alternatives and even identified some specific alternatives for SSA to
consider studying, including (1) reducing, rather than terminating,
benefits based on earnings; (2) lengthening the trial work period; (3)
decreasing the 24-month waiting period for Medicare benefits; (4) altering
program administration; (5) earlier referral of beneficiaries for
rehabilitation; and (6) using employers and others to stimulate new forms
of vocational rehabilitation. The projects SSA has conducted thus far have
focused predominantly on the latter category of issues involving
vocational rehabilitation and have focused to a lesser extent-or not at
all-on other key policy issues affecting return to work (see table 1).
More specifically, examination of policy alternatives dealing with the
provision of vocational rehabilitation and employment services has been
the primary objective of four of the five completed or nearly completed
demonstrations. Although two of these projects also examined other DI
return-to-work policy issues-such as the possible effects of changes in
program work incentives and alterations in the provision of medical
benefits-they did so to only a limited extent. None of the projects looked
at other potentially significant DI policy issues, such as the possibility
of changing SSA's benefit structure to allow for a reduction in benefits,
rather than a complete cutoff of benefits, based on earnings.
18SSA also initiated three other projects under its DI demonstration
authority, which were subsequently terminated. In addition, SSA has
sometimes conducted demonstration projects involving DI beneficiaries
under other research authorities (rather than under its DI demonstration
authority). Although our discussion in this report focuses only on those
projects that were conducted under SSA's DI demonstration authority, our
review of these other demonstration projects suggests that our findings
would apply similarly to them.
19We were not able to obtain complete cost figures for these projects
because of the limited information maintained by SSA. Because most of
these projects involved both DI and SSI beneficiaries, they were funded
through the use of OASDI Trust Funds (under SSA's DI demonstration
authority) and general appropriations (as provided under SSA's SSI
demonstration authority).
Table 1: Completed or Nearly Completed Projects Conducted under SSA's DI
Demonstration Authority
Project Policy issues studied Year initiated Year completed Cost
Research Demonstration Mostly focused on provision of 1987
Program (RDP) vocational rehabilitation and employment services but also
touched on other policy issues such as changes in SSA work incentives
Pain Assessment Focused on the testing of pain 1990
Instruments assessment questionnaires for
Development Project determining the impact of pain
Most projects under the $32 million, about half of RDP appear to have
which was funded through been completed by the the Trust Funds, with the
early 1990s. other half coming from
appropriations
1994 A $3.8 million contract was awarded by SSA on this project, but it
is unclear if there were any additional project costs. All of the costs of
this project appear to have been funded through the Trust Funds.
on an applicant's or beneficiary's ability to return to work
Project Focused on provision of 1991 1999 $25.4 million, about
Network $13.7
vocational rehabilitation million of it funded
and through
employment services the Trust Funds, with
the
remainder coming from
appropriations
Project Referral Focused on 1997 No No information
System provision of identifiable available
for Vocational vocational completion
rehabilitation date; project
Rehabilitation was eventually
Providers
(Project RSVP) superseded by
the
Ticket to Work
program
Mostly focused on 1998 Expected Projected to cost
State Partnership provision of completion in $46.1
Initiative (SPI) vocational 2006 million, about
rehabilitation and $8.7 million of
employment it funded through
services but also the Trust
touched on other Funds and $37.4
policy issues million
such as increased coming from
availability of appropriations
health insurance
Source: GAO analysis of SSA data.
Note: Most of these projects involved both DI and SSI beneficiaries and
therefore were jointly conducted and funded under SSA's DI demonstration
authority and its SSI demonstration authority.
Furthermore, SSA has not used its DI demonstration authority to
comprehensively examine issues involving vocational rehabilitation,
including key policy issues with which the agency is currently grappling.
For example, SSA did not extensively test key elements of what eventually
became the Ticket to Work program.20 Although the ticket program was
20Under this program, beneficiaries are issued a "ticket," or voucher,
which they can use to obtain vocational rehabilitation, employment, or
other return-to-work services from an approved public or private provider
of their choice.
not formally proposed by SSA in a legislative package until 1997, as early
as 1989, in an annual report to the Congress on SSA's demonstration
activities, SSA noted that among its ideas for improving SSA's ability to
assist beneficiaries in returning to work was a voucher program that could
be used to pay for vocational rehabilitation services from private
providers. SSA told the Congress that such a program, as well as other
possible policy changes, would need to be thoroughly tested as a
prerequisite to developing a new nationwide program. However, only one
project completed under SSA's DI demonstration authority-Project Referral
System for Vocational Rehabilitation Providers (Project RSVP), initiated
in 1997-addressed an issue directly relevant to the ticket program,
namely, the use of a contractor to perform certain administrative
functions for an expanded vocational rehabilitation referral and
reimbursement program. But our review of project documentation and our
discussions with SSA officials indicate that Project RSVP was more of an
effort to make an operational change in the way SSA managed its vocational
rehabilitation program than a study to evaluate the advantages and
disadvantages of such a change. In fact, we could not identify any end
product or final results for this project.
SSA also made another attempt, ultimately unsuccessful, to directly
address issues related to establishment of a ticket program. In the
Omnibus Budget Reconciliation Act of 1990, the Congress mandated that SSA
use its DI demonstration authority to assess the advantages and
disadvantages of permitting DI beneficiaries to select from among both
public and private vocational rehabilitation providers. But in January
1993, SSA reported to the Congress that it would be unable to conduct this
demonstration because of an insufficient number of providers willing to
participate in the project. SSA explained that the performance-based
reimbursement provisions of the proposed project appeared to be the reason
why providers were reluctant to participate.21 Despite the Congress'
expressed interest in these issues, SSA did not attempt to identify
alternative ways to carry out such a demonstration. In particular, given
that SSA remained very interested in the expanded use of private
rehabilitation providers for the DI program, the difficulties encountered
in recruiting providers for the demonstration should have highlighted the
need for SSA to further study the issue of provider reimbursement before
21These provisions permitted reimbursement only for rehabilitation
services which result in a beneficiary's performance of substantial
gainful activity for a continuous period of 9 months. The provisions were
identical to those in place for reimbursing state vocational
rehabilitation agencies.
proceeding with any policy initiatives in this area. SSA's current Deputy
Commissioner for Disability and Income Security Programs told us that if
SSA had used its demonstration authority to study these types of issues in
the 1990s, SSA might have been able to identify and possibly resolve these
issues then rather than struggling to do so now. In addition, such
information could have been helpful in the Congress' consideration of the
ticket legislation's merits as it deliberated whether to enact this
program.
SSA's Recently Proposed Demonstrations Address a Broader Range of Policy
Issues
In contrast to the completed and nearly completed demonstration projects,
SSA's more recent projects, which are generally in the early planning or
proposal stages, represent a much more wide-ranging set of demonstrations
(see table 2). For example, the projects, as currently described, will
deal with a variety of issues such as early provision of cash and medical
benefits and a change in the benefit payment structure to allow a benefit
offset for beneficiaries earning above the SGA level. This more
comprehensive approach to demonstrations is due in part to legislative
changes. For example, the Ticket to Work Act mandated that SSA conduct a
benefit offset demonstration and also permitted SSA, for the first time,
to conduct demonstrations involving DI applicants, thereby allowing SSA to
test ideas such as early provision of cash and medical benefits and
vocational rehabilitation services to individuals who have not yet entered
the disability rolls. In addition, SSA has recently placed a high priority
on conducting disability demonstration projects that examine the key
issues affecting beneficiaries' return to work. This priority was
reflected in the SSA Commissioner's September 25, 2003, testimony before
the House Committee on Ways and Means, Subcommittee on Social Security, in
which she announced several new demonstrations as part of a broader
strategy to improve the DI and SSI programs. SSA estimates that these
recently proposed and initiated projects will cost about $357 million,
$293 million of which will be paid for from the OASDI Trust Funds.
Table 2: Recently Proposed or Initiated Projects Conducted under SSA's DI
Demonstration Authority
Actual/planned Expected project Project Policy issues studied year of
initiation duration Expected cost
DI Benefit Offset Focused on assessment of a benefit offset that would
allow beneficiaries to retain their eligibility for benefits when earning
above the SGA level while offsetting these benefits by $1 for every $2 in
earnings; it would also assess various employment support interventions in
conjunction with the offset
2004 6 years $106 million, all from the Trust Funds
Early Focused on Pilot is $34.4 million, of
Intervention provision of 2005 expected to which
cash
and medical last 4 years; $28.4 million is
benefits and duration expected to
employment of full be funded through
supports to demonstration the
DI
applicants (if conducted) Trust Funds
is
unknown
Interim
Medical Focused on $59.8 million, of
Benefits provision of Project in 6 years which
medical preliminary $54.9 million is
benefits to stage; expected to
DI
applicants be funded through
with no too early to the
medical
insurance determine
whose start Trust Funds
condition is
likely to (These costs apply
improve with date to both the
treatment Interim and
Ongoing
Medical Benefits
demonstrations)
Ongoing Medical Benefits Focused on provision of Project in Project in
preliminary See above medical benefits to DI preliminary stage; stage; too
early to beneficiaries who want to too early to determine duration work
but have no affordable determine start access to health insurance date,
although
SSA has proposed starting a pilot project in 2005
Mental Health Focused on provision $66.9
Treatment of Project in 6 years million, of
which
outpatient mental preliminary $59.7 million
Study health stage; is expected
to
treatment and too early to be funded
vocational through the
rehabilitation determine start Trust Funds
services to DI
beneficiaries for date
whom a
mental health
disorder is the
primary diagnosis
Temporary Focused on provision of Project in 5 years $32 million,
Allowance all from the
immediate cash and preliminary Trust Funds
medical stage;
benefits to DI too early to
applicants likely
to benefit from determine start
aggressive
medical care date
Actual/planned Expected project Project Policy issues studied year of
initiation duration Expected cost
a
Predictive Modeling Focused on analyses to develop screening tools that
will be used to identify candidates for several of SSA's other
demonstration projects
2004 2 years $9 million, of which $7.2 million is expected to be funded
through the Trust Funds
Youth Transition Process Focused on states' 2003 6 years $48.5 million, of which
$4.9
b
Demonstration
development of integrated service delivery systems to improve employment
outcomes for youth transitioning to adulthood million is expected to be
funded through the Trust Funds
Source: GAO analysis of SSA data.
Note: Some of these projects are expected to involve both DI and SSI
beneficiaries and therefore will be jointly conducted and funded under
SSA's DI demonstration authority and its SSI demonstration authority.
aThis is not a demonstration project designed to assess policy
alternatives. Instead, this project is intended to develop a specific
product for use in certain other SSA demonstrations.
bThis project is focused on the SSI program. However, SSI beneficiaries
who are dually eligible for DI benefits may also participate in this
demonstration.
SSA Lacks a Formal Process for Establishing a Demonstration Research Agenda
Despite SSA's recent broadening of the scope of its projects, the agency
does not have in place any systematic processes for identifying and
assessing potential issues that could be well suited for study under SSA's
demonstration project authority. Therefore, there is no assurance that the
agency will, in future demonstration efforts, maintain its current focus
on a broad array of return-to-work policy issues. Our discussions with SSA
officials and review of a study examining earlier demonstration efforts
indicate that the agency's agenda for demonstration projects is subject to
significant change over time resulting, in part, from changes in executive
branch and SSA leadership and senior management. The effects of such
changes may include termination of projects or significant delays and
modifications in their planning and implementation. For example, in its
1994 report examining SSA's Research Demonstration Program, the agency's
Inspector General noted that changes in SSA leadership had disrupted the
accomplishment of RDP objectives. The disability research and advisory
officials we spoke with also indicated that SSA's project priorities and
decisions are significantly influenced by larger political and
organizational changes, which may prevent SSA from focusing on longterm
research objectives. One advisory official noted that these difficulties
in long-term planning have occurred despite the fact that the Congress-in
making SSA an independent agency and establishing a 6-year term for the
SSA Commissioner-intended that SSA would be better able to engage in the
type of long-range planning required to address its program needs.
SSA's approach for identifying and prioritizing demonstrations has varied
through the years. Soon after being granted DI demonstration authority in
1980, SSA developed a detailed demonstration research plan to directly
address the policy issues identified in SSA's authorizing legislation.
However, our discussions with SSA officials and review of internal agency
documents indicate that the plan was never acted upon because of competing
organizational priorities and concerns over the potential cost of the
demonstrations and possible technical limitations, such as the adequacy of
systems support. Consequently, as its DI demonstration authority was due
to expire in 1985, SSA had not used it to conduct any demonstrations. In
the second half of the 1980s, after its demonstration authority was
renewed, SSA changed course. Partly on the basis of solicitation of ideas
from the public, SSA identified priority areas dealing mostly with
vocational rehabilitation and employment services issues for which it
would issue grants to public and private organizations to conduct
demonstrations. The specific priority areas identified changed from year
to year as SSA attempted to stimulate, test, and coordinate effective
approaches toward employment assistance.
In its required 1991 annual report to the Congress on its DI demonstration
activities, SSA said that it was proceeding with broader testing of key
elements of a comprehensive employment and rehabilitation system. But our
review of agency documents and discussions with SSA officials indicate
that SSA has not developed a formal, comprehensive, long-term agenda for
conducting DI demonstration projects. Senior SSA officials told us that
the agency's current demonstration project decisions are, to some extent,
based on discussions with outside research, advocacy, and other groups.
But SSA has no formal mechanisms and requirements in place to ensure that
the agency obtains such input and to decide how such input should be
factored in with other considerations in determining the agency's
demonstration priorities.
The need for explicit planning concerning SSA research, including
demonstrations, has been identified in past reviews of SSA's disability
programs. For example, in 1998, the Social Security Advisory Board (SSAB)
noted the need for SSA to develop a comprehensive, long-range research and
program evaluation plan for DI and SSI that would guide the
agency's research and define priorities.22 SSAB also said that SSA's
research plan should reflect broad consultation with the Congress, other
agencies, SSAB, and others and recommended the establishment of a
permanent research advisory panel to advise in the development of a
longrange plan. In a 1996 report on SSA's disability programs, the
National Academy of Social Insurance noted the "dearth of rigorous
research on the disability benefit programs" since the 1980s and said that
SSA needed a comprehensive, long-range research program to address this
deficiency.23 In addition, officials from disability research, advisory,
and advocacy groups told us that they believe the establishment of a
formal research agenda or an advisory panel with regard to demonstration
projects would be helpful in ensuring that SSA adequately identifies its
demonstration priorities and maintains its commitment to these priorities
even in the face of political or administrative changes.
SSA's demonstration projects have had little influence on the agency's and
the Congress' consideration of DI policy issues. This is due, in part, to
methodological limitations that have prevented SSA from producing project
results that are useful for reliably assessing DI policy alternatives. In
addition, SSA lacks a formal process for fully considering the potential
policy implications of its demonstration results. Furthermore, SSA's
reports on demonstration projects have not fully apprised the Congress of
project results and their policy implications.
SSA's Demonstration Projects Have Had Little Influence on Consideration of DI
Policy Changes
Limitations in Project The demonstration projects SSA has conducted under
its DI
Design, Implementation, demonstration authority have generally not been
designed, implemented,
and Evaluation Lessen or evaluated in a rigorous enough manner to allow
the agency to reliably
Usefulness of assess the advantages and disadvantages of specific policy
alternatives.
While SSA's major DI demonstrations have varied significantly in
theirDemonstration Results methodological rigor, all of them have
experienced at least some
22Social Security Advisory Board, Strengthening Social Security Research:
The Responsibilities of the Social Security Administration, p. 6-7
(Washington, D.C.: January 1998).
23National Academy of Social Insurance, The Environment of Disability
Income Policy: Programs, People, History, and Context, p. 108-109
(Washington, D.C.: 1996).
significant methodological limitations.24 For example, SSA's first major
DI demonstration, the Research Demonstration Program, was characterized by
a number of fundamental design and evaluation flaws such as the limited
scope and small sample sizes of the RDP projects and the limited use of
control groups. In its 1994 report on the RDP, the Department of Health
and Human Services' (HHS) Inspector General noted that because of such
limitations, "grantees were unable to conduct research that SSA deemed
necessary for definitive tests of alternatives to help beneficiaries
obtain work."25 In addition, SSA did not develop a plan for evaluating the
overall RDP results as part of its initial project design. In a required
1994 annual report to the Congress on its demonstration activities, SSA
acknowledged that the lack of a rigorous project design and the omission
of a strong evaluation component limited the ways in which the project
results could be generalized. But SSA also described a number of
"observations" that resulted from the RDP and noted that this project had
helped to identify the agency's future demonstration priorities. However,
given the significant limitations of the RDP, it is unlikely that its
results could have provided a reliable basis for effectively establishing
such priorities.
In its next major DI demonstration effort, Project Network, which was
initiated as the RDP projects were being completed, SSA avoided many of
the major shortcomings of the RDP. For example, Project Network was
rigorously designed, using an experimental approach based on the random
assignment of beneficiaries to treatment and control groups. As a result,
this project produced some reasonably clear results, which SSA thoroughly
evaluated in an effort to assess the overall impact of the tested policy
alternatives. Despite its generally rigorous design, Project Network
24Our discussion in this section is based on our review of SSA's three
major DI demonstrations conducted since 1980-the Research Demonstration
Program, Project Network, and the State Partnership Initiative. For the
two other projects SSA has conducted under its DI demonstration
authority-the Pain Assessment Instruments Development Project and Project
RSVP-SSA was unable to provide, and we were unable to otherwise identify,
any documentation or information other than relatively brief project
descriptions provided in SSA reports or notices. However, these project
descriptions as well as information obtained in discussions with SSA
officials who were knowledgeable of these projects indicate that these
projects were not research efforts designed to assess particular
return-to-work policy options but instead represented attempts by SSA to
implement operational changes in the DI program.
25Department of Health and Human Services, Office of Inspector General,
Impact of SSA's Disability Research Demonstration Program,
OEI-04-91-01660, pp. 5 (Washington, D.C.: September 1994).
also had some limitations that may have, to some extent, limited its
usefulness for policy consideration. For example, in examining the effects
of a case management approach for providing vocational rehabilitation
services, Project Network used four different service delivery models.26
Although the Project Network evaluation provided information on the
overall effects of a case management approach, it did not provide a basis
for reliably assessing and comparing the separate effects of the four
models even though such an assessment may have provided useful information
for policy considerations. In addition, Project Network did not produce
results that could be generalized to the larger population of
beneficiaries, which, in turn, limited SSA's ability to assess whether the
tested policy should be implemented on a nationwide basis.27
As was the case with Project Network, SSA has made a significant effort
under its State Partnership Initiative demonstration to avoid some of the
problems encountered under the RDP. For example, SSA contracted with two
research institutions to design an evaluation plan for the demonstration
and to provide assistance with technical issues and data collection to the
various states conducting this demonstration. Our discussions with SSA and
contractor officials who have been involved in this demonstration as well
as our own review of SPI project documents indicate that the efforts of
the contractors appear to have introduced a certain degree of rigor in the
design, implementation, and, potentially, evaluation of this
demonstration.28 For example, SSA's contractors have indicated that the
SPI "core evaluation" will likely produce useful results regarding the
effects on beneficiary employment of the overall package of policy
alternatives tested under the demonstration. But despite these efforts,
the SPI design also has a number of limitations that could substantially
reduce the usefulness of its results for evaluating the effects
26The researchers involved in Project Network described case management as
"a tool for facilitating employment-oriented interventions customized for
each individual, given that persons with disabilities face a range of
barriers to work." The four models for providing case management services
used in Project Network included one that was operated by a private
contractor and another operated by state vocational rehabilitation
agencies, as well as two other models that were operated by SSA but
differed in terms of the intensity of services offered.
27Because participation in the demonstration was voluntary, the project's
results could not be generalized to the broader population of
beneficiaries, most of whom did not volunteer. For example, the project's
volunteers were generally healthier and had fewer work limitations than
those who did not volunteer.
28The SPI demonstration is still in process and is expected to be
completed in 2006.
of the demonstration's individual policy alternatives. For example, SSA
gave each of the 12 participating states significant discretion in
designing and conducting projects, which resulted in 12 distinct state
projects. Each project tested different combinations of policy
alternatives, applied different research methods to study these
alternatives, and used varying approaches to select beneficiaries for
participation in the project. SSA officials told us that such differences
across projects make it unlikely that SPI will produce final results that
allow for reliable evaluations of specific policy alternatives on a
national level. SSA and one of its SPI contractors have also noted other
potential limitations in the design and implementation of SPI, such as
problems with the quality of states' data collection, that may detract
from SSA's ability to evaluate specific policy alternatives.29
SSA officials currently responsible for planning and conducting DI
demonstrations acknowledged that the agency's past demonstrations have
generally not provided useful information for policy making largely
because of the limited rigor with which these projects were conducted.
However, they emphasized that the agency has, over the past couple of
years, placed a new emphasis on ensuring that DI demonstrations are
rigorously designed so that the results can be used to effectively
evaluate specific policy options and develop recommendations. In
particular, the officials noted the importance of using, whenever
feasible, an experimental approach in its demonstration projects and of
ensuring that demonstration results can be generalized to the larger
population of DI beneficiaries.30 The officials also emphasized the need
for SSA to hire additional staff with the expertise needed to carry out
methodologically rigorous demonstration projects.
29Because the SPI demonstration is not yet complete, it is possible that
SSA and its contractors will take additional steps to address these
limitations. However, information obtained through our discussions with
SSA and SPI contractor officials provided little indication that such
problems were likely to be addressed.
30The officials noted that being able to generalize to the larger DI
population does not necessarily mean that a policy tested through a
demonstration has to be applicable to all or most DI beneficiaries, but
could instead be applicable to a subgroup of beneficiaries. For example,
an SSA demonstration might evaluate the costs and benefits of a policy
option that is targeted toward beneficiaries with mental impairments. In
this case, SSA would want to be assured that the project's results are
generalizable to the larger population of beneficiaries with mental
illnesses, but it would not expect generalizability to other DI
beneficiaries (i.e., those with other types of impairments).
Aside from the SPI demonstration, all of SSA's other current DI
demonstrations are in the early design phase or have been proposed only
recently. Therefore, we were not able to assess the methodological rigor
of these projects. However, our review of SSA's request for proposal (RFP)
for its Benefit Offset demonstration indicates that SSA is making a
serious effort to comprehensively and rigorously study this policy issue.
For example, SSA has proposed using an experimental design with random
assignment to treatment and control groups. Nevertheless, the scope and
complexity of SSA's proposal suggest that this will be a very challenging
project for SSA to carry out successfully, and that the agency will need
to ensure that its project design avoids some of the pitfalls that have
limited the usefulness of past demonstrations, such as insufficient sample
size and lack of uniformity in tested interventions across sites.
SSA Does Not Ensure That Project Results Are Adequately Considered and
Communicated
SSA does not have procedures or processes in place to ensure that project
results-regardless of any limitations that they may have-are fully
considered by senior officials within the agency for their policy
implications or their implications for future SSA research and
demonstrations. Without such processes, projects that begin with the
support of senior managers under one administration may not receive
adequate attention from a new group of senior managers under a future
administration. Our discussions with current and former SSA officials and
with officials from disability research, advocacy, and advisory
organizations indicate that such shifting priorities have been the norm
for SSA's DI demonstration projects. For example, several of these
officials told us that when Project Network was completed in 1999, its
results were not formally reviewed and considered by senior SSA managers,
in part because of the changes in presidential administrations and in
senior agency leadership that had occurred since the start of the project.
Officials from one of the groups we spoke with told us that SSA's
consideration of project results could be improved by the establishment of
a panel to review project results and explore their policy implications.
An additional factor that could limit SSA's consideration of demonstration
results is the lack of an adequate historical record-reflecting the
outcomes and the problems or issues encountered-of the various projects
that the agency has conducted under its demonstration
authority.31 SSA has not maintained a formal record of its disability
demonstration project activities and results, so basic information on
these projects-such as project notices, design documents, and evaluation
documents-is in some cases no longer available. As a result, information
on some projects can be obtained only by relying on the recollection of
SSA employees who were around when the study was conducted. While formal
document retention requirements may not dictate that SSA maintain such
information, several SSA officials told us that the agency would benefit
from an institutional record of demonstration activity. According to these
officials, such a record would constitute a body of knowledge that the
agency should be building to improve DI return-towork policies. This
becomes even more important in light of the expected retirement of a large
percentage of SSA staff during this decade.32
In addition to having shortcomings in its consideration of demonstration
results, SSA has not sufficiently communicated the status and results of
its demonstration projects to the Congress. Although SSA had been required
to issue various reports to the Congress regarding its DI demonstration
projects, it has not always produced such reports. For example, although
SSA was required to submit final reports on the use of its demonstration
authority in 1985, 1990, 1993, and 1996, the only final report that SSA
submitted was in 1996. In addition, SSA did not submit annual reports on
its demonstration activities in 7 of the 16 years in which these reports
were required. Furthermore, when these reports have been produced, they
have not provided all of the information needed to fully inform the
Congress of demonstration activities and results. For example, our review
of these reports indicates that they have frequently lacked key
information such as a discussion of a project's potential policy
implications, its limitations, and the costs of conducting the project.
Conclusions In allowing SSA to waive program provisions and use OASDI
Trust Fund dollars, SSA's DI demonstration authority provides the agency
with a special, and potentially very valuable, means of studying policy
31Although our focus in this report is on DI demonstrations, during the
course of our work, we had also requested information on SSI
demonstrations and were similarly unable to identify a historical record
at SSA.
32In its strategic plan for fiscal years 2003-2008, SSA said that during
this decade, over 28,000 of its federal employees will retire and another
10,000 will leave the agency for other reasons, which represents
approximately 59 percent of the agency's current workforce.
alternatives to improve the agency's return-to-work programs. SSA has
spent tens of millions of dollars from the OASDI Trust Funds to conduct
these projects-in addition to tens of millions of dollars from SSA's
general appropriations-and expects to spend hundreds of millions more
within the next 10 years. While these amounts may be small as a percentage
of the total Trust Funds, they nevertheless represent a substantial use of
increasingly limited federal resources. After having this authority for
more than two decades, SSA has yet to use it to propose or assess major
policy options that could result in savings to the Trust Funds. Because
SSA's use of its DI demonstration authority has yet to achieve the
Congress' intended results-and because SSA is permitted to draw on
increasingly limited Trust Funds to conduct these demonstrations-we
believe it is important for the Congress to maintain close oversight of
SSA's use of this authority. We also believe that such oversight would be
a greater challenge if the Congress were to grant this demonstration
authority on a permanent basis.
As the DI Trust Fund approaches exhaustion, the need for programmatic
improvements becomes greater and greater. As part of a broader effort to
address this need, SSA has recently initiated or proposed a number of DI
demonstration projects that, according to SSA officials, are geared toward
producing useful and methodologically sound results. Such results could
provide an important basis for SSA to address some of the long-standing
issues that have led GAO to identify federal disability programs as a
highrisk area. However, the challenges SSA has historically faced in
conducting demonstration projects and the potential for changing
priorities to adversely affect long-range research plans suggests that, in
the long run, SSA may be unable to fulfill these demonstration goals. This
is especially likely if SSA continues its informal approach to
prioritizing and planning demonstrations and assessing their results.
Without more formal mechanisms for establishing its commitment to
effective and thorough DI demonstrations-including the submission of
regular reports to the Congress on the results and implications of its
demonstration projects- SSA will be unable to ensure that the extensive
amount of time, effort, and funding devoted to these demonstrations is
well spent.
Recommendations To help ensure the effectiveness of SSA's DI
demonstration projects, we recommend that the Commissioner of Social
Security take the following actions:
o Develop a formal agenda reflecting the agency's long-term plans and
priorities for conducting DI demonstration projects. In establishing this
agenda, SSA should consult broadly with key internal and external
stakeholders, including SSA advisory groups, disability researchers, and
the Congress.
o Establish an expert panel to review and provide regular input on the
design and implementation of demonstration projects from the early stages
of a project through its final evaluation. Such a panel should include
SSA's key research personnel as well as outside disability experts and
researchers. SSA should establish guidelines to ensure that its project
plans and activities adequately address the issues or concerns raised by
the panel or provide a clear rationale for not addressing such issues.
o Establish formal processes to ensure that, at the conclusion of each
demonstration project, SSA fully considers and assesses the policy
implications of its demonstration results and clearly communicates SSA's
assessment to the Congress. Such processes should ensure that SSA consults
sufficiently with internal and external experts in its review of
demonstration project results and that SSA issues a report to the Congress
clearly identifying (1) major project outcomes, (2) major project
limitations, (3) total project costs, (4) any policy options or
recommendations, (5) expected costs and benefits of proposed options or
recommendations, and (6) any further research or other actions needed to
clarify or support the project's results. Another key aspect of such
formal processes should be a requirement that SSA maintain a comprehensive
record of DI demonstration projects. This record would help SSA in
establishing an empirically based body of knowledge regarding possible
return-to-work strategies and in deriving the full value of its
substantial investments in demonstration projects.
To facilitate close congressional oversight and provide greater assurance
that SSA will make effective use of its DI demonstration authority, the
Congress should consider the following actions:
o Continue to provide DI demonstration authority to SSA on a temporary
basis but allow SSA to complete all projects that have been initiated
prior to expiration of this authority. This would provide SSA with greater
certainty and stability in its efforts to plan and conduct demonstration
projects while preserving the Congress' ability to periodically reassess
and reconsider SSA's overall use of DI demonstration authority.
o Require that SSA periodically provide a comprehensive report to the
Congress summarizing the results and policy implications of all of its
Matters for Congressional Consideration
DI demonstration projects. The due date for this report could either
coincide with the expiration of SSA's DI demonstration authority or, if
this authority is made permanent or extended for a period greater than 5
years, be set for every 5 years. Such reports could serve as a basis for
the Congress' assessment of SSA's use of its demonstration authority and
its consideration of whether this authority should be renewed.
o Establish reporting requirements that more clearly specify what SSA is
expected to communicate to the Congress in its annual reports on DI
demonstrations. Among such requirements could be a description of all SSA
projects that the SSA Commissioner is considering conducting or is
conducting some preliminary work on. For each demonstration project that
the agency is planning or conducting, SSA should provide clear information
on the projects' specific objectives, potential costs, key milestone dates
(e.g., actual or expected dates for RFP, award of contracts or grants,
start of project operations, completion of operations, completion of
analysis, and final report), potential obstacles to project completion,
and the types of policy alternatives that SSA might consider pursuing
depending on the results of the demonstration. This would provide the
Congress with a more complete understanding of the direction and progress
of SSA in its efforts to fulfill its DI demonstration requirements.
o More clearly specify the methodological and evaluation requirements
for DI demonstrations to better ensure that such projects are designed in
the most rigorous manner possible and that their results are useful for
answering specific policy questions and for making, where appropriate,
well-supported policy recommendations. Such requirements should not be
entirely prescriptive given the need for SSA to have sufficient
flexibility for choosing the right methodological approach based on the
specific circumstances and objectives of a particular demonstration
project. However, the requirements could call for SSA to choose, to the
extent practical and feasible, the most rigorous methods possible in
conducting these demonstrations. Whatever methods are ultimately selected,
SSA should be sure that the methods used will allow for a reliable
assessment of the potential effect on the DI program of the individual
policy alternatives being studied. Finally, SSA's legislative requirements
could be revised to include a more explicit list of project
objectives-such as assessments of specific employment outcomes, costs and
benefits, and Trust Fund savings-similar to the language that was included
under Sections 302(b)(1) and (b)(2) of the Ticket to Work and Work
Incentives Improvement Act.
Agency Comments
and Our Evaluation
In commenting on a draft of this report, SSA agreed with our
recommendations. SSA agreed that in the past it has not used its
demonstration authority to extensively evaluate DI policy but noted that
its recently initiated or proposed demonstrations will play a vital role
in testing program and policy changes. SSA also agreed that the use of
experts in developing demonstration projects is very useful and commented
that it has used the expertise of particular individuals on an ad hoc
basis and plans to continue to use the advice and recommendations of
experts in the development of future demonstrations. Finally, SSA agreed
that a central source of information regarding the results and policy
implications of disability demonstrations needs to be established and
stated that it planned to fully analyze the results of demonstration
projects to inform DI policy decisions. SSA's comments appear in appendix
II.
Copies of this report are being sent to the Commissioner of SSA,
appropriate congressional committees, and other interested parties. The
report is also available at no charge on GAO's Web site at
http://www.gao.gov. If you have any questions about this report, please
contact me at (202) 512-7215. Other contacts and staff acknowledgments are
listed in appendix III.
Robert E. Robertson Director, Education, Workforce, and Income Security
Issues
Appendix I: Scope and Methodology
To address the mandated objectives, we reviewed legislation authorizing
the Social Security Administration (SSA) to conduct Disability Insurance
(DI) demonstration projects, Congressional reports related to this
legislation, and SSA regulations governing DI demonstration activities. We
also examined internal SSA memorandums and planning documents discussing
proposals to conduct demonstration projects and the nature, purpose,
requirements, and distinguishing features of SSA's demonstration
authority. We interviewed a wide range of current and former SSA officials
who have had involvement in or responsibility for conducting disability
program demonstration projects, including officials from the Office of
Disability and Income Security Programs (ODISP) and two offices operating
under ODISP-the Office of Program Development and Research and the Office
of Employment Support Programs-as well as officials from the Office of the
Chief Actuary, the Office of Acquisition and Grants, the Office of Budget,
the Office of Strategic Management, and the Office of Research,
Evaluation, and Statistics. We also interviewed officials from disability
research, advisory, and advocacy organizations. In addition, we examined
other reviews of SSA's disability demonstration and research programs,
including prior GAO and Inspector General reports and reports from
disability research and advisory groups. We also reviewed SSA budget
documents identifying agency spending on disability program demonstrations
and SSA testimony describing agency priorities related to the DI program
in general and demonstration projects in particular. In addition, we
examined SSA's strategic plan, annual performance plans, and annual
accountability reports.
To obtain detailed information on SSA's DI demonstration projects, we
reviewed various documents related to SSA's design, implementation, and
evaluation of demonstration projects including agency reports to the
Congress; public notifications of demonstration projects issued in the
Federal Register; contract, grant, and cooperative agreement solicitation
and award notices issued in the Federal Register or in the Commerce
Business Daily; and project reports submitted to SSA by grantees or
contractors, including project design and evaluation documents. We used
information from these sources to identify key characteristics and
outcomes of each project, including its broad goals, specific study
objectives, types of program waivers applied, methodology, actual or
expected costs, funding sources, major project milestones including actual
or expected initiation and completion dates, project duration, involvement
of outside contractors and grantees, key project strengths and
limitations, and final project results, including any recommendations that
may have been made. The type and extent of information we obtained for
each demonstration project varied widely, in large part because SSA has
not
Appendix I: Scope and Methodology
maintained comprehensive documentation on its prior demonstrations. In
addition, documentation on SSA's more recent demonstrations was very
limited given that these projects are in the early planning and design
stages.
To provide a broader context for understanding SSA's use of its
demonstration authority, we reviewed other federal agencies' legislative
authorities for conducting demonstration and research activities. We also
examined reports from GAO and other organizations that evaluated
demonstration and research projects conducted by other federal agencies or
that identified key evaluation and methodological issues related to such
projects.
We performed our work at SSA headquarters in Baltimore, Maryland, and at
various locations in Washington, D.C. We conducted our work between
October 2003 and August 2004 in accordance with generally accepted
government auditing standards.
Appendix II: Comments from the Social Security Administration
Appendix II: Comments from the Social Security Administration
Appendix II: Comments from the Social Security Administration
Appendix II: Comments from the Social Security Administration
Appendix III: GAO Contacts and Staff Acknowledgments
GAO Contacts
Staff Acknowledgments
(130323)
Shelia D. Drake, Assistant Director, (202) 512-7172 Mark Trapani,
Analyst-in-Charge, (202) 512-6513
The following individuals also made important contributions to this
report: Jacquelyn D. Stewart, Erin M. Godtland, Corinna A. Nicolaou,
Daniel A. Schwimer, Ronald La Due Lake, Michele C. Fejfar.
GAO's Mission The Government Accountability Office, the audit, evaluation
and investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of The fastest and easiest way to obtain copies of GAO
documents at no cost
is through GAO's Web site (www.gao.gov). Each weekday, GAO postsGAO
Reports and newly released reports, testimony, and correspondence on its
Web site. To Testimony have GAO e-mail you a list of newly posted products
every afternoon, go to
www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
To Report Fraud, Contact:
Waste, and Abuse in Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.govFederal Programs Automated answering system: (800)
424-5454 or (202) 512-7470
Congressional Gloria Jarmon, Managing Director, JarmonG@gao.gov (202)
512-4400 U.S. Government Accountability Office, 441 G Street NW, Room 7125
Relations Washington, D.C. 20548
Public Affairs Susan Becker, Acting Manager, BeckerS@gao.gov (202)
512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
*** End of document. ***