Social Security Administration: More Effort Needed to Assess
Consistency of Disability Decisions (02-JUL-04, GAO-04-656).
Each year, about 2.5 million people file claims with the Social
Security Administration (SSA) for disability benefits. If the
claim is denied at the initial level, the claimant may appeal to
the hearings level. The hearings level has allowed more than half
of all appealed claims, an allowance rate that has raised
concerns about the consistency of decisions made at the two
levels. To help ensure consistency, SSA began a "process
unification" initiative in 1994 and recently announced a new
proposal to strengthen its disability programs. This report
examines (1) the status of SSA's process unification initiative,
(2) SSA's assessments of possible inconsistencies in decisions
between adjudication levels, and (3) whether SSA's new proposal
incorporates changes to improve consistency in decisions between
adjudication levels.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-04-656
ACCNO: A10714
TITLE: Social Security Administration: More Effort Needed to
Assess Consistency of Disability Decisions
DATE: 07/02/2004
SUBJECT: Claims adjudicators
Claims processing
Claims settlement
Decision making
Disability benefits
Disability insurance
Eligibility determinations
Federal social security programs
Quality control
Social security benefits
Internal controls
Appeals
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-04-656
United States General Accounting Office
GAO
Report to the Chairman, Subcommittee on Social Security, Committee on Ways and
Means, House of Representatives
July 2004
SOCIAL SECURITY
ADMINISTRATION
More Effort Needed to Assess Consistency of Disability Decisions
GAO-04-656
Highlights of GAO-04-656, a report to the Chairman, Subcommittee on Social
Security, Committee on Ways and Means, House of Representatives
Each year, about 2.5 million people file claims with the Social Security
Administration (SSA) for disability benefits. If the claim is denied at
the initial level, the claimant may appeal to the hearings level. The
hearings level has allowed more than half of all appealed claims, an
allowance rate that has raised concerns about the consistency of decisions
made at the two levels. To help ensure consistency, SSA began a "process
unification" initiative in 1994 and recently announced a new proposal to
strengthen its disability programs. This report examines (1) the status of
SSA's process unification initiative, (2) SSA's assessments of possible
inconsistencies in decisions between adjudication levels, and (3) whether
SSA's new proposal incorporates changes to improve consistency in
decisions between adjudication levels.
To build an effective strategy to address possible inconsistencies in its
decisions, we recommend that SSA quickly expand its assessment of
inconsistency by implementing several specific enhancements. In its
comments, SSA had some reservations concerning our findings, conclusions,
and recommendations but agreed to pilot one recommendation and consider
the others as it refines its new proposal. We continue to believe that SSA
should implement our recommendations without delay to ensure the
effectiveness of its new proposal.
July 2004
SOCIAL SECURITY ADMINISTRATION
More Effort Needed to Assess Consistency of Disability Decisions
SSA has only partially implemented its process unification initiative.
Although the agency initially made improvements in its policies and
training intended to address inconsistency in decisions made at the two
adjudication levels, it has not continued to actively pursue these
efforts. Further, as part of this initiative, the agency implemented a
review of hearings level decisions to identify ways to improve training
and policies, but no new improvements were made as a result of the review.
Finally, the agency began tests of two process changes intended to improve
the consistency of decision making between the two adjudication levels.
One test, which is ongoing, was not well designed and therefore will not
provide conclusive results. The other test was abandoned because of
implementation difficulties.
SSA's assessments have not provided a clear understanding of the extent
and causes of possible inconsistencies in decisions between adjudication
levels. The two measures SSA uses to monitor inconsistency of decisions
have weaknesses, such as not accounting for the many factors that can
affect decision outcomes, and therefore do not provide a true picture of
the changes in consistency. Furthermore, SSA has not sufficiently assessed
the causes of possible inconsistency. For example, SSA conducted an
analysis in 1994 that identified potential areas of inconsistency, but it
did not employ more sophisticated techniques-such as multivariate
analyses, followed by in-depth case studies-that would allow the agency to
identify and address the key areas and leading causes of possible
inconsistency. SSA has yet to repeat or expand upon this 10-year-old
study.
SSA's new proposal incorporates changes intended to improve consistency in
decisions between levels. However, challenges may hinder its
implementation. Most stakeholder groups for adjudicators and claimant
representatives told us that a number of aspects of the proposal hold
promise for improving consistency. These included one change, being tested
as part of the process unification initiative, that requires state
adjudicators to more fully develop and document their decisions, as well
as several new changes, such as providing both adjudication levels with
equal access to medical expertise. However, stakeholder groups also told
us that insufficient resources and other obstacles might hinder the
implementation of some changes. Adding to uncertainties about the
proposal's overall success is its dependence on a new electronic folder
system that would allow cases to be easily accessed by various
adjudicators across the country. However, this technically complex project
has not been fully tested.
www.gao.gov/cgi-bin/getrpt?GAO-04-656.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Robert Robertson,
202-512-7215, [email protected].
Contents
Letter
Results in Brief
Background
SSA Has Partially Implemented Its Process Unification Initiative
SSA Lacks a Clear Understanding of the Extent and Causes of
Inconsistency between Levels
SSA's New Proposal Incorporates Efforts to Improve the Consistency of
Decisions, but Challenges May Impede Successful Implementation
Conclusions
Recommendations
Agency Comments and Our Evaluation
1
3 5 14
20
26 32 33 34
Appendix I Excerpt of SSA's Testimony Announcing Its New Proposal to Improve
Its Disability Decision-Making Process
Appendix II Comments from the Social Security Administration 44
GAO Comments 50
Appendix III GAO Contacts and Staff Acknowledgments 53
GAO Contacts 53
Staff Acknowledgments 53
Tables
Table 1: SSA's Process Unification Efforts 9
Table 2: Newly Proposed Changes to the Disability Decision-
Making Process 13
Figures
Figure 1: SSA's Disability Decision-Making Process and Outcomes for Fiscal
Year 2003 7 Figure 2: SSA Allowances at Initial Level versus Hearings
Level, by Proportion of Allowances 21
Abbreviations
AeDib Accelerated Electronic Disability System
ALJ Administrative Law Judge
DDS Disability Determination Services
DI Disability Insurance
OASDI Old-Age, Survivors and Disability Insurance
OHA Office of Hearings and Appeals
OQA Office of Quality Assurance and Performance Assessment
RO Reviewing Official
SSA Social Security Administration
SSI Supplemental Security Income
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
United States General Accounting Office Washington, DC 20548
July 2, 2004
The Honorable E. Clay Shaw, Jr.
Chairman, Subcommittee on Social Security
Committee on Ways and Means
House of Representatives
Dear Mr. Chairman:
The Social Security Administration (SSA) is the nation's largest provider
of income assistance to individuals with disabilities, paying $91 billion
in
federal benefits to 11.4 million beneficiaries with a disability and their
families in 2003.1 Each year, about 2.5 million people file claims with
SSA
for disability benefits. State agencies called Disability Determination
Services (DDS) decide whether claimants meet SSA's definition of
disability by applying SSA's decision-making criteria. If a DDS ultimately
decides, after an initial determination and then a reconsideration of this
decision, that a claimant does not meet SSA's definition for disability,
the
claimant may appeal to the hearings level, where an SSA Administrative
Law Judge (ALJ) reviews the claim to decide if the claimant should be
allowed benefits. About one-third of disability claims denied at the state
level were appealed to the hearings level; of these, SSA's ALJs have
allowed over one-half, with annual allowance rates fluctuating between 58
percent and 72 percent since 1985. While it is appropriate that some
appealed claims, such as those in which a claimant's impairment has
worsened and prohibits work, be allowed benefits, representatives from
SSA, the Congress, and interest groups have long been concerned that the
high rate of claims allowed at the hearings level may indicate that
decision
makers at the two levels are interpreting and applying SSA's criteria
differently. If this is the case, adjudicators at the two levels may be
making
inconsistent decisions that result in similar cases receiving dissimilar
decisions.
Concerned about the possibility that adjudicators are making inconsistent
decisions, SSA embarked on a "process unification" initiative in 1994 with
1The figures include federal payments for the Disability Insurance and the
Supplemental Security Income programs to beneficiaries who have a
disability or are blind and their families.
the goal of ensuring that adjudicators at both levels consistently apply
SSA's policy guidance and make similar decisions on similar cases. Partly
on the basis of early studies of potential causes of inconsistent
decisions, SSA included in its process unification initiative efforts to
provide consistent guidance to all adjudicators, clarify policy, provide
training, test potential process changes, and perform a new quality review
of allowances decided by ALJs. However, SSA continues to face challenges
in its efforts to provide consistent disability decisions. These
challenges, and others associated with modernizing its disability
programs, contributed to our decision to include federal disability
programs on our list of high-risk government programs.2 In September 2003,
SSA's Commissioner unveiled a new proposal that laid out the vision of a
long-term strategy for improving the disability decision-making process
and helping people with disabilities return to work. Several of the
changes in the new proposal are intended to improve the accuracy,
timeliness, and consistency of decisions, such as having the DDS decision
makers more fully develop and document their decisions, providing for
centralized quality review of all decisions, and providing both
adjudication levels equal access to medical expertise.
In response to your interest in the effectiveness of SSA's past and future
efforts to improve and assess the consistency of decisions between levels,
we evaluated these agency efforts. Specifically, we examined (1) the
status of SSA's process unification initiative, (2) SSA's assessments of
possible inconsistencies in decisions between adjudication levels, and (3)
whether SSA's new proposal incorporates changes to improve consistency in
decisions between adjudication levels.
To assess the extent to which SSA has implemented its planned activities
under the initiative, we evaluated agency documentation describing SSA's
process unification efforts. To evaluate SSA's efforts to assess
consistency in decisions between levels, we interviewed officials from
SSA's Office of Quality Assurance and Performance Assessment (OQA) and
reviewed summary data and reports from SSA's quality assurance and
performance management systems, including findings from SSA's Disability
Hearings Quality Review, which is a quality review of DDS adjudicators'
and ALJs' decisions and the associated case files. While we evaluated
SSA's methods
2For additional information on the major problems facing federal
disability programs and our decision to designate them as high-risk, see
U.S. General Accounting Office, High-Risk Series: An Update, GAO-03-119,
(Washington, D.C.: January 2003).
Results in Brief
and approaches for assessing the consistency of decisions, we generally
did not trace figures cited by SSA back to their original source
documents.3 To further evaluate SSA's process unification efforts and its
new proposal, we interviewed selected officials who have firsthand
knowledge about these issues, including SSA officials, staff of the Social
Security Advisory Board, and leaders of SSA stakeholder groups
representing state and hearings office adjudicative staff and claimant
attorneys.4 We also reviewed recent testimony and other documents on the
consistency of SSA's decisions from the Social Security Advisory Board and
other stakeholder groups. We conducted our work between February 2003 and
March 2004 in accordance with generally accepted government auditing
standards. Our ability to evaluate SSA's new proposal has been limited by
a lack of detailed information, such as specific information on how
changes will be implemented and their costs, because the agency is still
in the process of developing and refining its proposal.
SSA has partially implemented its process unification initiative. At the
beginning of its process unification efforts, the agency took decisive
steps by issuing clarifying guidance in a number of key policy areas to
all adjudicators. However, SSA ultimately abandoned its plans to update
older policy guidance and to provide a single policy guide for both
initial- and hearings-level adjudicators, concluding that these efforts
would not be cost-effective. Similarly, while SSA initially provided
extensive process unification training to adjudicators at both levels, the
scope of SSA's training efforts on this issue has since diminished. Also
as part of this initiative, SSA implemented a quality review of ALJ
allowances, in part to identify the need for new policies and training to
improve consistency, but no new improvements were made as a result of
findings from the review. Finally, the agency began two tests to determine
if process changes would improve the consistency of decisions. However,
the test of having DDS
3We took additional steps to assess the reliability of data included in
our report. For the proportion of initial and hearings levels allowances,
we verified SSA's calculations of summary data from its workload reporting
systems. For the information from the Disability Hearings Quality Review,
we reviewed the weights and calculations used by SSA to determine national
support rates for reviewed decisions and found them to be correctly
calculated and reliable.
4These groups included the National Association of Disability Examiners,
the National Council of Disability Determination Directors, the
Association of Attorney Advisors, the Association of Administrative Law
Judges, the National Treasury Employee Union, the Public Employees
Federation, the Social Security Section of the Federal Bar Association,
and the National Organization of Social Security Claimants'
Representatives.
adjudicators more fully document decisions-which is ongoing-will, because
of design flaws, be unable to provide conclusive information on how this
change might affect the consistency of decisions. SSA abandoned its other
test of having the initial level reevaluate appealed cases for which new
medical information was submitted prior to the hearing, because of several
major difficulties encountered during testing, such as difficulty
identifying cases to be sent back to the initial level.
SSA's assessments have not provided the agency with a clear understanding
of the extent and causes of possible inconsistencies in decisions between
adjudication levels. The two measures SSA uses to monitor changes in the
extent of inconsistency of decisions have weaknesses, such as not
accounting for the many factors that can affect decision outcomes, and
therefore do not provide a true picture of changes in consistency. SSA has
also not sufficiently assessed causes of possible inconsistency. For
example, 10 years ago SSA analyzed cases in which reviewers representing
the initial and hearings level disagreed over the final decision and
identified two prevalent areas of disagreement: assessments of claimants'
mental impairments and assessments of claimants' ability to work. Although
SSA continues to collect information that would support this analysis, it
has not repeated this initial effort, nor has it expanded on it by
employing more sophisticated techniques-such as multivariate analyses,
followed by in-depth case studies-that would allow the agency to identify
and address the key areas and leading causes of possible inconsistency.
SSA's recent proposal to improve the disability decision-making process
incorporates efforts intended to address inconsistencies in decisions
between levels. However, challenges may hinder the implementation of the
proposal. Most stakeholder groups for SSA adjudicators and claimant
representatives told us that several aspects of the proposal hold promise
for improving consistency between adjudication levels. For example, they
thought that requiring state adjudicators to more fully develop and
document their decisions-a process change that SSA is still testing under
its process unification initiative-might improve consistency. In addition,
they said that other new ideas proposed by SSA, such as centralizing
medical expertise to give both adjudication levels equal access to it, may
also improve consistency. However, stakeholders told us that insufficient
resources and other obstacles might hinder the implementation of some
changes. For example, they were concerned that, as has happened in the
past, limited resources would hinder state adjudicators' ability to more
fully document their decisions. Stakeholders also questioned SSA's ability
to attract and retain sufficient medical expertise. Adding to
uncertainties
Background
about the proposal is its dependence upon the successful implementation of
a new electronic folder system that would allow cases to be easily
accessed by adjudicators across the country. However, this technically
complex project has not been fully tested.
GAO is making several recommendations in this report to the Commissioner
of Social Security to improve SSA's methods for assessing the
inconsistency between DDS and ALJ decisions and thereby build a more
effective strategy to address potential inconsistencies. In commenting on
the draft of this report, SSA had several reservations concerning the
report's findings, conclusions, and recommendations. Although SSA agreed
to pilot one of our recommendations, the agency believes our other
recommendations need to be reevaluated as the design of its Commissioner's
new approach to disability decision making matures. We continue to believe
that SSA should begin implementing our recommendations without delay so
that it has the critical information needed to build a new approach to
decision making that will improve the consistency of decisions between
adjudication levels.
SSA operates the Disability Insurance (DI) and Supplemental Security
Income (SSI) programs-the two largest programs providing cash benefits to
people with disabilities. The law defines disability for both programs as
the inability to engage in any substantial gainful activity by reason of a
severe physical or mental impairment that is medically determinable and is
expected to last at least 12 months or result in death. The programs have
grown substantially, from 10.7 million beneficiaries and $61 billion in
benefits in 1995 to 11.4 million beneficiaries and $91 billion in federal
benefits to individuals with disabilities in 2003.5 While disability
benefits account for only 15 percent of SSA's total benefit payments for
its Old-Age, Survivors and Disability Insurance (OASDI) programs,
administering the disability benefits accounted for 45 percent of the
agency's annual administrative expenses.6 The relatively high cost of
administering the DI program reflects the complex and demanding nature of
making disability decisions. SSA estimates that the cost of the disability
programs will rise
5The figures include federal payments for the Disability Insurance and the
Supplemental Security Income programs to beneficiaries who have a
disability or are blind and their families. The figures were calculated
based on statistical information from SSA's web site.
6These figures are based on information provided on SSA's web site from
the 2004 OASDI Trustees Report, Part III. Financial Operations of the
Trusts Funds and Legislative Changes in the Last Year.
substantially in the near future as the baby boom generation reaches its
disability-prone years.
The disability determination process begins at a field office, where an
SSA representative determines whether a claimant meets the programs'
nonmedical eligibility criteria. Claims meeting these criteria are
forwarded to the state DDS to determine if a claimant meets the agency's
definition of disability. At the DDS, the disability examiner takes the
lead, or works as a team with the medical or psychological consultants, to
analyze a claimant's documentation, gather additional evidence as
appropriate, and approve or deny the claim. A denied claimant may ask the
DDS to reconsider its finding, at which point a different DDS team reviews
the claim. If the claim is denied again, the claimant may appeal the
determination to SSA's Office of Hearings and Appeals (OHA), where it will
be reviewed by an ALJ. The ALJ usually conducts a hearing in which the
claimant and others may testify and present new evidence.7 In making the
disability decision, the ALJ uses information from the hearing and from
the state DDS, including the findings of the DDS medical consultant. A
claimant whose appeal is denied may request a review by SSA's Appeals
Council and, if denied again, may file suit in federal court. Figure 1
provides an overview of SSA's disability decision-making process and
outcomes for 2003.
7Others who may testify at ALJ hearings frequently include vocational and
medical experts.
Figure 1: SSA's Disability Decision-Making Process and Outcomes for Fiscal
Year 2003
Source: SSA data.
Notes:
The data provided by SSA did not include the number or rate of claims
appealed to the next decision step. Without this information, we cannot
determine the extent to which claimants appealed or abandoned their denied
claims.
Twenty-five percent of the initial DDS determinations are subject to an
alternative process that does not include the reconsideration step.
Under certain specified circumstances, ALJs and Appeals Council judges can
dismiss a claim. For example, an ALJ can dismiss a claim if the claimant's
request for a hearing is not timely and lacks a good cause for the delay.
The Appeals Council can remand a claim by returning it to an ALJ for
further proceedings and a new hearing decision.
Because of rounding, decisional outcomes may not equal 100 percent.
SSA uses a sequential evaluation process when determining disability.
First, SSA field office representatives determine whether a claimant is
performing substantial gainful work.8 If not, DDS or ALJ adjudicators will
assess the severity of a claimant's medical condition(s) to determine
whether it meets or equals the criteria in SSA's regulations (commonly
referred to as the medical listings). For a claimant whose conditions do
not meet or equal the listings, adjudicators then focus on the functional
consequences of the claimant's medically determined impairments-that is,
whether the claimant can perform work he or she has done in the past, and,
if not, whether the claimant can perform other work in the national
economy.
Concerns about the rate of appeals for hearings, ALJs' allowance rates,
and the accuracy and consistency of ALJ decisions led the Congress to
direct SSA to conduct a study in 1980 to determine the extent to which
hearings decisions conformed to legal requirements and binding SSA
policy.9 Since the allowance rates at the hearings level could be
influenced by many factors, such as the introduction of new evidence, the
purpose of the 1980 study was to present the same evidence on cases to
different reviewers representing different adjudication levels. In
determining the extent to which decision makers agreed on whether to allow
or deny benefits, the study concluded that different levels of decision
makers had significantly different allowance rates. Specifically, the ALJs
decided to allow 64 percent of the cases, whereas the SSA's central office
quality assurance reviewers, comprising medical consultants and disability
examiners, decided that only 13 percent of cases should be allowed. The
study identified several possible causes of the disparity, including
inconsistency in the standards and procedures, interpretation of the
standards, and weight given to the evidence. The study also found that
disability decisions are complex and necessarily involve some degree of
subjectivity by adjudicators.
8Substantial gainful work is a level of work activity that involves doing
significant physical or mental work, or a combination of both, that is
productive. SSA has established earning criteria as a reasonable
indication of whether claimants are able to engage in substantial gainful
activity. In 2004, SSA generally considered claimants to be engaging in
substantial gainful activity if their earnings averaged over $810 a month.
9See Implementation of Section 304 (g) Public Law 96-265, Social Security
Disability Amendments of 1980 (the Bellmon Report), Secretary of Health
and Human Services (Washington, D.C.: January 1982).
To help address concerns raised by this and other studies,10 SSA began its
process unification efforts to ensure that both levels more consistently
interpreted and applied SSA's policy guidance. SSA's plans for its process
unification initiative were part of SSA's larger effort to redesign its
disability claims process and were modified over time. SSA's process
unification plans included six major efforts, as described in table 1.
Table 1: SSA's Process Unification Efforts
Process unification effort Planned activities
Develop a single presentation Publish all new guidance in the same wording
to all adjudicators. of policy Revise older guidance.
Provide one policy manual for adjudicators.
Create additional policy Publish nine process unification rulings to
clarify policy areas contributing to inconsistent DDS and guidance ALJ
decisions that are binding on all SSA adjudicators.
Publish regulations and instructions to clarify selected process
unification rulings, including (1) the weight to be given to the treating
physician's opinion when evaluating a claim, (2) the weight to be given to
the DDS medical consultant's opinion, and (3) the evaluation of the
residual functional capacity for claimants who are limited to performing
less than a full range of sedentary work.
Publish a regulation to clarify the agency's process for acquiescing to
court decisions and an action plan to implement the regulation.
Provide training Provide ongoing training to all adjudicators to increase
their understanding of the three most complex disability areas-assessing
(1) the opinion evidence from physicians and others, (2) the claimant's
symptoms, and (3) the claimant's remaining capacity to work (i.e.,
residual functional capacity).
Fully document DDS decisions Test procedures for fully developing and
documenting DDS decisions to determine their impact on DDS accuracy,
allowance rates, and other aspects of the claims process.
Remand cases awaiting a hearing to a DDS when new medical evidence is
received Test the impact of returning cases to a DDS that are awaiting a
hearing and have received new medical information to determine if a
reevaluation of the cases by medical consultants residing at a DDS will
help to improve the consistency of decisions between levels, with a 1-year
goal of remanding 100,000 cases.
Review selected ALJ
allowance decisions
Conduct a joint Appeals Council and OQA review of ALJ allowances that have
not yet received a final decision (i.e., the claimant has not been awarded
benefits) to identify policy areas leading to inconsistent decisions
between levels.
Source: SSA documents and prior GAO reports.
10Findings from SSA's 1994 Disability Hearings Quality Review Process
report provided the agency with additional information on potential causes
of inconsistency. The report identified two assessment areas associated
with inconsistent decisions. In addition, quality reviewers found that
when applying standards used by the initial level to adjudicate claims, 29
percent of the appealed DDS reconsideration decisions and 51 percent of
ALJ decisions were not supported by the decision makers. These findings
help to support SSA's decision to include efforts to have DDSs more fully
develop and document their decisions and to assess ALJ decisions as part
of its process unification initiative.
In 1997, we reported on the possible reasons for the inconsistency of
decisions between the initial and hearings levels. Our report found that
differences in state DDSs' and ALJs' views on the claimants' functional
abilities was a key factor in explaining why ALJs allowed benefits on
appealed cases.11 We also reported that poorly documented state DDS
evaluations of the claims were of limited use to ALJs and SSA quality
reviews did not focus on identifying inconsistency in decisions. To
support SSA's process unification efforts, the report recommended that
SSA, using available systems and data collected so far, move quickly ahead
to implement its quality assurance initiative to provide consistent
feedback to DDS and ALJ adjudicators as soon as possible. In addition, we
recommended that SSA expand its effort to return cases to a DDS for review
when new evidence is introduced on appeal. Last, we recommended that SSA
set goals for measuring the effectiveness of process unification in
reducing inconsistent decisions.
More recently, the Social Security Advisory Board issued a 2001 report
that identified many factors that could potentially affect the overall
consistency of disability decision making between adjudication levels.12
Some of the factors the board suggested as potentially affecting
consistency included:
o the fact that most claims are decided based on a paper review of case
evidence without face-to-face contact with an adjudicator until a claimant
has an ALJ hearing,
o involvement of attorneys and other claimant representatives at the ALJ
hearing,
o the fact that claimants are allowed to introduce new evidence and
allegations at each stage of the appeals process,
o differences in quality assurance procedures applied to initial-and
hearings-level decisions,
o differences in the training given to ALJs and state examiners, and
o lack of clear and unified policy guidance from SSA.
11U.S. General Accounting Office, Social Security Disability: SSA Must
Hold Itself Accountable for Continued Improvement in Decision-Making,
GAO/HEHS-97-102 (Washington, D.C.: Aug. 12, 1997).
12The board also identified factors that could affect consistency within
adjudication levels, as opposed to strictly between adjudication levels.
For more information see the Social Security Advisory Board, Disability
Decision Making: Data and Materials, January 2001, pp. 5-6.
Despite SSA's process unification efforts and related studies to improve
the consistency of decisions, recent ALJ allowance rates-which declined
after process unification began, but started increasing in 1999 to reach
61 percent in fiscal year 2003-still raise questions as to whether
initialand hearings-level decision makers are consistently applying the
agency's guidance. In addition to inconsistent application of SSA's policy
guidance, there are several other reasons why a large number of ALJ
allowances are made. For example, some ALJ allowances should be expected
because, by law, cases can remain open throughout the hearings process,
allowing new evidence to be submitted that may not have been available to
the state adjudicators. Such new evidence could show that the claimant's
condition has worsened and prohibits work. Also, SSA's decision-making
criteria require that a great deal of professional judgment be applied. As
a result, some allowances at the hearings level could simply reflect the
differing judgments of two adjudicators reviewing a case. While a
claimant's deteriorating health, changes in the characteristics of a claim
over time, and the complexity of disability decisions may help to explain
some of the ALJ allowances, studies have not sufficiently explained why
consistently over half the cases appealed to the hearings level are
allowed. Instead, studies indicate that systemic differences in the
assessment of claims at both adjudication levels are contributing to the
ALJ allowance rate. For example, our 1997 report noted a difference in
state DDSs' and ALJs' views on the claimant's functional abilities was a
key factor in explaining why ALJs allowed cases on appeal.
Inconsistency in decisions may create several problems. High hearings
allowance rates may create the perception that the hearings level is
applying SSA's criteria less strictly than the initial level and create an
incentive for claimants to appeal to an ALJ for a more favorable
decision.13 If deserving claimants must appeal to the hearings level for
benefits, this situation increases the burden on claimants, who must wait,
on average, almost a year for a hearing decision and frequently incur
extra costs to pay
13An appeal adds significantly to costs associated with making a decision.
According to SSA's Performance and Accountability Report for fiscal year
2001, the average cost per claim for an initial DDS disability decision
was about $583, while the average cost per claim of an ALJ decision was an
estimated additional $2,157.
for legal representation.14 In addition, to the extent that the ALJ
allowance rates include inappropriate allowances, SSA could be incurring
unwarranted program costs. Although SSA has tried to address these
problems, its inability to resolve them has contributed to our decision to
include federal disability programs on our list of high-risk government
15
programs.
Renewing its effort to address long-standing and critical problems with
the disability programs, SSA's Commissioner recently announced a new
proposal to improve these programs. (See app. I for an excerpt of the
announcement that describes the newly proposed decision-making process.)
In addition to proposing demonstration projects that provide work
incentives and supports to help people with disabilities return to work,
SSA has proposed significant changes to both the process of adjudicating
disability claims and the structure and management of the agency's quality
management system to improve the timeliness, accuracy, and consistency of
the disability decision-making process. The agency believes that several
of these changes will help to improve consistency between DDS and ALJ
decisions. For example, SSA plans to provide more centralized end-of-line
quality reviews. According to SSA, the proposed quality reviews should
help to hold adjudicators more accountable for their decisions and ensure
that they consistently apply SSA's policies as well as help the agency
detect and amend those policy areas leading to inconsistent decisions.
Table 2 provides a description of SSA's proposed changes to improve the
disability decision-making process.
14An appeal also significantly increases the time required to reach a
decision. According to SSA's Performance and Accountability Report for
fiscal year 2003, the average number of days that claimants waited for an
initial decision was 97 days, while the number of days they waited for an
appealed decision was 344 days. However, the time a claimant waits for a
decision should not impact the amount of benefits received, as benefits
are based on the date the claimant becomes disabled.
15U.S. General Accounting Office, Performance and Accountability Series:
Major Management Challenges and Program Risks: Social Security
Administration,
GAO-03-117 (Washington, D.C.: January 2003).
Table 2: Newly Proposed Changes to the Disability Decision-Making Process
Proposed change Purpose
Centralize medical expertise from the states To make quick decisions on
initial claims for individuals who are obviously disabled into regional
offices and organize experts by (e.g., those with aggressive cancers and
end-stage renal disease) and to provide medical specialty equal access to
medical expertise for all adjudicators and more consistent medical
review of claims.
Require DDSs to fully document and explain To hold DDS adjudicators
accountable for providing higher-quality and more
the basis for their decisions consistent decisions and to have better
information to identify and correct problem areas leading to incorrect
decisions.
Eliminate DDS reconsiderations To reduce time taken to process claims and
avoid having claimants who are disabled drop out of the disability claims
process.
Create a reviewing official position To evaluate all appealed DDS
decisions and prepare either (1) an on-the-record allowance, (2) a
recommended disallowance detailing reasons for a denial, or (3) a
prehearing report outlining the evidence needed to fully support the
claim.
Require ALJs to address the reviewing officials' To improve accountability
and consistency by having ALJs either (1) describe in
reports detail the basis for rejecting the reviewing official's
recommended disallowance or (2) provide detailed information on the
evidence used to support allowances made in response to the reviewing
official's prehearing report.
Eliminate the claimant's ability to appeal ALJ To reduce processing time
for claims and use resources more effectively. decisions to the Appeals
Council
Use of in-line quality control To build quality into each level of the
decision-making process rather than rely too heavily on case reviews
performed by end-of-line quality reviewers or by the next adjudication
level in response to claimants appealing denied decisions.
Use of centralized quality control unit To perform end-of-line reviews
for all decisions, thereby replacing regional reviews of DDS decisions,
and to provide a more balanced review of both DDS and ALJ decisions to
ensure that all adjudicators are consistently applying SSA's policies and
to detect and amend those policy areas leading to inconsistent decisions.
Create oversight panels that include two ALJs To review and either affirm
or reverse an ALJ decision referred by the centralized and one
Administrative Appeals Judge from the quality control unit when the unit
disagrees with the ALJ decision. Appeals Council
Sources: SSA documents and agency interviews.
Note: Under the new proposal, when the agency's centralized quality
control unit and oversight panel review an ALJ decision, a claimant cannot
submit any new information to be considered by the agency.
SSA does not plan to implement its proposed changes before it has
successfully implemented its Accelerated Electronic Disability (AeDib)
system. This major initiative should allow adjudication staff in states
and throughout the agency, regardless of geographic location, to access
case information electronically through the use of an electronic
disability folder. The initiative is intended to reduce delays that result
from mailing, locating, and organizing paper folders. SSA also expects
this new system to provide critical management information for analyzing
and reducing inconsistencies in disability decisions. SSA is implementing
the new system and plans to give adjudicators time to adjust to this
change before
SSA Has Partially Implemented Its Process Unification Initiative
implementing its new proposal. SSA's implementation of the new proposal
will therefore be no earlier than October 2005. In the meantime, SSA
continues to discuss the proposal with stakeholders and plans to further
refine it before implementation.
SSA has partially implemented its process unification initiative. Although
the agency initially made improvements in its policies and training
intended to improve the consistency of decisions between adjudication
levels, it has not continued to actively pursue these efforts. As part of
the initiative, the agency also implemented a review of ALJs' allowance
decisions to identify additional ways to improve training and policies,
but no new changes were made as a result of findings from the review.
Finally, the agency also began two tests of process changes to help
improve the consistency of decisions, but one ongoing test with design
problems is not likely to lead to any conclusive results and the other
test has been abandoned.
SSA Made Early Progress Improving Policies and Training, but Has Not
Actively Pursued These Efforts
While SSA initially made progress carrying out efforts to improve policies
and training to better ensure the consistency of decisions, the agency has
not continued to actively pursue these efforts. SSA quickly accomplished
most of its planned efforts to clarify policy guidance. In 1996, SSA
issued nine process unification rulings to clarify policy areas it found
to be contributing to inconsistent decisions. For example, one ruling
provided all adjudicators with guidance on how to weigh and document their
evaluation of the treating physician's opinions when making a disability
decision.16 SSA successfully went through the regulatory process several
years later and published three new regulations to strengthen its process
unification rulings, but was unable to agree on a fourth regulation
regarding the weight to be given to the treating physician's opinion when
evaluating a claim.17
SSA planned to develop a single presentation of policy guidance to replace
the different sources used by each level, but has since abandoned full
implementation of these plans in favor of a more limited approach. DDS
16Social Security Ruling 96-2: Policy Interpretation Ruling Titles II and
XVI: Giving Controlling Weight to Treating Source Medical Opinions,
effective July 2, 1996.
17Since SSA rulings are binding only on SSA adjudicators and do not have
to be followed by the courts, SSA planned to strengthen the impact of
several rulings by creating regulations that would be followed by the
courts.
adjudicators currently follow a detailed set of policy and procedural
guidelines,18 whereas ALJs rely directly on statutes, regulations, and
rulings for guidance in making disability decisions. To help ensure that
inconsistent guidance was not contributing to inconsistent DDS and ALJ
decisions, SSA began issuing guidance in the same wording to all
adjudicators in 1996. Although SSA had also planned to address differences
in policy guidance issued before 1996 and to eventually combine existing
adjudication policy documents into a single document, it ultimately
decided not to take these additional steps. According to SSA, further
efforts to unify the policy guidance used by both levels would be a
massive undertaking and not worth the cost because the guidance issued
since 1996 had already addressed important policy areas that were leading
to inconsistent decisions. While some stakeholder groups representing
adjudicators tended to agree with SSA's position, the Social Security
Advisory Board and other groups still believe the agency should take
additional steps to provide a unified policy guide to all adjudicators.
Instead of creating one policy manual for all adjudicators, SSA told us
that it plans to undertake a comprehensive effort to evaluate and improve
its disability policies to make them less susceptible to differing
interpretations and to ensure they are up to date. A more comprehensive
approach could address key weaknesses in SSA's disability program that we
previously highlighted in our performance and accountability series, and
thereby help to modernize federal disability programs to better meet the
needs of Americans with disabilities.
Early on, SSA also provided extensive cross-training of DDS and ALJ
adjudicators, although the scope of its efforts has since diminished. To
help all adjudicators understand how to appropriately apply process
unification rulings, SSA provided extensive and mandatory training in 1996
and 1997 to 15,000 disability adjudicators (including DDS examiners,
physicians, ALJs, and quality assurance staff). The training was provided
to adjudicators at all levels of the process in three of the most complex
disability areas-assessment of symptoms, treatment of expert opinions, and
assessment of claimants' remaining capacity to work (i.e., residual
functional capacity). While this training was intended to be ongoing,
SSA's training efforts have diminished significantly since 1997.
Stakeholder groups representing DDS adjudicators told us that SSA's
training does not
18These guidelines-called the Program Operations Manual System
(POMS)-contain, within an estimated 30,000 pages, interpretations of
relevant statutes, regulations, and rulings and procedural information.
sufficiently cover process unification issues. In addition, our review of
DDS and OHA participation in video training revealed inconsistent
participation in training by adjudicators. To provide ongoing training to
both adjudication levels and other components involved in the claims
process, SSA has used interactive video technology. Almost all the state
DDS sites and about 85 percent of OHA offices have this technology.
However, in reviewing participation for two recent courses, we found for
those sites with this interactive technology only 31 percent of DDS sites
and 16 percent of OHA sites logged on for a course on the role of
consultative examinations, and 18 percent of DDS sites and 4 percent of
OHA sites logged on for a monthly disability hour training class.19
According to SSA, neither DDS nor OHA adjudicators are generally required
to attend courses. In line with these findings, our recent report on the
human capital challenges facing DDSs found gaps in the key knowledge and
skills of their adjudicators in the same areas SSA had earlier identified
as critical to making consistent decisions, and we recommended that SSA
work with DDSs to close these gaps.20
Despite SSA's early efforts to improve policy guidance and provide
training, stakeholder groups representing state adjudicators told us that
many states are not performing the additional development and
documentation of decisions required by the process unification rulings.
They also told us that the rulings have added significantly to the time,
complexity, and subjectivity of the decision-making process, while
insufficient resources have limited their ability to fully implement the
rulings' requirements. In addition, claimant lawsuits against three state
DDSs have alleged that DDS adjudicators were not following SSA's rulings
or other decision-making guidance. In settling these lawsuits, SSA agreed
to have these states fully develop and document cases. However, according
to DDS stakeholder groups, SSA has not ensured that states have sufficient
resources to meet ruling requirements, which they believe may lead to
inconsistency in decisions among states. Furthermore, SSA's quality
assurance process does not help ensure compliance because reviewers of DDS
decisions are not required to identify and return to the
19Additional DDS and OHA sites may have taped and viewed this training,
but SSA's monitoring of training provides only the number of sites logged
on to the interactive video training.
20U.S. General Accounting Office, Social Security Administration:
Strategic Workforce Planning Needed to Address Human Capital Challenges
Facing the Disability Determination Services, GAO-04-121 (Washington,
D.C.: Jan. 27, 2004).
DDSs cases that are not fully documented in accordance with the rulings.
SSA's procedures require only that the reviewers return cases that have a
deficiency that could result in an incorrect decision.
SSA's Review of ALJ Allowances Has Not Resulted in Improvements to Policy
and Training
As part of its initiative, the agency has also implemented a quality
review of ALJ decisions, but the review has not proved useful for
identifying any new changes to SSA's policies or training that would help
to address the inconsistency of decisions. This review-referred to as the
ALJ Preeffectuation Review-involves a sequential review by SSA's OQA and
the Appeals Council of certain ALJ allowances that have not yet been
finalized (i.e., the claimant has not yet been awarded benefits). In
selecting allowances for review, OQA uses an error-prone profile developed
from its analysis of errors detected when reviewing DDS allowances. SSA
began testing the new review of ALJs' decisions in 1996 and implemented it
as an annual review in 1998. From fiscal years 1998 through 2002, OQA
reviewed 27,148 ALJ allowances and of these, OQA found fault with about 35
percent and referred them to the Appeals Council.21 The Appeals Council
screens the allowances for its own review and selects those in which the
prior actions may not have been proper, fair, or in accordance with the
law or the ALJ's decision was not supported by substantial evidence.22 If
the council finds fault with the ALJ's decision, it will deny the claimant
benefits or return the claim to the ALJ to have the identified problems
corrected. If the council does not find fault with the ALJ's decision, the
claimant will be awarded benefits.
In addition to identifying inappropriate ALJ allowances, SSA intended to
use the new quality review to identify areas of inconsistency between
adjudication levels and ways to improve policies and training to address
those inconsistencies. Specifically, OQA identified cases where it found
fault with the ALJ decision, but the Appeals Council, after screening
them, did not accept them for review. OQA then forwarded these cases to a
panel of staff from the various components involved in SSA's claims
process to determine whether the inconsistent assessment of these cases
21When reviewing an ALJ allowance, OQA uses the preponderance of evidence
standard, which requires that the reviewer fairly consider all evidence
and decide whether the weight of that evidence supports the allowance.
22The Appeals Council uses a substantial evidence standard of review that
requires the reviewer to determine that the evidence in the case is
sufficient to convince a reasonable mind of the credibility of the
allowance decision, and that there is no opposing evidence that clearly
compels another finding or conclusion.
by OQA and the Appeals Council indicated the need to clarify policies,
issue new policies, or provide training to improve the consistency of
decisions. However, according to a SSA official, this review did not
identify any new areas of inconsistency that required improvements to
policy and training. Weaknesses in the design of the review may have
contributed to SSA's inability to identify new policy areas contributing
to inconsistency. For example, rather than reviewing a random sample of
all ALJ decisions, this review focused on allowances. Further, the review
looked only at ALJ allowances that were selected using a DDS error-prone
profile, i.e., a profile that is based upon cases in which quality
reviewers did not agree with the DDS adjudicators' decisions. As a result,
SSA selected and reviewed nonrandom allowance decisions with case
characteristics that the agency may have already suspected were associated
with inconsistent decisions. In 1999, the panel was disbanded because
members had other priorities needing attention. OQA told us that it
continued to perform a limited review of cases viewed differently by OQA
and the Appeals Council. More recently, OQA began an effort to summarize
the results of its review and expected to issue a report of its findings
in April 2004. As of April 2004, this report had not been issued.
SSA Began Two Tests of Process Changes to Improve the Consistency of
Decisions, but Neither Test Was Successfully Completed
SSA began two tests of potential changes to the process to help improve
the consistency of decisions, but neither test was successfully completed.
The changes tested were (1) more fully developing and documenting
decisions made at the initial level and (2) sending appealed cases that
involve new medical information back to the initial level to be
reevaluated.
SSA wanted to test having DDSs more fully develop and document decisions
because it believed that DDS decisions, especially denials, are often not
well documented. SSA wanted to test whether better explanations of why
benefits were denied would improve the accuracy of DDS decisions and
consistency of decisions between adjudication levels. SSA first
implemented a pilot of this change to explore alternatives for developing
and documenting decisions. Then SSA tested this change, along with other
process changes, in a larger test, called the prototype initiative.
Concurrently, SSA tested other process changes, such as the elimination of
a reconsideration step and a predecision DDS interview with the claimant.
The prototype test had limitations for predicting the impact of documented
decisions. For example, SSA's decision to test several changes together
left the agency without clear information on what impact fully developed
decisions would have on the decision-making process
without the other process changes. SSA's test design also did not build in
an ALJ feedback mechanism to provide sufficient information on the
usefulness of more fully documented decisions. SSA continues to test this
change along with other changes and, despite limited information on the
best approach for and impact of this change, currently plans to implement
more fully documented decisions as part of the Commissioner's new proposal
to improve SSA's disability programs.
SSA also began, but ultimately abandoned, a test in which appealed cases
with new medical information submitted prior to the hearing were to be
sent back to the initial level so that the evidence could be evaluated by
medical consultants residing at the DDSs. Since medical expertise resides
in the DDS and not at the hearings level,23 SSA decided to test whether
"remanding," or sending cases to the DDS for evaluation, might result in a
more consistent review of medical evidence. SSA believed that this change,
in turn, could help improve the consistency of decisions because the new
medical information might be contributing to ALJ allowances. However, the
change also had the potential to increase the time claimants with remanded
claims would have to wait for final decisions because claims that were not
allowed by the DDSs had to be returned to OHA for hearings. SSA began
remanding cases in July 1997, with a 1-year goal of remanding 100,000
cases, but after 10 months, it had remanded fewer than 9,000. In
implementing this test, SSA encountered several difficulties. For example,
it had difficulty identifying the claims to be remanded and ensuring the
ALJs, who had authority over the claims, would remand the claims to the
DDSs. The ALJs' resistance to remanding claims to the DDSs may be due in
part to concerns that remanding would not lead to many allowances by the
DDSs and would result in many claims being returned to OHA, thereby
increasing the time many claimants would have to wait for a final decision
from OHA. Realizing that the agency would not be able to reach its
remanding goal, the agency decided to discontinue this test.
23At the hearings level, ALJs can purchase medical evidence, at their
discretion.
SSA Lacks a Clear Understanding of the Extent and Causes of Inconsistency
between Levels
SSA's assessments have not provided the agency with a clear understanding
of the extent and causes of possible inconsistencies in decisions between
adjudication levels. The two measures SSA uses to monitor changes in the
extent of inconsistency of decisions have weaknesses and therefore do not
provide a true picture of the changes in consistency. In addition, SSA has
not sufficiently assessed the causes of possible inconsistency. The agency
conducted an analysis in 1994 that identified some potential areas of
inconsistency. However, although SSA continues to collect information that
would support this analysis, it has not repeated this initial effort, nor
has it expanded on it by employing more sophisticated assessment
techniques.
SSA Attempts to Monitor Changes in the Extent of Inconsistency, but the
Measures It Uses Provide an Incomplete Picture
SSA has made some efforts to monitor changes in the extent of
inconsistency between the initial and hearings levels, including tracking
trends in allowance rates at different levels and conducting special
reviews of ALJ decisions. Together, according to SSA, these measures and
assessments suggest that the consistency between levels has improved since
the agency began implementing its process unification initiative. However,
because of methodological weaknesses, these measures provide, at best, a
partial picture of trends in the consistency of decisions between
adjudication levels.
SSA tracks trends in the proportion of all allowances decided at each
level to assess the consistency of decisions between levels. The agency
collects information on the number of allowances granted to claimants at
each level of the process, tracks the proportion of claims allowed at the
initial level relative to the hearings level, and looks at trends in these
proportions over a period of several years. According to data from SSA,
the proportion of overall allowances that occurred at the initial level
has increased since process unification was implemented. As shown in
figure 2, in fiscal year 1996, 72 percent of all allowances were granted
at the initial level. This proportion increased in most subsequent years,
and by fiscal year 2003, 77 percent of all allowances were granted at the
initial level. Officials from OQA, the office responsible for reviewing,
evaluating, and assessing the integrity and quality of the administration
of SSA's programs, view the relative shift toward earlier allowances as an
indicator that consistency between adjudication levels has improved, and
they believe that process unification efforts have contributed to these
results.
Figure 2: SSA Allowances at Initial Level versus Hearings Level, by
Proportion of Allowances
Percentage of all allowances
100
80
60
40
20
0 1996 1997 1998 1999 2000 2001 2002 2003 Fiscal year
Hearings level allowances
Initial level allowances
Source: SSA.
Note: Hearings level allowances include allowances made by ALJs and judges
from the Appeals Council.
However, SSA's measure of tracking yearly changes in the proportion of
allowances at each level is a simplistic and inconclusive indicator of
trends in the consistency of decisions because it does not control for the
multitude of factors that can affect allowance rates at either
adjudication level in any given year and over time. For example, SSA uses
"snapshot" data in looking at the proportion of allowances granted at each
level, meaning that it looks at the number of claimants and allowances at
each level during a given year, rather than following a 1-year cohort of
initial claimants through the entire process and capturing the proportion
of allowances for that cohort decided at each level. Because SSA uses data
that illustrate allowance rates at a given moment in time, it captures a
different pool of claimants in the process at each level, and the
resulting allowance rates are subject to a different set of demographic
and case characteristics. Over time, the pool of claimants may change
because of factors such as a downturn in the economy, which can cause more
people with less severe impairments to claim benefits or appeal initial
denial
decisions. In addition, snapshot data may be significantly affected by
fluctuations in productivity at either adjudication level caused by
process changes that are unrelated to process unification and that affect
only one level.
SSA has collected other data to further assess trends in the consistency
of decisions. Since 1993, the agency has conducted a biennial case review
as part of its Disability Hearing Quality Review process.24 This review
consists of medical consultants and disability examiners in SSA's central
office evaluating a sample of ALJs' decisions plus supporting
documentation to determine whether the ALJ has adequately supported his or
her decision. In evaluating the ALJ decisions, these medical consultants
and disability examiners use the same standards as those used by
initial-level adjudicators to adjudicate claims, which are from the
official SSA program policy and operations guidance found in POMS. To some
degree, therefore, the medical examiners and disability reviewers serve as
a proxy for initial-level adjudicators, and their decisions are
representative of how initial-level examiners should be deciding claims.
While unpublished results from the biennial case reviews indicate an
increase in supportable ALJ allowances, such findings focus on the ALJ
level and therefore provide only a partial picture of trends in
consistency. The reviews indicated that medical consultants and disability
examiners have found that supportable ALJ allowances increased from 36
percent in fiscal year 1993-94 to 57 percent in fiscal year 1999-2000. OQA
officials told us that this increase suggests an improvement in
consistency between adjudication levels because it indicates that
disability examiners using initial-level standards and ALJs increasingly
agree on how like cases should be decided.25 However, SSA's assessment
provides only a partial picture because it does not reflect trend
information on the extent to which ALJs have found DDS decisions to be
supportable, to ensure that
24See the Social Security Administration, Office of Program and Integrity
Reviews, Findings of the Disability Hearings Quality Review Process: An
Assessment of the Quality of Hearing Decisions and Appealed
Reconsideration Denial Determinations, September 1994.
25An OQA official also said that results from another review bolster the
conclusion that the quality of ALJ decisions has improved. Specifically,
in the ALJ peer review, in which ALJs evaluate a random sample of other
ALJs' decisions, the reviewing ALJs have found an increasing percentage of
ALJ decisions to be supportable. The percent of decisions found to be
supportable increased from 81 percent to 90 percent from the reviews of
decisions issued from fiscal years 1993 through 1994 compared with
decisions from fiscal years 2001 through 2002.
both levels are making more consistent decisions. Although the 1994 report
of findings from the initial biennial case review included the results of
a special probe in which ALJs reviewed 165 DDS reconsideration denial
decisions, the sample was not representative, and therefore results could
not serve as a baseline for developing trend information. In 2003, SSA
began another probe, in which ALJs reviewed 400 DDS reconsideration denial
determinations, but the agency does not plan to release its findings until
summer 2004.
Although SSA has limited information on how ALJs view DDS decisions, other
information collected by the agency suggests that consistency of decision
making at the initial level might not be improving. For example, OQA
reviewers routinely assess the accuracy and supportability of DDS
decisions. A recent SSA study of these data shows that the accuracy of DDS
denial decisions-those decisions most likely to be appealed to the
hearings level-has declined by 4 percentage points over a 1-year period.26
Another review of DDS decisions by OQA reviewers also suggests a lack of
improvement at the initial level. Specifically, the extent to which
quality reviewers found that DDS reconsideration denials appealed to the
hearings level were supported declined from 71 percent in fiscal year
1993-94 to 68 percent in fiscal year 1999-2000.
SSA's Assessments Have Not Identified the Areas and Causes of Possible
Inconsistency in Decisions between Adjudication Levels
Despite some efforts to assess inconsistency in decisions, shortcomings in
SSA's analyses also limit its ability to identify areas and causes of
possible inconsistency. Most notably, over the last 10 years, SSA has not
updated its prior analyses of information from its initial biennial case
review that helped identify problem areas. In addition, SSA has not
improved on its case review and analysis by ensuring that reviewers assess
all relevant case evidence used to make decisions, or performed more
sophisticated analysis to identify the areas and causes of inconsistency
in decisions. Other efforts-including the review of ALJ allowances and a
probe of DDS reconsideration denials-have yet to yield useful information.
In 1994, for its initial biennial case review report, the agency took its
first step in identifying areas of possible inconsistency by identifying
two characteristics about the claimants and their cases over which
initial-level
26See the Social Security Administration, Office of Quality Assurance and
Performance Assessment, Quality Assurance Group I Initial Denial Accuracy
Report: August-October 2002, July 2003.
reviewers tended to disagree with ALJs. Specifically, the 1994 report
concluded that teams of reviewing medical consultants and disability
examiners sometimes viewed cases involving mental impairments differently
than the reviewing ALJs. In addition, these two sets of reviewers tended
to have different views on the severity of claimants' impairments and
their resulting capacity to work. According to the official responsible
for overseeing the review, the findings in this initial report provided
important support for SSA's process unification efforts as well as the
agency's efforts to redesign the disability claims process.
SSA continues to conduct the biennial case reviews; however, the agency
has not continued to analyze and identify areas that are viewed
differently by different adjudication levels. Specifically, SSA no longer
identifies the particular case characteristics over which reviewers from
the two levels tend to disagree. As a result, SSA does not know whether
previously identified problem areas are still present. Moreover, SSA no
longer publishes any information from the medical consultant and
disability examiner biennial case reviews, even though it has performed
some limited analysis of the supportability of decisions made by
adjudicators. By not continuing to publish its analysis and findings, the
agency makes it difficult to ensure the reliability of its methods and
results, and leaves stakeholders outside the agency, including disability
groups, without a means for understanding SSA's assessment efforts and
progress in improving the consistency of decisions. The SSA office
conducting the study has told us that, because of downsizing and competing
priorities, it has no current plans to further analyze and publish these
data.
Further, in its ongoing biennial case reviews, SSA does not make full use
of available case information that would be useful in identifying areas
and causes of inconsistency. Specifically, medical consultants and
disability reviewers do not listen to tapes of the hearings and therefore
do not review the entire case as presented to the original ALJ. Although
reviewing medical consultants and disability examiners read the ALJs'
explanations for their original decisions, which should include the most
important factors behind the ALJs' decisions, the reviewers do not
evaluate the oral evidence independently. An SSA official with whom we
spoke indicated that some evidence entered by witnesses at the hearing
might not be accompanied by other hard copy sources of the same
information. Therefore, reviewers would not consider information
potentially relevant to the ALJ's decision that could be used to identify
areas and causes of inconsistency.
SSA also does not make full use of the information it collects because it
has not employed analytical tools that would improve its ability to
identify areas and causes of inconsistency. For example, SSA's biennial
case reviews provide a rich dataset that lends itself to regression
analysis to identify areas and possible causes of inconsistency between
levels. Regression analysis would allow the agency to better pinpoint any
significant case characteristics affecting decisions and to more clearly
identify the underlying causes of inconsistency. Specifically, among the
data collected in this review are such variables as the types of
impairments the claimant has, the types of relevant medical evidence, and
additional impairments presented at the hearing. Multivariate analysis,
such as a multiple regression model, could allow SSA to assess how these
and many other factors, relative to one another, contribute to whether a
case results in a similar outcome at both levels. However, SSA has not
employed this more sophisticated multivariate technique, citing resource
constraints and competing priorities. We recognize the methodological
complexities of analyzing disability decisions, and we previously
recommended that SSA establish an advisory panel of external experts from
a range of disciplines to provide leadership, oversight, and technical
assistance to the agency.27 Otherwise, in forgoing such analysis, the
agency will continue to miss an opportunity to better pinpoint areas and
some possible causes of inconsistency in decisions between the two
adjudication levels, and to lay the foundation for further investigation.
Another tool SSA has not sufficiently employed for identifying areas and
causes of inconsistency is in-depth case studies involving both levels of
adjudication. Case studies, in which different adjudicators review the
same test case, can be a means for unearthing causes for inconsistency by
getting adjudicators from both levels to acknowledge and address
discrepancies in the ways they view cases. SSA has performed case studies
in the past to ascertain differences in policy interpretation between DDS
examiners and quality reviewers. However, SSA does not routinely have both
DDS examiners and ALJs perform in-depth review of the same sample of
cases, despite this method's potential for helping identify causes
27In addition, the report recommended that SSA include cases appealed to
its Appeals Council in the sample for its biennial case review to
eliminate the systematic bias in that sample and make it representative of
all cases that receive a decision from the hearings level. For more
information on this recommendation and others, see U.S. General Accounting
Office, SSA Disability Decision Making: Additional Steps Needed to Ensure
Accuracy and Fairness of Decisions at the Hearings Level, GAO-04-14
(Washington, D.C.: Nov. 12, 2003).
of inconsistency between the two adjudication levels. OQA officials told
us that case studies are a very resource-intensive tool because they need
a sufficient number of cases from which to generalize. Therefore, the
agency is reluctant to use this approach to help it understand the causes
of inconsistency between adjudication levels. However, using multivariate
analyses of the biennial case review data could help the agency to more
effectively target its in-depth case studies on those areas found to be
leading to inconsistent decisions and thereby increase its success at
identifying the causes of inconsistency.
SSA conducts other analyses of inconsistency between levels, but to date
these efforts have yielded limited information concerning areas and
possible causes of inconsistency. For example, as part of SSA's ALJ
Pre-Effectuation Review, two different levels of reviewers have evaluated
thousands of cases. However, limitations in the review methodology, such
as not using a random sample of ALJ decisions, do not allow the agency to
use this review to identify the leading causes of inconsistency. SSA
recently began an evaluation of this effort and plans to publish its
findings and recommendations in April 2004. Another analysis currently
under way, a special 400-case review, might help identify areas of
inconsistency at the initial level, but it has yet to be completed. Begun
in 2003, this review by ALJs of DDS reconsideration denial determinations
is expressly aimed at assessing inconsistency between adjudication levels.
SSA expects to gain some understanding of why about 60 percent of cases
denied by the initial level and appealed to the hearings level are
allowed. The agency plans to publish its findings in summer 2004.
Some changes included in SSA's new proposal to overhaul its disability
claims process may improve the consistency of DDS and ALJ decisions, but
challenges may hinder the implementation of the proposal. The new proposal
includes several changes to the disability claims process that the agency
and stakeholder groups representing adjudicators and claimant
representatives believe offer promise for improving the consistency of DDS
and ALJ decisions. However, past difficulties in improving the process, as
well as stakeholder concerns about limited resources and other obstacles,
indicate that some difficulties may arise in the development and
implementation of SSA's new proposal.
SSA's New Proposal Incorporates Efforts to Improve the Consistency of
Decisions, but Challenges May Impede Successful Implementation
The Agency and Most Stakeholders Viewed Several Aspects of the New Proposal
as Offering Promise for Improving Consistency
SSA told us that several aspects of the new proposal may improve the
consistency of decisions, and although opinions varied among stakeholder
groups, most thought the following four proposed changes have the
potential to improve the consistency of decisions between adjudication
levels: (1) requiring state adjudicators to more fully develop and
document their decisions, (2) centralizing the agency's approach to
quality control, (3) providing both adjudication levels with equal access
to more centralized medical expertise, and (4) requiring ALJs to address
agency reports that either recommend denying the claim or outline the
evidence needed to fully support the claim.
Representatives from most stakeholder groups with whom we spoke told us
that having state adjudicators more fully develop and document their
decisions may help to improve the consistency of DDS and ALJ decisions.
Specifically, stakeholders said that more developed decisions may provide
ALJs with a better understanding of the DDS decision and enable them to
more fully consider this information when evaluating a case. According to
the agency and stakeholders, this change may contribute to a more
consistent interpretation and application of SSA's decision-making
criteria. They also mentioned that well-developed decisions by DDS
examiners could assist SSA in holding adjudicators accountable for case
development and decisions, such as enabling quality reviewers to more
effectively assess the appropriateness of the DDSs decisions. Unlike SSA's
earlier attempt at more fully developing decisions as part of process
unification, SSA plans to incorporate a reviewing official into the
process whose assessment of all appealed DDS decisions can provide
feedback on the extent to which cases are being fully developed.
In addition, the agency and many stakeholders told us that they believe
centralizing the agency's quality control system may help resolve some
problems contributing to inconsistent decisions between the two levels.
For example, they believed that it may help ensure a more consistent
review of cases across the country and between adjudication levels.
According to both stakeholders and other experts within and outside of SSA
(including SSA's Deputy Commissioner of Disability and Income Security and
a consulting group that reviewed SSA's quality assurance system),28 the
current quality control and case review process encourages
28See the Lewin Group, Inc., et al., Evaluation of SSA's Disability
Quality Assurance (QA) Processes and Development of QA Options That Will
Support the Long-Term Management of the Disability Program, (Falls Church,
VA: Mar. 16, 2001), pp. 168 and E-9.
adjudicators at the initial level to inappropriately deny cases, while
encouraging adjudicators at the hearings level to inappropriately allow
cases. Specifically, by overemphasizing a review of DDS allowances to help
control the cost of benefits, the agency has unintentionally encouraged
DDS examiners to deny cases. Conversely, SSA's review of ALJ decisions
consists mostly of SSA's Appeals Council reviewing cases denied by ALJs,
thereby providing an incentive for ALJs to allow cases. By centralizing
the quality control system and making other changes to the process, SSA
believes that it can remove the current incentives that contribute to
inconsistency.
The third proposed change that the agency and most stakeholder groups
believe may improve consistency is SSA's plan to provide both adjudication
levels with equal access to more centralized medical experts, organized by
clinical specialty. Although located in the regions, these experts should
be able to review cases from across the country with the successful
completion of SSA's AeDib initiative-an electronic folder initiative for
exchanging case information currently being implemented by SSA. By making
experts in a range of specialties available to assist both levels of
adjudicators in their decision making, SSA and stakeholders believe that
adjudicators could more consistently apply SSA's decisionmaking criteria,
in addition to acquiring better medical evidence.
Finally, the agency and most stakeholder groups told us that the
requirement to have an ALJ's decision address the recommendations from a
reviewing official's report to either deny or more fully develop the claim
may increase consistency between levels. Under the new proposal, SSA plans
to introduce a reviewing official into the process to evaluate all
appealed DDS claims. The official will allow claims that meet SSA's
definition of disability and, for the remaining claims, will develop a
report that either (1) contains reasons for denying the claim or (2)
outlines the evidence needed to fully support the claim. The ALJ's
decision must address issues raised in the reviewing official's report.
Stakeholders believed that this change could, as intended by SSA, hold
adjudicators more accountable for their decisions and provide adjudicators
with feedback on the reasons decisions tend to differ between levels to
improve the quality and consistency of their decisions.
Although there was less agreement among stakeholder groups on the
potential effect that other aspects of the new proposal may have on the
consistency of decisions, some groups thought that other changes could
result in improved consistency between DDS and ALJ decisions. For example,
the Social Security Advisory Board and two groups representing
the DDSs thought that the proposed in-line quality control, if implemented
effectively at all levels, could have a positive impact on consistency by
ensuring that adjudicators adhere to the rulings and regulations
throughout the decision-making process. One stakeholder group added that
in-line quality control could also help the agency identify problem areas,
including areas in which policy is applied inconsistently or where more
training is needed.
Resource and Other Constraints May Limit SSA's Ability to Successfully
Implement Some Changes in the New Proposal
According to stakeholder groups-and based on SSA's prior experience with
making significant changes to its claims process-insufficient resources
and other obstacles may prove to be major challenges for the agency in
developing and implementing aspects of its new proposal. For example,
experience with the process unification initiative has shown that limited
state resources have hindered the agency's ability to have state
adjudicators fully document decisions. To address this issue, SSA plans to
reduce the states' workloads by decreasing the number of claims to be
decided by the DDSs. Specifically, SSA expects that establishing regional
expert review units to make quick decisions for claimants who are
obviously disabled will substantially decrease the states' workloads.
However, SSA has not developed and provided stakeholders with estimates of
the administrative cost for more fully documenting decisions and other
planned changes, and stakeholder groups were not convinced that the
reduction in claims was sufficient to offset resources needed to fully
document their decisions. Although the agency has had some recent success
in increasing its 2004 administrative budget, and is confident that it
will be successful in acquiring the resources it needs to implement the
proposal, the significance of stakeholders' concerns about funding cannot
be assessed until SSA fully develops its proposal and associated cost
estimates.
Experience has also shown that another proposed change, developing a
centralized quality control system for both adjudication levels, could be
a major challenge for the agency. In 1994, SSA began efforts to create a
unified and comprehensive quality control system as part of its redesign
efforts, but made little progress, in part because of considerable
disagreement among internal and external stakeholders on how to accomplish
this difficult objective. To get external assistance in developing an
effective quality assurance system, SSA contracted with an independent
consulting firm to assess SSA's quality assurance practices used in the
disability claims process. In 2001, concluding that SSA could achieve its
quality objectives for the disability program only by adopting a broad,
modern view of quality management, the consulting firm
recommended SSA abandon its current system and design a new quality
management system focused on building quality into the process. The agency
agreed that it was appropriate to transform the existing quality assurance
system and established an executive work group to decide a future course
of action. The agency is working with another consulting group to further
develop the changes recently proposed by the Commissioner. However, after
10 years of efforts to develop a more unified quality review system, SSA
has not yet formulated changes to its quality review system, beyond the
brief and general descriptions provided in the Commissioner's new
proposal.
Other obstacles also add to the complexity and difficulty of implementing
the proposal. For example, stakeholder groups have raised concerns about
SSA's ability to successfully implement its proposed change to provide
equal access for all adjudicators to more centralized medical expertise by
removing medical expertise from the state DDSs and providing expertise in
regional offices. Stakeholder groups were concerned that SSA would not be
able to attract and retain sufficient medical experts to meet the agency's
needs. They told us that states are currently experiencing problems
attracting medical experts because SSA's compensation rates are too low.
State adjudicators, who currently work with medical experts directly at
DDS offices, were also concerned that removing these experts and placing
experts in SSA regional offices would impair the states' effectiveness and
efficiency. By placing experts in regional offices, state disability
examiners would no longer have on-site access to these experts who help
facilitate the states' adjudication of claims and provide on-thejob
training and mentoring to DDS examiners.
Stakeholders have also raised questions about SSA's ability to ensure that
ALJs' decisions fully respond to the reviewing officials' reports and the
ultimate effectiveness of this change. Stakeholder groups representing
ALJs and claimant representatives believed that the requirements may have
the potential to impinge on an ALJ's legal responsibility to ensure a
claimant receives a fair hearing and an independent decision. Other groups
have raised concerns about SSA's ability to ensure that ALJs will
adequately address recommendations in the reviewing officials' reports to
help ensure that this requirement leads to more consistent decisions.
Although these concerns have been raised, the Commissioner has clearly
stated that the intent of the proposal is to improve service to claimants,
including providing fair and accurate decisions, and that changes will not
impinge on the independence of ALJs.
In addition, several stakeholder groups also told us that staffing the new
reviewing official positions with attorneys, as SSA intends to do, would
be expensive. To the extent that SSA has difficulty filling these
positions, the agency could create a slowdown or bottleneck in the process
that could increase the time claimants must wait for a decision.
Furthermore, according to one stakeholder group, SSA's new quality
assurance process will need to ensure that this new position does not
create another source of inconsistent interpretation and application of
SSA's decision-making criteria.
Several groups representing hearings level adjudicators and claimant
representatives were also concerned about other aspects of the
Commissioner's new proposal, such as the proposed elimination of the
Appeals Council and the claimants' loss of the right to appeal an ALJ
decision to the council. The Appeals Council currently reviews about
100,000 appealed ALJ decisions annually. For these claims, the council
provides an additional appellate step for addressing claimants' objections
to the ALJs' decisions, reviewing new medical information on the claims
and reducing the number of claims appealed directly to the federal courts.
According to one stakeholder group, the council also performs other
important functions, such as reviewing claims for surviving children or
spouses of workers who were insured under the disability insurance and
retirement program. The council also reviews cases remanded from federal
courts. This stakeholder group also told us that as SSA refines its
proposal it will need to articulate how all of the council's functions
will be handled under the new process.
Adding to uncertainties about the proposal's success is its dependence on
the successful development and implementation of the AeDib system-a highly
complex and as yet unproven system using electronic folders to share
information with all entities involved in disability determinations. SSA
does not plan to implement its newly proposed changes before it has
completed a national rollout of its electronic disability system,
scheduled to be completed by October 2005. The new electronic disability
system represents an important step toward a paperless and more efficient
sharing of information by multiple partners involved in the disability
claims process, including SSA and state officials, as well as physicians
and other members of the medical community who provide needed medical
evidence. SSA also expects this new system to provide critical management
information for analyzing and reducing inconsistencies in disability
decisions. As we previously reported, SSA has made progress developing the
new system. However, its approach involves risks that could jeopardize the
agency's successful transition to an electronic
Conclusions
disability claims process.29 For example, SSA recently began a national
rollout of the electronic disability system without fully evaluating pilot
test results or ensuring the resolution of all critical problems. Skipping
such important steps in development and implementation leaves the new
system vulnerable to problems in its performance and reliability. In
addition, problems with implementation of this system could delay the
implementation of SSA's new proposal.
SSA recognizes that transforming its massive and complex disability
programs and achieving the benefits envisioned by the Commissioner will be
a challenging undertaking. The agency is refining its proposal and, as
part of this process, is actively seeking input from stakeholder groups.
The Commissioner and her staff have met directly with stakeholder groups
to understand and begin to address their concerns. As the agency refines
its proposal, the significance of both stakeholder concerns and previous
problems SSA has experienced improving its programs should become clearer.
When SSA's Commissioner announced her new proposal to overhaul the
disability programs, the agency acknowledged the importance of making
similar decisions on similar cases and making the right decision as early
in the process as possible. SSA has good cause to focus on the consistency
of decisions between adjudication levels. Incorrect denials at the initial
level that are appealed increase both the time claimants must wait for a
decision and the cost of deciding cases. Incorrect denials that are not
appealed may leave needy individuals without a financial or medical safety
net. Conversely, incorrect allowances at any adjudication level could
substantially increase the cost of providing disability benefits.
While the agency has made some effort to assess the inconsistency in
decisions between levels, its efforts have not provided the agency with a
clear understanding of the extent and leading causes of possible
inconsistencies in the interpretation and application of disability
guidance.
29For additional information on SSA's progress and its risky strategy for
implementing its accelerated disability claims system, see U.S. General
Accounting Office, Electronic Disability Claims Processing: Social
Security Administration's Accelerated Strategy Faces Significant Risks,
GAO-03-984T (Washington, D.C.: July 24, 2003); Social Security
Administration: Subcommittee Questions Concerning Efforts to Automate the
Disability Claims Process, GAO-03-1113R (Washington, D.C.: Sept. 5, 2003);
and Electronic Disability Claims Processing: SSA Needs to Address Risks
Associated with Its Accelerated Systems Development Strategy, GAO-04-466
(Washington, D.C.: Mar. 26, 2004).
For example, SSA's assessment of ALJ error-prone allowances has not proven
to be effective at identifying new areas and causes of inconsistency. SSA
also has not updated its more effective approach of analyzing its
Disability Hearings Quality Review data to identify problem areas and help
improve its understanding of the factors that may be contributing to
inconsistency. Further, SSA's analysis lacked sophisticated statistical
techniques and in-depth analysis of cases by adjudicators at both levels,
which together would have allowed SSA to better identify and address the
areas and leading causes of inconsistency. Moreover, by not having
examiners and medical consultants perform a complete review of all
relevant information before an ALJ, SSA has limited its ability to
understand the areas and causes of possible inconsistency.
Without better information on the areas and causes of possible
inconsistency, the agency cannot ensure that the Commissioner's new
proposal will help to resolve this complex and long-standing concern. By
taking immediate actions to improve its understanding of the leading
causes of possible inconsistency in decisions, the agency will have
information needed to evaluate and possibly refine its new proposal,
including its plans to build an effective quality assurance system that
can both detect and prevent inconsistencies in decisions. This information
will help the agency to target its limited resources and take decisive
steps to build a claims process that provides claimants with the accurate,
consistent, and timely decisions they deserve, as envisioned in the
Commissioner's proposal.
To move successfully forward with agency efforts to make more consistent
decisions, including efforts incorporated in the Commissioner's proposal
for an improved disability claims process and quality assurance system, we
recommend that SSA quickly expand its assessment of the areas and causes
of inconsistency in decisions between adjudication levels. In doing so,
SSA should consider making near-term and costeffective enhancements to its
current approach for assessing the consistency of decisions, including:
1. Reestablish ongoing analyses of case characteristics as part of its
biennial case review, in line with efforts undertaken for the review
report published in 1994.
2. Perform more sophisticated multivariate analysis on the biennial case
review data in order to pinpoint the most significant case characteristics
influencing allowance decisions and to distinguish
Recommendations
3.
4.
5.
6.
factors that might be contributing either appropriately or inappropriately
to allowance decisions.
Expand the biennial case review by requiring disability examiners and
medical consultants to review the hearing tapes to ensure that reviewers
have the complete case before them (including the types and sources of
testimonial evidence provided during the hearings) when evaluating the
ALJs' decisions.
Have adjudicators and reviewers from each level study cases in depth to
help pinpoint the causes of inconsistency, once potential areas of
inconsistency between levels are identified.
Publish the methods and findings of all analyses, to keep internal and
external stakeholders aware of the agency's efforts to assess consistency
and demonstrate improvement over time.
Use the information from these improved analyses to develop a more focused
and effective strategy for ensuring uniform application of SSA's guidance
and to improve the consistency of decisions. To accomplish this, SSA
should clarify guidance for making disability decisions and develop
mandatory training for adjudicators on issues identified as contributing
to inconsistency.
Agency Comments
and Our Evaluation
We provided a draft of this report to SSA for comment. SSA expressed
several reservations about the recommendations, findings, and conclusions
of our report. Primarily, SSA took issue with: (1) our characterization of
the agency's progress over the past several years in analyzing and
reducing the inconsistency of decisions, (2) our recommendation that the
agency incorporate multivariate analysis into its assessments, and (3) our
finding that the agency has not acted on the results of its reviews of
decisions. SSA indicated that it would reevaluate our recommendations as
the design of its Commissioner's new approach to disability decision
making evolves. However, the agency did agree to pilot one
recommendation-that quality reviewers assess hearing tapes when evaluating
the ALJs' decisions-as part of a quality review.
One of SSA's main concerns was that our report did not fully discuss the
progress SSA had achieved in analyzing and reducing the inconsistency in
decision making between adjudication levels. For example, SSA commented
that our report dismissed the 21-percentage point increase in the quality
reviewers' support rate of ALJ decisions, conducted as part of
SSA's biennial case reviews over the last 10 years. SSA also pointed to
findings from its ALJ peer reviews as additional evidence that the quality
and consistency of SSA's decisions had improved. In addition, SSA asserted
that its comparison of the relative proportion of allowances at the DDS
and ALJ levels, along with high accuracy rates, indicated that
adjudicators were making the right decisions sooner in the process-a goal
of both process unification and the Commissioner's new disability
approach. Although our report incorporates results from the analyses cited
by SSA, our conclusion about the improvement in consistency between levels
is not as optimistic as SSA's because of weaknesses in SSA's assessments.
As we reported, SSA's analysis of the quality reviewers' assessment of ALJ
cases has been limited for 10 years to calculating ALJ support rates. SSA
has not used available data to determine the potential areas of
inconsistency between levels or the extent to which changes in the ALJ
support rate is related to improvements in consistency of decisions
between adjudication levels. SSA's assessment also lacks a reliable method
for determining whether DDS decisions are more consistent with ALJ
decisions, for example, by having ALJs regularly review a statistical
sample of DDS decisions. Lastly, as we pointed out, changes in the
proportion of overall allowances made by the DDS and ALJ levels cannot
serve as a reliable indicator for measuring the consistency of decisions
between levels, because many factors can affect these proportions, such as
significant fluctuations in the number of decisions made at each
adjudication level.
SSA also expressed its reservations about the benefits of multivariate
analysis in its evaluation of decision making. SSA asserted that its
analyses over the past 10 years have provided the agency with a solid
understanding of how certain variables influence disability decision
making and that the multivariate analyses we recommended would not
identify the causes and effects of inconsistent decision making at
different levels of this complex process. We agree with SSA that the
disability decision-making process is complex and that multivariate
analysis alone cannot establish all the causes and effects of inconsistent
decision making. However, because multivariate analysis takes into account
the influence of a number of relevant variables for each decision, this
analytical technique can provide a more accurate understanding of areas
and causes of inconsistency in decisions than methods previously employed
by SSA. Such analyses, followed by in-depth case studies by adjudicators
at both levels, which we also recommended, would bring SSA closer to
understanding and resolving the inconsistency of decisions between
adjudication levels. Therefore, we continue to believe that by performing
the analyses we recommend, the agency will have a better understanding
of the extent and causes of inconsistency, and that SSA's Commissioner
should quickly implement our recommendations to ensure that her new
approach effectively addresses the consistency of decisions between
adjudication levels.
Finally, SSA disagreed with our finding that it has not acted on the
results of its reviews of decisions. SSA noted that it has made changes to
address training needs that have been identified by its reviews.
Specifically, SSA indicated that it has provided a series of interactive
video training (IVT) sessions focusing on problematic areas noted in the
ALJ peer review reports. We acknowledge that SSA has conducted ALJ peer
reviews and used findings from its reviews to develop and provide training
to ALJs. However, we did not include these findings in our report, because
our objectives were limited to reporting efforts undertaken by SSA to
assess or improve the consistency of decisions between adjudication levels
or to implement its process unification initiative. SSA's ALJ peer review
is conducted to identify problems with the quality of ALJ hearing process
and decisions, not to identify problems with the inconsistency of
decisions between levels. Conversely, our report included information on
SSA's ALJ pre-effectuation review, because it was part of SSA's process
unification initiative. According to information provided to us by SSA
during our audit, although this review was intended to help identify
policy and training areas that were associated with inconsistent decisions
between adjudication levels, it was not effective at identifying any new
areas to be pursued by the agency. This finding, along with those provided
throughout the report, supports our recommendations to SSA that the agency
perform additional analysis to determine the causes of potential
inconsistency between adjudication levels and to clarify guidance and
provide mandatory training to address any identified causes.
In addition, SSA provided several other general and technical comments
about the draft report. These additional comments, as well as our response
to them, are provided in appendix II.
Copies of this report are being sent to the Commissioner of SSA,
appropriate congressional committees, and other interested parties. The
report is also available at no charge on GAO's Web site at
http://www.gao.gov. If you have any questions about this report, please
contact me at (202) 512-7215. Other contacts and staff acknowledgments are
listed in appendix III.
Sincerely yours,
Robert E. Robertson Director, Education, Workforce, and Income Security
Issues
Appendix I: Excerpt of SSA's Testimony Announcing Its New Proposal to Improve
Its Disability Decision-Making Process
In designing my approach to improve the overall disability determination
process, I was guided by three questions the President posed during our
first meeting to discuss the disability programs. 1
o Why does it take so long to make a disability decision?
o Why can't people who are obviously disabled get a decision
immediately?
o Why would anyone want to go back to work after going through such a
long process to receive benefits?
I realized that designing an approach to fully address the central and
important issues raised by the President required a focus on two
overarching operational goals: (1) to make the right decision as early in
the process as possible; and (2) to foster return to work at all stages of
the process. I also decided to focus on improvements that could be
effectuated by regulation and to ensure that no SSA employee would be
adversely affected by my approach. My reference to SSA employees includes
State Disability Determination Service employees and Administrative Law
Judges (ALJs).
As I developed my approach for improvement, I met with and talked to many
people-SSA employees and other interested organizations, individually and
in small and large groups-to listen to their concerns about the current
process at both the initial and appeals levels and their recommendations
for improvement. I became convinced that improvements must be looked at
from a system-wide perspective and, to be successful, perspectives from
all parts of the system must be considered. I believe an open and
collaborative process is critically important to the development of
disability process improvements. To that end, members of my staff and I
visited our regional offices, field offices, hearing offices, and State
Disability Determination Services, and private disability insurers to
identify and discuss possible improvements to the current process.
Finally, a number of organizations provided written recommendations for
changing the disability process. Most recently, the Social Security
1This excerpt is taken from a statement by the Honorable Jo Anne B.
Barnhart, Commissioner, Social Security Administration, Testimony before
the Subcommittee on Social Security of the House Committee on Ways and
Means, September 25, 2003.
Appendix I: Excerpt of SSA's Testimony Announcing Its New Proposal to
Improve Its Disability Decision-Making Process
Advisory Board issued a report prepared by outside experts making
recommendations for process change. My approach for changing the
disability process was developed after a careful review of these
discussions and written recommendations. As we move ahead, I look forward
to working within the Administration and with Congress, as well as
interested organizations and advocacy groups. I would now like to
highlight some of the major and recurring recommendations made by these
various parties.
The need for additional resources to eliminate the backlog and reduce the
lengthy processing time was a common theme. This important issue is being
addressed through my Service Delivery Plan, starting with the President's
FY 2004 budget submission which is currently before Congress. Another
important and often heard concern was the necessity of improving the
quality of the administrative record. DDSs expressed concerns about
receiving incomplete applications from the field office; ALJs expressed
concerns about the quality of the adjudicated record they receive and
emphasized the extensive pre-hearing work required to thoroughly and
adequately present the case for their consideration. In addition, the
number of remands by the Appeals Council and the Federal Courts make clear
the need for fully documenting the administrative hearing record.
Applying policy consistently in terms of: 1) the DDS decision and ALJ
decision; 2) variations among state DDSs; and 3) variations among
individual ALJs-was of great concern. Concerns related to the
effectiveness of the existing regional quality control reviews and ALJ
peer review were also expressed. Staff from the Judicial Conference
expressed strong concern that the process assure quality prior to the
appeal of cases to the Federal Courts.
ALJs and claimant advocacy and claimant representative organizations
strongly recommended retaining the de novo hearing before an ALJ.
Department of Justice litigators and the Judicial Conference stressed the
importance of timely case retrieval, transcription, and transmission.
Early screening and analysis of cases to make expedited decisions for
clear cases of disability was emphasized time and again as was the need to
remove barriers to returning to work.
My approach for disability process improvement is designed to address
these concerns. It incorporates some of the significant features of the
current disability process. For example, initial claims for disability
will continue to be handled by SSA's field offices. The State Disability
Appendix I: Excerpt of SSA's Testimony Announcing Its New Proposal to
Improve Its Disability Decision-Making Process
Determination Services will continue to adjudicate claims for benefits,
and Administrative Law Judges will continue to conduct hearings and issue
decisions. My approach envisions some significant differences.
I intend to propose a quick decision step at the very earliest stages of
the claims process for people who are obviously disabled. Cases will be
sorted based on disabling conditions for early identification and
expedited action.
Examples of such claimants would be those with ALS, aggressive cancers,
and end-stage renal disease. Once a disability claim has been completed at
an SSA field office, these Quick Decision claims would be adjudicated in
Regional Expert Review Units across the country, without going to a State
Disability Determination Service. This approach would have the two-fold
benefit of allowing the claimant to receive a decision as soon as
possible, and allowing the State DDSs to devote resources to more complex
claims.
Centralized medical expertise within the Regional Expert Review Units
would be available to disability decision makers at all levels, including
the DDSs and the Office of Hearings and Appeals (OHA). These units would
be organized around clinical specialties such as musculoskeletal,
neurological, cardiac, and psychiatric. Most of these units would be
established in SSA's regional offices.
The initial claims not adjudicated through the Quick Decision process
would be decided by the DDSs. However, I would also propose some changes
in the initial claims process that would require changes in the way DDSs
are operating. An in-line quality review process managed by the DDSs and a
centralized quality control unit would replace the current SSA quality
control system. I believe a shift to in-line quality review would provide
greater opportunities for identifying problem areas and implementing
corrective actions and related training. The Disability Prototype would be
terminated and the DDS Reconsideration step would be eliminated. Medical
expertise would be provided to the DDSs by the Regional Expert Review
units that I described earlier.
State DDS examiners would be required to fully document and explain the
basis for their determination. More complete documentation should result
in more accurate initial decisions. The increased time required to
accomplish this would be supported by redirecting DDS resources freed up
by the Quick Decision cases being handled by the expert units, the
elimination of the Reconsideration step, and the shift in medical
expertise responsibilities to the regional units.
Appendix I: Excerpt of SSA's Testimony Announcing Its New Proposal to
Improve Its Disability Decision-Making Process
A Reviewing Official (RO) position would be created to evaluate claims at
the next stage of the process. If a claimant files a request for review of
the DDS determination, the claim would be reviewed by an SSA Reviewing
Official. The RO, who would be an attorney, would be authorized to issue
an allowance decision or to concur in the DDS denial of the claim. If the
claim is not allowed by the RO, the RO will prepare either a Recommended
Disallowance or a Pre-Hearing Report. A Recommended Disallowance would be
prepared if the RO believes that the evidence in the record shows that the
claimant is ineligible for benefits. It would set forth in detail the
reasons the claim should be denied. A Pre-Hearing Report would be prepared
if the RO believes that the evidence in the record is insufficient to show
that the claimant is eligible for benefits but also fails to show that the
claimant is ineligible for benefits. The report would outline the evidence
needed to fully support the claim. Disparity in decisions at the DDS level
has been a long-standing issue and the SSA Reviewing Official and creation
of Regional Expert Medical Units would promote consistency of decisions at
an earlier stage in the process.
If requested by a claimant whose claim has been denied by an RO, an ALJ
would conduct a de novo administrative hearing. The record would be closed
following the ALJ hearing. If, following the conclusion of the hearing,
the ALJ determines that a claim accompanied by a Recommended Disallowance
should be allowed, the ALJ would describe in detail in the written opinion
the basis for rejecting the RO's Recommended Disallowance. If, following
the conclusion of the hearing, the ALJ determines that a claim accompanied
by a Pre-Hearing Report should be allowed, the ALJ would describe the
evidence gathered during the hearing that responds to the description of
the evidence needed to successfully support the claim contained in the
Pre-Hearing Report.
Because of the consistent finding that the Appeals Council review adds
processing time and generally supports the ALJ decision, the Appeals
Council stage of the current process would be eliminated. Quality control
for disability claims would be centralized with end-of-line reviews and
ALJ oversight. If an ALJ decision is not reviewed by the centralized
quality control staff, the decision of the ALJ will become a final agency
action. If the centralized quality control review disagrees with an
allowance or disallowance determination made by an ALJ, the claim would be
referred to an Oversight Panel for determination of the claim. The
Oversight Panel would consist of two Administrative Law Judges and one
Administrative Appeals Judge. If the Oversight Panel affirms the ALJ's
decision, it becomes the final agency action. If the Panel reverses the
ALJ's decision, the oversight Panel decision becomes the final agency
action. As is
Appendix I: Excerpt of SSA's Testimony Announcing Its New Proposal to
Improve Its Disability Decision-Making Process
currently the case, claimants would be able to appeal any final agency
action to a Federal Court.
At the same time these changes are being implemented to improve the
process, we plan to conduct several demonstration projects aimed at
helping people with disabilities return to work. These projects would
support the President's New Freedom Initiative and provide work incentives
and opportunities earlier in the process.
Early Intervention demonstration projects will provide medical and cash
benefits and employment supports to Disability Insurance (DI) applicants
who have impairments reasonably presumed to be disabling and elect to
pursue work rather than proceeding through the disability determination
process. Temporary Allowance demonstration projects will provide immediate
cash and medical benefits for a specified period (12-24 months) to
applicants who are highly likely to benefit from aggressive medical care.
Interim Medical Benefits demonstration projects will provide health
insurance coverage to certain applicants throughout the disability
determination process. Eligible applicants will be those without such
insurance whose medical condition is likely to improve with medical
treatment or where consistent, treating source evidence will be necessary
to enable SSA to make a benefit eligibility determination. Ongoing
Employment Supports to assist beneficiaries to obtain and sustain
employment will be tested, including a Benefit Offset demonstration to
test to effects of allowing DI beneficiaries to work without total loss of
benefits by reducing their monthly benefit $1 for every $2 of earnings
above a specified level and Ongoing Medical Benefits demonstration to test
the effects of providing ongoing health insurance coverage to
beneficiaries who wish to work but have no other affordable access to
health insurance.
I believe these changes and demonstrations will address the major concerns
I highlighted earlier. I also believe they offer a number of important
improvements:
o People who are obviously disabled will receive quick decisions.
o Adjudicative accountability will be reinforced at every step in the
process.
o Processing time will be reduced by at least 25%.
Appendix I: Excerpt of SSA's Testimony Announcing Its New Proposal to
Improve Its Disability Decision-Making Process
o Decisional consistency and accuracy will be increased.
o Barriers for those who can and want to work would be removed.
Describing my approach for improving the process is the first step of what
I believe must be-and will work to make-a collaborative process. I will
work within the Administration, with Congress, the State Disability
Determination Services and interested organizations and advocacy groups
before putting pen to paper to write regulations. As I said earlier, and I
say again that to be successful, perspectives from all parts of the system
must be considered.
Later today, I will conduct a briefing for Congressional staff of the Ways
and Means and Senate Finance Committees. I will also brief SSA and DDS
management. In addition, next week I will provide a video tape of the
management briefing describing my approach for improvement to all SSA
regional, field, and hearing offices, State Disability Determination
Services, and headquarters and regional office employees involved in the
disability program. Tomorrow, I will be conducting briefings for
representatives of SSA employee unions and interested organizations and
advocacy groups, and I will schedule meetings to provide an opportunity
for those representatives to express their views and provide assistance in
working through details, as the final package of process improvements is
fully developed.
I believe that if we work together, we will create a disability system
that responds to the challenge inherent in the President's questions. We
will look beyond the status quo to the possibility of what can be. We will
achieve our ultimate goal of providing accurate, timely service for the
American people.
Appendix II: Comments from the Social Security Administration
Note: GAO comments supplementing those in the report text appear at the
end of this appendix.
Appendix II: Comments from the Social Security Administration
See comment 1.
See comment 2.
See comment 3.
See comment 4.
Appendix II: Comments from the Social Security Administration
See comment 5.
See comment 6.
See comment 7.
Appendix II: Comments from the Social Security Administration
Appendix II: Comments from the Social Security Administration
See comment 8.
See comment 9.
See comment 10.
Appendix II: Comments from the Social Security Administration
Now on p. 11.
Now on page 11, footnote 13.
Appendix II: Comments from the Social Security Administration
GAO Comments
1. We maintain that our report fully and fairly describes SSA's progress
in analyzing and addressing the underlying causes of inconsistent
decisions between state DDS examiners and ALJs. Our research included an
extensive review of agency documentation and interviews with SSA
officials, as well as stakeholder groups for adjudicators and claimant
representatives, to develop a complete understanding of the agency's
efforts to assess and improve the consistency of decisions between
adjudication levels. Also, in agreement with our requestor, we sought to
expand the review to include SSA's new approach to improving its
disability programs, so that we could provide the Congress with an
understanding of how SSA's future plans may help to address this issue.
2. We provided information on the various reviews and analyses of
disability decisions to assess the consistency of decisions between
adjudication levels conducted by SSA over the last 10 years, but none of
these reviews have clearly identified the causes of inconsistency in
decisions between adjudication levels.
3. Our report has not overlooked the data cited by SSA. Nevertheless, our
conclusion about the improvement in consistency between levels indicated
by the data is not as optimistic as SSA's because of weaknesses in SSA's
assessments. As we reported, for 10 years SSA's analysis of the quality
reviewers' assessment of ALJ cases has been limited to calculating ALJ
support rates. SSA has not used available data to determine the potential
areas of inconsistency between levels nor the extent to which changes in
the ALJ support rate are related to improvements in consistency of
decisions between adjudication levels. SSA's assessment only provides a
general indication of overall changes in consistency at one adjudication
level.
4. Our report recognizes that SSA's disability decision-making process is
complex. Because of this complexity, we believe that multivariate analysis
is an appropriate assessment tool that would allow SSA to assess the
effect of multiple factors. In recommending this sophisticated tool, we
were careful not to imply that causes and effects of inconsistent decision
making can be established with certainty. However, we believe that such an
analysis will help SSA understand the relative importance of the variety
of factors that affect its decisionmaking process. After identifying areas
of inconsistency, SSA can target these areas with in-depth case analyses
to pinpoint the causes of inconsistency and develop a more effective
strategy for addressing inconsistency. On the basis of our review of SSA's
analyses to date, we do not agree with the implications of SSA's comments
that it has a
Appendix II: Comments from the Social Security Administration
solid understanding of how certain variables influence disability decision
making, and therefore does not need to conduct additional, more
sophisticated analyses.
5. We agree with SSA that the proportion of allowances made at each level
can provide some insight into the allowance rate dynamic. However, as we
reported, we do not believe that it can serve as a reliable indicator of
the agency's progress in achieving more consistent decisions between the
DDS and OHA levels. The allowance data provided by SSA simply show that
the relative proportion of allowances made at the DDS level increased in
comparison with the OHA level, but SSA has not performed any additional
analysis to show that these changes have any relationship to improved
consistency in decision making between the two adjudication levels.
Additional analysis is needed because a myriad of factors, such as changes
in the economy, can affect allowance rates. Although SSA claims that over
this period of time the economy has been "relatively stable," without
performing any additional analysis it cannot eliminate changes in the
economy or demographics of claimants as an influence on the allowance
rates at each level. In addition, SSA has not analyzed how other factors,
such as changes in productivity and total number of decisions made at each
level, may be influencing the allowance data.
6. The allowance rate data provided by SSA in its comments is very
similar to that provided by SSA earlier to us and included in our report
in figure 2. The figures we reported for the proportion of allowances made
by the DDS and OHA levels for fiscal years 1997 and 1998 vary in
comparison with those provided by SSA by one percentage point. We have not
changed the figures in our report because we believe that these slight
differences simply reflect that we reported data based upon fiscal, not
calendar, years.
7. In our report, our statements that SSA has not made changes as a
result of findings from its reviews were specifically related to SSA's ALJ
pre-effectuation review. We included information on this review because it
was part of SSA's process unification initiative and was intended to
identify policy and training areas associated with inconsistent decisions
between adjudication levels. During our review, we were told by an SSA
official that the ALJ pre-effectuation review was not successful at
identifying new areas of inconsistency to be addressed by SSA. In its
comments, SSA cites a review unrelated to assessing the inconsistency of
decisions between levels, the ALJ peer review, to assert that it has used
reviews to identify training issues to improve the quality of decisions.
The lack of success with the ALJ pre-
Appendix II: Comments from the Social Security Administration
effectuation review-along with other findings showing a limited
understanding of the cause of inconsistency-supports our recommendations
to SSA to perform additional analysis and to clarify guidance and provide
mandatory training to address any identified causes of inconsistency
between adjudication levels.
8. We applaud SSA's plans to use the electronic disability system to
capture critical management information to address decisional variance or
inconsistency, which could provide a wealth of useful information for the
agency. We have adjusted our report's text to reflect this additional
purpose. We continue to believe that SSA should not wait for the
development of this system, but should proceed to perform multivariate
analysis, using available data from its biennial case reviews, to start
identifying areas of potential inconsistency between adjudication levels.
9. We applaud SSA's deep commitment to improving the disability
decision-making process, but believe that additional efforts to understand
the causes of potential inconsistencies in decision making would help to
inform the design of the Commissioner's new approach and should,
therefore, be undertaken immediately.
10. We generally agree with the technical comments provided and changed
the text accordingly.
Appendix III: GAO Contacts and Staff Acknowledgments
GAO Contacts
Staff Acknowledgments
(130219)
Michele Grgich, Assistant Director (415) 904-2183 Julie DeVault, Analyst
in Charge (415) 904-2269
In addition to the individuals mentioned above, the following staff
members made major contributions to this report: Michael Morris, Corinna
Nicolaou, Walter Vance, and Rebecca Woiwode. Douglas Sloane provided
assistance with methodological issues, and Daniel Schwimer provided legal
support.
GAO's Mission
Obtaining Copies of GAO Reports and Testimony
The General Accounting Office, the audit, evaluation and investigative arm
of Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability of
the federal government for the American people. GAO examines the use of
public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
The fastest and easiest way to obtain copies of GAO documents at no cost
is through the Internet. GAO's Web site (www.gao.gov) contains abstracts
and fulltext files of current reports and testimony and an expanding
archive of older products. The Web site features a search engine to help
you locate documents using key words and phrases. You can print these
documents in their entirety, including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document files.
To have GAO e-mail this list to you every afternoon, go to www.gao.gov and
select "Subscribe to e-mail alerts" under the "Order GAO Products"
heading.
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:
U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C.
20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
To Report Fraud, Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in E-mail: [email protected]
Federal Programs Automated answering system: (800) 424-5454 or (202)
512-7470
Jeff Nelligan, Managing Director, [email protected] (202) 512-4800
Public Affairs U.S. General Accounting Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
*** End of document. ***