SSA's Disability Programs: Improvements Could Increase the
Usefulness of Electronic Data for Program Oversight (10-DEC-04,
GAO-05-100R).
In 2003, we added the federal government's disability programs to
our high-risk list in part because of difficulties agencies faced
in managing these programs and the expected growth in the rolls
as baby boomers reach their disability-prone years. The Social
Security Administration (SSA) manages the federal government's
two largest disability programs, the Disability Insurance (DI)
program and the Supplemental Security Income (SSI) program, which
together paid out $91 billion in federal benefits to 11.4 million
individuals with disabilities in 2003. To help address management
difficulties and prepare for expected growth in the rolls, SSA
must have reliable administrative data from its disability
decision-making process to adequately understand the population
it serves and the possible effect of proposed program changes on
this population. However, in a prior study, we identified
potential problems with the reliability of SSA's electronic
administrative data. This report examines (1) the extent to which
SSA collects useful and reliable electronic administrative data
in order to effectively manage its DI and SSI programs and (2)
whether ongoing and planned changes to SSA's computer systems and
internal controls will address any weaknesses that we identified.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-05-100R
ACCNO: A14399
TITLE: SSA's Disability Programs: Improvements Could Increase
the Usefulness of Electronic Data for Program Oversight
DATE: 12/10/2004
SUBJECT: Claims processing
Data collection
Data integrity
Disability benefits
Disability insurance
Eligibility determinations
Federal social security programs
Information resources management
Internal controls
Management information systems
Persons with disabilities
Records management
Program management
Decision making
SSA Disability Insurance Program
SSA Supplemental Security Income
Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-05-100R
* Results in Brief
* Background
* While Most Types of Information Collected Are Useful, Unknown Accuracy
Detracts from Their Value
* SSA Captures Important Information for Program Management, but
Additional Information Could Improve Program Oversight
* SSA Does Not Have Adequate Internal Controls to Ensure That the
Electronic Data Accurately Reflect the Facts of the Case
* SSA's Ongoing and Planned Changes May Reduce the
* SSA's Ongoing and Planned Changes May Reduce the
* SSA's Plans Do Not Include Strategies for Ensurin
* Conclusion
* Recommendations for Executive Action
* Agency Comments and Our Evaluation
* Appendix I: Comments from the Social Security Administration
* GAO Comments
United States Government Accountability Office Washington, DC 20548
December 10, 2004
The Honorable Jo Anne B. Barnhart Commissioner, Social Security
Administration
Subject: SSA's Disability Programs: Improvements Could Increase the
Usefulness of Electronic Data for Program Oversight
Dear Ms. Barnhart:
In 2003, we added the federal government's disability programs to our
high-risk list in part because of difficulties agencies faced in managing
these programs and the expected growth in the rolls as baby boomers reach
their disability-prone years. The Social Security Administration (SSA)
manages the federal government's two largest disability programs, the
Disability Insurance (DI) program and the Supplemental Security Income
(SSI) program, which together paid out $91 billion in federal benefits to
11.4 million individuals with disabilities in 2003. To help address
management difficulties and prepare for expected growth in the rolls, SSA
must have reliable administrative data from its disability decision-making
process to adequately understand the population it serves and the possible
effect of proposed program changes on this population. However, in a prior
study, we identified potential problems with the reliability of SSA's
electronic administrative data. 1
This report examines (1) the extent to which SSA collects useful and
reliable electronic administrative data in order to effectively manage its
DI and SSI programs and (2) whether ongoing and planned changes to SSA's
computer systems and internal controls will address any weaknesses that we
identified. To determine the adequacy of SSA's electronic administrative
data for management of its disability programs, we reviewed SSA, Office of
the Inspector General (OIG), and our prior reports regarding these data;
interviewed users of these data; and performed a literature search on
criteria for establishing information databases. To identify the nature
and extent of data problems, we interviewed staff
1GAO, SSA Disability Decision Making: Additional Steps Needed to Ensure Accuracy
and Fairness of Decisions at the Hearings Level, GAO-04-14, Washington, D.C.:
Nov. 12, 2003.
Results in Brief
responsible for managing and using the data to assess the controls and
processes in the disability system, conducted data reliability tests on
selected variables in the disability records for calendar year 2003, and
reviewed relevant reports by OIG and SSA contractors for their assessment
of the data quality. To determine the extent to which SSA has plans or
efforts under way to address these problems, we interviewed SSA officials
and reviewed SSA documents and OIG reports. We conducted our work between
December 2003 and September 2004 in accordance with generally accepted
government auditing standards.
While most types of information collected by SSA are useful for program
management purposes, the agency lacks sufficient internal controls to
ensure that these data are reliable. In its electronic records, SSA
collects information related to the claimant, process, and decision-all of
which are important for informing aspects of program management. For
example, SSA electronically collects information on the claimant's
impairment that is used to identify beneficiaries who should be reviewed
for medical improvement and continuing eligibility for disability
benefits. However, SSA does not collect some information that would
enhance program oversight. For example, without an indicator on claimant's
race, SSA lacks a complete basis for assessing the equity of disability
decisions. At the same time, SSA collects other information in its
electronic disability record that the agency considers of limited value.
For example, codes entered for the claimant's prior occupation and
industry are based on outdated lists, and the agency acknowledges that the
value of that information is therefore limited. Although the types of
information SSA collects in its electronic records generally contribute to
the management of the disability programs, the accuracy of the records is
unknown. For example, inadequate data entry controls permit the entry of
invalid impairment codes-codes that do not reflect a meaningful
impairment. In addition, inadequate data entry controls have resulted in
missing data for some information, such as the claimant's educational
level, which is a critical factor in the disability decision and is
therefore important in a study of the disability decision-making process.
Most important, because SSA's policy does not require that the electronic
record be verified against the information in the case file, the agency
does not know the extent to which codes that appear valid reflect the
claimant's actual information, such as the claimant's actual impairment or
education level.
While SSA's ongoing and planned changes to computer systems and internal
controls may reduce the chance for some inaccuracies, SSA's plans do not
provide adequate assurance of the accuracy of its electronic
Background
administrative data. SSA's ongoing transition from a paper-based
disability system to an electronic one should reduce some of the
inaccuracies we identified by limiting the amount of data re-entered at
various levels in the process. However, SSA's planned changes will not
address data entry problems found by GAO that could be prevented with
additional data entry controls. SSA's current plans also do not include an
internal control strategy for ensuring that electronic data match the
information in the case file, nor do they provide for corrective action
when inaccuracies are found. Finally, although SSA has proposed
far-reaching changes to its disability decision-making process and is
currently reassessing the processes for ensuring the quality of its
disability decisions, the agency has not yet made any plans for evaluating
the types of information it currently collects and whether other types of
information would improve program management and oversight.
To improve the value of SSA's electronic administrative data for managing
its disability programs, we recommend that the agency establish a
costeffective internal control strategy to ensure that those data are
reliable and that they accurately reflect the information in the case
file. In addition, the agency should take steps to review the usefulness
of the types of information collected and consider whether additional
types of information could improve program oversight.
In commenting on the draft of this report, SSA generally agreed with our
conclusions and recommendations. SSA acknowledged that the report made
valid points about steps SSA can take to increase the usefulness of
electronic data for program oversight, and that the agency would consider
how best to incorporate improvements in its data collection activities.
DI and SSI are the two largest federal programs providing cash assistance
to people with disabilities. Established in 1956, DI provides monthly
payments to workers with disabilities (and their dependents) under the age
of 65 who have enough work experience to qualify for disability benefits.
Created in 1972, SSI is a means-tested income assistance program that
provides monthly payments to adults or children who are blind or who have
other disabilities and whose income and assets fall bellow a certain
level. To be considered eligible for either program as an adult, a person
must be unable to perform any substantial gainful activity because of a
medically determinable physical or mental impairment that is expected to
result in death or that has lasted or can be expected to last for a
continuous period of at least 12 months.
The disability determination process is complex, involving more than one
office and decision maker. The process begins at a field office, where an
SSA representative determines whether a claimant meets the programs'
nonmedical eligibility criteria. Claims meeting these criteria are
forwarded to a state's disability determination service (DDS) office to
determine if a claimant meets the agency's medical eligibility criteria.
At the DDS, the disability examiner and the medical or psychological
consultants work as a team to analyze a claimant's documentation, gather
additional evidence as appropriate, and approve or deny the claim. A
denied claimant may ask the DDS for a reconsideration of its finding, at
which point a different DDS team will review the claim. 2 If the claim is
denied again, the claimant may appeal the determination to SSA's Office of
Hearings and Appeals (OHA), where an administrative law judge (ALJ) will
review it. The ALJ usually conducts a hearing in which the claimant and
others may testify and present new evidence. In making the disability
decision, the ALJ uses information from the hearing and from the state
DDS, including the findings of the DDS's medical consultant. A claimant
whose appeal is denied may request a review by SSA's Appeals Council and,
if the claim is denied again, may file suit in federal court.
At each step of the disability determination process, information
collected is entered into the claimant's case file as well as into an
electronic database. SSA field office staff create the case file, which is
the legal record for the disability determination and contains all
documents related to the case. If the claimant meets the nonmedical
eligibility criteria, the field office staff forwards the case file to the
appropriate state DDS office. DDS staff add the information collected at
the DDS level and the initial disability determination to the case file
and enter selected data from the case file into an electronic record known
as the Form 831. Once the DDS submits these 831 records to SSA, the
records become part of SSA's 831 file. 3 If the claimant asks for a
reconsideration, another DDS team will place the new information in the
case file and enter selected data from the case file into a new 831 record
reflecting the reconsideration decision, which will also become part of
SSA's 831 file. If the claimant appeals the
2
While most claimants may request a reconsideration, at the time of our
study, SSA was testing an initiative that eliminates the reconsideration
step from the DDS decision-making process.
3
SSA's database of 831 records is called the National Disability
Determinations Service System (NDDSS) but is generally referred to as the
"831 file." Therefore, in this report, we use the term "831 file" or "831
database" to refer to the NDDSS.
Page 4 GAO-05-100R SSA's Disability Programs
reconsideration by requesting a hearing by an administrative law judge at
the Office of Hearings and Appeals, the ALJ conducts a new review of the
claimant's file and any additional evidence the claimant submitted since
the DDS determination. The ALJ may also hear testimony from medical or
vocational experts and the claimant regarding the claimant's medical
condition and ability to work. As with the DDS, OHA staff place the new
information in the case file and enter selected data from the case file
into an electronic record. These electronic records become part of a
nationwide electronic database referred to as the Case Control System
(CCS) database.
It has been well established that adequate internal controls over
electronic data systems are necessary to effectively manage programs, such
as SSA's disability programs. In 1998, the Social Security Advisory Board
noted that it is essential for policy makers have accurate, balanced, and
objective information to help them determine the extent to which the
program is meeting the long-standing objectives of social adequacy and
individual equity, the nature and extent of changes that may be needed,
and the impact of proposals for change. SSA stated in its 2003 Performance
and Accountability Report that the agency was committed to providing clear
and reliable data to those who use the data for managing, decision making,
and oversight of SSA's programs. Finally, GAO's standards state that
adequate internal controls should be established to provide reasonable
assurance that agency objectives are being achieved. These internal
control standards define the minimum level of quality acceptable for
internal controls in the government and include controls over information
processing and management review of performance.
SSA is implementing significant changes to its disability determination
process, and considering numerous others, in order to improve speed and
efficiency. Significant changes currently under way include the Electronic
Disability System (eDib)-which SSA considers to be a foundation for other
initiatives currently under consideration. With eDib, all components
involved in disability claims adjudication and review-including SSA field
offices, DDSs, OHA, and its Office of Quality Assurance-will use an
electronic disability folder that replaces the paper case file. The
electronic folder will contain an electronic copy of everything that was
formerly included in the paper case file, such as electronic images of
medical
While Most Types of Information Collected Are Useful, Unknown Accuracy Detracts
from Their Value
evidence and other documents related to the disability determination. 4
The electronic folder will also contain electronic images of the 831 or
CCS records. 5 When the process is fully implemented, each component will
be able to electronically view and process claims using the electronic
disability folder. Other initiatives under consideration include
centralizing medical expertise in regional expert review teams to expedite
claims where the claimant is clearly disabled, requiring the DDSs to more
fully document decisions and provide in-line quality review, eliminating
the DDS reconsideration and Appeals Council decisional levels, and
establishing end-of-line quality review at OHA.
Although, for the most part, the types of information SSA collects in its
electronic records contribute to the management of the disability
programs, the accuracy of the records is unknown. In its electronic
records, SSA collects information related to the claimant, process, and
decision, which informs aspects of program management. However, SSA does
not collect some information that would enhance program oversight. At the
same time, SSA enters codes in its electronic disability records for the
claimant's prior occupation and industry, even though these codes are
based on outdated lists of occupations and industries, and the agency
acknowledges that the value of that information is therefore limited.
While SSA has some internal controls to help capture reliable data, these
controls are not sufficient to ensure those data are reliable. SSA's
policy does not require that the electronic record be verified against the
information in the case file, and inadequate data entry controls have also
resulted in some inaccurate and missing data.
4
Electronic images are like electronic photocopies of the documents. They
can be retrieved or viewed in their entirety, but the elements that
constitute the image-such as words or numbers-are not in a format that
allows them to be identified using electronic search mechanisms or readily
retrieved and manipulated for reporting or analytical purposes.
5
Although these electronic images cannot be manipulated, data keyed into
the 831 and CCS databases are in a format that allows for easy
identification, retrieval, and manipulation for reporting or analytical
purposes.
SSA Captures Important Information for Program Management, but Additional
Information Could Improve Program Oversight
In its electronic disability records, SSA collects important information
about the claimant, the process, and the decision. For example, SSA
electronically collects information on individuals who apply for
disability benefits, such as the applicant's name, Social Security number,
gender, date of birth, state of residence, and type of benefits requested.
It also collects information on the various events in the disability
determination process, such as the application and decision dates, as well
as the office processing the application and the adjudicative level
involved. Finally, it collects information on the disability decision,
such as the date the disability began, the primary impairment used in the
medical determination of eligibility, and the regulation basis code for
the disability decision that indicates the specific DI or SSI program
requirements and whether they were met.
This electronic disability information is analyzed and used to manage the
DI and SSI programs. For example, SSA uses the data to monitor the time it
takes the agency to process a claim and to determine the number of cases
allowed and denied at each adjudicative level. From the 831 and CCS
databases, SSA selects (randomly or based on particular characteristics)
cases for review to help the agency ensure the accuracy of disability
determinations. For each selected case, staff from SSA's Office of Quality
Assurance obtain and review the paper case file to ensure that the DDS or
ALJ disability decision is accurate. 6 SSA also uses the data to determine
when beneficiaries should be reviewed for continuing disability and what
type of review is cost-effective. The disability data may also be used to
study specific aspects of its disability determination process, and other
issues, such as the prevalence or handling of specific illnesses, like
chronic fatigue syndrome.
The electronic disability data are also used to inform the Congress and
the general public about the administration of the DI and SSI programs.
For example, SSA uses these data to prepare the Annual Statistical
Supplement, which is a major data resource on SSA-administered programs
that serves as a foundation for various research and policy analyses and
helps SSA to respond to requests for program data from congressional
committees, government agencies at all levels, and the research community.
Further, SSA uses these data to provide the Congress
6
Once eDib is fully implemented, instead of reviewing a paper case file,
quality assurance staff will review electronic documents in the electronic
disability folder, including medical records and other information related
to the disability determination.
Page 7 GAO-05-100R SSA's Disability Programs
with an annual report on the status of the Supplemental Security Income
program, including 25-year projections of program participation prepared
by SSA's Office of the Chief Actuary. SSA's Office of the Chief Actuary
uses these data to analyze demographic trends in mortality and morbidity
rates, in order to project program growth and financial status.
SSA shares its electronic disability data with other entities for research
and analysis purposes. For example, the Disability Research Institute,
which is funded through a cooperative agreement between SSA and the
College of Applied Life Studies at the University of Illinois at
Urbana-Champaign, uses SSA's data to conduct in-depth research to directly
inform disability policy decision makers. In addition, SSA's
administrative and program data have also been used by outside researchers
and academics to study relationships between and among various
characteristics of people with disabilities, including their work
histories, impairments, and outcomes. 7 SSA's databases have the advantage
of being large enough to allow researchers to make statistically
meaningful distinctions among subgroups of disabled individuals.
While SSA has many uses for the information it collects, there is some
information not being collected that could enhance program oversight. For
example, after identifying unexplained racial differences in decisions
made at the OHA level, we recommended that SSA collect information on race
so that the agency can better monitor the equity of its decisions. 8 In
this same study, we also recommended that SSA collect information on the
source of medical evidence (such as specialist, primary physician,
emergency physician) and type of medical evidence (such as blood test,
xray, observation, opinion), which could be used to analyze and better
understand the role the agency and claimant representatives play in
developing evidence for the disability determination.
Outside researchers and academics using SSA disability data include Kajal
Lahiri, Denton
R. Vaughan, and Bernard Wixon, "Modeling Social Security's Sequential
Disability Determination Using Matched SIPP Data," Social Security
Bulletin, 58, 3-42; Congressional Research Service Report to the Congress
(1992), "Status of the Disability Programs of the Social Security
Administration," 92-691 EPW, the Library of Congress, Washington, D.C.;
and Eddy Bresnitz, Howard Frumkin, Lawrence Goldstein, David Neumark,
Michael Hodgson, and Carolyn Needleman, "Occupational Impairment and
Disability among Applicants for Social Security Disability Benefits in
Pennsylvania," American Journal of Public Health 84 (November 1994),
1786-1790.
8GAO-04-14. SSA officials told us that the agency formed a work group in
September 2003 to develop a plan for collecting race/ethnicity
information.
An SSA contractor also found the agency's electronic medical information
lacking for other purposes. In 2001, PricewaterhouseCoopers (PwC) reported
on the quality of electronic disability data used to target cases for
continuing disability reviews (CDRs)-i.e., reviews of medical improvement
of beneficiaries to determine their continuing eligibility for disability
benefits. In its report, PwC stated that although SSA collects a large
amount of medical evidence for SSA disability decisions, very little of
this medical information is captured in SSA's electronic disability
databases. Specifically, PwC noted that only 8 of the 162 variables
captured in the 831 record relate directly to medical condition. PwC
stated that despite the importance of medical information to any decision
on continuing eligibility, a relatively small number of medical variables
are available in SSA's 831 record for use in determining the most
cost-effective method to target cases for CDRs. 9
In addition to medical information, SSA lacks electronic data on
claimants' vocational and functional capacity, which are critical factors
in the disability decision and are therefore important in a study of the
disability decision-making process. 10 Nearly 60 percent of disability
decisions cannot be made based solely on medical considerations and must
rely on these additional factors. Nevertheless, in the CCS database, we
found that no variable is related to vocational and functional capacity.
For the DDS level, we found one variable in the 831 database that provides
limited vocational and functional information. 11 The 831 database has no
other variables that capture additional factors often considered in making
disability decisions (such as mental conditions, fine motor skills, and
stamina). Moreover, for DDS decisions made in 2003, we found that the one
variable related to vocational and functional factors is captured in only
10 percent of the cases in which such factors had to be considered for the
disability
9
In targeting cases for CDRs, SSA also uses information contained in the
Master Beneficiary Record system and the Supplemental Security Record
system, as well as information from Medicare claim forms and individual
responses to prior CDR mailer forms.
10
Vocational factors are age, education and work experience. A claimant's
residual functional capacity is the individual's maximum sustained
capability for sedentary, light, medium, heavy, or very heavy work.
11
The vocational rule number variable indicates certain vocational and
functional patterns that include functional capacity, age, work history,
and education level.
determination. 12 In the remaining 90 percent of these cases, DDS staff
did not capture any vocational or functional factors used in the decision.
Without this type of information, SSA lacks electronic data on critical
factors considered in the disability decision and important to any study
of the disability decision-making process.
Mathematica-a contractor hired by SSA to evaluate new strategies for
promoting employment among people with disabilities-also noted that
important vocational and functional information was not being collected
electronically by SSA. Specifically, Mathematica reported that certain
characteristics (such as household composition, occupation, industry, and
the presence of functional limitations) are, according to the research
community, important predictors of employment, but they are not available
in SSA's electronic databases.
While SSA does not collect many types of information that contractors and
others believe would be useful for managing the agency's programs, SSA may
be collecting other data that are not particularly useful. SSA officials
told us that they believe information on the claimant's prior occupation
and industry could be useful, but because the lists of industries and
occupations used by SSA are largely outdated, the agency considers these
variables to be of limited value. Nevertheless, some DDS staff members
still collect information on the claimant's prior occupation and industry.
13
12
SSA uses a five-step disability determination process whereby the agency
considers vocation and functional capacity in steps 4 and 5, i.e., when it
has determined that the claimant has a severe, medically determinable
physical or mental impairment, but the impairment is not severe enough for
the individual to be determined eligible based on medical factors alone.
In steps 4 and 5, SSA uses these additional factors to determine whether
the claimant's impairment(s) prevents the performance of his or her past
work or other work in the national economy. Federal regulations contain
rules (vocational rule numbers) that direct a finding of disabled or not
disabled based on vocational factors (such as age, education, and work
experience) in combination with the individual's residual functional
capacity.
13
In 1992, when the DDS workload was heavy, SSA informed the DDSs that the
variables for industry, occupation, and years in occupation did not need
to be filled in. Although this was intended to be a temporary measure,
SSA's incentive for re-instituting the policy diminished as the job
classifications grew more outdated over time.
SSA Does Not Have Adequate Internal Controls to Ensure That the Electronic Data
Accurately Reflect the Facts of the Case
While SSA emphasizes the importance of prompt and accurate disability
determinations, the agency has not put sufficient emphasis on creating a
positive control environment that would ensure a reasonable level of
quality for its electronic disability data. SSA officials told us that
although the DDSs and OHA are required to ensure that disability
determinations are accurate, there is no specific requirement that
electronic records provided to SSA be accurate. Regulations specify only
that the DDSs must "provide the organizational structure, qualified
personnel, medical consultant services, and a quality assurance function
sufficient to ensure that disability determinations are made accurately
and promptly." SSA officials told us that while some DDS offices may
verify the accuracy of electronic data, SSA does not require or suggest
that the DDSs follow a specific strategy or format for doing so. Some DDS
and OHA staff and supervisors told us that they were unaware of how SSA
used the 831 and CCS records or why it was important that they be
accurate. If SSA required verification on at least a sample of cases, it
would indicate to DDS and OHA staff the importance of accuracy in the
electronic data.
SSA has some electronic controls in place to help ensure the quality of
data. These front-end data entry controls help ensure data quality by
limiting what can be keyed into certain variables. For example, some
variables accept only a numeric entry, and others, such as the Social
Security number, must be filled in or the record will be rejected. Many
date variables require a month/date/year entry, and certain dates must
fall within a prescribed range. For example, the application date must be
on or after a person's date of birth. In addition to these front-end data
entry controls, SSA also reviews the 831 file by checking for unacceptable
entries for certain variables. If SSA finds an increasing trend of
unacceptable entries being keyed into those variables, the agency may
issue additional guidance to staff in the field. On the basis of the
results of this review, SSA officials told us that while the data may not
be 100 percent accurate, they believe the data are sufficiently accurate.
However, while such controls may reduce data entry errors, we believe the
data entry controls are inadequate. In reviewing SSA's electronic records
for disability decisions made by the DDSs and OHA in 2003, we found that
even though information keyed into the record had not been rejected by the
data entry controls, the record contained information that was clearly
wrong because it did not agree with other information in the same record.
14 This type of comparison was possible for only a few variables-
impairment code, body system, the code indicating the basis for the
disability decision, and the disability decision itself-but these are
important variables that contain information crucial to SSA's management
of its disability programs. Although our analysis identified only a small
number of errors, the errors were in key variables. For example, GAO found
that
o More than 4 percent of DDS records for claims processed in 2003
contained codes for the body system and impairment that were
incompatible, indicating that either the impairment code or the body
system code was incorrect.
o For 211 claims allowed by the DDSs and 1,541 claims allowed by OHA,
the impairment code indicated an unknown impairment for which there
was no medical evidence.
o For 239 claims allowed by the DDSs and 519 claims allowed by OHA, the
impairment code indicated an unknown impairment for which there was
insufficient medical evidence.
o For nearly 4 percent of OHA's electronic disability records for
allowed claims, the impairment codes did not appear in the OHA list of
impairment codes.
o OHA's records indicated that 2,367 applicants, who applied for both
the DI and SSI programs, met SSA's medical criteria for one program
but not the other-a situation not allowed under SSA regulations, which
require that the same medical criteria apply to both programs.
Nevertheless, for each of these applicants, OHA recorded different
decision codes, primary impairment codes, or regulation basis codes
for the two programs.
Inadequate controls have also resulted in missing data. In reviewing the
831 and CCS records, we found that for many of the variables, leaving the
variable blank is considered a valid entry. Therefore, when variables are
not filled in, it's not possible to determine if staff had the information
but failed to key it in, or if staff did not have the information and
correctly left
We reviewed 3.9 million 831 records for disability decisions made in 2003,
of which 31 percent (1.2 million) were allowed. We also reviewed over
400,000 CCS records for disability decisions made in 2003, of which 70
percent (nearly 282,000) were allowed.
Page 12 GAO-05-100R SSA's Disability Programs
it blank. Although for most variables we were unable to determine that the
variable should have been filled in, the education variable was an
exception. For cases in which the adjudicator is required to consider
educational level in making the disability determination, the information
should have been obtained and keyed into the record. 15 Yet, for 25
percent of such cases, that variable was left blank. This raises the
question of how much of the missing data for other variables was available
and should have been keyed into the record.
Other organizations have also found weaknesses in SSA's data entry
controls. In its 2001 study for SSA, PwC reported that poor data quality
could present a significant barrier to efficient CDR case selection. As a
result, it recommended that the agency improve system data edits, stating
that such data entry controls would improve the integrity of disability
programs, reduce inaccuracies and inconsistencies in data collection and
reporting, and reduce the risk of error or erroneous data collection and
reporting. PwC estimated that between 2001 and 2005, poor and missing
disability data would cause SSA to improperly target CDRs and thereby
incur additional administrative and program costs. An OIG report in 2000
reported that missing or invalid diagnostic codes had contributed to SSA's
failing to perform mandatory reviews of certain disability cases and
prevented the agency from accumulating and disseminating more accurate
disability statistics.
Even when variables have been filled in and accepted by existing data
entry controls, SSA does not know the extent to which the data reflect the
information in the case file because the agency's policy does not require
that the electronic record be verified against the information in the case
file. SSA's case files contain all documents related to the claim (such as
medical records) and represent the legal record of the claim. Although we
did not perform a comparison of electronic records against actual case
files, a study by SSA's OIG in 2000 provides an example of the type of
inaccuracies that may exist in disability records, even when the entries
have been accepted by data entry controls. The OIG reviewed case files for
a sample of 132 disability beneficiaries whose electronic records
contained questionable impairment codes. It found 63 electronic records
that contained an impairment code indicating, "diagnosis established-no
predetermined list code of medical nature applicable," although the case
Over 50 percent of all claims processed by DDSs in 2003 required that
educational level be considered in making the disability determination.
Page 13 GAO-05-100R SSA's Disability Programs
SSA's Ongoing and Planned Changes May Reduce the Chance for Some Inaccuracies,
but the Agency Has No Plans for Improving the Accuracy or Usefulness of
Electronic Data
file contained enough information to determine the proper impairment code.
At the same time, the OIG sample was quite small and not intended to be
representative. It is not cost-effective to check all, or even a large
percentage of, electronic records against case files, and perfect
reliability is not possible. However, without comparing a more
representative sample of electronic records against case files, SSA cannot
reliably estimate the true extent of inaccuracies in the electronic data.
SSA's ongoing and planned changes to computer systems and internal
controls may reduce the chance for some inaccuracies. However, these
planned changes will not address data entry problems found by GAO that
could be prevented with additional data entry controls. SSA's current
plans also do not include an internal control strategy for ensuring that
electronic data match the information in the case file, nor do they
provide for corrective action when inaccuracies are found. Finally,
although SSA has proposed far-reaching changes to its disability
decision-making process and is currently reassessing the processes for
ensuring the quality of its disability decisions, the agency has not yet
made any specific plans for evaluating the types of information it
currently collects and whether other types of information would improve
program management and oversight.
SSA's Ongoing and Planned Changes May Reduce the Chance for Some Inaccuracies
Although most of SSA's ongoing and planned changes do not involve aspects
of the electronic data systems touched on in this report, a few will alter
the way information is transmitted and, in this way, could reduce the
number of chances for unintentional errors. Under eDib, SSA field office
staff keying in the disability history for an applicant will now use the
Electronic Data Collect System (EDCS). Once the field office staff have
keyed the Social Security number into the EDCS record, EDCS automatically
verifies that there is a disability application in existing SSA databases.
Information such as the application date, disability allegations, and
disability onset date are automatically propagated into the EDCS record.
This will improve the accuracy of the information keyed into the EDCS
record, which goes into the electronic disability folder used at all
levels of the disability determination process. Some additional data entry
controls that are part of the new OHA case tracking software (the Case
Processing Management System-CPMS) that was implemented this year as part
of eDib may also reduce data entry errors.
SSA's Plans Do Not Include Strategies for Ensuring Data Accuracy or Evaluating
Types of Information Needed
SSA's current plans do not include an overall internal control strategy
for ensuring the accuracy of electronic disability data. While SSA has
included some front-end data entry controls, it does not address problems
with electronic data we found that could be prevented with additional
controls. SSA also has no plans to improve its system for the back-end
review of electronic disability records, so that the agency can estimate
the accuracy of its electronic disability records and provide feedback to
staff on accuracy. Further, the agency has no specific plans to evaluate
the types of information collected during the disability determination
process.
Most of the changes currently under way-such as creating a shared
electronic folder for each disability case, scanning evidence for
inclusion in the electronic folder, providing new software systems, and
improving interfaces between systems-were intended to improve the speed
and efficiency of the disability claims process and not necessarily to
improve the reliability or usefulness of the data in SSA's 831 and CCS
databases. To accelerate the rollout of eDib, SSA carried over the
requirements from the old systems to the new systems. Although the agency
did introduce some additional front-end data entry controls to reduce the
entry of invalid data, the agency did not add sufficient controls to
address the problem with missing and invalid data we found. An OHA
official told us that even the additional data entry controls available in
the new CPMS software are not adequate to prevent data entry errors we
identified in this study. For example, under eDib, SSA's data entry
control still allows any four digits for the impairment variable in the
CPMS, as well as the 831 file, rather than limiting data entry to one of
the 240 acceptable impairment codes. These officials said that limited
resources have prevented the agency from establishing and maintaining
additional or more exacting controls-such as a current list of acceptable
impairment codes. Moreover, SSA officials told us that additional data
entry controls would require staff to determine the correct entry when
errors were identified, which could slow case processing.
SSA also has no plans to improve its system for the back-end review of
electronic disability records, so that the agency can estimate the
accuracy of its electronic disability records and provide feedback to
staff on accuracy. SSA already has a process in place for reviewing a
random sample of case files as part of its quality assurance for
disability decisions, which allows the agency to make estimations of the
accuracy of decisions at both the DDS and the OHA levels. In the course of
this quality assurance review, staff are supposed to verify the electronic
831 record against the information in the case file and note errors. 16 If
a significant error is found, staff are directed to fax a corrected 831
form to a central location so that the correction can be made in the 831
file. However, SSA officials told us that corrections don't always end up
in the 831 file, and the agency does not track input errors because such
errors are low on the priority list. Because there is no tracking system
that would collect statistical data on the number or type of errors, SSA
is unable to determine the level of accuracy for this electronic data. A
tracking system that provided feedback to the field would also send a
message to staff that the items being tracked are important-a basic
element in any quality assurance system. Without the statistical data and
feedback to staff, the reviews for data quality affect only those records
reviewed and do not affect the possible quality of the rest of the
records. Finally, the back-end review covers only 831 records, not CCS
records.
SSA has not yet made concrete plans for evaluating the types of
information it currently collects and whether other types of information
would improve program management and oversight. Users of administrative
data that we interviewed indicated that they have not been surveyed as to
their data needs, and some indicated that their requests for additional
data collection have not yet been implemented. However, the agency may
take such steps after other priorities are accomplished. An SSA official
stated that SSA's first priority was to implement eDib, in order to
modernize its core business process of taking and adjudicating claims. The
official also stated that after eDib is fully implemented, SSA will assess
whether the information collected is sufficient.
In more recent interviews, high-level SSA officials validated the
importance to the agency of reliable and useful data for program
management purposes, and indicated a strong interest in developing a
comprehensive information management plan that would allow the agency to
systematically and regularly assess its information needs and make
appropriate adjustments to its computer systems. According to agency
officials, this effort would include a review of best practices in
information management. However, the ideas expressed were still being
formulated, and the agency has not yet taken any concrete actions.
Under eDib, such a back-end review would involve matching the 831 or CCS
record against the electronic images of medical records and other
documents related to the disability determination.
Page 16 GAO-05-100R SSA's Disability Programs
Conclusion
SSA lacks a comprehensive strategic plan for collecting useful and
reliable data for effective oversight and program planning. Despite the
fact that the reliability of its data has been an ongoing issue, the
agency currently has no specific plans to implement additional front-end
data entry controls or for a tracking and feedback system for the back-end
verification of the electronic records. In addition, the agency has no
specific plans for evaluating whether the types of information it
currently collects continue to be useful for program oversight and whether
other types of information would contribute to oversight.
While stretched resources and other priorities may have prevented SSA from
addressing these issues to date, SSA plans to make more changes to its
disability processes, which will create additional opportunities and a
sense of urgency for improving the usefulness and reliability of its
electronic disability data. For example, in September 2003, SSA's
Commissioner proposed significant changes to its disability decisionmaking
process-such as the proposed use of centralized medical expertise and full
documentation of initial-level decisions-that will, if implemented,
translate into new requirements for collecting administrative data to
monitor and assess these changes. Moreover, as SSA continues to transition
from a paper-based case-processing system to an electronic one, SSA will
have additional opportunities to institute changes that would improve upon
the usefulness and reliability of the data that it routinely collects.
Finally, SSA has been considering fundamental changes to its quality
assurance processes, at both the front and back ends of its disability
determination process. Combined, these changes, each significant and
interrelated, create a compelling case for SSA to broadly assess and
improve the nature and reliability of the data it uses to manage its
disability programs.
Recommendations for Executive Action
To improve the value of SSA's electronic administrative data for managing
its disability programs, we recommend that the Commissioner of Social
Security develop a comprehensive strategic plan to ensure the reliability
and usefulness of the data the agency collects. In doing so, the agency
should take the following steps:
1. Establish a cost-effective internal control strategy for ensuring the
reliability of data in the electronic disability records that would
include both front-end controls on data entry and a tracking and feedback
system for back-end verification of the electronic records.
Agency Comments
and Our Evaluation
2. Take steps to review the usefulness of the types of information
collected and consider whether additional types of information could
improve program oversight. This effort could include a survey of users of
electronic disability data.
We provided a draft of this report to SSA for comment. SSA generally
agreed with our conclusions and recommendations, acknowledging that the
report made valid points about steps SSA can take to increase the
usefulness of electronic data for program oversight and that the agency
would consider how best to incorporate improvements in its data collection
activities.
SSA agreed in theory with our first recommendation and noted that the
agency is continuing to examine future software enhancements to improve
front-end controls on data entry and back-end review. SSA also noted that
its back-end review of cases may provide some check on data entry. We
agree and modified our report and first recommendation to further clarify
this. However, SSA expressed concern regarding the cost and technical
feasibility of implementing this recommendation. We agree that the agency
needs to balance enhanced internal controls with operational costs and
productivity, but we believe some of the specific concerns raised by SSA
are unwarranted. For example, as discussed in our response to SSA's
comment in appendix I, we believe that the internal controls proposed by
GAO would not require additional "structured" data or a data collection
system "robust enough to catalog and aggregate the information."
In its response to our second recommendation, SSA agreed that the
disability program would be well served by reviewing all the information
that SSA could potentially collect on a disability claim and determining
which of these data elements might be beneficial for program oversight and
policy development. SSA also stated that the agency is drafting a
statement of work to contract for an assessment of data needs for managing
the disability programs, which will identify possible sources and methods
for obtaining data and make recommendations for improving existing data
and data retrieval. The agency further observed that it would have to
consider the potential adverse effect of increased electronic data
collection on productivity and timeliness.
SSA provided additional general comments, which we have included (along
with our responses to them) in appendix I and addressed in the body of our
report where appropriate. SSA also provided technical comments that we
have incorporated in the report as appropriate.
We are sending copies of this report to the appropriate congressional
committees and other interested parties. We will also make copies
available to others on request. In addition, the report will be available
at no charge on GAO's Web site at http://www.gao.gov.
If you or your staff have any questions concerning this report, please
contact me or Michele Grgich, Assistant Director, at (202) 512-7215. You
may also reach us by e-mail at [email protected] or [email protected].
Other major contributors to this assignment were Ann T. Walker, Jill D.
Yost, and Corinna Nicolaou.
Sincerely yours,
Robert E. Robertson
Director, Education, Workforce, and Income Security Issues
Appendix I: Comments from the Social Security Administration
Note: GAO comments supplementing those in the report text appear at the
end of this appendix.
See comment 1.
See comment 2.
See comment 3.
Page 21 GAO-05-100R SSA's Disability Programs
See comment 4.
See comment 5.
See comments 1, 2, 6 and 7.
See comment 6.
Page 22 GAO-05-100R SSA's Disability Programs
See comment 7.
See comment 6. See comments 1 and 2.
Page 23 GAO-05-100R SSA's Disability Programs
Page 24 GAO-05-100R SSA's Disability Programs
Appendix I: Comments from the Social Security Administration
1. We acknowledge that the Disability Quality Branches' (DQB) back-end
GAO Comments
review of cases may provide some check on data entry, and we have adjusted
our recommendation and text in our report to clarify this. However, SSA
officials told us that tracking of input errors is low on DQB's priority
list, and DQB changes don't always end up in the 831 file. Further, as SSA
indicated, it does not track input errors for statistical purposes;
therefore SSA does not know the level of data accuracy. A tracking system
that also provides feedback to the field would also send a message to
staff that the items being tracked are important-a basic element in any
quality assurance system. Without the statistical data and feedback to
staff, the reviews for data quality affect only those records reviewed,
and do not affect the possible quality of the rest of the records.
Finally, the back-end review by DQB covers only 831 records (i.e., DDS
decisions), not CCS records (i.e., ALJ decisions).
1. Our report recognizes that as part of the implementation of eDib, SSA
is making software enhancements that may address some data entry
problems. However, we believe that the eDib enhancements fall short of
an effective overall strategy. Our previous comment explains why we
believe SSA's back-end reviews do not sufficiently ensure that
electronic data match the information in the case file.
2. We amended the report to include information about the work group.
3. We have incorporated into our report the additional sources of
information used in the CDR targeting process.
4. We recognize that the agency has improved data capture policies and
practices over the years, but in our study, we examined electronic
disability data for 2003, the most recent year available, and found
problems with that data. In addition, without an effective system for
verifying the electronic record against the case file and a tracking
system for errors, neither SSA nor we can determine the extent to
which the electronic disability data are accurate. Further,
PricewaterhouseCoopers' projection that SSA would incur hundreds of
millions of dollars in CDR administrative and program costs between
2001 and 2005 because of poor and missing electronic disability data
was made despite its acknowledgment that SSA's electronic disability
data had improved. In order to not overstate PwC's findings, we
adjusted the text of our report to say that the agency would incur
"additional administrative and program costs" because of problems with
the quality of its electronic disability data.
Appendix I: Comments from the Social Security Administration
1. We do not agree that additional "structured" data are necessary before
data quality can improve. The 831 record is an electronic version of
the SSA-831 form, in a structured data format. For cases processed
prior to the use of the electronic case folder, a printout of the
completed SSA831 form was placed in the paper case file to document
the final decision. Under eDib, which uses an electronic case folder,
an electronic image ("tiff" image) of the SSA-831 form is created and
placed in the electronic case folder (along with other electronic
images such as medical records) to document the final decision.
Regardless of this 831 image, the original 831 record still exists as
structured data and becomes part of a national database known as the
"831 file." SSA is still able to aggregate and report on these data.
Therefore, it is possible for SSA to implement our recommendation that
it compare the information in the electronic 831 record against the
information in the case file or the electronic case folder, and
correct the electronic 831 record as needed. Even after the agency
eliminates paper source documents and relies totally on electronic
documents in the case folder, SSA will still be able to compare the
electronic 831 record against the electronic images of medical records
and other documents related to the disability determination that are
in the case folder.
2. We agree that it may be difficult to determine the primary impairment.
However, our concern is that once a disability examiner makes the
decision, the 831 record should accurately reflect the disability
decision. For example, if the primary impairment is determined to be
diabetes, then the 831 record should contain the four-digit impairment
code for diabetes and the appropriate body system code for that
primary impairment. SSA could develop data entry controls that would
allow only acceptable impairment codes and the corresponding body
system code. The agency would have to update those listings only when
changes in policy dictated changes in impairment codes or body system
codes. This type of front-end data entry control should not require a
"data collection system robust enough to catalog and aggregate the
information."
(130348)
*** End of document. ***