Social Security Administration: Subcommittee Questions Concerning
Efforts to Automate the Disability Claims Process (05-SEP-03,
GAO-03-1113R).
This letter responds to a Congressional request on August 12,
2003 that we provide answers to questions relating to our July
24, 2003, testimony. In that testimony, we discussed the risks
that the Social Security Administration (SSA) faces in its
efforts to automate its disability claims process. The questions
concerned money saved by implementing an electronic disability
folder, SSA's development of a risk management plan, areas which
SSA could improve, and difficulties in completing the project.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-03-1113R
ACCNO: A08352
TITLE: Social Security Administration: Subcommittee Questions
Concerning Efforts to Automate the Disability Claims Process
DATE: 09/05/2003
SUBJECT: Data bases
Risk management
Cost analysis
Strategic planning
Systems development life cycle
SSA Accelerated Electronic Disability
Initiative
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-03-1113R
A
September 5, 2003 The Honorable E. Clay Shaw, Jr. Chairman, Subcommittee
on Social Security Committee on Ways and Means House of Representatives
Subject: Social Security Administration: Subcommittee Questions Concerning
Efforts to Automate the Disability Claims Process
Dear Mr. Chairman: This letter responds to your August 12, 2003, request
that we provide answers to questions relating to our July 24, 2003,
testimony. 1 In that testimony, we discussed the risks that the Social
Security Administration (SSA) faces in its efforts to automate its
disability claims process. Your questions, along with our responses,
follow. 1. The Social Security Administration (SSA) has indicated that the
agency could potentially save $1 billion, at an estimated cost of
approximately $900 million, by implementing an electronic disability
folder. In your opinion, does the $900 million cost of this project appear
to be a reasonable estimate? Is the $1 billion in savings a reasonable
estimate? Is there a possibility that the cost of the project could
balloon?
Our work to date on SSA*s February 2003 cost- benefit analysis raises
concerns that SSA may have underestimated its accelerated electronic
disability (AeDib) system costs. For example, the cost- benefit analysis
did not fully consider the costs associated with certain critical
information technology infrastructure elements supporting the nationwide
rollout, such as scanning and imaging by the outsourced vendor,
telecommunications, disaster recovery, and on- site retention and
destruction of source documents, such as medical records. Because SSA has
not yet fully estimated these costs, we are unclear about their magnitude.
1 U. S. General Accounting Office, Electronic Disability Claims
Processing: Social Security Administration*s Accelerated Strategy Faces
Significant Risks, GAO- 03- 984T (Washington, D. C.: July 24, 2003).
We are also concerned that the corresponding benefits cited in SSA*s
costbenefit analysis may be overstated. Specifically, our review found
that certain assumptions used in the analysis could be too optimistic. For
example, SSA estimated benefits based on an assumption that state
Disability Determination Services (DDS) would receive 30 percent of all
medical evidence in electronic form by 2004. However, state DDS officials
with whom we spoke contend that their offices currently receive about 11
percent or less of medical evidence electronically, and disagree with the
30percent assumption.
As to whether the costs of AeDib could balloon, the possibility of cost
increases exists for any project the size and magnitude of AeDib. As
mentioned previously, the existing estimates do not include certain costs,
such as some costs for outsourced scanning and imaging and the cost of
disaster recovery. Including these could add to SSA*s overall cost
estimate. Our work analyzing the costs and benefits of AeDib is ongoing;
the final
results will be included in our report to be issued to you later this
year. 2. According to your testimony, the SSA has developed a risk
management plan, but is still without a comprehensive assessment of risk
that could affect the electronic disability folder. Can you explain the
difference between the risk management plan and a comprehensive assessment
of risks? How does their risk management plan fall short of what GAO
recommends? Why is this plan so important?
A risk management plan provides guidance to project management teams and
requires them to proactively identify facts and circumstances that could
increase the probability of failing to meet project commitments, and take
steps to prevent this from occurring. 2 A comprehensive assessment of
risks, which is completed according to the risk management plan, is the
process of identifying risks with a high probability and cost of failure,
and developing strategies for mitigating those risks. 3 2 U. S. General
Accounting Office, Information Technology: Greater Use of Best Practices
Can Reduce Risks in Acquiring Defense Health Care System, GAO- 02- 345
(Washington, D. C.: September 26, 2002). 3 Social Security Administration,
SSA AeDib Risk Management Plan, Version 2.01 (Baltimore, MD, January 21,
2003).
Based on our work to date, we do not have concerns about SSA*s risk
management plan, which was developed in accordance with our own and Office
of Management and Budget (OMB) guidance. However, as mentioned in our
testimony, we were concerned that SSA had not completed a subsequent
comprehensive risk assessment. SSA officials agreed with the need for a
comprehensive risk assessment, and since our testimony, the agency has
provided us with detailed risk assessments for four of the five AeDib
projects* the Electronic Disability Collect System, the Document
Management Architecture, a DDS systems migration and electronic folder
interface, and Internet disability applications. SSA has not yet provided
us with a time frame for completing the remaining risk assessment for its
Hearings and Appeals case processing management system.
A risk management plan is an essential tool used to guide the development
of a comprehensive assessment of risks and mitigation strategies. A
comprehensive risk assessment is equally critical. Both tools can help SSA
avoid potential problems before they manifest themselves in cost,
schedule, and performance shortfalls.
3. The GAO has identified a number of areas where the SSA could improve.
For instance, the agency needs to develop risk assessment tools and could
take additional measures to involve key stakeholders in the systems
development process. Is there any indication that the SSA is acting on
your recommendations?
SSA has acknowledged our concerns relative to the areas of improvement we
identified in our testimony, and has taken some action to address them.
However, more work remains to fully address these issues.
Specifically, we expressed the need for SSA to perform a comprehensive
risk assessment to identify project risks and establish mitigation
strategies for them. As noted in our response to question 2, SSA recently
provided us with detailed risk assessments for four of the five AeDib
projects, but has not yet provided a detailed assessment of risks for its
remaining project.
We also commented on the need for SSA to perform end- to- end testing
prior to implementation, to ensure that the system it is developing will
perform as intended. To date, SSA has not yet finalized its test and
evaluation strategy. Therefore, it is unclear whether SSA intends to
include end- to- end testing.
Finally, we commented on the need for SSA to resolve stakeholder concerns
to ensure program acceptance and use, and take additional steps to consult
with the medical community. SSA acknowledged the importance of ensuring
sound relations with stakeholders and the need to take additional actions,
where necessary, to ensure that all stakeholder concerns have been
adequately addressed. SSA also stated that additional steps would be taken
to keep stakeholders involved in the initiative, and that plans were being
made for a meeting with state DDS representatives to
discuss the AeDib initiative. However, SSA has not yet provided us with a
copy of its communications plan for dealing with stakeholder issues,
including its plans for consultation with the medical community. As part
of our ongoing work, we will continue to monitor SSA*s progress in
addressing these issues. 4. The SSA has worked 11 years now to implement
an electronic disability folder. The automation project started in 1992 as
the
Modernized Disability System (later renamed the Reengineered Disability
System). After this project failed in 1999, the SSA immediately began work
on the accelerated electronic disability initiative. Why is this project
so difficult for the agency to complete? Has the SSA learned from its
mistakes along the way?
Software development is one of the riskiest areas of systems development.
4 We have reported that SSA*s software development efforts have been
problematic and plagued with delays because SSA has not consistently
followed sound practices in developing systems designed to automate its
disability claims process; thus, it has experienced numerous software
development problems over the past 11 years. For example, in September
1996 we reported that software development problems had delayed the
scheduled implementation of the Reengineered Disability System (RDS) by
more than 2 years. 5 An assessment of the development activity revealed a
number of factors as having contributed to that delay, including (1) using
programmers with insufficient experience, (2) using software development
4 U. S. General Accounting Office, Social Security Administration:
Information Technology Challenges Facing the Commissioner, GAO/ T- AIMD-
98- 109 (Washington, D. C.: March 12, 1998). 5 U. S. General Accounting
Office, Social Security Administration: Effective Leadership Needed to
Meet Daunting Challenges, GAO/ HEHS- 96- 196 (Washington, D. C.: September
12, 1996).
tools that did not perform effectively, and (3) establishing initial
software development schedules that were too optimistic. We reported
again, in June 1998, that SSA had encountered performance problems during
its RDS pilot tests. 6 In response to these performance problems, SSA
delayed its plans for expanding the pilot to other offices, and obtained a
contractor to independently evaluate and recommend options for proceeding
with the
initiative. SSA*s contractor reported that RDS software had defects that
would diminish the case- processing rate at DDS sites, and that SSA had
not been timely in addressing software defects. For example, 90 software
problems identified by SSA staff remained unresolved for more than 120
days. As a result, the contractor recommended that SSA discontinue the RDS
initiative and focus on an alternative solution involving the use of an
electronic folder to replace the paper- based case folder in the
disability determination process. In another example, we reported in
August 2001 on weaknesses regarding
SSA*s adherence to key software development procedures for several
projects, including its electronic disability system. 7 We noted that SSA
did not consistently adhere to its software development procedures in the
areas of requirements management, software project planning, software
quality assurance, and software configuration management.
In our ongoing review, we have found that SSA has taken important steps to
mitigate past software development problems. It has generally addressed
its contractor*s recommendations aimed at automating its disability claims
process, and has generally been applying key software process improvement
practices to its development of AeDib projects, such as developing plans
to manage the projects, tracking and overseeing the initiatives to measure
progress, performing quality assurance reviews to determine that the
project is complying with its policies and procedures, and performing
configuration management activities. While continually applying these
software development practices is no guarantee of AeDib
6 U. S. General Accounting Office, Social Security Administration:
Technical and Performance Challenges Threaten Progress of Modernization,
GAO/ AIMD- 98- 136 (Washington, D. C.: June 19, 1998). 7 U. S. General
Accounting Office, Information Technology Management: Social Security
Administration Practices Can Be Improved, GAO- 01- 961 (Washington, D. C.:
August 21, 2001).
success, these practices nevertheless should improve SSA*s capability to
develop high- quality software in support of AeDib, thereby avoiding
mistakes experienced in the past.
Agency Comments and Our Evaluation
The Commissioner of Social Security provided comments on a draft of this
correspondence, which are reproduced in full as the attachment. In her
comments, the Commissioner offered clarifications to our responses to
questions 1 through 3.
Regarding question 1, the Commissioner said that we were incorrect in
stating that SSA*s cost- benefit analysis had not considered all costs of
certain critical IT infrastructure elements supporting the nationwide
AeDib rollout, and that, in fact, SSA had included the costs of scanning
and imaging by the outside vendor in its estimates.
We have revised our response to clarify that SSA*s cost- benefit analysis
considered some, but not all, of the key cost elements that could affect
the initiative. SSA*s February 2003 cost- benefit analysis noted, for
example, that the agency had not considered as part of its scanning and
imaging functions, the keying in of indexing information (for case folder
identification) by the outsourced scanning vendor, although this is deemed
critical to the implementation of SSA*s document management capability.
Further, during our review, SSA officials told us that certain costs
associated with the scanning and imaging functions were not expected to be
identified until the agency performed its pilot tests (now ongoing) for
its document management architecture.
The Commissioner added that other AeDib- related costs associated with
telecommunications, disaster recovery, and on- site retention and
destruction of source documents should more appropriately be accounted
for as part of the agency*s ongoing operations, and therefore were covered
in the agency*s regular infrastructure costs, rather than the AeDib cost
estimates. However, we disagree. OMB guidance states that cost- benefit
analyses should include comprehensive estimates of all direct and indirect
costs associated with a project. 8 As such, a sound cost- benefit analysis
for
8 Office of Management and Budget Circular No. A- 94 Revised (Transmittal
Memo No. 64), Guidelines and Discount Rates for Benefit- Cost Analysis of
Federal Programs, October 29, 1992.
AeDib will depend on SSA*s fully considering the project- related costs
for these critical elements supporting the development, operation, and
maintenance of the electronic disability system.
Further, the Commissioner expressed concern about the example that we
provided in noting that SSA*s estimate of AeDib benefits could be
overstated. Based on our interviews with SSA and state DDS
representatives, our response highlighted the possibility that AeDib
benefits could be overstated because of differences in SSA*s and the DDSs*
assumptions about the extent that medical evidence may be received
electronically. We are continuing to assess SSA*s cost- benefit analysis
as part of our ongoing review.
In commenting on our response to question 2, the Commissioner stated that
SSA had expected to receive draft risk assessments by June 30, 2003, and
either planned to or already had shared some of its risk assessments with
us. We have revised our response to reflect that SSA recently provided us
with detailed risk assessments for four of the five AeDib projects.
Finally, regarding question 3, the Commissioner stated that SSA had
provided us with information concerning stakeholder issues and the
agency*s plans for consulting with the medical community. She further
noted that outreach to the medical community was occurring. We are
encouraged that SSA is taking steps to ensure productive communications
with its key stakeholders and the medical community, and look forward to
reviewing documented evidence of the agency*s actions. To date, however,
we have not received the additional information on SSA*s plans for
addressing stakeholder issues or consulting with the medical community,
that the Commissioner mentions in her letter. In responding to these
questions, we relied on past work and our ongoing
review of SSA*s efforts to automate its disability claims process. We
discussed our assessment of the cost- benefit analysis with SSA*s Office
of Disability Programs, Office of Systems, and Office of the Chief
Information Officer, and with the supporting contractor and key
stakeholders. We reviewed and analyzed the most recent agency documents
associated with
SSA*s risk management efforts. We also discussed with officials in the
Office of Disability Programs SSA*s efforts to develop a strategy for
resolving stakeholder concerns, as well as a more aggressive approach for
consultation with the medical community. Finally, we reviewed and
analyzed SSA*s software process improvement documentation, as well as
past assessments of the agency*s failed attempts to automate its
disability claims process to determine lessons learned, and whether SSA is
avoiding past software development problems. We conducted our work in
accordance with generally accepted government auditing standards, during
August 2003.
We are sending copies of this letter to the Honorable Jo Anne B. Barnhart,
Commissioner of Social Security, and other interested parties. Copies will
also be available at our Web site at www. gao. gov.
If you have any questions on matters discussed in this letter, please
contact me at (202) 512- 6240 or Valerie Melvin, Assistant Director, at
(202) 512- 6304. We can also be reached by E- mail at koontzl@ gao. gov
and melvinv@ gao. gov, respectively. Key contributors to this
correspondence include Michael A. Alexander, Tonia B. Brown, and Mary J.
Dorsey.
Sincerely yours, Linda D. Koontz Director, Information Management Issues
Encl osure Comments from Social Security
(310386)
Page 1 GAO- 03- 1113R SSA Electronic Disability System United States
General Accounting Office Washington, D. C. 20548
Page 1 GAO- 03- 1113R SSA Electronic Disability System
A
Page 2 GAO- 03- 1113R SSA Electronic Disability System
Page 3 GAO- 03- 1113R SSA Electronic Disability System
Page 4 GAO- 03- 1113R SSA Electronic Disability System
Page 5 GAO- 03- 1113R SSA Electronic Disability System
Page 6 GAO- 03- 1113R SSA Electronic Disability System
Page 7 GAO- 03- 1113R SSA Electronic Disability System
Page 8 GAO- 03- 1113R SSA Electronic Disability System
Page 9 GAO- 03- 1113R SSA Electronic Disability System
Enclosure
Enclosure Comments from Social Security
Page 10 GAO- 03- 1113R SSA Electronic Disability System
This is a work of the U. S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
GAO s Mission The General Accounting Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional
responsibilities and to help improve the performance and accountability of
the federal government for the American people. GAO examines the use of
public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO s commitment to
good government is reflected in its core values of accountability,
integrity, and reliability. The fastest and easiest way to obtain copies
of GAO documents at no cost is
through the Internet. GAO s Web site ( www. gao. gov) contains abstracts
and full- text files of current reports and testimony and an expanding
archive of older products. The Web site features a search engine to help
you locate documents using key words and phrases. You can print these
documents in their entirety,
including charts and other graphics. Each day, GAO issues a list of newly
released reports, testimony, and correspondence. GAO posts this list,
known as Today s Reports, on its Web site daily. The list contains links
to the full- text document files. To have GAO e- mail this list to you
every afternoon, go to www. gao. gov and select Subscribe to e- mail
alerts under the Order GAO Products heading.
The first copy of each printed report is free. Additional copies are $ 2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:
U. S. General Accounting Office 441 G Street NW, Room LM Washington, D. C.
20548
To order by Phone: Voice: ( 202) 512- 6000 TDD: ( 202) 512- 2537 Fax: (
202) 512- 6061
Contact: Web site: www. gao. gov/ fraudnet/ fraudnet. htm E- mail:
fraudnet@ gao. gov Automated answering system: ( 800) 424- 5454 or ( 202)
512- 7470 Jeff Nelligan, Managing Director, NelliganJ@ gao. gov ( 202)
512- 4800 U. S. General Accounting Office, 441 G Street NW, Room 7149
Washington, D. C. 20548 Obtaining Copies of
GAO Reports and Testimony Order by Mail or Phone To Report Fraud, Waste,
and Abuse in Federal Programs Public Affairs
*** End of document. ***