Electronic Disability Claims Processing: SSA Needs to Address
Risks Associated with Its Accelerated Systems Development
Strategy (26-MAR-04, GAO-04-466).
The Social Security Administration's (SSA) AeDib initiative is
designed to provide SSA with a more efficient, paperless system
that will enable its disability components to electronically view
and share claims data and process claims electronically. Yet
previous GAO reviews found that SSA's accelerated strategy to
develop AeDib involved risks that could threaten a complete and
successful transition to this capability. At the Subcommittee's
request, GAO reviewed AeDib to assess (1) SSA's progress and
strategy, (2) the adequacy of measures taken to avoid software
development problems similar to those encountered in SSA's
previous efforts, (3) the adequacy of cost/benefit analyses, and
(4) SSA's consultation with stakeholders.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-04-466
ACCNO: A09599
TITLE: Electronic Disability Claims Processing: SSA Needs to
Address Risks Associated with Its Accelerated Systems Development
Strategy
DATE: 03/26/2004
SUBJECT: Claims processing
Disability benefits
Electronic government
Program evaluation
Strategic planning
Systems design
Systems analysis
Risk management
SSA AeDib
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-04-466
United States General Accounting Office
GAO Report to the Chairman, Subcommittee on Social Security, Committee on
Ways and Means, House of Representatives
March 2004
ELECTRONIC DISABILITY CLAIMS PROCESSING
SSA Needs to Address Risks Associated with Its Accelerated Systems Development
Strategy
a
GAO-04-466
Highlights of GAO-04-466, a report to the Chairman, Subcommittee on Social
Security, Committee on Ways and Means, House of Representatives
The Social Security Administration's (SSA) AeDib initiative is designed to
provide SSA with a more efficient, paperless system that will enable its
disability components to electronically view and share claims data and
process claims electronically. Yet previous GAO reviews found that SSA's
accelerated strategy to develop AeDib involved risks that could threaten a
complete and successful transition to this capability.
March 2004
ELECTRONIC DISABILITY CLAIMS PROCESSING
SSA Needs to Address Risks Associated with Its Accelerated Systems Development
Strategy
SSA is continuing its work on the AeDib initiative and is in various
stages of completing its electronic disability system; however, its
accelerated strategy continues to involve risks. Specifically, GAO found
that the agency is relying on limited pilot testing to help guide business
and technical decisions and ensure that technology supporting the
electronic disability system will work as intended (see table). Further,
it is beginning its national rollout without ensuring that all critical
problems identified in the pilot testing have been resolved and without
conducting testing adequate to evaluate the performance of all system
components collectively. Without resolution of critical problems and full
testing, SSA cannot be assured that interrelated components will work
together successfully.
Pilot Tests are Limited in Scope, Involving Few Examiners
At the Subcommittee's request, State/ Number/percentage of GAO reviewed AeDib to
assess (1) pilot start date examiners participating
SSA's progress and strategy, (2) the adequacy of measures taken to avoid
software development problems similar to those encountered in SSA's
previous efforts, (3) the adequacy of costbenefit analyses, and (4) SSA's
consultation with stakeholders.
GAO recommends that before proceeding with national rollout of AeDib, the
Commissioner of Social Security, among other steps, ensure that critical
problems have been resolved and full testing completed, expedite
completion of riskmitigation strategies, and validate cost-benefit
estimates. In commenting on a draft of this report, SSA stated that it
would conduct studies to help validate AeDib cost assumptions but
disagreed with GAO's other recommendations. In GAO's view, it is essential
that SSA fully address all recommendations.
www.gao.gov/cgi-bin/getrpt?GAO-04-466.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Linda D. Koontz at (202)
512-6240 or [email protected].
North Carolina/ 21 of 254/ July 2003 8 percent Illinois/ 15 of 250/
September 2003 6 percent California/ 2 of 620/ October 2003 0.3 percent
Source: GAO analysis of SSA data.
While SSA has procedures to guide its software development, it could
provide no evidence that it was consistently applying them in this case.
In addition, while SSA has identified AeDib system and security risks, it
has not finalized mitigation strategies. Without these measures being in
place, SSA stands at greater vulnerability to circumstances that could
impede project success.
The agency also has not validated its analysis to ensure the
reasonableness of estimated AeDib costs and benefits. While indicating
that it would use pilot test results to validate cost-benefit estimates,
officials have not indicated when this will be accomplished. This leaves
SSA without a validated cost-benefit analysis, and the assurance that its
AeDib cost estimates are reliable and that anticipated benefits will
therefore be realized.
Finally, SSA reports that it has increased its communications with AeDib
stakeholders and users; however, state officials dealing with disability
determinations have varying perspectives. A national organization
representing these state officials continues to voice concerns about SSA's
approach. And while the Commissioner states that SSA is consulting with
stakeholders and the medical community, the agency has not articulated a
comprehensive plan for ensuring that the concerns of this population are
addressed.
Contents
Letter 1
Recommendations for Executive Action 3
Agency Comments and Our Evaluation 4
Appendixes
Appendix I: GAO's January 30, 2004, Briefing 8
Comments from the Social Security 52
Appendix II: Administration
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
A
United States General Accounting Office Washington, D.C. 20548
March 26, 2004
The Honorable E. Clay Shaw, Jr.
Chairman, Subcommittee on Social Security
Committee on Ways and Means
House of Representatives
Dear Mr. Chairman:
As you know, the Social Security Administration (SSA) is currently
pursuing a major initiative to establish an electronic disability claims
processing capability. With this initiative-known as AeDib-SSA aims to
develop and implement a more efficient, paperless system that will enable
its disability processing components to electronically view and share
claims information, including large volumes of medical images, files, and
other documents currently maintained in paper folders. SSA's pursuit of an
electronic disability process represents a positive and necessary step
toward more efficient delivery of benefits payments and services to an
increasing beneficiary population. However, as we have previously
reported,1 the agency's accelerated strategy to develop the electronic
disability system involves risks that could pose a threat to the complete
and
successful transition to this capability.
At your request, we have been reviewing SSA's continued actions on the
AeDib initiative. Specifically, our objectives were to
o assess SSA's progress toward and strategy for achieving AeDib,
o determine the adequacy of measures SSA is taking with AeDib to avoid
the kinds of software development problems that it encountered with its
prior disability system initiative and to identify and mitigate risks to a
successful development,
o assess the adequacy of SSA's analysis of AeDib costs and benefits, and
o assess SSA's consultation with and support from key stakeholders.
1U.S. General Accounting Office, Electronic Disability Claims Processing:
Social Security Administration's Accelerated Strategy Faces Significant
Risks, GAO-03-984T (Washington, D.C.: July 24, 2003) and Social Security
Administration: Subcommittee Questions Concerning Efforts to Automate the
Disability Claims Process, GAO-03-1113R (Washington, D.C.: Sept. 5, 2003).
To address these objectives, we analyzed project management and technical
documentation describing SSA's plans, strategies, and progress related to
developing and implementing the electronic disability system. We
identified and evaluated measures-including institutional software
capability mechanisms-that the agency had taken to guide its development
of the AeDib software and mitigate program and project risks. We also
assessed the adequacy of SSA's analyses of AeDib costs and benefits,
including evaluating them against federal guidance and industry best
practices and analyzing the impact of critical pilot evaluation data on
costs and benefits. In addition, we analyzed SSA's plans and actions for
consulting with and identifying and resolving issues of key disability
stakeholders (i.e., state Disability Determination Services [DDS] offices,
medical providers, and SSA field offices). We performed our work in
accordance with generally accepted government auditing standards, from
August 2003 through January 2004.
On January 30, 2004, we provided your staff with a briefing on the results
of our study. The slides from that briefing are included as appendix I to
this report. The purpose of this report is to provide the published
briefing slides to you and to officially transmit our recommendations to
the Commissioner of Social Security.
In summary, our briefing made four main points:
o While the AeDib initiative is important to achieving more efficient
delivery of disability payments, SSA's accelerated strategy continues to
involve risks that threaten its ultimate success. SSA is relying on
limited pilot testing to help guide business and technical decisions and
ensure that technology supporting the electronic disability folder will
work as intended. Further, the agency is beginning its national rollout
without ensuring that all critical problems identified during the pilot
tests have been resolved and without conducting end-to-end tests to
evaluate the functionality and performance of all electronic disability
system components collectively. SSA has maintained that its pilot tests of
the components would be sufficient to evaluate the system; however,
without end-to-end testing, SSA lacks assurance that the interrelated
components will work together successfully.
o While SSA has established processes and procedures to guide its
software development activities, such as project management plans, it
could not provide evidence that it was consistently applying these
procedures to AeDib. In addition, while SSA has identified AeDib system
and security risks, it has not finalized mitigation strategies. Without
concurrence on software validation and system certifications, or risk
mitigation strategies, SSA lacks assurance that its system will be
acceptable to end users and it will not be positioned to effectively
prevent circumstances that could impede project success.
o SSA has not validated its analysis to ensure the reasonableness of
estimated AeDib costs and benefits. The agency stated that it would use
pilot test results to validate its cost-benefit estimates; however, its
officials have not indicated when this will be accomplished. Without a
validated cost-benefit analysis, SSA lacks assurance that its AeDib cost
estimates are reliable and that anticipated benefits will be realized.
o While SSA has stated that it has increased its communications with
AeDib stakeholders and users and instituted mechanisms to facilitate its
communications, state DDSs have varying views regarding their involvement.
Specifically, while state disability offices participating in the AeDib
pilots agree that communications have improved, a national organization
representing these offices continues to have concerns about SSA's
approach. In addition, although the Commissioner has stated that SSA is
consulting with stakeholders and the medical community, SSA has not
articulated a comprehensive plan for ensuring that all of these concerns
are addressed.
In light of the risks associated with the AeDib initiative, SSA's
successful transition to an electronic disability claims process could be
jeopardized unless it addresses the weaknesses described above.
Recommendations for Executive Action
To reduce the risks associated with SSA's strategy to develop an
electronic disability claims processing system, we recommend that the
Commissioner of Social Security, before continuing with the AeDib national
rollout,
o ensure that all critical problems identified in pilot testing of the
electronic disability system components are resolved and that end-toend
testing of the interrelated systems is performed;
o ensure that users have approved the software developed and that systems
have been certified for production;
o establish a revised time frame for and expedite actions toward
finalizing AeDib risk mitigation strategies;
o validate all AeDib cost and benefit estimates; and
o implement a communications plan that clearly and comprehensively
conveys SSA's approach for effectively addressing disability stakeholders'
and users' concerns and ensuring their full involvement in the AeDib
initiative.
Agency Comments and Our Evaluation
In providing written comments on a draft of this report (reprinted in app.
II), the Commissioner of Social Security reiterated the department's
general disagreement with the findings discussed in our briefing slides.
The Commissioner added that, in considering our recommendations, the
agency had not been compelled to change its direction on the AeDib
initiative.
In particular, SSA continued to disagree with the need for end-to-end
testing, stating that to perform such testing would delay the agency's
ability to realize benefits from this initiative. SSA estimated that
following our recommendation would have delayed the project, and the
resulting benefits, by at least 3 years. Given the technological
complexity of the AeDib project, we remain convinced that end-to-end
testing is essential to ensuring that the multiple interrelated components
of the electronic disability system will operate as intended. As our
report noted, at the time that SSA began national implementation of the
electronic disability system, it had conducted only limited pilot testing
of the Document Management Architecture that is fundamental to the
electronic disability folder. It also had not fully evaluated the results
of its pilot tests or ensured that all critical problems were resolved.
Further, we disagree with SSA's contention that end-to-end testing would
delay the AeDib project by 3 years. In referring to this time frame, SSA
has emphasized the processing of its most comprehensive cases-those in
which individuals pursue their disability claims through all levels of
agency appeal-which reportedly take an average of 1,153 days to reach a
final decision. However, not every case that SSA processes requires this
level of appeal; thus, some cases can be processed in less time. In
addition, SSA has reported that, due to backlogs, cases that go through
all levels of appeal spend nearly 50 percent of that time in queue,
waiting for agency action. Given these factors, we believe SSA has an
opportunity to apply a more expedited approach to end-to-end testing,
focused on evaluating certain claims that require some, but not all levels
of agency appeal, and processing them outside the normal backlog of cases.
Without full awareness and resolution of critical problems and endto-end
testing, SSA cannot ensure that the electronic disability system
components will work together effectively. Thus, SSA could incur future
unanticipated costs to correct the system, and stands to increase its risk
of not realizing the very benefits that it seeks, by proceeding with AeDib
without this critical testing.
In addition, SSA said it considered moot our recommendation that the
agency ensure that users have approved new software and that it certify
its systems for production, stating that the agency has always performed
these tasks. In addition, SSA said that our concern in this area had
pertained to the lack of wet signatures2 on its project documents. In
speaking to this issue, our primary concern was not that SSA lacked wet
signatures on its software development documentation. Rather, during the
course of our review, agency officials were unable to provide readily
verifiable evidence that users had approved software developed for the
electronic disability system, or that systems had been certified for
production. At the end of our study, SSA provided certain documentation
supporting its software validation and certification efforts; however, the
documents lacked sufficient details for us to fully evaluate and reach
conclusions about the extent to which all software had been validated by
the users and certified for production. For any project, and especially
one of AeDib's size and complexity, having readily verifiable evidence
that systems were developed in accordance with specified requirements is
essential to ensuring that users' needs will be met by the software
developed and that systems can be certified for production.
Regarding AeDib program and project risks, SSA stated that it believed the
success of AeDib over the last 26 months had attested to the agency's
diligent and ongoing efforts to mitigate risks. Nonetheless, as our report
stresses, strategies for addressing program-level and security-level risks
are needed to help prevent circumstances that could impede successful
project outcomes. In the absence of explicit mitigation strategies, SSA
increases the probability of failing to meet project commitments before
they can manifest themselves in cost, schedule, and performance
shortfalls. Accordingly, it is essential that SSA establish a time frame
for and expedite actions toward finalizing risk-mitigation strategies for
the AeDib project.
2A wet signature refers to an original signature written on a piece of
paper, as opposed to a fax copy or an agreement offered verbally or
electronically.
Regarding our recommendation that SSA validate all AeDib cost and benefit
estimates, the agency reiterated its intent to conduct studies to help
validate planning assumptions, and noted that work supporting this action
was under way.
Finally, SSA stated that it continually reassesses its communications, and
that while it can always improve, it is satisfied that information is
being shared, and that stakeholder voices are being heard. Further, SSA
stated that the stakeholders' differing opinions about how to design the
AeDib system do not reflect an issue of communication, but rather, the
reality of bringing change into a very complex and diverse environment.
Our report noted that SSA had taken additional actions to work
collaboratively with stakeholders. Further, we recognize that changing
conditions can elicit varying opinions regarding the actions being taken
on a project. Nonetheless, stakeholders continue to voice serious concerns
regarding the manner in which SSA is communicating with them about AeDib,
necessitating that the agency remain diligent in pursuing a mutually
agreedupon understanding with stakeholders on systems development and
implementation issues. SSA's success in fully implementing AeDib depends
heavily on resolving all outstanding issues and concerns that could affect
the ultimate use and outcome of the intended electronic capability. By
implementing a communications plan that clearly and comprehensively
conveys its approach for effectively addressing disability stakeholders'
and users' concerns and ensuring their full involvement in the AeDib
initiative, SSA could strengthen the likelihood that it will achieve a
successful project outcome that effectively meets the needs of all
critical players in the disability claims process.
We are sending copes of this report to the Commissioner of Social
Security, and to the Director, Office of Management and Budget. Copies
will also be available at no charge on our Web site at www.gao.gov.
Should you have any question on matters contained in this report, please
contact me at (202) 512-6240, or Valerie Melvin, Assistant Director, at
(202) 512-6304. We can also be reached by e-mail at [email protected] and
[email protected], respectively. Other key contributors to this report were
Michael A. Alexander, Tonia B. Brown, Harold J. Brumm, Mary J. Dorsey, and
Michael P. Fruitman.
Sincerely yours,
Linda D. Koontz Director, Information Management Issues
Appendix I
GAO's January 30, 2004, Briefing
Update on the
Social Security Administration's
Accelerated Electronic Disability Strategy
Briefing for the Staff of the Subcommittee on Social Security House Ways
and Means Committee
January 30, 2004
Purpose
To provide an updated assessment of SSA's accelerated electronic
disability (AeDib) claims processing initiative
Outline
o Introduction
o Objectives
o Scope and Methodology
o Results in Brief
o Background
o Objective 1: Strategy
o Objective 2: Software Development and Risk Management
o Objective 3: Cost-Benefit Analysis
o Objective 4: Stakeholder/User Involvement
o Conclusions
o Recommendations
o Agency Comments and Our Evaluation
Over the years, SSA has faced increasing challenges in ensuring acceptable
levels of service to the millions of disabled individuals and
beneficiaries. A major barrier to improving efficiency in the disability
determination process has been SSA's continuing reliance on paper folders
to process and adjudicate claims.
As a result, SSA is currently pursuing a major systems-related initiative
to enhance its disability claims processing capability. A key goal of this
initiative, known as the Accelerated Electronic Disability (AeDib) System,
is to develop by late January 2004, a disability claims process with the
capability to electronically view and share claims information among all
processing components.
GAO previously reported1 on SSA's progress in developing the AeDib
initiative, noting that its strategy involved risks that could jeopardize
the agency's successful transition to an electronic disability process.
1U.S. General Accounting Office, Electronic Disability Claims Processing:
Social Security Administration's Accelerated Strategy Faces Significant
Risks, GAO-03-984T (Washington, D.C.: July 24, 2003) and U.S. General
Accounting Office, Social Security Administration: Subcommittee Questions
Concerning Efforts to Automate the Disability Claims Process, GAO-03-1113R
(Washington, D.C.: Sept. 5, 2003).
As requested by the Chairman of the subcommittee, our objectives were to
o assess SSA's progress toward and strategy for achieving AeDib,
o determine the adequacy of measures SSA is taking with AeDib to avoid
the kinds of software development problems that it encountered with its
prior disability system initiative and to identify and mitigate risks to a
successful development effort,
o assess the adequacy of SSA's analysis of AeDib costs and benefits, and
o assess SSA's consultation with and support from key stakeholders.
Analyzed project management and technical documentation describing SSA's
plans, strategies, and progress related to developing and implementing
AeDib
Evaluated the extent to which SSA has implemented measures-including
institutional software capability mechanisms-to avoid the types of
software development problems identified in prior reviews and established
strategies for mitigating program and project risks
Assessed the adequacy of SSA's analyses of AeDib costs and benefits,
including evaluating them against federal guidance and industry best
practices and analyzing the impact of critical pilot evaluation data on
costs and benefits
Assessed SSA's plans and actions to effectively consult with and identify
and resolve issues raised by key disability stakeholders (i.e., state
Disability Determination Services [DDS] offices, medical providers, and
SSA field offices)
Conducted site visits at the North Carolina DDS office and SSA field
offices in Raleigh and Durham to observe pilot tests supporting the
electronic folder capability, and teleconferences with officials in the
California and Illinois DDSs and the SSA field office in Hillside,
Illinois to discuss pilot test progress and results
Interviewed officials in selected DDS offices that had been included in
our prior review2 (Delaware, Iowa, Mississippi, Nebraska, New York,
Virginia, and Wisconsin) to update our assessments of SSA's involvement of
stakeholders in the AeDib initiative
Interviewed headquarters officials in SSA's Offices of Disability
Programs, Operations, Systems, Hearings and Appeals, and the Inspector
General to obtain their perspectives on the AeDib project
Conducted our review in accordance with generally accepted government
auditing standards, from August 2003 through January 2004
2GAO-03-984T.
AeDib represents an important step toward more efficient delivery of
disability payments and SSA is in various stages of completing key
components of the initiative. SSA's strategy, however, involves risks that
could threaten project success. Specifically, SSA is relying on limited
pilot tests to help guide business and technical decisions and ensure that
technology supporting the electronic disability folder will work as
intended. Further, SSA is beginning its national rollout without ensuring
that all critical problems identified in its pilot testing have been
resolved and without conducting end-to-end testing to evaluate the
functionality and performance of all electronic disability system
components collectively. SSA maintained that its pilot tests of the
electronic disability system components would be sufficient to evaluate
the system. However, in the absence of end-to-end testing, SSA lacks
assurance that the interrelated electronic disability system components
will work together successfully.
SSA has established processes and procedures to guide its software
development activities, including using plans to manage projects and
tracking and overseeing initiatives to measure progress. However, SSA
could not provide evidence that it is consistently applying these
procedures to the AeDib initiative. For example, our review found no
evidence of approved software validation plans and system certifications
to show that users' requirements were satisfied. In addition, while SSA
had identified AeDib system and security risks, it had not finalized
mitigation strategies needed to help ensure a successful project outcome.
Without concurrence on software validation and system certifications, or
risk mitigation strategies, SSA lacks assurance that its system will be
acceptable to end users and it will not be positioned to effectively
prevent circumstances that could impede project success.
SSA has not validated its cost-benefit analysis to ensure the
reasonableness of estimated AeDib costs and benefits. SSA stated that it
would use pilot test results to validate its cost-benefit estimates;
however, its officials have not indicated when this will be done. Without
a validated cost-benefit analysis, SSA lacks assurance that its AeDib cost
estimates are reliable and that anticipated benefits will be realized.
SSA stated that it has increased its communications with AeDib
stakeholders/users and has instituted some mechanisms to facilitate its
communications. However, state DDSs have varying views regarding their
involvement. Specifically, while state disability offices involved in
AeDib pilots agreed that communications have improved, a national
organization representing these offices continues to have concern about
SSA's approach. In addition, although the Commissioner has stated that SSA
is consulting with stakeholders and the medical community, SSA has not
articulated a comprehensive plan for ensuring that all of their concerns
are addressed.
Given the importance of the AeDib initiative to SSA's future service
delivery capability, it is essential that the agency satisfy itself that
the electronic disability system will perform as intended with minimal
risk before it is deployed nationwide. Therefore, to help improve the
success of this initiative, we are making recommendations to the
Commissioner of Social Security on (1) resolving critical problems
identified and conducting end-to-end testing, (2) ensuring user
concurrence on software validation and systems certifications, (3)
finalizing AeDib risk mitigation strategies, (4) validating AeDib costs
and benefits, and (5) improving agency communications with stakeholders
and users.
SSA officials reviewed a draft of our briefing slides, and their comments
have been incorporated where appropriate. SSA took issue with many of our
findings. For example, it disagreed with the need for end-to-end testing,
and also disagreed with our finding that it had not followed all agency
processes and procedures for software development. We discuss these
comments and our response to them in the Agency Comments and Our
Evaluation section.
The Disability Insurance and Supplemental Security Income programs are the
nation's largest providers of federal income assistance to disabled
individuals, with SSA making payments of approximately $86 billion to
about 10 million beneficiaries in 2002.
SSA's process of approving or denying disability benefits is complex and
involves multiple partners at both the state and federal levels in
determining a claimant's eligibility.
o SSA's 1,300 field offices and its Office of Hearings and Appeals, along
with 54 state DDSs, are the primary players in processing disability
claims.
o Physicians and other members of the medical community support
disability determinations by providing the medical evidence needed to
evaluate disability claims.
To address concerns regarding the disability program's efficiency, in 1992
SSA initiated a redesign of the disability claims process, emphasizing the
use of automation to achieve an electronic (paperless) processing
capability. However, SSA encountered problems with this prior initiative,
the Reengineered Disability System (RDS), and in 1999 suspended the
project after approximately 7 years and about $71 million reportedly spent
on the initiative.3
3U.S. General Accounting Office, Social Security Administration: Technical
and Performance Challenges Threaten the Progress of Modernization,
GAO/AIMD-98-136 (Washington, D.C.: June 19, 1998).
In August 2000 SSA renewed its plan to achieve paperless disability
decision making through the use of an electronic disability folder and
automated case processing systems for disability claims
adjudication/review by the end of 2005. In the spring of 2002, the
Commissioner of Social Security accelerated the strategy to more quickly
automate the disability claims process. Under the accelerated
strategy-called AeDib-SSA planned to begin implementing its electronic
disability system by January 2004.
SSA is undertaking five key projects to support the AeDib strategy:
1. an Electronic Disability Collect System, giving SSA field offices the
capability to electronically capture and store in an electronic folder
information about a claimant's disability;
2. a Document Management Architecture (data repository and scanning and
imaging capabilities) to allow claimant information and medical evidence
to be captured, stored, indexed, and shared electronically among
disability processing components;
3. Internet applications to enable SSA to obtain disability claims and
medical information from claimants via the Internet;
4. a DDS systems migration and electronic folder interface to enhance
existing case processing systems and enable state DDS offices to operate
on a common platform and share information in the electronic folder; and
5. a Case Processing and Management System for the Office of Hearings and
Appeals that will interface with the electronic folder and enable staff to
track, manage, and complete case-related tasks electronically.
The AeDib strategy focuses on developing the capability for claimant
information and large volumes of medical images, files, and other
documents that are currently maintained in paper folders to be stored in
electronic folders, and then accessed, viewed, and shared by the
disability processing offices.
According to SSA, the Electronic Disability Collect System and the
Document Management Architecture are the two fundamental elements needed
to achieve the electronic disability folder.
SSA is using an incremental, phased approach to implementing AeDib.
The agency planned to rely on pilot tests and evaluations to help guide
business and technical decisions about the electronic disability folder,
including critical decisions regarding the Document Management
Architecture.
SSA had planned to complete its pilot tests of the Document Management
Architecture in December 2003, and begin its national implementation of
the electronic disability folder in late January 2004, with scanning and
imaging capabilities and interface software being rolled out over 18
months.
AeDib is a major capital investment project, and its February 2003
cost-benefit analysis estimates 10-year life-cycle costs of about $900
million and benefits of approximately $1.3 billion.
SSA reported actual AeDib costs of approximately $91 million for fiscal
year 2000 through fiscal year 2003.
SSA identified two AeDib funding sources for IT hardware, software, and
services:
o information technology systems/automation investment fund, and
o limitation on administrative expenses budget.
In July 2003 testimony4 we noted that SSA
o had performed important tasks for the initial electronic capability,
but had substantial work to accomplish in order to begin implementing the
electronic folder by late January 2004;
o was not expected to complete pilot tests that were to provide critical
information about the electronic folder's performance until late December
2003 and had not developed a strategy for conducting end-to-end testing to
demonstrate that the individual components would work together reliably;
o had not comprehensively assessed AeDib project risks or developed risk
mitigation strategies; and
o had taken some steps to involve key stakeholders in the system's
development, but had not resolved all outstanding concerns that could
affect the use and ultimate success of the intended electronic capability.
4GAO-03-984T.
In September 20035 we reported that SSA
o may have underestimated the accelerated electronic disability system's
costs and overstated corresponding benefits in its AeDib cost-benefit
analysis, and
o had applied software development practices that could help the agency
avoid software development problems experienced with its prior electronic
disability initiative-the Reengineered Disability System.
5GAO-03-1113R.
Since July, SSA has continued its work toward completing the AeDib
initiative, and is in various stages of development.
o SSA has implemented all planned releases of the Electronic Disability
Collect System (as of January 20, 2004).
o SSA has completed and placed into production three Internet
applications to aid claimants in filing for disability benefits and
services online.
o SSA has enhanced the DDS legacy systems by installing hardware upgrades
and moving systems to a common platform; it has also migrated legacy
systems claims processing software in all but 3 of the 54 state DDS
offices. However, it has yet to complete the legacy system electronic
folder interface software that will enable existing SSA and DDS systems to
interface with the electronic folder. SSA reported that it plans to
complete its implementation of the legacy system electronic folder
interface software by October 2004.
o The Case Processing Management System is being pilot tested in a
standalone environment at five Office of Hearings and Appeals sites.
Consistent with generally accepted best practices, SSA should ensure that
the system it delivers successfully meets key business and technical
requirements for reliably exchanging data among disability processing
components and is protected from errors and vulnerabilities that can
disrupt service.
Accomplishing this requires SSA to conduct complete and thorough testing
to provide reasonable assurances that systems perform as intended. This
includes tests and evaluations of pilot projects to obtain data on the
system's functional performance, and end-to-end tests to ensure that the
interrelated systems will operate together effectively.
SSA planned to rely on the pilot tests and evaluations to guide critical
decisions about its document management technology and verify that this
system component works in the actual user environment. For example, SSA
stated that Document Management Architecture pilots would be used to test
electronic folder interface and DDS site configurations for the electronic
disability system's national implementation. SSA and DDS offices stated
that they intended to expand the size and scope of the pilot tests to
analyze the system's performance.
SSA had planned to complete these pilot tests in December 2003.
Since last July, SSA has been pilot testing the Document Management
Architecture in three locations-North Carolina, Illinois, and California.
SSA has been incrementally providing increased electronic folder
functionality.
These pilot tests have demonstrated that the DDSs can view medical
evidence captured in the Document Management Architecture; however, the
pilots also identified numerous technical/operational problems, which SSA
is in the process of resolving:
o DDS pilot systems experienced software glitches (i.e., screen lock-ups,
inability to save highlighted text, and inability to enhance images) after
new releases.
o Scanning quality for some documents, such as graphical images, was
poor.
o Response times for retrieving medical evidence stored in the data
repository was slow (California/Midas).
o 2-D barcode resolution diminished with multiple faxing and copying,
preventing effective identification of the case file and the assigned DDS.
o Existing 19-inch monitors proved too small for DDS claims examiners to
simultaneously view two different documents (e.g., forms and medical
evidence) on split screens.
Although the pilot tests are proceeding, they are limited in scope,
involving a relatively small number of examiners and disability cases:
North Carolina (pilot started July 29, 2003)
o 21 of 254 examiners participating (8 percent)
o Has closed 561 of 1,246 cases received for the pilot
o Closes about 147,000 cases annually Illinois (pilot started Sept. 8,
2003)
o 15 of 250 examiners participating (6 percent)
o Has closed 282 of 1,165 cases received for the pilot
o Closes about 164,000 disability cases annually California (pilot
started Oct. 15, 2003)
o 2 of 620 examiners participating (0.3 percent)
o Has closed 47 of 160 cases received for the pilot
o Closes about 400,000 disability cases annually
Although SSA had planned to expand the pilot tests to additional users,
the technical/operational problems encountered during the pilots (e.g.,
software glitches experienced after new releases) have limited the
expansion, and consequently, the opportunity to analyze system performance
under increasingly heavy workloads (e.g., higher transaction rates.)
The pilot tests, originally scheduled to be completed in December 2003,
are also behind schedule. Further, it is not clear when SSA plans to
complete the pilot tests. On January 13, SSA's Deputy Commissioner for
Systems stated that, beginning on January 20, SSA planned to release to
the three pilot states, the interface software and enhancements that would
provide disability users the capability to electronically process an
entire disability case.
The Deputy Commissioner added that, barring significant problems, SSA
planned to begin the national rollout of this functionality in Mississippi
on January 26, 2004. Thus, SSA would begin its national rollout before
fully evaluating pilot test results and ensuring that all critical
problems are resolved.
Without fully evaluating pilot test results and ensuring the resolution of
all critical problems before national rollout of the electronic disability
system, SSA not only risks being unable to ensure that the electronic
folder will perform as intended, but also risks being unable to rely on
the new system to sustain current disability workloads.
Given AeDib's technological complexity and multiple interrelated
components, coupled with limited pilot testing of the Document Management
Architecture and the numerous problems encountered during that testing,
end-to-end testing remains essential to helping SSA reduce system risks
prior to national roll-out.
Our prior work has noted the need for such testing to ensure that
interrelated systems that collectively support a core business area or
function will work as intended in a true operational environment.6
End-to-end testing evaluates both the functionality and performance of all
systems components, enhancing an organization's ability to trust the
system's reliability.
In addition to its limited pilot testing of the Document Management
Architecture, performance and other problems that SSA has encountered
while developing and implementing other critical AeDib components further
underscore the need for endto-end testing. For example:
6U.S. General Accounting Office, Year 2000 Computing Crisis: A Testing
Guide, GAO/AIMD-10.1.21 (Washington, D.C.: November 1998).
o since implementing the Electronic Disability Collect System, SSA has
encountered software performance issues, such as slow response times when
moving from screen to screen, which have contributed to longer waits in
reception areas, scheduling fewer claimant appointments, and backlogs in
post-entitlement work; and
o edits for claimant information provided on Internet applications
generate additional edits after the data are input to the Electronic
Disability Collect System. Having to respond to the additional edits has
contributed to increased SSA field office claims representatives'
workloads.
Resolving all critical problems identified in its testing of the AeDib
components and then assessing how these components function in an
integrated environment before proceeding to national roll-out are
essential to gauging the system's overall successful operation.
Nonetheless, SSA does not plan to conduct end-to-end testing of the
electronic disability system prior to national implementation. SSA has
maintained that its pilot tests of each component of the electronic
disability system would be sufficient to evaluate the system's
functionality and performance.
Without resolution of all critical problems and end-to-end testing, SSA
cannot make fully informed decisions about the system it delivers, risks
having a system that will not work effectively, and could incur future
unanticipated costs for systems adjustments or corrections.
Our prior reports7 have noted that effective software development
processes and procedures are essential to producing high-quality software
products and ensuring that the software works as intended.
In response to problems encountered during its prior effort with the
Reengineered Disability System, SSA established key software process
improvement procedures to guide its software development activities. These
procedures include
o using plans to manage projects,
o tracking and overseeing initiatives to measure progress,
o performing quality assurance reviews to ensure compliance with policies
and procedures, and
o conducting software validation and systems certifications to ensure the
effectiveness of and users' satisfaction with the software.
Implementation of these processes should help the agency ensure that the
system
being developed is ready for production and will meet users' requirements.
7U.S. General Accounting Office, Customs Service Modernization: Serious
Management and Technical Weaknesses Must Be Corrected, GAO/AIMD-99-41
(Washington, D.C., Feb. 26, 1999); Customs Service Modernization:
Ineffective Software Development Processes Increase Customs System
Development Risks, GAO/AIMD-99-35 (Washington, D.C., February 11, 1999);
and Air Traffic Control: Immature Software Acquisition Processes Increase
FAA System Acquisition Risks, GAO-AIMD-97-47 (Washington, D.C., March 21,
1997).
SSA is adhering to some of its software development processes and
procedures for the AeDib initiative. For example, it is using plans to
monitor software development activities and is performing quality
assurance reviews. However, the agency has not consistently implemented
certain key procedures. For example, we found no evidence that
o users approved software validation plans to show that they agreed with
what was being produced, and
o all systems currently in production were certified for release, to
indicate that the systems were developed in accordance with specified
requirements.
Without software validation and systems certification, SSA lacks assurance
that the system is ready for production and will be acceptable to its end
users.
Best practices and federal guidance advocate risk management to identify
facts and circumstances that decrease the probability of failing to meet
project commitments before they manifest themselves as cost, schedule, and
performance shortfalls.
SSA has developed a risk management plan to guide the identification and
mitigation of AeDib program and project risks. The plan requires SSA to
ensure that the necessary risk assessments and mitigation strategies are
developed and implemented to reduce risks and achieve schedule and
performance goals.
SSA has identified system and security risks for the five AeDib projects.
For example, it noted
o the need for Document Management Architecture interfaces with other
AeDib components to be properly defined, tested, integrated, and installed
to avoid jeopardizing AeDib success, and
o the need for completion and approval of systems security plans for all
five projects.
However, SSA has not yet finalized the mitigation strategies needed to
help prevent circumstances that could impede successful project outcomes.
SSA had previously stated that its mitigation strategies would be
completed by December 15, 2003. However, according to agency officials,
the draft strategies are currently under review by the Office of the Chief
Information Officer and a date for finalizing them has not been
established. Until the strategies are finalized, SSA will not be in a
position to cost-effectively plan for and prevent circumstances that could
impede project success.
Office of Management and Budget guidance8 states that the purpose of a
costbenefit analysis is to promote efficient resource allocation through
well-informed decision making when initiating, renewing, or expanding
programs or projects that would result in a series of measurable benefits
or costs extending for 3 or more years.
SSA's guidance states that a cost-benefit analysis is to be conducted when
there is a major capital investment, potential major human resources
impact, and the investment is in competition with other projects for
funding and resources.
We previously reported that SSA's estimates of costs and benefits were
based on critical assumptions that were not substantiated:
o Critical infrastructure costs, such as outsourced scanning and imaging
(keying and indexing), telecommunications, and disaster recovery, were not
included in the analysis.
o The estimated electronic receipt of 30 percent of medical evidence by
2004 was not based on sound data or tested in pilots.
8Office of Management and Budget Circular A-94 (Revised Transmittal Memo
No. 64), October 29, 1992.
Validating the estimates is necessary to review assumptions that were
made, and to continuously update and ensure that costs and benefits
reflect the current situation and are accurate and realistic.
Our review determined that SSA's analysis had not been validated to ensure
the reasonableness of estimated AeDib costs and benefits. SSA agreed with
our assessment and stated that it planned to use the results of pilot
tests to validate cost and performance data included in the analysis.
However, agency officials could not state when the validation of costs and
benefits would be performed.
Without a validated cost-benefit analysis, SSA lacks assurance that its
AeDib cost estimates are reliable and that anticipated benefits will be
realized.
As partners in the disability determination process, stakeholders can
offer valuable insight regarding existing work processes and information
technology needs, and their involvement in the systems development
initiative is essential for ensuring AeDib's acceptance and use.
Our work on cross-organizational collaborative projects identified
effective communication and outreach as a key practice that contributes to
success.9 The research indicated that to achieve this, a specific outreach
plan specifying tasks and mechanisms may be needed.
In July we reported on the need for SSA to resolve stakeholder concerns to
ensure AeDib's acceptance and use, and to take additional steps to consult
with the medical community.10 SSA acknowledged the importance of ensuring
sound relations with stakeholders and the need to take additional actions
toward addressing stakeholders' concerns.
9U.S. General Accounting Office, Electronic Government: Potential Exists
for Enhancing Collaboration on Four Initiatives, GAO-04-6 (Washington,
D.C.: Oct. 10, 2003).
10GAO-03-1113R.
SSA stated that it has increased its communications about AeDib with state
DDSs and medical providers, and the agency has established some specific
communication mechanisms (e.g., ad hoc meetings, Internet television
broadcasts, and an AeDib intranet site).
State DDSs, however, have varying views and concerns about their
involvement with the AeDib initiative.
o DDSs associated with the Document Management Architecture pilot (North
Carolina, Illinois, and California) agreed that SSA has increased
communications and is working more closely with them.
o Four states that are not participating in the pilot (Iowa, Nebraska,
New York, and Virginia) cited no improvement in SSA's overall
communications with them about the development and implementation of the
electronic disability system.
o The National Council of Disability Determination Directors, which
represents the DDSs, continues to have concerns about SSA's communications
approach, stating that the AeDib steering committee is only being used as
a vehicle for reporting decisions already made, and that improved
communication with DDSs is still needed.
While the Commissioner previously stated that SSA was consulting with
stakeholders and the medical community, agency officials could not
articulate a comprehensive plan for effectively involving all stakeholders
and consulting with the medical community.
Without a clear and comprehensive plan for communicating with its
stakeholders and the medical community, SSA risks not obtaining vital
end-user buy-in and acceptance of the electronic disability system and the
medical community's support in providing electronic medical evidence that
is crucial to achieving anticipated benefits.
SSA is beginning its implementation of an electronic disability system
without fully evaluating pilot test results or ensuring that all critical
problems have been addressed, and with no plans for end-to-end testing to
guide its system implementation. Thus, SSA lacks assurance that the system
will operate as intended in a national environment-making the agency
highly susceptible to failure in its current effort to achieve an
electronic disability claims processing capability.
While SSA has established software process improvement procedures,
critical user concurrence for all software validations and systems
certifications are not evident. Without such concurrence, SSA lacks
assurance that it is building a system that will meet users' requirements.
Further, although SSA had planned to complete strategies for mitigating
program and project risks by last December, it has yet to finalize them.
SSA is beginning its national rollout of the electronic disability system
without a validated cost-benefit analysis. Lacking a reliable analysis
based on validated data, the agency is assuming the risk that its
anticipated costs may be inconsistent with planned spending for the
initiative and that anticipated measurable benefits may not be realized.
Finally, SSA has taken some steps to improve its communications with key
AeDib stakeholders and users; however, the agency has not yet articulated
a comprehensive plan for ensuring that all stakeholders and users are
fully involved in the initiative and that their concerns are adequately
addressed. State DDSs currently have mixed views regarding their
involvement, with DDS representatives continuing to see a need for
improved communications with SSA about critical AeDib decisions.
Given the technological complexity, scope, and size of the AeDib
initiative, it is essential that SSA take the necessary steps to avoid the
substantial risks of moving into production with a system that does not
perform as intended. Accordingly, we recommend that the Commissioner of
Social Security, before proceeding with the AeDib national rollout,
o ensure that all critical problems identified in pilot testing of the
electronic disability system components are resolved and that end-to-end
testing of the interrelated systems is performed;
o ensure that users have approved of software developed and that systems
have been certified for production;
o establish a revised time frame for and expedite actions toward
finalizing AeDib risk mitigation strategies,
o validate all AeDib cost and benefit estimates, and
o implement a communications plan that clearly and comprehensively
conveys SSA's approach for effectively addressing disability stakeholders'
and users' concerns and ensuring their full involvement in the AeDib
initiative.
SSA's Deputy Commissioner for Finance, Assessment and Management provided
an e-mail message containing the agency's comments on a draft of the
briefing slides. In its comments, SSA took issue with many of our
findings.
SSA disagreed with the need for end-to-end testing, stating that such
testing would have been too slow for its accelerated strategy. In
addition, it stated that examples discussed in our slides did not support
our position, noting, for example, that the Electronic Disability Collect
System performance issues highlighted in our slides had not appeared until
the software was in full production across the country.
We maintain that end-to-end testing is essential for AeDib. The
acceleration of such a technologically complex initiative, with its
multiple interrelated components, elevates the need for end-to-end testing
to help SSA reduce system risks, many of which were identified during
pilot tests of each component. The fact that performance issues did not
appear until the Electronic Disability Collect System software was in
production, and that high edit rates were discovered when data from
Internet applications were input into that system, is further evidence of
the need for end-to-end testing to provide SSA with the opportunity to
evaluate both the functionality and performance of all systems components
prior to national rollout.
Regarding examples of technical problems that we noted for the Document
Management Architecture pilots, SSA stated that (1) it had completed a
technical evaluation of the Document Management Architecture in three
pilot states and (2) that four of five examples identified in our slides
were resolved before SSA made a final decision to roll out to Mississippi.
In addition, SSA stated that our finding regarding the 19-inch monitor
size prejudged a conclusion that the screens were too small.
While SSA stated that it had completed technical evaluations of the
Document Management Architecture pilot efforts in the three pilot offices,
it did not provide evidence of its complete evaluations during our review.
Specifically, SSA provided evaluation results covering tests performed in
the North Carolina and Illinois DDSs only during the July through
September 2003 time frame. Further, on January 20, SSA provided a software
release supporting the electronic folder capability to the pilot offices,
leaving little time to ensure that the software was fully tested and
evaluated and that all critical problems were resolved prior to beginning
its national rollout.
In addition, while we acknowledge that SSA may have resolved four of the
five problems noted in our slides, the examples that we cited constituted
only a subset of the numerous problems that were noted in the Document
Management Architecture evaluations provided for our review. Our analysis
of the North Carolina and Illinois pilot evaluations and SSA's problem
tracking system report, and our discussions with officials of all three
state DDSs participating in the pilots, revealed that, as certain software
performance issues were resolved, other problems appeared, and that DDS
staff participating in the pilots were overwhelmed as workloads and
backlogs increased.
Finally, it was not our judgment or conclusion that SSA's monitor size was
too small. Rather, DDS claims examiners informed us that the 19-inch
monitors were too small for viewing two documents on split screen. For
example, during our visit to the North Carolina pilot site, claims
examiners participating in tests of the Document Management Architecture
and Electronic Disability Collect System stated that the small size of the
monitors had affected their ability to effectively adjudicate a claim. In
discussing this matter with SSA's Deputy Associate Commissioner for
Systems and Associate Commissioner for Disability Programs, both stated
that the agency was aware of DDS staffs' concerns about the size of the
monitors and that SSA planned to send an ergonomics team to those sites to
investigate this and other issues affecting staff using the system. At the
time of review, the study had not been completed.
In discussing the October 2004 completion date of its legacy systems
migration, SSA stated that, while technically correct, our discussion did
not acknowledge that DDS legacy software migration had been completed in
all but 3 of the 54 state DDS offices and that those three offices would
be completed in February 2004.
We have modified our slides to reflect that SSA has completed legacy
software migration in all but three DDS offices.
SSA also stated that the DDSs have or will shortly have the Electronic
Disability Collect System electronic folder interface (EFI). It noted that
its more expansive EFI software is currently running in 11 states, and
expansion of that will occur in concert with the Document Management
Architecture rollout.
SSA's recent documentation stated that it has made the Electronic
Disability Collect System electronic folder interface software available
to 15 DDS states, including the three Document Management Architecture
pilot offices. However, SSA has yet to complete the necessary DDS legacy
systems software-including its critical Document Management Architecture
interface software-to enable these components to process disability
claims. Until this is accomplished, existing disability processing
components will not have the capability to interface with or share
information captured in the full electronic folder.
SSA stated that it has followed processes and procedures for software
development and disagreed with our findings that it did not have approved
software validation plans and had not properly certified all software
releases.
While we agree that SSA has established key software process improvement
procedures to guide its software development activities, we found that SSA
was not consistently adhering to its established procedures. At the time
of our review, SSA could not provide the necessary documentation to
demonstrate that users had approved of the AeDib software's being
produced. SSA's Inspector General had reported the same findings in July
2003, and in discussions with us last December, reiterated this continuing
concern.
SSA stated that its security risk assessment documentation is no longer in
draft and can be made available to us for review.
SSA's actions toward finalizing its security level risk assessments for
AeDib are commendable, and we look forward to reviewing this
documentation. However, our recent reviews of AeDib found that the agency
had not finalized mitigation strategies for its program-level or
security-level risk assessments. SSA officials informed us that the draft
mitigation strategies were under review by the Office of the Chief
Information Officer and that no date for finalizing them had been
established. We continue to believe that these strategies are needed to
help prevent circumstances that could impede successful project outcomes.
SSA believes that its 30-percent electronic medical evidence is a viable
goal, and continued to disagree with our view that its cost-benefit
analysis did not consider proper infrastructure costs. However, SSA did
agree that validation of its cost and benefit assumptions is important and
stated that it is actively planning that work.
We are encouraged that SSA is planning to validate its cost and benefit
assumptions. In noting SSA's 30-percent electronic medical evidence goal,
our intent was to highlight the importance of SSA's validating its
cost-benefit analysis to ensure the reasonableness of its estimated costs
and benefits. The assumption that we highlighted was one of many included
in SSA's costbenefit analysis that had not been validated. Further, it
continues to be our position that all relevant costs need to be reflected
in SSA's cost-benefit analysis. Office of Management and Budget guidance
states that cost-benefit analyses should include comprehensive estimates
of all direct and indirect costs associated with a project. Without a
validated cost-benefit analysis, and the inclusion of missing critical
cost elements, SSA lacks assurance that its AeDib cost estimates are
reliable and that anticipated benefits will be realized.
Finally, regarding our findings on communication and consultation with
disability stakeholders, SSA stated that AeDib information is pervasive
and available to all stakeholders/users, and that the agency continues to
work collaboratively with them. It added that outreach to external groups
will ramp up in relation to national rollout.
We acknowledge that SSA has made additional efforts to work
collaboratively with its stakeholders regarding AeDib. However, we found
during our analyses of documentation and in discussions with disability
stakeholders that there are still varying views and concerns about SSA's
approach for implementing AeDib. Due to the complexity of the AeDib
initiative, and the many concerns that continue to be voiced about SSA's
approach, we continue to believe that having a clear and comprehensive
plan before national rollout is vital to ensuring end-user buy-in and
acceptance of the electronic disability system, and to realizing
anticipated benefits.
Appendix II
Comments from the Social Security Administration
Appendix II
Comments from the Social Security
Administration
Appendix II
Comments from the Social Security
Administration
Appendix II
Comments from the Social Security
Administration
Appendix II
Comments from the Social Security
Administration
GAO's Mission The General Accounting Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony
The fastest and easiest way to obtain copies of GAO documents at no cost
is through the Internet. GAO's Web site (www.gao.gov) contains abstracts
and fulltext files of current reports and testimony and an expanding
archive of older products. The Web site features a search engine to help
you locate documents using key words and phrases. You can print these
documents in their entirety, including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document files.
To have GAO e-mail this list to you every afternoon, go to www.gao.gov and
select "Subscribe to e-mail alerts" under the "Order GAO Products"
heading.
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:
U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C.
20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
To Report Fraud, Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htmWaste, and Abuse in E-mail:
[email protected] Federal Programs Automated answering system: (800)
424-5454 or (202) 512-7470
Public Affairs Jeff Nelligan, Managing Director, [email protected] (202)
512-4800 U.S. General Accounting Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
Presorted Standard
Postage & Fees Paid
GAO
Permit No. GI00
United States
General Accounting Office
Washington, D.C. 20548-0001
Official Business
Penalty for Private Use $300
Address Service Requested
*** End of document. ***