Social Security Administration: Software Development Process Improvements
Started But Work Remains (Letter Report, 01/28/98, GAO/AIMD-98-39).
Pursuant to a congressional request, GAO reviewed the status of the
Social Security Administration's (SSA) software development process
improvement efforts.
GAO noted that: (1) SSA has initiated a number of actions to improve its
software development capability; (2) among other things, it has: (a)
launched a formal software process improvement program and initiated
pilot projects to test improved software development processes; (b)
acquired the assistance of the Software Engineering Institute (SEI) to
help it assess the strengths and weaknesses in its current software
development processes and to assist in implementing the improvement
program; and (c) established a management steering committee and a
software engineering process group within the Office of Systems to
oversee software process improvement activities; (3) these are positive
steps that should help position SSA to improve its software development
capability; (4) although these initiatives are under way, SSA has not
yet established key elements of its software process improvement program
that are needed to measure the progress and success of its improvement
efforts; (5) in particular, SSA has not yet defined specific, measurable
goals for its software process improvement program or established the
baseline data that it will use to assess its progress in achieving these
goals; and (6) without this essential information, SSA cannot be assured
of the extent to which its improvement efforts will result in the
consistent and cost-effective production of high-quality products.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: AIMD-98-39
TITLE: Social Security Administration: Software Development
Process Improvements Started But Work Remains
DATE: 01/28/98
SUBJECT: Management information systems
Computer software verification and validation
Social security benefits
Computer software
Systems conversions
Federal agency reorganization
Computer networks
IDENTIFIER: Software Capability Maturity Model
CMM
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to the Chairman, Subcommittee on Social Security, Committee on
Ways and Means, House of Representatives
January 1998
SOCIAL SECURITY ADMINISTRATION -
SOFTWARE DEVELOPMENT PROCESS
IMPROVEMENTS STARTED BUT WORK
REMAINS
GAO/AIMD-98-39
Software Process Improvements
(511216)
Abbreviations
=============================================================== ABBREV
CMM - Capability Maturity Model
DDS - Disability Determination Service
IWS/LAN - Intelligent Workstation/Local Area Network
RDS - Reengineered Disability System
SEI - Software Engineering Institute
SSA - Social Security Administration
Letter
=============================================================== LETTER
B-276524
January 28, 1998
The Honorable Jim Bunning
Chairman, Subcommittee on Social Security
Committee on Ways and Means
House of Representatives
Dear Mr. Chairman:
As you know, the Social Security Administration (SSA) is in the
process of redesigning its work processes and modernizing its
computer systems in order to better serve an increasing beneficiary
population and achieve improvements in productivity. A key aspect of
this modernization program is the agency's transition from its
current centralized mainframe-based computer processing environment
to a client/server processing environment\1 in which its Intelligent
Workstation/Local Area Network (IWS/LAN)\2 will serve as the basic
automation infrastructure.
Software developed for the new client/server systems will be critical
to ensuring that the modernized processes function as intended and
achieve the desired productivity outcomes. However, software
development has been identified by many experts as one of the
riskiest and most costly aspects of systems development. SSA has
recognized weaknesses in its own software development capability and
has begun taking steps to improve the processes and methods that it
uses to develop software. As requested, our objective was to review
the status of SSA's software development process improvement efforts.
--------------------
\1 In a client/server environment, servers and individual
workstations are all capable of performing tasks that previously only
the mainframe computer could accomplish. This can sometimes result
in improvements over mainframe performance.
\2 In June 1996, SSA awarded a national IWS/LAN contract to modernize
and standardize the distributed processing environment in its
headquarters and field components and in state Disability
Determination Services (DDS). Phase I of the IWS/LAN initiative will
provide 56,000 workstations and 1,742 local area networks to SSA and
DDS offices nationwide.
RESULTS IN BRIEF
------------------------------------------------------------ Letter :1
SSA has initiated a number of actions to improve its software
development capability. Among other things, it has (1) launched a
formal software process improvement program and initiated pilot
projects to test improved software development processes, (2)
acquired the assistance of the Software Engineering Institute (SEI)\3
to help it assess the strengths and weaknesses in its current
software development processes and to assist in implementing the
improvement program, and (3) established a management steering
committee and a software engineering process group within the Office
of Systems to oversee software process improvement activities. These
are positive steps that should help position SSA to improve its
software development capability.
Although these initiatives are underway, SSA has not yet established
key elements of its software process improvement program that are
needed to measure the progress and success of its improvement
efforts. In particular, SSA has not yet defined specific, measurable
goals for its software process improvement program or established the
baseline data that it will use to assess its progress in achieving
these goals. Without this essential information, SSA cannot be
assured of the extent to which its improvement efforts will result in
the consistent and cost-effective production of high-quality
products.
--------------------
\3 SEI is a nationally recognized, federally funded research and
development center established at Carnegie Mellon University in
Pittsburgh, Pennsylvania, to address software development issues.
BACKGROUND
------------------------------------------------------------ Letter :2
SSA is currently undertaking a multiyear, multibillion dollar systems
modernization effort that is intended to replace aging equipment,
support current and future redesigned work processes, and improve
productivity. The cornerstone of this modernization effort is the
agency's transition from its current centralized, mainframe-based
computer processing environment to a highly distributed client/server
processing environment. The IWS/LAN infrastructure--consisting of
networks of intelligent workstations connected to each other and to
SSA's mainframe computers--is intended to provide SSA with the
initial computing framework for using client/server technology to
achieve cost savings and improve customer service by distributing
selected processes and information closer to where they are needed.
Through fiscal year 1997, SSA had reported spending approximately
$565 million on acquiring workstations, local area networks, and
other services to support the IWS/LAN infrastructure.\4
Software development is a critical component of the modernization
initiative. SSA's Office of Systems, with contractors' assistance,
is designing and developing a new generation of software that is
scheduled to operate on the IWS/LAN to support redesigned work
processes in a client/server environment. It has selected the
disability claims process as the first major redesign effort and is
currently developing the software--referred to as the Reengineered
Disability System (RDS)--that is intended to automate this redesigned
process. RDS is scheduled for national implementation on the IWS/LAN
from July 1999 through May 2001.
To help SSA's software development efforts succeed, however, it is
important that the agency have disciplined and consistent software
development practices that produce high-quality software within
budget and on schedule. In September 1996, we reported that SSA had
experienced problems in developing RDS, which contributed to a delay
of more than 2 years in its scheduled implementation.\5
These problems included (1) using programmers with insufficient
experience, (2) using software development tools that did not perform
effectively, and (3) establishing initial software development
schedules that were too optimistic. During that same month, a
contractor's preliminary risk assessment of SSA's client/server
transition strategy identified various risks associated with the
existing software development processes, including ineffective
requirements definition and inadequate configuration management.
In addition, SSA is currently facing the critical challenge of
ensuring that its information systems are Year 2000 compliant. By
the end of this century, SSA must review all of its computer software
and make the changes needed to ensure that its systems can correctly
process information relating to dates. These changes affect not only
its new network but computer programs operating on both its mainframe
and personal computers. We recently reported that while SSA has made
significant progress in its Year 2000 efforts, it faces the risk that
not all of its mission-critical systems will be corrected by the turn
of the century.\6 At particular risk are the systems used by state
DDSs to help SSA process disability claims.
Making software process improvements to address problems such as
those SSA faces is considered to be a challenging undertaking for any
organization. To guide agencies in assessing the strengths and
weaknesses of their software development processes, SEI, in the late
1980s, developed the Capability Maturity Model (CMM). CMM is
organized into five levels that characterize an organization's
software process maturity. As shown in table 1, these levels range
from initial (level 1), characterized by ad hoc and chaotic
processes, to optimizing (level 5), characterized by continuous
process improvement based upon analysis and quantitative data.
Table 1
Software Capability Maturity Model
Levels and Descriptions
Leve
l Name Description
---- ------------ --------------------------------------------------
5 Optimizing Continuous process improvement is enabled by
quantitative feedback from the process and from
piloting innovative ideas and technologies.
4 Managed Detailed measures of the software process and
product quality are collected. Both the software
process and products are quantitatively understood
and controlled.
3 Defined The software process for both management and
engineering activities is documented,
standardized, and integrated into a standard
software process for the organization. All
projects use an approved, tailored version of the
organization's standard software process for
developing and maintaining software.
2 Repeatable Basic project management processes are established
to track cost, schedule, and functionality. The
necessary process discipline is in place to repeat
earlier successes on projects with similar
applications.
1 Initial The software process is characterized as ad hoc
and occasionally even chaotic. Few processes are
defined and success depends on individual effort.
----------------------------------------------------------------------
Source: Software Engineering Institute, Carnegie Mellon University,
1993.
Further, to assist agencies in implementing effective software
process improvement programs, SEI developed the IDEAL\SM model, which
defines five phases of process improvement activity.\7 These phases
are:
Initiating. Management determines that there is a business reason
for improving their processes, sets general process improvement
goals, and sponsors a process improvement program.
Diagnosing. Using CMM, the current practices of the organization are
appraised and characterized. Results of the assessment are
documented and recommendations are made regarding areas in which to
focus improvement efforts.
Establishing. Based on the results of the diagnosing phase and the
general goals that were defined in the initiating phase, the
organization develops a strategy for improvement, prioritizes
activities, and formulates measurable goals. Process action teams
are established to develop action plans for improvement.
Acting. Action plans are implemented through pilots. The results of
pilots are evaluated and action plans are modified as appropriate.
When proven effective, action plans are implemented throughout the
organization.
Learning. After the new processes have been in place for some time,
their effectiveness is evaluated, communicated throughout the
organization, and, as appropriate, used to formulate new action plans
to ensure that goals are achieved.
--------------------
\4 The other services include site preparation, support services and
training, and telecommunications and maintenance.
\5 Social Security Administration: Effective Leadership Needed to
Meet Daunting Challenges (GAO/AIMD-96-196, September 12, 1996).
\6 Social Security Administration: Significant Progress Made in Year
2000 Effort, But Key Risks Remain (GAO/AIMD-98-6, October 22, 1997).
\7 IDEAL\SM: A User's Guide for Software Process Improvement, SEI,
February 1996.
SCOPE AND METHODOLOGY
------------------------------------------------------------ Letter :3
To determine the status of SSA's efforts to improve its software
development processes, we analyzed key documents, including SSA's
Software Process Improvement Program Management Plan, dated April
1997, Client/Server Transition Strategy: Preliminary Risk
Assessment, dated September 1996, and relevant systems and strategic
planning documents, such as the Information Systems Plan. In
addition, to determine the status of specific projects being
undertaken by contractors in support of the improvement initiatives,
we reviewed the statements of work for contractor services and final
deliverables, such as baseline assessments and software process
improvement reports. We did not independently verify the accuracy of
information reported in the contractors' assessments of SSA's
software development processes.
We analyzed SEI's IDEAL\SM: A User's Guide for Software Process
Improvement, dated February 1996, which SSA is using to implement and
manage its process improvement program, to determine whether SSA's
current and planned software development practices are consistent
with this guidance. We reviewed additional SEI reports, including
Moving On Up: Data and Experience Doing CMM-Based Process
Improvement, dated August 1995, to identify successful practices of
organizations applying CMM-based process improvements. In addition,
we reviewed documents discussing the implementation schedule,
technical strategies, and risks associated with SSA's development of
the RDS software application to obtain information on the agency's
experiences in software development. However, we did not
specifically evaluate the progress of SSA's ongoing effort to develop
RDS.
To further support our assessment of the actions that SSA is taking
to improve its software development capability, we interviewed the
Deputy Commissioner for Systems and other systems officials directly
involved in implementing the improvement initiative, the General
Services Administration official responsible for administering the
support contract for SSA's client/server software development
assessment, and representatives of the contractors involved in this
initiative.
We performed our work from March 1997 through November 1997 in
accordance with generally accepted government auditing standards.
SSA provided written comments on a draft of this report. These
comments are discussed in the "Agency Comments and Our Evaluation"
section and are reprinted in appendix I.
SSA HAS INITIATED ACTIONS TO
IMPROVE ITS SOFTWARE
DEVELOPMENT CAPABILITY
------------------------------------------------------------ Letter :4
Recognizing the need to improve its software development capability,
SSA has launched a formal software process improvement program and
initiated pilot projects to test improved software development
processes. In doing so, it is seeking to achieve a repeatable (level
2) level of software capability maturity. SSA acquired the
assistance of SEI to help formulate and implement the improvement
program. It adopted SEI's CMM as the framework for assessing the
current state of its software development capability and establishing
improvement priorities and is following SEI's IDEAL\SM model as the
methodology for implementing and managing its software process
improvement actions. Further, SSA has developed a software process
improvement schedule identifying the specific phases and tasks that
it plans to undertake to complete the implementation of its improved
software development processes by June 2000.
Consistent with SEI's IDEAL\SM model, SSA has performed a number of
the steps recommended for the initiating, diagnosing, and
establishing phases of its software process improvement program and
is beginning to implement steps under the acting phase. It has put
in place the initial management infrastructure to support and
facilitate its software process improvement initiatives by
establishing a management steering committee to raise organizational
awareness of the improvement program and a software engineering
process group to oversee the development and implementation of the
software process activities. It has also established various work
groups to carry out the specific process improvement initiatives.\8
SSA also has undertaken two assessments of the maturity of its
existing software development processes, focusing on identifying (1)
effective software development policies and procedures already being
used within the agency and (2) key software development process areas
needing improvement. SEI's CMM specifies key process areas and
criteria that must be addressed to achieve a particular software
development maturity level. The key process areas for level 2
(repeatable) maturity--the level that SSA is currently seeking to
achieve--are (1) requirements management, (2) software project
planning, (3) software project tracking and oversight, (4) software
subcontract management, (5) software quality assurance, and (6)
software configuration management.
With SEI's assistance, SSA conducted a self-assessment to determine
the strengths and weaknesses of its current processes for developing
and maintaining software, which are primarily mainframe-oriented.
This self-assessment identified 22 weaknesses in the level 2 key
process areas. For example, the assessment found within the area of
software project planning that risks had not been identified,
assessed, or documented for some projects, and within the area of
software project tracking and oversight, that the results and
performance of some projects had not been tracked against key
elements--such as costs, schedules, and risks--of SSA's software
project plans.
In addition, a support contractor hired to identify client/server
best practices and assist SSA in transitioning to a client/server
environment performed an independent assessment of the agency's
software development processes. This assessment focused specifically
on identifying the strengths and weaknesses in SSA's ability to
develop client/server software. It, too, identified weaknesses in
SSA's software development practices. For example, the assessment
identified within the area of requirements management a need to
improve practices for defining system requirements and
specifications, and in the area of project planning, a need to
improve practices for scheduling and estimating the cost of software
development efforts.
Based on the findings identified in both assessments, SSA developed a
software process improvement program implementation plan. This plan
will be used during the acting phase to, among other things, direct
the software development activities of three pilot projects that the
Office of Systems intends to undertake to help institutionalize the
software process improvements.\9 According to the implementation
plan, project teams for the three pilots are expected to test and
evaluate improved software development processes to address the
identified weaknesses. SSA initiated its pilot activities in
September 1997 and, at the conclusion of our review, had begun
developing the policies and procedures that it will use to test each
of the key process areas. SSA expects to complete all of the pilots
by March 1999, after which it will finalize its strategy for
implementing the improved software development processes.
--------------------
\8 The management steering committee is comprised primarily of senior
managers, including the deputy and assistant deputy commissioners for
systems and associate commissioners within the Office of Systems; the
software engineering process group is comprised primarily of division
director-level representatives within the Office of Systems; and
technical working groups include line staff from the various systems
components.
\9 The three projects that SSA plans to pilot are (1) the Customer
Help and Information Program, a workstation-based, client/server
system that is to assist SSA in providing customers with consistent,
accurate responses to telephone requests; (2) the Consolidated
Development Worksheet, a project in which software will be developed
to consolidate information, such as diaries, alerts, and actions
items, that is used by various SSA systems in processing claims,
post-entitlement, and other workloads; and (3) the Earnings
Management Information Operational Data Store that in its first
release is to provide an integrated suite of applications for
accessing and updating earnings information.
SOFTWARE PROCESS IMPROVEMENT
PROGRAM LACKS MEASURABLE GOALS
------------------------------------------------------------ Letter :5
Although SSA has made important progress in its efforts to improve
its software development processes, its improvement program does not
yet include specific, measurable goals and baseline data that are
essential to helping it achieve a repeatable (level 2) software
development capability. Without measurable goals and baseline data,
SSA does not yet have critical information needed to guide its
improvement efforts and to provide evidence that the efforts are
resulting in more consistent, cost-effective, and timely production
of higher quality products.
According to SEI's IDEAL\SM: A User's Guide for Software Process
Improvement, clearly defined and measurable goals are necessary to
provide guidance and to assist in developing tactics for improving
the software development process. They also allow for objective
measurement of the improvement results. SEI prescribes that general
goals of the improvement program be defined during the initiating
phase based on the business needs of the organization. These general
goals are used, in conjunction with baseline data on the agency's
existing processes (such as software size estimates, defects
identified, and calendar time for project completion), to develop
specific short- and long-term measurable goals during the
establishing phase. For example, one general goal could be to make
software projects more predictable in terms of cost and schedule. If
the measurement baseline established that 80 percent of the
organization's current projects exceed their original cost and
schedule estimates by more than 25 percent, then the specific,
measurable goal could be to improve that measure such that 80 percent
of all projects are completed within 10 percent of their original
cost and schedule estimates within 2 years.
At the conclusion of our review, SSA had established general goals
for its improvement program that included (1) achieving a repeatable
(level 2) software capability maturity and (2) creating a software
development environment that encourages continuous improvements in
quality, productivity, and timeliness. However, it had not yet
established specific, measurable improvement goals for its software
development processes, nor had it defined the actual baseline data
that it will use to monitor progress in achieving its goals.
SSA's Deputy Commissioner for Systems and the director of the
software process improvement program told us that they recognized the
importance of and need for establishing specific, measurable goals
for the agency's improvement program. However, they stated that the
agency has not yet been able to define such goals for the improvement
initiatives because it has not traditionally maintained baseline data
on its software development projects that would be required to make
such determinations. These officials told us that they intend to
develop the necessary measures based on data compiled during the
pilots being conducted to test improved software development
processes.
While SSA officials stated that they intend to develop the necessary
measures during the pilots, at the time of our review they did not
have a detailed strategy explaining how the pilots will be used to
generate these measures. SSA contracted with the Gartner Group to
develop a strategic measurement plan which recommends a general
framework and steps for establishing consistent, repeatable processes
to collect and track measurements; it also has outlined within its
implementation plan the general strategy it intends to use for the
three pilots. However, neither of these documents provided detailed
information on how and in what time frames baseline data and
specific, measurable goals will be derived from the pilots. Without
an explicit strategy and time frames for generating baseline data and
measurable goals, SSA cannot ensure that it will have essential
information to monitor the progress of its improvement efforts.
CONCLUSIONS
------------------------------------------------------------ Letter :6
Recognizing the need to reassess its software development processes
in light of transitioning to a client/server processing environment,
SSA is taking important steps to improve its software development
capability. If successfully completed, these actions should help
position the agency to strengthen its processes for developing
quality software products. However, until SSA develops baseline data
and establishes specific measurable goals for its improvement
initiatives as part of its pilot projects, it will not have necessary
information to monitor its progress toward achieving its intended
software process improvements.
RECOMMENDATION
------------------------------------------------------------ Letter :7
To strengthen SSA's software process improvement program, we
recommend that, as part of its recently initiated pilot projects, the
Commissioner of Social Security direct the Deputy Commissioner for
Systems to develop and implement plans that explicitly articulate
SSA's strategy and time frames for (1) developing baseline data, (2)
identifying specific, measurable goals for its improvement
initiative, and (3) monitoring and measuring progress in achieving
these goals.
AGENCY COMMENTS AND OUR
EVALUATION
------------------------------------------------------------ Letter :8
In commenting on a draft of this report, SSA agreed with our
recommendation and described actions that it is undertaking to
develop a plan for its measurement activities. These actions include
obtaining support for its pilot projects from the Gartner Group and
working with SEI to formulate a plan that will include (1) tasks and
time frames for developing baseline data, (2) measurable goals for
the implementation of CMM-compliant processes, and (3) methods for
measuring progress against established goals.
We are encouraged by SSA's response and will continue to monitor the
agency's progress in implementing its software improvement effort.
---------------------------------------------------------- Letter :8.1
As agreed with your office, unless you publicly announce the contents
of this report earlier, we plan no further distribution until 30 days
from its date. At that time, we will send copies to the Commissioner
of Social Security; the Director of the Office of Management and
Budget; appropriate congressional committees; and other interested
parties. Copies will also be made available to others upon request.
Please contact me at (202) 512-6253 or by e:mail at
[email protected] if you have any questions concerning this
report. Major contributors to this report are listed in appendix II.
Sincerely yours,
Joel C. Willemssen
Director, Civil Agencies Information Systems
(See figure in printed edition.)Appendix I
COMMENTS FROM THE SOCIAL SECURITY
ADMINISTRATION
============================================================== Letter
(See figure in printed edition.)
MAJOR CONTRIBUTORS TO THIS REPORT
========================================================== Appendix II
ACCOUNTING AND INFORMATION
MANAGEMENT DIVISION, WASHINGTON,
D.C.
Valerie C. Melvin, Assistant Director
Leonard J. Latham, Technical Assistant Director
Michael A. Alexander, Senior Information Systems Analyst
Gwendolyn M. Adelekun, Business Process Analyst
*** End of document. ***