HUD Information Systems: Immature Software Acquisition Capability
Increases Project Risks (14-SEP-01, GAO-01-962).
The Department of Housing and Urban Development (HUD) routinely
acquires new information systems and enhancements to manage and
support its various programs and operations. GAO has designated
HUD's major program areas as high risk, in part because the
department's information and financial management systems were
poorly integrated, ineffective, and generally unreliable. HUD has
been endeavoring to improve its systems to better support its
missions and management reforms. HUD did not fully satisfy the
requirements for any of the "repeatable" key process areas GAO
reviewed. Although HUD has several software acquisition process
strengths, GAO found a large number of software process
weaknesses in all key process areas evaluated: requirements
development and management, project management, contract tracking
and oversight, and software evaluation. As a result, its
processes for acquiring software are immature and can be
characterized as ad hoc, sometimes chaotic, and not repeatable
across projects. HUD is aware that it has software acquisition
weaknesses, has stated its commitment to improving its software
and system acquisition processes, and will begin a process
improvement effort in the near future.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-01-962
ACCNO: A01787
TITLE: HUD Information Systems: Immature Software Acquisition
Capability Increases Project Risks
DATE: 09/14/2001
SUBJECT: Computer software
Financial management systems
Integrated software
ADP procurement
Information systems
HUD Central Accounting and Program
System
HUD Empowerment Information System
HUD Public and Indian Housing
Information Center System
HUD Real Estate Management System
HUD Resident Assessment Subsystem
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Testimony. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-01-962
Report to the Ranking Minority Member, Subcommittee on Housing, and
Transportation, Committee on Banking, Housing and Urban Affairs, U. S.
Senate
United States General Accounting Office
GAO
September 2001 HUD INFORMATION SYSTEMS
Immature Software Acquisition Capability Increases Project Risks
GAO- 01- 962
Page i GAO- 01- 962 HUD Information Systems Letter 1
Executive Summary 2
Chapter 1 Introduction 7 HUD Relies on Information Systems 7 The Software
Acquisition Capability Maturity Model Provides a
Means of Assessing an Organization?s Ability to Manage Software Acquisitions
9 Objective, Scope, and Methodology 12
Chapter 2 HUD?s Requirements Development and Management Processes Are Not
Repeatable 15
Chapter 3 HUD?s Project Management Processes Are Not Repeatable 28
Chapter 4 HUD?s Contract Tracking and Oversight Processes Are Not Repeatable
41
Chapter 5 HUD?s Software Evaluation Processes Are Not Repeatable 54
Chapter 6 Conclusions, Recommendations, and Agency Comments 67
Appendix I Comments From the Department of Housing and Urban Development 69
Contents
Page ii GAO- 01- 962 HUD Information Systems Appendix II GAO Contact and
Staff Acknowledgments 71
Tables
Table 1: Total Number of Key Process Area Strengths, Weaknesses, and
Observations for the Five Projects 5 Table 2: Key Process Areas for SA- CMM
Level 2 11 Table 3: Requirements Development and Management Findings for
the Public and Indian Housing Information Center System 18 Table 4:
Requirements Development and Management Findings for
the Real Estate Management System 20 Table 5: Requirements Development and
Management Findings for
the Resident Assessment Subsystem 22 Table 6: Requirements Development and
Management Findings for
HUD?s Central Accounting and Program System 24 Table 7: Requirements
Development and Management Findings for
the Empowerment Information System 26 Table 8: Project Management Findings
for the Public and Indian
Housing Information Center System 31 Table 9: Project Management Findings
for the Real Estate
Management System 33 Table 10: Project Management Findings for the Resident
Assessment Subsystem 35 Table 11: Project Management Findings for HUD?s
Central
Accounting and Program System 37 Table 12: Project Management Findings for
the Empowerment
Information System 39 Table 13: Contract Tracking and Oversight Findings for
the Public
and Indian Housing Information Center System 44 Table 14: Contract Tracking
and Oversight Findings for the Real
Estate Management System 46 Table 15: Contract Tracking and Oversight
Findings for the
Resident Assessment Subsystem 48 Table 16: Contract Tracking and Oversight
Findings for HUD?s
Central Accounting and Program System 50 Table 17: Contract Tracking and
Oversight Findings for the
Empowerment Information System 52 Table 18: Software Evaluation Findings for
the Public and Indian
Housing Information Center System 57 Table 19: Software Evaluation Findings
for the Real Estate
Management System 59
Page iii GAO- 01- 962 HUD Information Systems
Table 20: Software Evaluation Findings for the Resident Assessment Subsystem
61 Table 21: Software Evaluation Findings for HUD?s Central
Accounting and Program System 63 Table 22: Software Evaluation Findings for
the Empowerment
Information System 65
Figures
Figure 1: SA- CMM Levels and Descriptions 10 Figure 2: Requirements
Development and Management Summary 17 Figure 3: Project Management Summary
30 Figure 4: Contract Tracking and Oversight Summary 43 Figure 5: Software
Evaluation Summary 56
Abbreviations
CIO Chief Information Officer EIS Empowerment Information System GAO General
Accounting Office HUD Department of Housing and Urban Development HUDCAPS
HUD Central Accounting and Program System PIC Public and Indian Housing
Information Center system RASS Resident Assessment Subsystem REMS Real
Estate Management System SA- CMM Software Acquisition Capability Maturity
Model SEI Software Engineering Institute
Page 1 GAO- 01- 962 HUD Information Systems
September 14, 2001 The Honorable Wayne Allard Ranking Minority Member
Subcommittee on Housing and Transportation Committee on Banking, Housing,
and Urban Affairs United States Senate
Dear Senator Allard: The accompanying report is the first in a series of
reports responding to your February 2001 request that we examine a range of
management issues at the Department of Housing and Urban Development. This
report discusses our assessment of the department?s capability to acquire
software. We are making recommendations to the Secretary of Housing and
Urban Development to assist the department in improving this capability.
We are sending copies of this report to the Secretary of Housing and Urban
Development, the Director of the Office of Management and Budget, and other
congressional committees. We will also make copies available to other
interested parties upon request.
If you have questions or wish to discuss the issues in this report, please
contact me at (202) 512- 6240 or by email at koontzl@ gao. gov. An
additional GAO contact and staff acknowledgements are listed in appendix II.
Sincerely yours, Linda D. Koontz Director, Information Management Issues
United States General Accounting Office Washington, DC 20548
Executive Summary Page 2 GAO- 01- 962 HUD Information Systems
The Department of Housing and Urban Development (HUD) routinely acquires new
information systems and enhancements to manage and support its various
programs and operations. GAO has designated HUD?s major program areas as
high risk, in part because the department?s information and financial
management systems were poorly integrated, ineffective, and generally
unreliable. 1 HUD has been endeavoring to improve its systems to better
support its missions and management reforms.
Because of the importance of information and financial management systems
and related improvement efforts to meeting HUD?s mission, the Ranking
Minority Member, Subcommittee on Housing and Transportation, Senate
Committee on Banking, Housing, and Urban Affairs, requested that GAO review
the maturity of HUD?s software acquisition processes. More mature processes
increase the likelihood that the software acquired will meet an
organization?s needs.
HUD is the principal federal agency responsible for housing and community
development programs. HUD?s mission includes making housing affordable
(through mortgage insurance for multifamily housing and through rental
assistance), helping to revitalize localities, and encouraging home
ownership. In fiscal year 2001, the department?s budget was about $32
billion, including $360 million for information technology investments.
High- quality software is essential for HUD?s information systems to provide
reliable management, financial, and administrative information and support
the department?s many programs. The quality of software is governed largely
by the quality of the processes involved in developing or acquiring it and
in maintaining it. Carnegie Mellon University?s Software Engineering
Institute (SEI), recognized for its expertise in software processes, has
developed models and methods that define and determine organizations?
software process maturity. Together, these models and
1 GAO High- Risk Program (GAO/ AIMD- 94- 72R, January 27, 1994); High- Risk
Series: Department of Housing and Urban Development (GAO/ HR- 95- 11,
February 1995); HighRisk Series: Department of Housing and Urban Development
(GAO/ HR- 97- 12, February 1997); Major Management Challenges and Program
Risks: Department of Housing and Urban Development (GAO/ OCG- 99- 8, January
1999); Major Management Challenges and Program Risks: Department of Housing
and Urban Development (GAO- 01- 248, January 2001). Executive Summary
Purpose Background
Executive Summary Page 3 GAO- 01- 962 HUD Information Systems
methods provide a logical framework for baselining an organization?s current
process capabilities (i. e., determining what practices are effectively
implemented (strengths), not effectively implemented (weaknesses), or
contain mixed or inconclusive evidence (observations)) and providing a
structured plan for incremental process improvement.
Using SEI?s software acquisition capability maturity model SM and SEI?s
software capability evaluation method, 2 GAO analysts trained at SEI
evaluated HUD?s software acquisition maturity in four of seven key process
areas 3 that are necessary to attain a ?repeatable? level of process
maturity. The ?repeatable? level of process maturity is the second level on
SEI?s five- level scale. An organization at the repeatable level of process
maturity has the necessary process discipline in place to repeat earlier
successes on similar projects. Organizations that do not satisfy the
requirements for the repeatable level are by default judged to be at the
?initial? level of maturity. This means that their processes are immature,
ad hoc, and sometimes even chaotic, with few of the processes defined and
success dependent mainly on the heroic efforts of individuals.
In this evaluation, GAO examined five ongoing projects selected by HUD as
being representative of the department?s current software acquisition
practices. 4 These were the Public and Indian Housing Information Center
system, the Real Estate Management System, the Resident Assessment
Subsystem, HUD?s Central Accounting and Program System, and the Empowerment
Information System.
HUD did not fully satisfy the requirements for any of the ?repeatable? key
process areas we reviewed. While HUD has several software acquisition
2 Capability Maturity Model SM is the service mark of Carnegie Mellon
University, and CMM is registered in the U. S. Patent and Trademark Office.
GAO used Software Acquisition Capability Maturity Model (SA- CMM) Version
1.2 (CMU/ SEI- 99- TR- 002, April 1999), the latest version of the model.
3 The four key process areas GAO evaluated are requirements development and
management, project management, contract tracking and oversight, and
software evaluation. GAO did not evaluate the remaining three key process
areas- software acquisition planning, solicitation, and transition to
support- because HUD?s project teams had not recently performed activities
in these areas.
4 GAO also asked that the five projects be (1) major efforts with large
software acquisition components, (2) managed by integrated project teams,
(3) at different stages of the life cycle, and (4) among HUD?s best managed
projects. Results in Brief
Executive Summary Page 4 GAO- 01- 962 HUD Information Systems
process strengths, GAO found a large number of software process weaknesses
in all key process areas evaluated: requirements development and management,
project management, contract tracking and oversight, and software
evaluation. As a result, its processes for acquiring software are immature
and can be characterized as ad hoc, sometimes chaotic, and not repeatable
across projects. These weaknesses can lead to systems that do not meet the
information needs of management and staff, do not provide support for needed
programs and operations, and cost more and take longer than expected to
complete. HUD is aware that it has software acquisition weaknesses, has
stated its commitment to improving its software and system acquisition
processes, and will begin a process improvement effort in the near future.
HUD did not meet the requirements for the repeatable level of maturity in
the four key process areas we reviewed. Of the 310 key practices GAO
evaluated, 50 percent constituted strengths (i. e., key practice was
effectively implemented), 39 percent were weaknesses (i. e., key practice
was not effectively implemented), 10 percent were rated as observations, and
1 percent were not rated.
Certain weaknesses were systemic, recurring in most or all of the key
process areas. For example, HUD has no overall policy for the acquisition of
software; the majority of project teams did not develop plans to conduct
various activities; none of the teams measured the status of activities; the
majority of the teams did not report regularly to management on progress and
problems; and most project managers did not require regular reports on
progress and problems. These weaknesses can lead to systems that do not meet
HUD?s information needs, do not effectively support HUD?s programs and
services, and cost more and take longer to complete.
To reach the repeatable level of maturity, HUD must overcome the key
practice weaknesses identified in this report. Table 1 summarizes the
strengths and weaknesses for the five projects. Principal Finding:
HUD?s Software Acquisition Processes Are Immature
Executive Summary Page 5 GAO- 01- 962 HUD Information Systems
Table 1: Total Number of Key Process Area Strengths, Weaknesses, and
Observations for the Five Projects
Key practice ratings Key process area Strengths Weaknesses Observations Not
rated
Requirements development and management 27 31 12 0 Project management 45 25
10 0 Contract tracking and oversight 45 33 4 3 Evaluation 37 33 5 0
Total 154 122 31 3
HUD acknowledged that it has software acquisition weaknesses and stated its
commitment to improving its software and system acquisition processes. The
department has prepared a statement of work to obtain contractor support to
begin a software process improvement effort. One of the tasks the contract
will include is development of a software process improvement plan.
To successfully complete such an effort, HUD?s plan must address several
important issues. To be comprehensive, this plan should include the results
of this review, measurable goals and time frames, estimates of resource
requirements, and time frames to reach the repeatable level. In addition, it
should address the systemic weaknesses that GAO found. Finally, the plan
should include steps to address the key process areas GAO did not review, as
well as steps to ensure that all ongoing and new software acquisition
projects adopt processes that meet the requirements for the repeatable
level.
To improve HUD?s software acquisition capabilities, GAO recommends that the
Secretary of Housing and Urban Development direct the HUD Chief Information
Officer to develop and implement a comprehensive plan for software
acquisition process improvement that is based on the software capability
results in this report and specifies measurable goals and time frames, sets
priorities for initiatives, estimates resource requirements (for trained
staff and funding), and defines a process improvement management structure.
Also, to address the systemic weaknesses mentioned above, the plan should
contain steps to Recommendations for
Executive Action
Executive Summary Page 6 GAO- 01- 962 HUD Information Systems
develop a comprehensive policy for the acquisition of software,
require plans for specific acquisition activities,
measure and track the status of activities performed by the software
acquisition teams, and
report regularly to management on progress and problems. In addition, to
ensure that all aspects of software acquisition are addressed, we recommend
that the Secretary direct the CIO to
assess HUD?s maturity in the three key process areas that could not be
evaluated by GAO and include any needed improvement actions in the
comprehensive plan for software acquisition process improvement;
ensure that all new software acquisition projects in HUD adopt processes
that satisfy at least SA- CMM level 2 requirements; and
ensure that process improvement activities are initiated for all ongoing
software acquisition projects.
In providing written comments on a draft of our report, HUD agreed with our
assessment of its software acquisition processes and our recommendations for
strengthening them, and stated that our assessment and recommendations will
be addressed as part of its process improvement effort. This effort should
help the department improve its ability to acquire systems that meet
management and staff information needs and efficiently support programs and
operations. Agency Comments
Chapter 1: Introduction Page 7 GAO- 01- 962 HUD Information Systems
The Department of Housing and Urban Development (HUD), established in 1965,
has as its primary mission ensuring a decent, safe, and sanitary home and
suitable living environment for every American. HUD?s mission includes
making housing affordable for about 4 million low- income households by
insuring loans for multifamily rental housing and by providing rental
assistance. Its mission also includes helping to revitalize over 4,000
localities through community development programs. The department encourages
homeownership by providing, through its Federal Housing Administration,
mortgage insurance for about 7 million homeowners who otherwise might not
have qualified for loans. HUD is one of the nation?s largest financial
institutions, responsible for managing about $508 billion in insured
mortgages and $570 billion in guarantees of mortgage- backed securities.
HUD?s budget in fiscal year 2001 was about $32 billion, including $360
million for information technology investments.
Like many federal agencies, HUD relies on its information and financial
management systems to help carry out its missions. However, past reports by
us and others showed that HUD was plagued by poorly integrated, ineffective,
and generally unreliable information systems that did not satisfy management
needs or provide adequate controls. 1 We designated HUD?s program areas as
high risk in part because of widespread problems with information and
financial management systems. In 1997, HUD began a management reform effort
designed to transform its culture, manage its programs and staff more
effectively, ultimately restore public trust in the department, and
modernize it for the 21st century.
In January 2001, we reported that HUD had been making credible progress
towards addressing its management reform goals. 2 However, we also reported
that information and financial systems remained an area of management
challenge because of unresolved issues. In addition, in its March 1, 2001,
financial statement audit report, the Office of the HUD Inspector General
listed two material internal control weaknesses related to systems. The
report stated that HUD needs to (1) complete improvements to its financial
management systems to meet its program and financial management needs and
(2) enhance the Federal Housing
1 GAO- 01- 248, January 2001; GAO/ OCG- 99- 8, January 1999; GAO/ HR- 97-
12, February 1997; GAO/ HR- 95- 11, February 1995; GAO/ AIMD- 94- 72R,
January 27, 1994. 2 GAO- 01- 248, January 2001. Chapter 1: Introduction
HUD Relies on Information Systems
Chapter 1: Introduction Page 8 GAO- 01- 962 HUD Information Systems
Administration?s information systems to more effectively support its
business practices.
HUD is aware that information systems remain an area of challenge. To help
address some of the challenges, HUD plans to (1) institute more disciplined
processes in its information technology capital investment planning reviews
(quarterly reviews that assess how well projects are functioning and attempt
to alert management to potential problems), (2) continue training its
project managers in new project management techniques, (3) complete efforts
to bring all software products under standard change control and
configuration management 3 and designate a separate organization to address
configuration management issues, (4) improve HUD?s end- to- end testing 4
capabilities, and (5) obtain and deploy new software for project- level
status reporting.
To manage specific software acquisitions, HUD uses integrated project teams.
These teams include representatives from the business organization( s) that
will use the acquired software to do their work; information technology
staff to oversee the contractor and review contractor work products; and
contract and procurement staff to assist the team in contract management.
Policy and procedures for software acquisition are the responsibility of
HUD?s Chief Information Officer.
3 Configuration management is a discipline applying technical and
administrative direction and surveillance to identify and document the
functional and physical characteristics of hardware or software, control
changes to these characteristics, record and report change processing, and
verify compliance of hardware or software with specified requirements.
4 End- to- end testing is done to verify that a defined set of interrelated
systems, which collectively support an organizational core business area or
function, interoperate as intended in an operational environment. This
includes not only those systems owned and managed by the organization but
also those external systems with which they interface.
Chapter 1: Introduction Page 9 GAO- 01- 962 HUD Information Systems
The software acquisition capability maturity model (SA- CMM), 5 developed by
Carnegie Mellon?s Software Engineering Institute (SEI), is used to measure
an organization?s capability to manage the acquisition of software. SEI?s
expertise in and methods for software process assessment are recognized and
accepted worldwide throughout the industry. The model defines five levels of
software acquisition maturity. Each level of maturity (except level 1)
indicates process capability and identifies key process areas. For a
maturity level to be achieved, all key process areas related to that level
must be implemented effectively.
The first level of process capability is level 2 (referred to as the
repeatable level), where basic management processes are established to track
performance, cost, and schedule, and the necessary discipline is in place to
repeat earlier successes on similar projects. Organizations that do not
effectively implement all key process areas for the repeatable level are, by
default, at level 1 or the initial level of maturity. Level 1 processes can
be described as immature, ad hoc, and sometimes chaotic; success in software
acquisition for these organizations is usually dependent upon the ability
and commitment of the staff involved. Figure 1 explains further the five-
level software acquisition model.
5 Software Acquisition Capability Maturity Model (SA- CMM), Version 1. 2,
Software Engineering Institute, CMU/ SEI- 99- TR- 002 (April 1999). The
Software
Acquisition Capability Maturity Model Provides a Means of Assessing an
Organization?s Ability to Manage Software Acquisitions
Chapter 1: Introduction Page 10 GAO- 01- 962 HUD Information Systems
Figure 1: SA- CMM Levels and Descriptions
Table 2 identifies each of the seven key process areas associated with the
repeatable level of maturity.
The software acquisition process is characterized as ad hoc, and
occasionally even chaotic. Few processes are defined and success depends on
individual effort.
Level 1: Initial
Basic project management processes are established to track performance,
cost, and schedule. The necessary process discipline is in place to repeat
earlier successes on projects in similar domains.
Level 2: Repeatable
The acquisition organization?s software acquisition process is documented,
standardized, and established as the standard software acquisition process.
All projects use an approved, tailored version of the organization?s
standard software acquisition process for acquiring their software products
and services.
Level 3: Defined Level 5: Optimizing
Level 4: Managed
Continuous process improvement is empowered by quantitative feedback from
the process and from piloting innovative ideas and technologies.
Detailed measures of quality of the software acquisition processes,
products, and services are collected. The software processes, products, and
services are quantitatively understood and controlled.
Disciplined process
Standard consistent process
Predictable process
Continuously improving process
Chapter 1: Introduction Page 11 GAO- 01- 962 HUD Information Systems
Table 2: Key Process Areas for SA- CMM Level 2 SA- CMM level 2 key process
areas Description
Software acquisition planning Ensuring that reasonable planning for the
software
acquisition is conducted and that all elements of the project are included.
Solicitation Ensuring that award is made to the contractor most
capable of satisfying the specified requirements. Requirements development
and management Establishing a common and unambiguous definition of
software acquisition requirements that is understood by the acquisition
team, system users, and contractor( s). Project management Managing the
activities of the project office and
supporting contractor( s) to ensure a timely, efficient, and effective
software acquisition. Contract tracking and oversight Ensuring that the
software activities under contract are
being performed in accordance with contract requirements and that products
and services will satisfy contract requirements. Evaluation Determining that
the acquired software products and
services satisfy contract requirements before acceptance. Transition to
support Providing for the transition of the software products being
acquired to their eventual support organization.
As established by the model, each key process area contains five common
features- commitment to perform, ability to perform, activities performed,
measurement and analysis, and verifying implementation- that indicate
whether the implementation and institutionalization of the key process area
can be effective, repeatable, and lasting. The common feature definitions
are as follows:
Commitment to perform: This feature describes the actions that the
organization takes to establish the process and ensure that it can endure.
Key practices typically involve establishing organizational policies and
sponsorship.
Ability to perform: This feature describes the preconditions that must
exist in the project or organization to competently implement the software
acquisition process. Key practices typically include assigning
responsibility and providing training.
Activities performed: This feature describes the roles and procedures
necessary to implement a key process area. Key practices typically involve
establishing plans and procedures, performing the work, tracking it, and
taking appropriate management actions.
Measurement and analysis: This feature describes the activities necessary
to measure progress and analyze the measurements. Key practices typically
involve defining the measurements to be taken and the
Chapter 1: Introduction Page 12 GAO- 01- 962 HUD Information Systems
analyses to be conducted to determine the status and effectiveness of the
activities performed.
Verifying implementation: This feature describes the steps the
organization must take to ensure that project activities are performed in
accordance with established processes. Key practices typically involve
regular reviews by management.
Each common feature consists of a number of key practices- specific
activities such as developing an organizational policy for software
acquisition, developing various plans for software acquisition activities,
and tracking a contractor?s progress. When an organization is evaluated
against the SA- CMM, comparisons of actual performance against a key
practice can result in one of four possible outcomes or ratings:
Strength: The key practice involved was effectively implemented.
Weakness: The key practice was ineffectively implemented or was not
implemented.
Observation: The key practice was evaluated, but cannot be characterized
as a strength because the (1) project team did not provide sufficient
evidence to support a strength rating or (2) key practice was only partially
performed.
Not rated: The key practice is not relevant to the project and was
therefore not rated.
To achieve the repeatable level, HUD would have to demonstrate that the key
practices related to that level were implemented effectively in its software
acquisition projects.
The Ranking Minority Member of the Subcommittee on Housing and
Transportation, Senate Committee on Banking, Housing, and Urban Affairs,
requested that we review HUD?s software acquisition capabilities. Our
objective for this review was to determine the maturity of HUD?s software
acquisition processes.
To determine HUD?s software acquisition process maturity, we applied the
Software Engineering Institute?s SA- CMM using our SEI- trained analysts. We
focused on the key process areas necessary to obtain a repeatable level of
maturity, the second level of SEI?s five- level model. We met with project
managers and project team members to determine whether and to Objective,
Scope, and
Methodology
Chapter 1: Introduction Page 13 GAO- 01- 962 HUD Information Systems
what extent they implemented each key practice and to obtain relevant
documentation. In accordance with the SEI model, for each key process area
we reviewed, 6 we evaluated HUD?s institutional policies and practices and
compared project- specific guidance and practices against the required key
practices.
More specifically, for each key practice we reviewed, we compared project-
specific documentation and practices against the criteria in the software
acquisition model. If the project met the criteria for the key practice
reviewed, we rated it as a strength. If the project did not meet the
criteria for the key practice reviewed, we rated it as a weakness. If the
evidence did not support a rating of a strength or a weakness, we rated it
as an observation. If the key practice was not relevant to a project, we
stated that the key practice was not rated.
We evaluated five HUD projects, each of which is described below. We asked
HUD to select five system acquisition projects that were representative of
HUD software acquisition practices and met the following criteria: all were
(1) major efforts with large software acquisition components, (2) managed by
integrated project teams, (3) at different stages of the life cycle, and (4)
among the best- managed acquisitions. HUD selected the following projects:
Public and Indian Housing Information Center system (PIC)- an
Internetbased system that collects information about the public housing
inventory managed by HUD and automates the processes for allocating program
funds to public housing authorities. It is a central repository for
information about public and Indian housing events and program areas as well
as for the housing inventory.
Real Estate Management System (REMS)- a system that provides automated
support to collect and maintain data on multifamily insured and assisted
properties and their ownership/ management entities and provides access to
data collected by HUD on physical and financial assessments.
6 We evaluated HUD on four of the seven key process areas- requirements
development and management, project management, contract tracking and
oversight, and software evaluation. We did not evaluate HUD on the three
other key process areas- software acquisition planning, solicitation, and
transition to support- because these were not recently performed for the
selected projects.
Chapter 1: Introduction Page 14 GAO- 01- 962 HUD Information Systems
Resident Assessment Subsystem (RASS)- a system that collects information
on public housing from residents and manages the resident assessment
process.
HUD Central Accounting and Program System (HUDCAPS)- HUD?s core accounting
and general ledger system. It serves as the standard accounting system for
all program areas.
Empowerment Information System (EIS)- an enterprisewide data warehouse
project that is to provide HUD users and business partners with access to
reports and analytical information across a large variety of program data
sources.
We performed our work from March 2001 through August 2001 in accordance with
generally accepted government auditing standards. We provided a draft of our
report to HUD for review, and the department?s comments are addressed in
chapter 6 and reprinted as appendix I.
Chapter 2: HUD?s Requirements Development and Management Processes Are Not
Repeatable
Page 15 GAO- 01- 962 HUD Information Systems
HUD did not satisfy the criteria for the repeatable level of maturity in
this key process area because of the number and severity of key practice
weaknesses found. As a result of these weaknesses, HUD is at greater risk of
acquiring software products that do not meet mission needs and that exceed
cost, schedule, and performance goals.
The purpose of requirements development and management is to establish and
maintain a common and unambiguous definition of software requirements 1
among the acquisition team, system users, and software development
contractor. Within this key process area, the SA- CMM specifies key
practices for each of the five common features that an organization must
implement effectively to achieve the repeatable level of maturity.
Generally, these practices include having a written organizational policy
for establishing and managing requirements allocated to software,
documenting plans for the development and management of requirements, having
documented processes for requirements development, including elicitation,
analysis, and verification, measuring and reporting on the status of
activities to management, appraising the impact of system- level
requirements changes on the software, and having a mechanism to ensure that
contractor- delivered work products meet the specified requirements.
Of the 70 key practices for this key process area, HUD had 27 strengths, 31
weaknesses and 12 observations. For the
Commitment to perform feature, there were two strengths, five weaknesses,
and three observations.
Ability to perform feature, there were nine strengths, five weaknesses,
and one observation.
Activities performed feature, there were ten strengths, 14 weaknesses, and
six observations.
Measurement and analysis feature, there were one strength, three
weaknesses, and one observation.
Verifying implementation feature, there were five strengths, four
weaknesses, and one observation.
In addition, none of the projects had strengths in all the key practices. 1
Requirements describe the functions that the software will perform to meet
user needs and provide support for business processes. Chapter 2: HUD?s
Requirements Development
and Management Processes Are Not Repeatable
Chapter 2: HUD?s Requirements Development and Management Processes Are Not
Repeatable
Page 16 GAO- 01- 962 HUD Information Systems
As a result of these weaknesses, HUD is exposed to increased risks that
acquired software will not meet its needs and that projects will exceed
cost, schedule, and performance goals. HUD acknowledges the need to improve
its requirements development and management processes and is committed to
improving its software and systems acquisition capability. The department
has prepared a statement of work to obtain contractor support to begin a
software process improvement effort. This document specifies that the
contractor would review HUD software acquisition processes, prioritize the
key processes and practices to address, and develop a plan to improve HUD?s
acquisition processes.
Details of our evaluation are provided in the following figure and tables.
Figure 2 provides a comprehensive listing of the strengths, weaknesses, and
observations for the requirements development and management key process
area. The specific findings supporting the practice ratings cited in figure
2 are in tables 3 through 7.
Chapter 2: HUD?s Requirements Development and Management Processes Are Not
Repeatable
Page 17 GAO- 01- 962 HUD Information Systems
Figure 2: Requirements Development and Management Summary Projects Common
features Key practices PIC REMS RASS HUDCAPS EIS
Commitment to perform 1 The acquisition organization has a written policy
for
establishing and managing the software- related contractual requirements.
Weakness Weakness Observation Observation Weakness
Commitment to perform 2 Responsibility has been designated for requirements
development and management activities. Weakness Strength Strength
Observation Weakness
Ability to perform 1 A group is responsible for performing requirements
development and management activities. Weakness Strength Strength
Observation Strength
Ability to perform 2 Adequate resources are provided for requirements
development and management activities. Weakness Weakness Strength Weakness
Weakness
Ability to perform 3 Individuals performing requirements development and
management activities have experience or receive training. Strength Strength
Strength Strength Strength
Activity performed 1 The project team performs its activities in accordance
with its documented requirements development and management plans. Weakness
Observation Strength Weakness Weakness
Activity performed 2 The project team develops, baselines, and maintains
software- related contractual requirements and places them under change
control early in the project, but not later than the release of the
solicitation package.
Observation Weakness Strength Strength Observation
Activity performed 3 The project team appraises system requirements
change requests for their impact on the software being acquired. Weakness
Weakness Strength Strength Weakness
Activity performed 4 The project team appraises changes to the
softwarerelated contractual requirements for their impact on
performance, architecture, supportability, system resource utilization, and
contract schedule and cost.
Weakness Weakness Strength Weakness Weakness
Activity performed 5 Bidirectional traceability between the contractual
requirements and the contractor team?s software products and services is
maintained throughout the effort.
Observation Weakness Strength Strength Weakness
Activity performed 6 The end user and other affected groups are involved in
the development of all software- related contractual requirements and any
subsequent change activity. Observation Strength Strength Observation
Weakness
Measurement and analysis 1 Measurements are made and used to determine the
status of the requirements development and management activities and
resultant products. Weakness Weakness Observation Weakness Strength
Verifying implementation 1 Requirements development and management
activities
are reviewed by the acquisition organization on a periodic basis. Strength
Observation Weakness Strength Strength
Verifying implementation 2 Requirements development and management
activities
are reviewed by the project manager on both a periodic and an event- driven
basis. Weakness Strength Weakness Strength Weakness
Key Strength Strength: Key practice effectively implemented.
Weakness Weakness: Key practice not effectively implemented.
Observation Observation: Key practice evaluated; evidence not sufficient to
rate as a strength, or practice only partially performed.
Chapter 2: HUD?s Requirements Development and Management Processes Are Not
Repeatable
Page 18 GAO- 01- 962 HUD Information Systems
Table 3: Requirements Development and Management Findings for the Public and
Indian Housing Information Center System Public and Indian Housing
Information Center system Common feature Key practice Finding Rating
Commitment to perform 1 The acquisition organization has a written policy
for establishing and managing the software- related contractual
requirements.
HUD has no written policy for establishing and managing software- related
contractual requirements.
Weakness Commitment to perform 2 Responsibility has been designated for
requirements development and management activities.
Responsibility for requirements development and management has not been
designated.
Weakness Ability to perform 1 A group is responsible for performing
requirements development and management activities.
No group is assigned responsibility for performing requirements development
and management.
Weakness Ability to perform 2 Adequate resources are provided for
requirements development and management activities.
There is no mechanism to identify resource needs and ensure that they are
provided to the project. Team members indicated that they do not have
adequate resources for performing these activities.
Weakness Ability to perform 3 Individuals performing requirements
development and management activities have experience or receive training.
Project team members performing requirements development and management
activities have experience.
Strength Activity performed 1 The project team performs its activities in
accordance with its documented requirements development and management
plans.
There is no requirements development and management plan, and so the
activities are not performed in accordance with it.
Weakness Activity performed 2 The project team develops, baselines,
and maintains software- related contractual requirements and places them
under change control early in the project, but not later than the release of
the solicitation package.
The project team developed and baselined software- related requirements
before the solicitation package was released. The project team provided no
evidence of change control.
Observation Activity performed 3 The project team appraises system
requirements change requests for their impact on the software being
acquired.
The project team does not assess system requirements change requests for
their impact on the software being acquired.
Weakness Activity performed 4 The project team appraises changes to
the software- related contractual requirements for their impact on
performance, architecture, supportability, system resource utilization, and
contract schedule and cost.
The project team does not assess the impact of software- related contractual
requirements changes on performance, architecture, supportability, and
resource utilization. Cost and schedule impacts are assessed.
Weakness Activity performed 5 Bidirectional traceability between the
contractual requirements and the contractor team?s software products and
services is maintained throughout the effort.
The project team maintains traceability between the contract deliverables
and the contractor?s task/ subtask specification numbering. Traceability to
HUD?s original requirements, however, is not maintained.
Observation Activity performed 6 The end user and other affected groups
are involved in the development of all software- related contractual
requirements and any subsequent change activity.
End users are involved in identifying software- related requirements for
development but are not involved in subsequent change activities.
Observation
Chapter 2: HUD?s Requirements Development and Management Processes Are Not
Repeatable
Page 19 GAO- 01- 962 HUD Information Systems
Public and Indian Housing Information Center system Common feature Key
practice Finding Rating
Measurement and analysis 1 Measurements are made and used to determine the
status of the requirements development and management activities and
resultant products.
No measurements are made of HUD?s requirements development and management
activities and resultant products.
Weakness Verifying implementation 1 Requirements development and
management activities are reviewed by the acquisition organization on a
periodic basis.
Project status, including requirements development and management
activities, is reviewed quarterly by acquisition management.
Strength Verifying implementation 2 Requirements development and
management activities are reviewed by the project manager on both a periodic
and an event- driven basis.
The project manager is not briefed on project activities and the project
manager?s reviews are not documented.
Weakness
Chapter 2: HUD?s Requirements Development and Management Processes Are Not
Repeatable
Page 20 GAO- 01- 962 HUD Information Systems
Table 4: Requirements Development and Management Findings for the Real
Estate Management System Real Estate Management System Common feature Key
practice Finding Rating
Commitment to perform 1 The acquisition organization has a written policy
for establishing and managing the software- related contractual
requirements.
HUD has no written policy for establishing and managing software- related
contractual requirements.
Weakness Commitment to perform 2 Responsibility has been designated for
requirements development and management activities.
Responsibility for requirements development and management activities is
designated in the project plan.
Strength Ability to perform 1 A group is responsible for performing
requirements development and management activities.
End users and project staff are responsible for performing requirements
development and management activities.
Strength Ability to perform 2 Adequate resources are provided for
requirements development and management activities.
There is no mechanism to identify resource needs and ensure that they are
provided to the project. The project manager said that the team does not
have adequate resources for performing these activities.
Weakness Ability to perform 3 Individuals performing requirements
development and management activities have experience or receive training.
Project team members performing requirements development and management
activities have experience.
Strength Activity performed 1 The project team performs its activities
in accordance with its documented requirements development and management
plans.
There is no requirements development and management plan, and so the
activities are not performed in accordance with it. The team does follow a
process for requirements gathering.
Observation Activity performed 2 The project team develops, baselines,
and maintains software- related contractual requirements and places them
under change control early in the project, but not later than the release of
the solicitation package.
The project team did not baseline requirements before solicitation. Task
orders do not show traceability to software related requirements.
Weakness Activity performed 3 The project team appraises system
requirements change requests for their impact on the software being
acquired.
The project team does not appraise system requirements change requests for
their impact on the software being acquired.
Weakness Activity performed 4 The project team appraises changes
to the software- related contractual requirements for their impact on
performance, architecture, support ability, system resource utilization, and
contract schedule and cost.
The project team does not assess the impact of software- related contractual
requirements changes on performance, architecture, supportability, and
resource utilization. Cost and schedule are assessed.
Weakness Activity performed 5 Bidirectional traceability between the
contractual requirements and the contractor team?s software products and
services is maintained throughout the effort.
The project team does not maintain bidirectional traceability between the
contractual requirements and contractor work products.
Weakness
Chapter 2: HUD?s Requirements Development and Management Processes Are Not
Repeatable
Page 21 GAO- 01- 962 HUD Information Systems
Real Estate Management System Common feature Key practice Finding Rating
Activity performed 6 The end user and other affected groups are involved in
the development of all software- related contractual requirements and any
subsequent change activity.
End users are involved in the development of and changes to software-
related contractual requirements.
Strength Measurement and analysis 1 Measurements are made and used to
determine the status of the requirements development and management
activities and resultant products.
No measurements are made of HUD?s requirements development and management
activities and resultant products.
Weakness Verifying implementation 1 Requirements development and
management activities are reviewed by the acquisition organization on a
periodic basis.
Requirements development and management activities are not routinely
reviewed by the acquisition organization. However, the project manager is a
senior official of the acquisition organization.
Observation Verifying implementation 2 Requirements development and
management activities are reviewed by the project manager on both a periodic
and an event- driven basis.
The project manager told us that he participates in biweekly status meetings
that include a review of requirements development and management issues. The
status meetings are documented.
Strength
Chapter 2: HUD?s Requirements Development and Management Processes Are Not
Repeatable
Page 22 GAO- 01- 962 HUD Information Systems
Table 5: Requirements Development and Management Findings for the Resident
Assessment Subsystem Resident Assessment Subsystem Common feature Key
practice Finding Rating
Commitment to perform 1 The acquisition organization has a written policy
for establishing and managing the software- related contractual
requirements.
HUD has no written policy for establishing and managing software- related
contractual requirements. The project team used HUD?s software development
methodology as its standard for establishing and managing software- related
contractual requirements.
Observation Commitment to perform 2 Responsibility has been designated
for requirements development and management activities.
Responsibility for requirements development and management is designated in
the project plan.
Strength Ability to perform 1 A group is responsible for performing
requirements development and management activities.
The project team is responsible for performing requirements development and
management activities.
Strength Ability to perform 2 Adequate resources are provided for
requirements development and management activities.
There is no mechanism to identify resource needs and ensure that they are
provided to the project. However, the project manager stated that they do
have adequate resources for performing requirements development and
management activities.
Strength Ability to perform 3 Individuals performing requirements
development and management activities have experience or receive training.
Project team members performing requirements development and management
activities have experience and have received training.
Strength Activity performed 1 The project team performs its
activities in accordance with its documented requirements development and
management plans.
The project team performs its activities in accordance with its requirements
development and management plan.
Strength Activity performed 2 The project team develops,
baselines, and maintains software related contractual requirements and
places them under change control early in the project, but not later than
the release of the solicitation package.
The project team developed and baselined contract requirements in its
functional requirements document and requirements traceability matrix, and
placed the requirements under change control.
Strength Activity performed 3 The project team appraises system
requirements change requests for their impact on the software being
acquired.
The project team assesses system requirements change requests for their
impact on the software being acquired.
Strength Activity performed 4 The project team appraises changes
to the software- related contractual requirements for their impact on
performance, architecture, support ability, system resource utilization, and
contract schedule and cost.
The project team appraises changes to the software- related contractual
requirements for impact on performance, architecture, support ability,
system resource utilization, and contract schedule and cost.
Strength
Chapter 2: HUD?s Requirements Development and Management Processes Are Not
Repeatable
Page 23 GAO- 01- 962 HUD Information Systems
Resident Assessment Subsystem Common feature Key practice Finding Rating
Activity performed 5 Bidirectional traceability between the contractual
requirements and the contractor team?s software products and services is
maintained throughout the effort.
The project team maintains bidirectional traceability between contractual
requirements and contractor work products.
Strength Activity performed 6 The end user and other affected
groups are involved in the development of all software- related contractual
requirements and any subsequent change activity.
End users are involved in the development of contractual requirements,
generation of system change requests, and testing.
Strength Measurement and analysis 1 Measurements are made and used to
determine the status of the requirements development and management
activities and resultant products.
Masurements are only made of HUD?s requirements development and management
products.
Observation Verifying implementation 1 Requirements development and
management activities are reviewed by the acquisition organization on a
periodic basis.
Requirements development and management activities are not routinely
reviewed by the acquisition organization. However, cost and schedule
performance is reviewed periodically by the acquisition organization.
Weakness Verifying implementation 2 Requirements development and
management activities are reviewed by the project manager on both a periodic
and an event- driven basis.
The project manager is not briefed on requirements development and
management activities, and the project manager?s reviews are not documented.
Weakness
Chapter 2: HUD?s Requirements Development and Management Processes Are Not
Repeatable
Page 24 GAO- 01- 962 HUD Information Systems
Table 6: Requirements Development and Management Findings for HUD?s Central
Accounting and Program System HUD?s Central Accounting and Program System
Common feature Key practice Finding Rating
Commitment to perform 1 The acquisition organization has a written policy
for establishing and managing the software- related contractual
requirements.
HUD has no written policy for establishing and managing the software-
related contractual requirements. The project team used HUD?s software
development methodology as its standard for establishing and managing
software- related contractual requirements.
Observation Commitment to perform 2 Responsibility has been designated for
requirements development and management activities.
The project team has been designated responsible for requirements
development and management, with no specific designation of what group or
person within the team.
Observation Ability to perform 1 A group is responsible for performing
requirements development and management activities.
The project manager told us that the project team is responsible for
requirements development and management activities.
Observation Ability to perform 2 Adequate resources are provided for
requirements development and management activities.
There is no mechanism to identify resource needs and ensure that they are
provided to the project. The project team indicated that they do not have
adequate resources for performing requirements development and management
activities.
Weakness Ability to perform 3 Individuals performing requirements
development and management activities have experience or receive training.
Project team members performing requirements development and management
activities have experience and have received training.
Strength Activity performed 1 The project team performs its activities
in accordance with its documented requirements development and management
plans.
There is no requirements development and management plan, and so the
activities are not performed in accordance with it.
Weakness Activity performed 2 The project team develops, baselines,
and maintains software- related contractual requirements and places them
under change control early in the project, but not later than the release of
the solicitation package.
The project team developed and baselined software- related contractual
requirements before the contractor began work, and has maintained the
requirements under configuration board control.
Strength Activity performed 3 The project team appraises system
requirements change requests for their impact on the software being
acquired.
The project team assesses all system requirements change requests for their
impact on the software being acquired.
Strength Activity performed 4 The project team appraises changes to
the software- related contractual requirements for their impact on
performance, architecture, support ability, system resource utilization, and
contract schedule and cost.
The project team does not assess the impact of software- related contractual
requirements changes on performance, architecture, supportability, and
system resource utilization. Cost is appraised.
Weakness
Chapter 2: HUD?s Requirements Development and Management Processes Are Not
Repeatable
Page 25 GAO- 01- 962 HUD Information Systems
HUD?s Central Accounting and Program System Common feature Key practice
Finding Rating
Activity performed 5 Bidirectional traceability between the contractual
requirements and the contractor team?s software products and services is
maintained throughout the effort.
The project team maintains bidirectional traceability between the
contractual requirements and the contractor team?s software products and
services.
Strength Activity performed 6 The end user and other affected groups
are involved in the development of all software- related contractual
requirements and any subsequent change activity.
End users and an independent verification and validation team were involved
in testing of changes. No evidence was provided to document user activity in
the development of requirements.
Observation Measurement and analysis 1 Measurements are made and used to
determine the status of the requirements development and management
activities and resultant products.
No measurements are made of HUD?s requirements development and management
activities or resultant products.
Weakness Verifying implementation 1 Requirements development and
management activities are reviewed by the acquisition organization on a
periodic basis.
Acquisition management is regularly updated on project status, including
requirements development and maintenance activities.
Strength Verifying implementation 2 Requirements development and
management activities are reviewed by the project manager on both a periodic
and an event- driven basis.
Project manager has regular, documented meetings to review requirements
development and management activities.
Strength
Chapter 2: HUD?s Requirements Development and Management Processes Are Not
Repeatable
Page 26 GAO- 01- 962 HUD Information Systems
Table 7: Requirements Development and Management Findings for the
Empowerment Information System Empowerment Information System Common feature
Key practice Finding Rating
Commitment to perform 1 The acquisition organization has a written policy
for establishing and managing the software- related contractual
requirements.
HUD has no written policy for establishing and managing softwarerelated
contractual requirements.
Weakness Commitment to perform 2 Responsibility has been designated for
requirements development and management activities.
Responsibility for requirements development and management has not been
designated.
Weakness Ability to perform 1 A group is responsible for performing
requirements development and management activities.
The project team is responsible for performing these activities. Strength
Ability to perform 2 Adequate resources are provided for requirements
development and management activities.
There is no mechanism to identify resource needs and ensure that they are
provided to the project. The project team stated that they do not have
adequate resources for performing requirements development and management
activities.
Weakness Ability to perform 3 Individuals performing requirements
development and management activities have experience or receive training.
Project team members have training in IT project management, including
requirements management.
Strength Activity performed 1 The project team performs its activities
in accordance with its documented requirements development and management
plans.
There is no requirements development and management plan, and so the
activities are not performed in accordance with it.
Weakness Activity performed 2 The project team develops, baselines,
and maintains software- related contractual requirements and places them
under change control early in the project, but not later than the release of
the solicitation package.
The project team developed high- level requirements. Detailed requirements
were developed by the contractor. No evidence was provided regarding change
control.
Observation Activity performed 3 The project team appraises system
requirements change requests for their impact on the software being
acquired.
The project team does not appraise system requirements changes for their
impact on the software being acquired.
Weakness Activity performed 4 The project team appraises changes to
the software- related contractual requirements for their impact on
performance, architecture, supportability, system resource utilization, and
contract schedule and cost.
The project team does not assess the impact of software- related contractual
requirements changes on performance, architecture, supportability, system
resource utilization, and contract schedule and cost.
Weakness Activity performed 5 Bidirectional traceability between the
contractual requirements and the contractor team?s software products and
services is maintained throughout the effort.
The project team does not maintain bidirectional traceability between the
contractual requirements and contractor work products.
Weakness
Chapter 2: HUD?s Requirements Development and Management Processes Are Not
Repeatable
Page 27 GAO- 01- 962 HUD Information Systems
Empowerment Information System Common feature Key practice Finding Rating
Activity performed 6 The end user and other affected groups are involved in
the development of all software- related contractual requirements and any
subsequent change activity.
No end users were involved in requirements development or subsequent change
activities.
Weakness Measurement and analysis 1 Measurements are made and used to
determine the status of the requirements development and management
activities and resultant products.
Measurements are made that identify the status of requirements development
and management activities.
Strength Verifying implementation 1 Requirements development and
management activities are reviewed by the acquisition organization on a
periodic basis.
Requirements development and management activities are reviewed periodically
by the acquisition organization.
Strength Verifying implementation 2 Requirements development and
management activities are reviewed by the project manager on both a periodic
and an event- driven basis.
The project manager told us that he is not briefed on activities, and the
project manager?s reviews are not documented.
Weakness
Chapter 3: HUD?s Project Management Processes Are Not Repeatable
Page 28 GAO- 01- 962 HUD Information Systems
HUD did not meet the criteria for the repeatable level in the project
management key process area because of the number and severity of key
practice weaknesses found. These weaknesses increase the risk that the
software acquisition project, the project team, and supporting contractors
will not be adequately managed, resulting in software that does not meet
mission needs or acquisitions that do not meet cost, schedule, and
performance goals.
The purpose of project management is to direct and oversee the activities of
the project team and supporting contractors to ensure a timely, efficient,
and effective software acquisition. Within this key process area, the SA-
CMM specifies key practices for each of the five common features that an
organization must implement effectively to achieve the repeatable level of
maturity. Generally, these practices include having a project team organized
to accomplish the project?s objective; having a written policy for the
management of the project; documenting plans for the activities of the
project team; measuring and controlling the cost, schedule, and performance
objectives throughout the software acquisition; and reporting to management
periodically on the status of project management activities.
Of the 80 key practices for this key process area, HUD had 45 strengths, 25
weaknesses, and 10 observations. For the
Commitment to perform feature, there were four strengths and six
weaknesses.
Ability to perform feature, there were 14 strengths, four weaknesses, and
two observations.
Activities performed feature, there were 22 strengths, seven weaknesses,
and six observations.
Measurement and analysis feature, there were no strengths, four
weaknesses, and one observation.
Verifying implementation feature, there were five strengths, four
weaknesses, and one observation.
In addition, none of the projects had strengths in all the key practices. As
a result of these weaknesses, HUD is exposed to increased risks that
software acquisition projects will exceed cost, schedule, and performance
goals, and that acquired software will not meet mission needs. HUD
acknowledges the need to improve its project management processes and is
committed to improving its software and system acquisition capability. The
department has prepared a statement of work to obtain contract Chapter 3:
HUD?s Project Management
Processes Are Not Repeatable
Chapter 3: HUD?s Project Management Processes Are Not Repeatable
Page 29 GAO- 01- 962 HUD Information Systems
support to begin a software process improvement effort. This document
specifies that the contractor would review HUD software acquisition
processes, prioritize the key processes and practices to address, and
develop a plan to improve HUD?s acquisition processes.
Details of our evaluation are provided in the following figure and tables.
Figure 3 provides a comprehensive listing of the strengths, weaknesses, and
observations for the project management key process area. The specific
findings supporting the practice ratings cited in figure 3 are in tables 8
through 12.
Chapter 3: HUD?s Project Management Processes Are Not Repeatable
Page 30 GAO- 01- 962 HUD Information Systems
Figure 3: Project Management Summary Projects Common features Key practices
PIC REMS RASS HUDCAPS EIS
Commitment to perform 1 The acquisition organization has a written policy
for
management of the software acquisition project. Weakness Weakness Weakness
Weakness Weakness
Commitment to perform 2 Responsibility has been designated for project
management activities. Weakness Strength Strength Strength Strength
Ability to perform 1 A group is responsible for managing software
acquisition activities. Observation Strength Strength Strength Strength
Ability to perform 2 Adequate resources for the project team are provided
for the duration of the project. Weakness Weakness Strength Weakness
Weakness
Ability to perform 3 When project trade- offs are necessary, the project
manager is permitted to alter the software acquisition cost, schedule, or
performance. Strength Strength Strength Strength Strength
Ability to perform 4 Individuals performing software acquisition
management activities have experience or receive training. Strength
Observation Strength Strength Strength
Activity performed 1 The project team performs its activities in accordance
with its documented software acquisition plans. Observation Strength
Strength Strength Strength
Activity performed 2 The roles, responsibilities, and authority of the
project
function are documented, maintained, and communicated to affected groups.
Weakness Observation Strength Strength Strength
Activity performed 3 The project team?s commitments, and changes to
commitments, are communicated to affected groups. Observation Strength
Strength Strength Strength
Activity performed 4 The project team tracks the cost, schedule, resource,
and technical risks of the project. Observation Weakness Strength Weakness
Strength
Activity performed 5 The project team tracks project issues, status,
execution, funding, and expenditures against project plans and takes action.
Observation Strength Strength Strength Strength
Activity performed 6 The project team implements a corrective action system
for the identification, recording, tracking, and correction of problems
discovered during the software acquisition. Weakness Observation Strength
Strength Strength
Activity performed 7 The project team keeps its plans current during the
life
of the project as replanning occurs, issues are resolved, and new risks are
discovered. Weakness Weakness Strength Weakness Strength
Measurement and analysis 1 Measurements are made and used to determine the
status of the project management activities and resultant products. Weakness
Weakness Observation Weakness Weakness
Verifying implementation 1 Project management activities are reviewed by the
acquisition organization on a periodic basis. Strength Weakness Strength
Strength Strength
Verifying implementation 2 Project management activities are reviewed by the
project manager on both a periodic and an event- driven basis. Weakness
Observation Weakness Strength Weakness
Key Strength Strength: Key practice effectively implemented.
Weakness Weakness: Key practice not effectively implemented.
Observation Observation: Key practice evaluated; evidence not sufficient to
rate as a strength, or practice only partially performed.
Chapter 3: HUD?s Project Management Processes Are Not Repeatable
Page 31 GAO- 01- 962 HUD Information Systems
Table 8: Project Management Findings for the Public and Indian Housing
Information Center System Public and Indian Housing Information Center
system Common feature Key practice Finding Rating
Commitment to perform 1 The acquisition organization has a written policy
for management of the software acquisition project.
HUD has no written policy for managing the acquisition of software products
and services.
Weakness Commitment to perform 2 Responsibility has been designated for
project management activities. The project team told us that project
management responsibilities had been
assigned, but no documentation was provided to support this statement.
Weakness Ability to perform 1 A group is responsible for managing
software acquisition activities. The project team told us that IT and
business project leads and support staff
are responsible for managing software acquisition activities, but no
documentation was provided to support this statement.
Observation Ability to perform 2 Adequate resources for the project team
are provided for the duration of the project.
There is no mechanism to identify resource needs and ensure that they are
provided to the project. Team members indicated that the team does not have
adequate resources for conducting project management activities.
Weakness Ability to perform 3 When project trade- offs are necessary,
the project manager is permitted to alter the software acquisition cost,
schedule, or performance.
The project manager is permitted to alter schedule or performance to ensure
that annual project budgets are not exceeded.
Strength Ability to perform 4 Individuals performing software
acquisition management activities have experience or receive training.
Project team members performing project management activities have
experience and project management training.
Strength Activity performed 1 The project team performs its activities in
accordance with its documented software acquisition plans.
The project team told us they did develop project management plans, but no
documentation was provided to support this statement.
Observation Activity performed 2 The roles, responsibilities, and authority
of the project function are documented, maintained, and communicated to
affected groups.
The project team has not documented the roles, responsibilities, and
authority of its members.
Weakness Activity performed 3 The project team?s commitments, and
changes to commitments, are communicated to affected groups.
The commitments and changes to commitments are communicated during small
informal group meetings, but they are not documented and no records are
kept.
Observation Activity performed 4 The project team tracks the cost,
schedule, resource, and technical risks of the project.
Cost and schedule are tracked, but resource and technical risks of the
project are not tracked.
Observation Activity performed 5 The project team tracks project issues,
status, execution, funding, and expenditures against project plans and takes
action.
The project team only tracks and takes action on changes to the project
schedule. Observation
Chapter 3: HUD?s Project Management Processes Are Not Repeatable
Page 32 GAO- 01- 962 HUD Information Systems
Public and Indian Housing Information Center system Common feature Key
practice Finding Rating
Activity performed 6 The project team implements a corrective action system
for the identification, recording, tracking, and correction of problems
discovered during the software acquisition.
The project team does not maintain a corrective action system for the
identification, recording, tracking, and correction of problems discovered
during the software acquisition.
Weakness Activity performed 7 The project team keeps its plans current
during the life of the project as replanning occurs, issues are resolved,
and new risks are discovered.
The project team does not keep its plans up to date over the life of the
project. Weakness
Measurement and analysis 1 Measurements are made and used to determine the
status of the project management activities and resultant products.
No measurements are made of HUD?s project management activities or resultant
products.
Weakness Verifying implementation 1 Project management activities are
reviewed by the acquisition organization on a periodic basis.
Acquisition management reviews project activities quarterly. Strength
Verifying implementation 2 Project management activities are reviewed by the
project manager on both a periodic and an event- driven basis.
The project manager stated that while he meets with members of his staff
daily to discuss the status of the project, meeting results and action items
are not documented.
Weakness
Chapter 3: HUD?s Project Management Processes Are Not Repeatable
Page 33 GAO- 01- 962 HUD Information Systems
Table 9: Project Management Findings for the Real Estate Management System
Real Estate Management System Common feature Key practice Finding Rating
Commitment to perform 1 The acquisition organization has a written policy
for management of the software acquisition project.
HUD has no written policy for managing the acquisition of software products
and services.
Weakness Commitment to perform 2 Responsibility has been designated for
project management activities. Responsibility for project management
activities has been assigned to the project
manager in the project management plan. Strength
Ability to perform 1 A group is responsible for managing software
acquisition activities. The project manager, along with business
and IT support staff, is responsible for project management activities.
Strength Ability to perform 2 Adequate resources for the project
team are provided for the duration of the project.
There is no mechanism to identify resource needs and ensure that they are
provided to the project. The project manager stated that the team does not
have adequate resources for conducting project management activities.
Weakness Ability to perform 3 When project trade- offs are
necessary, the project manager is permitted to alter the software
acquisition cost, schedule, or performance.
The project manager is permitted to alter schedule or performance to ensure
that annual project budgets are not exceeded.
Strength Ability to perform 4 Individuals performing software
acquisition management activities have experience or receive training.
The project team told us that individuals performing project management
activities have experience and have had training. However, the team did not
provide documentation to support this statement.
Observation Activity performed 1 The project team performs its activities
in accordance with its documented software acquisition plans.
Software acquisition project management plans are used to guide the
activities of the project team.
Strength Activity performed 2 The roles, responsibilities, and
authority of the project function are documented, maintained, and
communicated to affected groups.
Roles and responsibilities are defined in the project plan; however,
responsible team members are not named, which hinders effective
communication of this information.
Observation Activity performed 3 The project team?s commitments, and
changes to commitments, are communicated to affected groups.
The commitments, and changes to commitments, are communicated to affected
groups through weekly status meetings, emails, and telephone conversations.
Strength Activity performed 4 The project team tracks the cost,
schedule, resource, and technical risks of the project.
The project team does not track risks related to cost, schedule, resource,
and technical aspects of the project.
Weakness Activity performed 5 The project team tracks project issues,
status, execution, funding, and expenditures against project plans and takes
action.
The project manager tracks project status, execution, funding, and
expenditures, and changes the scope or schedule if necessary.
Strength
Chapter 3: HUD?s Project Management Processes Are Not Repeatable
Page 34 GAO- 01- 962 HUD Information Systems
Real Estate Management System Common feature Key practice Finding Rating
Activity performed 6 The project team implements a corrective action system
for the identification, recording, tracking, and correction of problems
discovered during the software acquisition.
The project team does not maintain a corrective action system for the
identification, recording, tracking, and correction of problems discovered
during the software acquisition. Minutes of biweekly status meetings
identify action items and who is responsible for each item.
Observation Activity performed 7 The project team keeps its plans
current during the life of the project as replanning occurs, issues are
resolved, and new risks are discovered.
The project team does not keep its plans up to date over the life of the
project. Weakness
Measurement and analysis 1 Measurements are made and used to determine the
status of the project management activities and resultant products.
No measurements are made of HUD?s project management activities or resultant
products.
Weakness Verifying implementation 1 Project management activities are
reviewed by the acquisition organization on a periodic basis.
Acquisition management does not regularly review project management
activities. Weakness
Verifying implementation 2 Project management activities are reviewed by the
project manager on both a periodic and an event- driven basis.
The project manager is involved with the project, and status meetings are
held regularly; however, the project manager?s reviews are not documented.
Observation
Chapter 3: HUD?s Project Management Processes Are Not Repeatable
Page 35 GAO- 01- 962 HUD Information Systems
Table 10: Project Management Findings for the Resident Assessment Subsystem
Resident Assessment Subsystem Common feature Key practice Finding Rating
Commitment to perform 1 The acquisition organization has a written policy
for management of the software acquisition project.
HUD has no written policy for managing the acquisition of software products
and services. The project team uses HUD?s software development methodology
where applicable.
Weakness Commitment to perform 2 Responsibility has been designated for
project management activities. Responsibility for project management
activities has been assigned to the IT and
business project leads. Strength
Ability to perform 1 A group is responsible for managing software
acquisition activities. The IT and business project leads and
support staff are responsible for project management activities.
Strength Ability to perform 2 Adequate resources for the project
team are provided for the duration of the project.
There is no mechanism to identify resource needs and ensure that they are
provided to the project. The project manager stated that the team has
adequate resources for conducting project management activities.
Strength Ability to perform 3 When project trade- offs are necessary,
the project manager is permitted to alter the software acquisition cost,
schedule, or performance.
The project manager is permitted to alter schedule or performance to ensure
that annual project budgets are not exceeded.
Strength Ability to perform 4 Individuals performing software
acquisition management activities have experience or receive training.
Project team members performing project management activities have
experience and have had training.
Strength Activity performed 1 The project team performs its activities
in accordance with its documented software acquisition plans.
Software acquisition project management plans are used to guide the
activities of the project team.
Strength Activity performed 2 The roles, responsibilities, and authority
of the project function are documented, maintained, and communicated to
affected groups.
Roles, responsibilities, and authority are defined in the project plan and
in the Configuration Control Board?s (CCB) role matrix.
Strength Activity performed 3 The project team?s commitments, and
changes to commitments, are communicated to affected groups.
The project team?s commitments and changes to commitments regarding this
project are communicated to affected groups.
Strength Activity performed 4 The project team tracks the cost,
schedule, resource, and technical risks of the project.
A risk management plan is maintained that tracks risks associated with
costs, schedule, resource, and technical risks.
Strength Activity performed 5 The project team tracks project issues,
status, execution, funding, and expenditures against project plans and takes
action.
The project team uses contractor weekly and monthly status reports to track
contractor costs and schedule issues. Also, a quarterly project control
review is done to review cost and schedule status.
Strength Activity performed 6 The project team implements a
corrective action system for the identification, recording, tracking, and
correction of problems discovered during the software acquisition.
A corrective action system has been implemented for the identification,
recording, tracking, and correction of problems discovered during the
software acquisition.
Strength
Chapter 3: HUD?s Project Management Processes Are Not Repeatable
Page 36 GAO- 01- 962 HUD Information Systems
Resident Assessment Subsystem Common feature Key practice Finding Rating
Activity performed 7 The project team keeps its plans current during the
life of the project as replanning occurs, issues are resolved, and new risks
are discovered.
The project plans are kept up to date as replanning occurs, issues are
resolved, and new risks are discovered. Monthly reports are provided that
show changes to scheduled software releases resulting from replanning.
Strength Measurement and analysis 1 Measurements are made and used to
determine the status of the project management activities and resultant
products.
Measurements are only made of HUD?s project management products. Observation
Verifying implementation 1 Project management activities are reviewed by the
acquisition organization on a periodic basis.
The acquisition organization reviews project management activities on a
quarterly basis. Strength
Verifying implementation 2 Project management activities are reviewed by the
project manager on both a periodic and an event- driven basis.
The project manager stated that while he meets with members of his staff
daily to discuss the status of the project, meeting results and action items
are not documented.
Weakness
Chapter 3: HUD?s Project Management Processes Are Not Repeatable
Page 37 GAO- 01- 962 HUD Information Systems
Table 11: Project Management Findings for HUD?s Central Accounting and
Program System HUD?s Central Accounting and Program System Common feature
Key practice Finding Rating
Commitment to perform 1 The acquisition organization has a written policy
for management of the software acquisition project.
HUD has no written policy for managing the acquisition of software products
and services. Weakness
Commitment to perform 2 Responsibility has been designated for project
management activities. Responsibility for project management activities
has been assigned to the project lead. Strength Ability to perform 1 A group
is responsible for managing
software acquisition activities. The IT and business project leads are
responsible for managing software acquisition
activities. Strength
Ability to perform 2 Adequate resources for the project team are provided
for the duration of the project.
There is no mechanism to identify resource needs and ensure they were made
available to the project. The project manager stated that the team does not
have adequate resources for performing project management activities.
Weakness Ability to perform 3 When project trade- offs are
necessary, the project manager is permitted to alter the software
acquisition cost, schedule, or performance.
The project manager is permitted to alter the schedule or performance to
ensure that annual project budgets are not exceeded.
Strength Ability to perform 4 Individuals performing software
acquisition management activities have experience or receive training.
Project team members performing project management activities have
experience and have had training.
Strength Activity performed 1 The project team performs its
activities in accordance with its documented software acquisition plans.
Software acquisition project management plans are used to guide the
activities of the project team.
Strength Activity performed 2 The roles, responsibilities, and
authority of the project function are documented, maintained, and
communicated to affected groups.
The roles, responsibilities, and authority of the project team are defined
in the software acquisition project plan.
Strength Activity performed 3 The project team?s commitments, and
changes to commitments, are communicated to affected groups.
The commitments and changes to commitments are communicated to affected
groups through weekly status meetings, emails, and telephone conversations.
Strength Activity performed 4 The project team tracks the cost,
schedule, resource, and technical risks of the project.
The project plans and exhibits identify risks associated with the project
and define mitigation plans for these risks. However, the project team does
not track the risks.
Weakness Activity performed 5 The project team tracks project
issues, status, execution, funding, and expenditures against project plans
and takes action.
The project team tracks project issues, status, execution, and funding
through weekly functional and technical status meetings. Funding is tracked
and actual costs compared to projected costs.
Strength Activity performed 6 The project team implements a
corrective action system for the identification, recording, tracking, and
correction of problems discovered during the software acquisition.
A corrective action system has been implemented for the identification,
recording, tracking, and correction of problems discovered during the
software acquisition.
Strength
Chapter 3: HUD?s Project Management Processes Are Not Repeatable
Page 38 GAO- 01- 962 HUD Information Systems
HUD?s Central Accounting and Program System Common feature Key practice
Finding Rating
Activity performed 7 The project team keeps its plans current during the
life of the project as replanning occurs, issues are resolved, and new risks
are discovered.
The project team does have a risk assessment and mitigation strategy.
However, the risks are not tracked, and plans are not kept current.
Weakness Measurement and analysis 1 Measurements are made and used to
determine the status of the project management activities and resultant
products.
No measurements are made of HUD?s project management activities or resultant
products. Weakness
Verifying implementation 1 Project management activities are reviewed by the
acquisition organization on a periodic basis.
The acquisition organization reviews project management activities on a
regular basis through meetings with the project manager.
Strength Verifying implementation 2 Project management activities are
reviewed by the project manager on both a periodic and an event- driven
basis.
The project manager reviews project management activities regularly through
briefings and document review.
Strength
Chapter 3: HUD?s Project Management Processes Are Not Repeatable
Page 39 GAO- 01- 962 HUD Information Systems
Table 12: Project Management Findings for the Empowerment Information System
Empowerment Information System Common feature Key practice Finding Rating
Commitment to perform 1 The acquisition organization has a written policy
for management of the software acquisition project.
HUD has no written policy for managing the acquisition of software products
and services. Weakness
Commitment to perform 2 Responsibility has been designated for project
management activities. Responsibility for project management activities
has been assigned to the project technical lead. Strength Ability to perform
1 A group is responsible for managing
software acquisition activities. The project lead and assigned staff are
responsible for performing project management
activities. Strength
Ability to perform 2 Adequate resources for the project team are provided
for the duration of the project.
There is no mechanism to identify resource needs and ensure that they are
made available to the project. The project manager stated that the team does
not have adequate resources for performing project management activities.
Weakness Ability to perform 3 When project trade- offs are
necessary, the project manager is permitted to alter the software
acquisition cost, schedule, or performance.
The project manager is allowed to alter schedule or performance to ensure
that annual project budgets are not exceeded.
Strength Ability to perform 4 Individuals performing software
acquisition management activities have experience or receive training.
Project team members performing project management activities have
experience and project management training.
Strength Activity performed 1 The project team performs its
activities in accordance with its documented software acquisition plans.
Software acquisition project management plans are used to guide the
activities of the project team.
Strength Activity performed 2 The roles, responsibilities, and
authority of the project function are documented, maintained, and
communicated to affected groups.
Roles, responsibilities, and authority of the project team are defined in
the project organization chart.
Strength Activity performed 3 The project team?s commitments,
and changes to commitments, are communicated to affected groups.
The project manager uses contractor status reports to communicate
commitments and changes to commitments to the project team and other
affected groups in meetings.
Strength Activity performed 4 The project team tracks the cost,
schedule, resource, and technical risks of the project.
The project team does assess and track risks for several areas, including
contract, funding, schedule, management, and technical areas.
Strength Activity performed 5 The project team tracks project
issues, status, execution, funding, and expenditures against project plans
and takes action.
The project team did conduct a review of the software developed by the
contractor. As a result of issues found during that review, modifications
were made to the contract.
Strength Activity performed 6 The project team implements a
corrective action system for the identification, recording, tracking, and
correction of problems discovered during the software acquisition.
A remedial action plan was developed and a corrective action system
implemented to record, track, and correct problems found during the software
acquisition.
Strength
Chapter 3: HUD?s Project Management Processes Are Not Repeatable
Page 40 GAO- 01- 962 HUD Information Systems
Empowerment Information System Common feature Key practice Finding Rating
Activity performed 7 The project team keeps its plans current during the
life of the project as replanning occurs, issues are resolved, and new risks
are discovered.
The replanning is documented in the remedial action plan. Strength
Measurement and analysis 1 Measurements are made and used to determine the
status of the project management activities and resultant products.
No measurements are made of HUD?s project management activities and
resultant products. Weakness
Verifying implementation 1 Project management activities are reviewed by the
acquisition organization on a periodic basis.
The acquisition organization reviews project management activities
quarterly. Strength
Verifying implementation 2 Project management activities are reviewed by the
project manager on both a periodic and an event- driven basis.
The project manager is not briefed by other members on either a regular or
event- driven basis, and project manager reviews are not documented.
Weakness
Chapter 4: HUD?s Contract Tracking and Oversight Processes Are Not
Repeatable
Page 41 GAO- 01- 962 HUD Information Systems
HUD?s contract tracking and oversight processes do not satisfy the criteria
for the repeatable level of maturity because of the number and severity of
key practice weaknesses found. These weaknesses increase the risk that
software activities will not be performed in accordance with contractual
requirements, resulting in software acquisitions being late, costing more
than intended, and not performing as intended.
The purpose of contract tracking and oversight is to ensure that the
software activities under contract are being performed in accordance with
contractual requirements. Within this key process area, the SA- CMM
specifies a number of key practices across the five common features that an
organization must implement effectively to achieve the repeatable level of
maturity. Generally, these practices include having a written organizational
policy for contract tracking and oversight, documenting plans for contract
tracking and oversight activities, reviewing contractor software planning
documents and using those documents to oversee the contractor?s efforts,
comparing actual cost and schedule to planned budgets and schedules, and
having a mechanism to ensure that contractordelivered work products meet
contract requirements.
For the 85 key practices in this key process area, HUD had 45 strengths, 33
weaknesses, four observations, and three practices not rated. For the
Commitment to perform feature, there were 12 strengths and three
weaknesses.
Ability to perform feature, there were ten strengths and five weaknesses.
Activities performed feature, there were 20 strengths, 15 weaknesses, two
observations, and three practices not rated.
Measurement and analysis feature, there were no strengths and five
weaknesses.
Verifying implementation feature, there were three strengths, five
weaknesses, and two observations.
In addition, none of the projects had strengths in all the key practices. As
a result of these weaknesses, HUD is exposed to increased risk that software
activities will not be performed in accordance with contractual
requirements, resulting in software acquisitions being late, costing more
than planned, and not performing as intended. HUD acknowledges the need to
improve its contract tracking and oversight processes and is committed to
improving its software and system acquisition capability. The department has
prepared a statement of work to obtain contractor support to begin a
software process improvement effort. This document Chapter 4: HUD?s Contract
Tracking and
Oversight Processes Are Not Repeatable
Chapter 4: HUD?s Contract Tracking and Oversight Processes Are Not
Repeatable
Page 42 GAO- 01- 962 HUD Information Systems
specifies that the contractor would review HUD software acquisition
processes, prioritize the key processes and practices to address, and
develop a plan to improve HUD?s acquisition processes.
Details of our evaluation are provided in the following figure and tables.
Figure 4 provides a comprehensive listing of the strengths, weaknesses, and
observations for the contract tracking and oversight key process area. The
specific findings supporting the practice ratings cited in figure 4 are in
tables 13 through 17.
Chapter 4: HUD?s Contract Tracking and Oversight Processes Are Not
Repeatable
Page 43 GAO- 01- 962 HUD Information Systems
Figure 4: Contract Tracking and Oversight Summary Projects Common features
Key practices PIC REMS RASS HUDCAPS EIS
Commitment to perform 1 The acquisition organization has a written policy
for
managing the contract tracking and oversight of the software acquisition
project. Weakness Strength Strength Strength Strength
Commitment to perform 2 Responsibility has been designated for contract
tracking
and oversight activities. Weakness Strength Strength Strength Strength
Commitment to perform 3 The project team includes contracting specialists in
the
management of the contract. Weakness Strength Strength Strength Strength
Ability to perform 1 A group is responsible for managing contract tracking
and oversight activities. Weakness Strength Strength Strength Strength
Ability to perform 2 Adequate resources are provided for contract tracking
and oversight activities. Weakness Weakness Strength Weakness Weakness
Ability to perform 3 Individuals performing contract tracking and oversight
activities have experience or receive training. Strength Strength Strength
Strength Strength
Activity performed 1 The project team performs its activities in accordance
with documented contract tracking and oversight plans. Weakness Weakness
Strength Weakness Weakness
Activity performed 2 The project team reviews required contractor software
planning documents which, when satisfactory, are used to oversee the
contractor team?s software effort. Weakness Weakness Weakness Weakness
Weakness
Activity performed 3 The project team conducts periodic reviews and
interchanges with the contractor team. Strength Strength Strength Strength
Strength
Activity performed 4 The actual cost and schedule of the contractor?s
software effort are compared to planned budgets and schedules, and issues
are identified. Strength Strength Strength Observation Strength
Activity performed 5 The size, critical computer resources, and technical
activities associated with the contractor team?s work products are tracked,
and issues are identified. Weakness Weakness Strength Strength Observation
Activity performed 6 The project team reviews and tracks the development of
the software engineering environment required to provide life- cycle support
for the acquired software, and issues are identified.
Strength Strength Strength Strength Strength
Activity performed 7 Any issues found by the project team during contract
tracking and oversight activities are recorded in the appropriate corrective
action system, action is taken, and issues are tracked to closure.
Weakness Weakness Strength Weakness Strength
Activity performed 8 The project team ensures that changes to the
softwarerelated contractual requirements are coordinated with
affected groups and individuals, such as the contracting official,
contractor, and end user.
Strength Not rated Not rated Not rated Weakness
Measurement and analysis 1 Measurements are made and used to determine the
status of the contract tracking and oversight activities and resultant
products. Weakness Weakness Weakness Weakness Weakness
Verifying implementation 1 Contract tracking and oversight activities are
reviewed
by acquisition management on a periodic basis. Weakness Weakness Weakness
Strength Strength
Verifying implementation 2 Contract tracking and oversight activities are
reviewed
by the project manager on both a periodic and an eventdriven basis. Weakness
Observation Weakness Strength Observation
Key Strength Strength: Key practice effectively implemented.
Weakness Weakness: Key practice not effectively implemented.
Observation Observation: Key practice evaluated; evidence not sufficient to
rate as a strength, or practice only partially performed.
Chapter 4: HUD?s Contract Tracking and Oversight Processes Are Not
Repeatable
Page 44 GAO- 01- 962 HUD Information Systems
Table 13: Contract Tracking and Oversight Findings for the Public and Indian
Housing Information Center System Public and Indian Housing Information
Center system Common feature Key practice Finding Rating
Commitment to perform 1 The acquisition organization has a written policy
for managing the contract tracking and oversight of the software acquisition
project.
HUD has no written policy for contract tracking and oversight of the
acquisition of software products and services, and the project team did not
establish one.
Weakness Commitment to perform 2 Responsibility has been designated for
contract tracking and oversight activities.
Responsibility for contract tracking and oversight activities is not
designated. Weakness
Commitment to perform 3 The project team includes contracting specialists in
the management of the contract.
The project team told us they did include contracting specialists in the
management of the contract, but no documentation was provided to support
this statement.
Weakness Ability to perform 1 A group is responsible for managing
contract tracking and oversight activities.
The project team told us there was a group assigned responsibility for
contract tracking and oversight activities, but no documentation was
provided to support this statement.
Weakness Ability to perform 2 Adequate resources are provided for
contract tracking and oversight activities.
There is no mechanism to identify resource needs and ensure that they are
provided to the project. Team members indicated that the resources for
performing contract tracking and oversight activities are not adequate.
Weakness Ability to perform 3 Individuals performing contract tracking
and oversight activities have experience or receive training.
Project team members performing contract tracking and oversight activities
have experience.
Strength Activity performed 1 The project team performs its activities
in accordance with its documented contract tracking and oversight plans.
There is no contract tracking and oversight plan; therefore, the activities
are not performed in accordance with it.
Weakness Activity performed 2 The project team reviews required
contractor software planning documents which, when satisfactory, are used to
oversee the contractor team?s software effort.
The contractor did not prepare software planning documents that the project
team could use to oversee the contractor?s software development effort.
Weakness Activity performed 3 The project team conducts periodic
reviews and interchanges with the contractor team.
The project team does have periodic reviews and discussions with the
contractor team that are documented.
Strength Activity performed 4 The actual cost and schedule of the
contractor?s software effort are compared to planned budgets and schedules,
and issues are identified.
The project team does compare the actual cost and schedule of the
contractor?s software engineering effort to planned schedules and budgets,
identifies issues, and documents results.
Strength Activity performed 5 The size, critical computer resources,
and technical activities associated with the contractor team?s work products
are tracked, and issues are identified.
The project team does not track the size, critical computer resources, and
technical activities associated with the contractor team?s work products or
identify issues.
Weakness
Chapter 4: HUD?s Contract Tracking and Oversight Processes Are Not
Repeatable
Page 45 GAO- 01- 962 HUD Information Systems
Public and Indian Housing Information Center system Common feature Key
practice Finding Rating
Activity performed 6 The project team reviews and tracks the development of
the software engineering environment required to provide life- cycle support
for the acquired software, and issues are identified.
The project team does review and track development of the software
engineering environment required to provide life- cycle support for the
acquired software, and issues are identified and documented.
Strength Activity performed 7 Any issues found by the project team
during contract tracking and oversight activities are recorded in the
appropriate corrective action system, action is taken, and issues are
tracked to closure.
The project team does not have a corrective action system to record or track
action taken on issues found by the project team during contract tracking
and oversight activities.
Weakness Activity performed 8 The project team ensures that changes
to the software- related contractual requirements are coordinated with
affected groups and individuals, such as the contracting official,
contractor, and end user.
The project team does ensure that changes to contractual requirements are
coordinated with all affected groups and individuals.
Strength Measurement and analysis 1 Measurements are made and used to
determine the status of the contract tracking and oversight activities and
resultant products.
No measurements are made of HUD?s contract tracking and oversight activities
or resultant products.
Weakness Verifying implementation 1 Contract tracking and oversight
activities are reviewed by acquisition management on a periodic basis.
The acquisition organization does not review contract tracking and oversight
activities on a periodic basis.
Weakness Verifying implementation 2 Contract tracking and oversight
activities are reviewed by the project manager on both a periodic and an
event- driven basis.
The project manager stated that while he discusses the status of the project
with team members, meeting results and action items are not documented.
Weakness
Chapter 4: HUD?s Contract Tracking and Oversight Processes Are Not
Repeatable
Page 46 GAO- 01- 962 HUD Information Systems
Table 14: Contract Tracking and Oversight Findings for the Real Estate
Management System Real Estate Management System Common feature Key practice
Finding Rating
Commitment to perform 1 The acquisition organization has a written policy
for managing the contract tracking and oversight of the software acquisition
project.
HUD has no written policy for contract tracking and oversight of the
acquisition of software products and services. The project team is following
the policy in HUD?s procurement manual for contract tracking and oversight.
Strength Commitment to perform 2 Responsibility has been designated for
contract tracking and oversight activities.
Responsibility for contract tracking and oversight activities has been
designated to contract specialists on the team.
Strength Commitment to perform 3 The project team includes contracting
specialists in the management of the contract.
The project team includes contract specialists to help manage the contract.
Strength
Ability to perform 1 A group is responsible for managing contract tracking
and oversight activities.
A group is assigned responsibility for contract tracking and oversight
activities. Strength
Ability to perform 2 Adequate resources are provided for contract tracking
and oversight activities.
There is no mechanism to identify resource needs and ensure that they are
provided to the project. Team members indicated that the resources for
performing contract tracking and oversight activities are not adequate.
Weakness Ability to perform 3 Individuals performing contract
tracking and oversight activities have experience or receive training.
Project team members performing contract tracking and oversight activities
have experience and have received training.
Strength Activity performed 1 The project team performs its activities
in accordance with its documented contract tracking and oversight plans.
There is no contract tracking and oversight plan; therefore, the activities
are not performed in accordance with it.
Weakness Activity performed 2 The project team reviews required
contractor software planning documents which, when satisfactory, are used to
oversee the contractor team?s software effort.
The contractor did prepare software planning documents that could be used to
oversee the contractor team?s software development effort, but no evidence
was provided that these were reviewed and/ or approved by the project team.
Weakness Activity performed 3 The project team conducts periodic
reviews and interchanges with the contractor team.
The project team does have periodic reviews and discussions with the
contractor team that are documented.
Strength Activity performed 4 The actual cost and schedule of the
contractor?s software effort are compared to planned budgets and schedules,
and issues are identified.
The project team does compare the actual cost and schedule of the
contractor?s software engineering effort to planned schedules and budgets,
identify issues, and document results.
Strength Activity performed 5 The size, critical computer resources,
and technical activities associated with the contractor team?s work products
are tracked, and issues are identified.
The project team told us they track the size, critical computer resources,
and technical activities associated with the contractor team?s work
products, but no documentation was provided to support this statement.
Weakness
Chapter 4: HUD?s Contract Tracking and Oversight Processes Are Not
Repeatable
Page 47 GAO- 01- 962 HUD Information Systems
Real Estate Management System Common feature Key practice Finding Rating
Activity performed 6 The project team reviews and tracks the development of
the software engineering environment required to provide life- cycle support
for the acquired software, and issues are identified.
The project team does review and track the development of the software
engineering environment required to provide life- cycle support for the
acquired software, and issues are identified and documented.
Strength Activity performed 7 Any issues found by the project team
during contract tracking and oversight activities are recorded in the
appropriate corrective action system, action is taken, and issues are
tracked to closure.
The project team does not have a corrective action system to record or track
action taken on issues found by the project team during contract tracking
and oversight activities.
Weakness Activity performed 8 The project team ensures that
changes to the software- related contractual requirements are coordinated
with affected groups and individuals, such as the contracting official,
contractor, and end user.
The project team stated that no changes to the software- related contractual
requirements have taken place because their contracts are broad enough to
cover changes to the contractual requirements.
Not rated Measurement and analysis 1 Measurements are made and used to
determine the status of the contract tracking and oversight activities and
resultant products.
No measurements are made of HUD?s contract tracking and oversight activities
or resultant products.
Weakness Verifying implementation 1 Contract tracking and oversight
activities are reviewed by acquisition management on a periodic basis.
The acquisition organization does not review contract tracking and oversight
activities on a periodic basis.
Weakness Verifying implementation 2 Contract tracking and oversight
activities are reviewed by the project manager on both a periodic and an
event- driven basis.
The project manager informally reviews the work of the project team;
however, the results of these reviews are not documented. Minutes of
biweekly status meetings with contractors are maintained.
Observation
Chapter 4: HUD?s Contract Tracking and Oversight Processes Are Not
Repeatable
Page 48 GAO- 01- 962 HUD Information Systems
Table 15: Contract Tracking and Oversight Findings for the Resident
Assessment Subsystem Resident Assessment Subsystem Common feature Key
practice Finding Rating
Commitment to perform 1 The acquisition organization has a written policy
for managing the contract tracking and oversight of the software acquisition
project.
HUD has no written policy for contract tracking and oversight of the
acquisition of software products and services. The project team is following
the policy in HUD?s Procurement Handbook for contract tracking and
oversight.
Strength Commitment to perform 2 Responsibility has been designated for
contract tracking and oversight activities.
Responsibility for contract tracking and oversight activities is designated
in the project plan.
Strength Commitment to perform 3 The project team includes contracting
specialists in the management of the contract.
The project team includes contract specialists to help manage the contract.
Strength
Ability to perform 1 A group is responsible for managing contract tracking
and oversight activities.
A group is assigned responsibility for contract tracking and oversight
activities. Strength
Ability to perform 2 Adequate resources are provided for contract tracking
and oversight activities.
There is no mechanism to identify resource needs and ensure that they are
provided to the project. The project manager stated that the team does have
adequate resources for performing contract tracking and oversight
activities.
Strength Ability to perform 3 Individuals performing contract
tracking and oversight activities have experience or receive training.
Project team members performing contract tracking and oversight activities
have experience and have received training.
Strength Activity performed 1 The project team performs its activities
in accordance with its documented contract tracking and oversight plans.
The project team performs its activities in accordance with its documented
contract tracking and oversight plan.
Strength Activity performed 2 The project team reviews required
contractor software planning documents which, when satisfactory, are used to
oversee the contractor team?s software effort.
The contractor did not prepare software planning documents that the project
team could use to oversee the contractor?s software development effort.
Weakness Activity performed 3 The project team conducts periodic
reviews and interchanges with the contractor team.
The project team does have periodic reviews and discussions with the
contractor team that are documented.
Strength Activity performed 4 The actual cost and schedule of the
contractor?s software effort are compared to planned budgets and schedules,
and issues are identified.
The project team does compare the actual cost and schedule of the
contractor?s software engineering effort to planned schedules and budgets,
identify issues, and document results.
Strength Activity performed 5 The size, critical computer resources,
and technical activities associated with the contractor team?s work products
are tracked, and issues are identified.
The project team does track the size, critical computer resources, and
technical activities associated with the contractor team?s work products and
identifies and documents issues.
Strength
Chapter 4: HUD?s Contract Tracking and Oversight Processes Are Not
Repeatable
Page 49 GAO- 01- 962 HUD Information Systems
Resident Assessment Subsystem Common feature Key practice Finding Rating
Activity performed 6 The project team reviews and tracks the development of
the software engineering environment required to provide life- cycle support
for the acquired software, and issues are identified.
The project team does review and track the development of the software
engineering environment required to provide life- cycle support for the
acquired software, and issues are identified and documented.
Strength Activity performed 7 Any issues found by the project team
during contract tracking and oversight activities are recorded in the
appropriate corrective action system, action is taken, and issues are
tracked to closure.
The project team records issues related to contract tracking and oversight
in a corrective action system, and tracks corrective actions to closure.
Strength Activity performed 8 The project team ensures that
changes to the software- related contractual requirements are coordinated
with affected groups and individuals, such as the contracting official,
contractor, and end user.
The project team stated that no changes to the software- related contractual
requirements have taken place.
Not rated Measurement and analysis 1 Measurements are made and used to
determine the status of the contract tracking and oversight activities and
resultant products.
No measurements are made of HUD?s contract tracking and oversight activities
and resultant products.
Weakness Verifying implementation 1 Contract tracking and oversight
activities are reviewed by acquisition management on a periodic basis.
The acquisition organization does not review contract tracking and oversight
activities on a periodic basis.
Weakness Verifying implementation 2 Contract tracking and oversight
activities are reviewed by the project manager on both a periodic and an
event- driven basis.
The project manager informally reviews the work of the project team.
However, the results of these reviews are not documented.
Weakness
Chapter 4: HUD?s Contract Tracking and Oversight Processes Are Not
Repeatable
Page 50 GAO- 01- 962 HUD Information Systems
Table 16: Contract Tracking and Oversight Findings for HUD?s Central
Accounting and Program System HUD?s Central Accounting and Program System
Common feature Key practice Finding Rating
Commitment to perform 1 The acquisition organization has a written policy
for managing the contract tracking and oversight of the software acquisition
project.
HUD has no written policy for contract tracking and oversight of the
acquisition of software products and services. The project team is following
the policy in HUD?s Procurement Handbook for contract tracking and
oversight.
Strength Commitment to perform 2 Responsibility has been designated for
contract tracking and oversight activities.
Responsibility for contract tracking and oversight activities is designated
in the organization chart.
Strength Commitment to perform 3 The project team includes contracting
specialists in the management of the contract.
The project team includes contract specialists to help manage the contract.
Strength
Ability to perform 1 A group is responsible for managing contract tracking
and oversight activities.
A group is assigned responsibility for contract tracking and oversight
activities. Strength
Ability to perform 2 Adequate resources are provided for contract tracking
and oversight activities.
There is no mechanism to identify resource needs and ensure that they are
provided to the project. The project manager stated that the team does not
have adequate resources for performing contract tracking and oversight
activities.
Weakness Ability to perform 3 Individuals performing contract
tracking and oversight activities have experience or receive training.
Project team members performing contract tracking and oversight activities
have experience and have received training.
Strength Activity performed 1 The project team performs its activities
in accordance with its documented contract tracking and oversight plans.
There is no contract tracking and oversight plan; therefore, the activities
are not performed in accordance with it.
Weakness Activity performed 2 The project team reviews required
contractor software planning documents which, when satisfactory, are used to
oversee the contractor team?s software effort.
The contractor did not prepare software planning documents that the project
team could use to oversee the contractor?s software development effort.
Weakness Activity performed 3 The project team conducts periodic
reviews and interchanges with the contractor team.
The project team does have periodic reviews and discussions with the
contractor team that are documented.
Strength Activity performed 4 The actual cost and schedule of the
contractor?s software effort are compared to planned budgets and schedules,
and issues are identified.
The project team told us they do compare the actual cost and schedule of the
contractor?s software engineering effort to planned schedules and budgets,
but no documentation was provided to support this statement.
Observation Activity performed 5 The size, critical computer resources,
and technical activities associated with the contractor team?s work products
are tracked, and issues are identified.
The project team does track the size, critical computer resources, and
technical activities associated with the contractor team?s work products and
identifies and documents issues.
Strength
Chapter 4: HUD?s Contract Tracking and Oversight Processes Are Not
Repeatable
Page 51 GAO- 01- 962 HUD Information Systems
HUD?s Central Accounting and Program System Common feature Key practice
Finding Rating
Activity performed 6 The project team reviews and tracks the development of
the software engineering environment required to provide life- cycle support
for the acquired software, and issues are identified.
The project team does review and track the development of the software
engineering environment required to provide life- cycle support for the
acquired software, and issues are identified and documented.
Strength Activity performed 7 Any issues found by the project team
during contract tracking and oversight activities are recorded in the
appropriate corrective action system, action is taken, and issues are
tracked to closure.
The project team does not have a corrective action system to record or track
action taken on issues found by the project team during contract tracking
and oversight activities.
Weakness Activity performed 8 The project team ensures that
changes to the software- related contractual requirements are coordinated
with affected groups and individuals, such as the contracting official,
contractor, and end user.
The project team stated that no changes to the software- related contractual
requirements have taken place because their contracts are broad enough to
cover changes to contractual requirements.
Not rated Measurement and analysis 1 Measurements are made and used to
determine the status of the contract tracking and oversight activities and
resultant products.
No measurements are made of HUD?s contract tracking and oversight activities
or resultant products.
Weakness Verifying implementation 1 Contract tracking and oversight
activities are reviewed by acquisition management on a periodic basis.
The acquisition organization regularly reviews contract tracking and
oversight activities.
Strength Verifying implementation 2 Contract tracking and oversight
activities are reviewed by the project manager on both a periodic and an
event- driven basis.
The project manager is briefed weekly on project status, including contract
tracking and oversight activities.
Strength
Chapter 4: HUD?s Contract Tracking and Oversight Processes Are Not
Repeatable
Page 52 GAO- 01- 962 HUD Information Systems
Table 17: Contract Tracking and Oversight Findings for the Empowerment
Information System Empowerment Information System Common feature Key
practice Finding Rating
Commitment to perform 1 The acquisition organization has a written policy
for managing the contract tracking and oversight of the software acquisition
project.
HUD has no written policy for contract tracking and oversight of the
acquisition of software products and services. The project team is following
the policy in HUD?s Procurement Handbook for contract tracking and
oversight.
Strength Commitment to perform 2 Responsibility has been designated for
contract tracking and oversight activities.
Responsibility for contract tracking and oversight activities has been
designated in the organization chart.
Strength Commitment to perform 3 The project team includes contracting
specialists in the management of the contract.
The project team includes contract specialists to help manage the contract.
Strength
Ability to perform 1 A group is responsible for managing contract tracking
and oversight activities.
A group is assigned responsibility for managing contract tracking and
oversight activities.
Strength Ability to perform 2 Adequate resources are provided for
contract tracking and oversight activities.
There is no mechanism to identify resource needs and ensure that they are
provided to the project. The project manager indicated that the team does
not have adequate resources for performing contract tracking and oversight
activities.
Weakness Ability to perform 3 Individuals performing contract
tracking and oversight activities have experience or receive training.
Project team members performing contract tracking and oversight activities
have experience and have received training.
Strength Activity performed 1 The project team performs its activities
in accordance with its documented contract tracking and oversight plans.
There is no contract tracking and oversight plan; therefore, the activities
are not performed in accordance with it.
Weakness Activity performed 2 The project team reviews required
contractor software planning documents which, when satisfactory, are used to
oversee the contractor team?s software effort.
The contractor did not prepare software planning documents that the project
team could use to oversee the contractor?s software development effort.
Weakness Activity performed 3 The project team conducts periodic
reviews and interchanges with the contractor team.
The project team does have periodic reviews and discussions with the
contractor team that are documented.
Strength Activity performed 4 The actual cost and schedule of the
contractor?s software effort are compared to planned budgets and schedules,
and issues are identified.
The project team does compare the actual cost and schedule of the
contractor?s software engineering effort to planned schedules and budgets,
identify issues, and document results.
Strength Activity performed 5 The size, critical computer resources,
and technical activities associated with the contractor team?s work products
are tracked, and issues are identified.
The project team reviewed the size, critical computer resources, and
technical activities associated with the contractor team?s work products
only during pre release testing.
Observation
Chapter 4: HUD?s Contract Tracking and Oversight Processes Are Not
Repeatable
Page 53 GAO- 01- 962 HUD Information Systems
Empowerment Information System Common feature Key practice Finding Rating
Activity performed 6 The project team reviews and tracks the development of
the software engineering environment required to provide life- cycle support
for the acquired software, and issues are identified.
The project does review and track development of the software engineering
environment required to provide life- cycle support for the acquired
software, and issues are identified and documented.
Strength Activity performed 7 Any issues found by the project team
during contract tracking and oversight activities are recorded in the
appropriate corrective action system, action is taken, and issues are
tracked to closure.
The project team records issues related to contract tracking and oversight
in a corrective action system, and tracks corrective actions to closure.
Strength Activity performed 8 The project team ensures that
changes to the software- related contractual requirements are coordinated
with affected groups and individuals, such as the contracting official,
contractor, and end user.
The project team stated that changes to software- related contractual
requirements were coordinated with all affected groups and individuals, but
they provided no documentation to support this statement.
Weakness Measurement and analysis 1 Measurements are made and used to
determine the status of the contract tracking and oversight activities and
resultant products.
No measurements are made of HUD?s contract tracking and oversight activities
or resultant products.
Weakness Verifying implementation 1 Contract tracking and oversight
activities are reviewed by acquisition management on a periodic basis.
The acquisition organization periodically reviews contract tracking and
oversight activities.
Strength Verifying implementation 2 Contract tracking and oversight
activities are reviewed by the project manager on both a periodic and an
event- driven basis.
The project manager works closely with and informally reviews the work of
the project team. However, the results of these reviews are not documented.
Observation
Chapter 5: HUD?s Software Evaluation Processes Are Not Repeatable
Page 54 GAO- 01- 962 HUD Information Systems
HUD did not satisfy the criteria for the repeatable level of maturity in
this key process area because of the number and severity of key practice
weaknesses found. These weaknesses result in increased risk that HUD?s
software acquisition projects will not be evaluated effectively, software
products will not work effectively or meet mission needs, and cost,
schedule, and other performance goals will not be met.
The purpose of evaluation is to determine that the acquired software
products and services satisfy contract requirements before acceptance.
Within this key process area, the SA- CMM specifies key practices for each
of the five common features that an organization must implement effectively
to achieve the repeatable level of maturity. Generally, these practices
include having a written organizational policy for evaluating acquired
software products and services, documenting plans for evaluating software
products and services, developing and managing evaluation requirements in
conjunction with software technical requirements, tracking contractor
evaluation activities for compliance with the contract, and measuring and
reporting on the status and results of evaluation activities to management.
For the 75 key practices in this key process area, HUD had 37 strengths, 33
weaknesses, and five observations. For the
Commitment to perform feature, there were four strengths, four weaknesses,
and two observations.
Ability to perform feature, there were 11 strengths and nine weaknesses.
Activities performed feature, there were 19 strengths, nine weaknesses,
and two observations.
Measurement and analysis feature, there were no strengths, four
weaknesses, and one observation.
Verifying implementation feature, there were three strengths and seven
weaknesses.
In addition, none of the projects had strengths in all the key practices. As
a result of these weaknesses, HUD is exposed to increased risk that software
acquisition projects will not be evaluated effectively, that acquired
software will not work efficiently or meet mission needs, and that cost,
schedule, and other performance goals will not be met. HUD acknowledges the
need to improve its software evaluation processes and is committed to
improving its software and system acquisition capability. The department has
prepared a statement of work to obtain contract support to begin a software
process improvement effort. This document Chapter 5: HUD?s Software
Evaluation
Processes Are Not Repeatable
Chapter 5: HUD?s Software Evaluation Processes Are Not Repeatable
Page 55 GAO- 01- 962 HUD Information Systems
specifies that the contractor would review HUD software acquisition
processes, prioritize the key processes and practices to address, and
develop a plan to improve HUD?s acquisition processes.
Details of our evaluation are provided in the following figure and tables.
Figure 5 provides a comprehensive listing of the strengths, weaknesses, and
observations in the evaluation key process area. The specific findings
supporting the practice ratings cited in figure 5 are in tables 18 through
22.
Chapter 5: HUD?s Software Evaluation Processes Are Not Repeatable
Page 56 GAO- 01- 962 HUD Information Systems
Figure 5: Software Evaluation Summary Projects Common features Key practices
PIC REMS RASS HUDCAPS EIS
Commitment to perform 1 The acquisition organization has a written policy
for
managing the evaluation of acquired software products and services. Weakness
Weakness Observation Observation Weakness
Commitment to perform 2 Responsibility has been designated for evaluation
activities. Weakness Strength Strength Strength Strength
Ability to perform 1 A group is responsible for planning, managing, and
performing evaluation activities. Weakness Strength Strength Strength
Strength
Ability to perform 2 Adequate resources are provided for evaluation
activities. Weakness Weakness Strength Weakness Weakness
Ability to perform 3 Individuals performing evaluation activities have
experience or receive training. Strength Strength Strength Strength Strength
Ability to perform 4 Members of the project team and groups supporting the
software acquisition receive orientation on the objectives of the evaluation
approach. Weakness Weakness Weakness Weakness Strength
Activity performed 1 The project team performs its activities in accordance
with its documented evaluation plans. Strength Strength Strength Weakness
Weakness
Activity performed 2 The project?s evaluation requirements are developed in
conjunction with the development of the system or software technical
requirements. Strength Observation Strength Strength Weakness
Activity performed 3 The project?s evaluation activities are planned to
minimize duplication and take advantage of all evaluation results, where
appropriate. Strength Strength Strength Strength Weakness
Activity performed 4 The project team appraises the contractor team?s
performance over the total period of the contract for compliance with
requirements. Weakness Observation Strength Strength Weakness
Activity performed 5 Planned evaluations are performed on the evolving
software products and services before acceptance for operational use.
Strength Strength Strength Strength Weakness
Activity performed 6 Results of the evaluations are analyzed and compared
to the contract?s requirements to establish an objective basis to support
the decision to accept the product or services or to take further action.
Weakness Strength Strength Strength Weakness
Measurement and analysis 1 Measurements are made and used to determine the
status of the evaluation activities and resultant products. Weakness
Weakness Weakness Weakness Observation
Verifying implementation 1 Evaluation activities are reviewed by acquisition
organization management on a periodic basis. Strength Strength Weakness
Weakness Weakness
Verifying implementation 2 Evaluation activities are reviewed by the project
manager on both a periodic and an event- driven basis. Weakness Strength
Weakness Weakness Weakness Key Strength Strength: Key practice effectively
implemented.
Weakness Weakness: Key practice not effectively implemented.
Observation Observation: Key practice evaluated; evidence not sufficient to
rate as a strength, or practice only partially performed.
Chapter 5: HUD?s Software Evaluation Processes Are Not Repeatable
Page 57 GAO- 01- 962 HUD Information Systems
Table 18: Software Evaluation Findings for the Public and Indian Housing
Information Center System Public and Indian Housing Information Center
system Common feature Key practice Finding Rating
Commitment to perform 1 The acquisition organization has a written policy
for managing the evaluation of acquired software products and services.
HUD has no written policy for managing the evaluation of acquired software
products and services.
Weakness Commitment to perform 2 Responsibility has been designated
for evaluation activities. The project team told us that responsibility for
evaluation activities has been assigned
to the lead technical person, but no documentation was provided to support
this statement.
Weakness Ability to perform 1 A group is responsible for planning,
managing, and performing evaluation activities.
The project team told us that evaluation activities are managed by the lead
technical person, but no documentation was provided to support this
statement.
Weakness Ability to perform 2 Adequate resources are provided for
evaluation activities. There is no mechanism to identify needed resources
and ensure that they are provided
to the project. The lead technical person indicated that the team does not
have adequate resources for performing evaluation activities.
Weakness Ability to perform 3 Individuals performing evaluation
activities have experience or receive training.
Project team members performing evaluation activities have experience.
Strength
Ability to perform 4 Members of the project team and groups supporting the
software acquisition receive orientation on the objectives of the evaluation
approach.
Members of the project team or others supporting the software acquisition
did not receive orientation on the objectives of the evaluation approach.
Weakness Activity performed 1 The project team performs its
activities in accordance with its documented evaluation plans.
The project team performs its evaluation activities in accordance with its
documented evaluation plans.
Strength Activity performed 2 The project?s evaluation requirements
are developed in conjunction with the development of the system or software
technical requirements.
Project evaluation requirements were developed in conjunction with software
technical requirements and are designated in the test plans and test
scripts.
Strength Activity performed 3 The project?s evaluation activities are
planned to minimize duplication and take advantage of all evaluation
results, where appropriate.
There appears to be no duplication of evaluation activities. Strength
Activity performed 4 The project team appraises the contractor team?s
performance over the total period of the contract for compliance with
requirements.
The project team stated that quarterly reports of contractor deliverables
were used to appraise contractor performance, but no documentation was
provided to support this statement.
Weakness Activity performed 5 Planned evaluations are performed
on the evolving software products and services before acceptance for
operational use.
Several evaluations are performed on evolving software products and services
before acceptance for operational use.
Strength
Chapter 5: HUD?s Software Evaluation Processes Are Not Repeatable
Page 58 GAO- 01- 962 HUD Information Systems
Public and Indian Housing Information Center system Common feature Key
practice Finding Rating
Activity performed 6 Results of the evaluations are analyzed and compared to
the contract?s requirements to establish an objective basis to support the
decision to accept the product or services or to take further action.
There is no analysis or comparison of evaluation results to contract
requirements to establish an objective basis supporting the decision to
accept the products and services or to take further action.
Weakness Measurement and analysis 1 Measurements are made and used to
determine the status of the evaluation activities and resultant products.
No measurements are made of HUD?s evaluation activities and resultant
products. Weakness
Verifying implementation 1 Evaluation activities are reviewed by acquisition
organization management on a periodic basis.
Acquisition organization management regularly reviews evaluation activities.
Strength
Verifying implementation 2 Evaluation activities are reviewed by the project
manager on both a periodic and an event- driven basis.
The project manager does not formally review evaluation activities, and the
project manager?s reviews are not documented.
Weakness
Chapter 5: HUD?s Software Evaluation Processes Are Not Repeatable
Page 59 GAO- 01- 962 HUD Information Systems
Table 19: Software Evaluation Findings for the Real Estate Management System
Real Estate Management System Common feature Key practice Finding Rating
Commitment to perform 1 The acquisition organization has a written policy
for managing the evaluation of acquired software products and services.
HUD has no written policy for managing the evaluation of acquired software
products and services.
Weakness Commitment to perform 2 Responsibility has been designated for
evaluation activities. Contractors and users have been assigned
responsibility for evaluation activities. Strength
Ability to perform 1 A group is responsible for planning, managing, and
performing evaluation activities.
The contractor and end users are responsible for evaluation activities.
Strength
Ability to perform 2 Adequate resources are provided for evaluation
activities. There is no mechanism to identify needed
resources and ensure that they are provided to the project. The project
manager stated that the team does not have adequate resources for performing
evaluation activities.
Weakness Ability to perform 3 Individuals performing evaluation
activities have experience or receive training.
Project team members performing evaluation activities have experience.
Strength
Ability to perform 4 Members of the project team and groups supporting the
software acquisition receive orientation on the objectives of the evaluation
approach.
Members of the project team or others supporting the software acquisition
did not receive orientation on the objectives of the evaluation approach.
Weakness Activity performed 1 The project team performs its activities
in accordance with its documented evaluation plans.
The team performs its evaluation activities in accordance with its
documented evaluation plans.
Strength Activity performed 2 The project?s evaluation requirements
are developed in conjunction with the development of the system or software
technical requirements.
Project evaluation requirements are documented in the project plan, but no
documentation was provided showing that they were developed in conjunction
with development of the software technical requirements.
Observation Activity performed 3 The project?s evaluation activities are
planned to minimize duplication and take advantage of all evaluation
results, where appropriate.
There appears to be no duplication of evaluation activities. Strength
Activity performed 4 The project team appraises the contractor team?s
performance over the total period of the contract for compliance with
requirements.
The project team assesses contractor team performance for compliance with
requirements only during user acceptance testing.
Observation Activity performed 5 Planned evaluations are performed on
the evolving software products and services before acceptance for
operational use.
Planned evaluations are performed before software releases are accepted for
operational use.
Strength
Chapter 5: HUD?s Software Evaluation Processes Are Not Repeatable
Page 60 GAO- 01- 962 HUD Information Systems
Real Estate Management System Common feature Key practice Finding Rating
Activity performed 6 Results of the evaluations are analyzed and compared to
the contract?s requirements to establish an objective basis to support the
decision to accept the product or services or to take further action.
The results of evaluations are analyzed and compared to the contract?s
requirements to establish an objective basis supporting the decision to
accept the products and services or to take further action.
Strength Measurement and analysis 1 Measurements are made and used to
determine the status of the evaluation activities and resultant products.
No measurements are made of HUD?s evaluation activities or resultant
products. Weakness
Verifying implementation 1 Evaluation activities are reviewed by acquisition
organization management on a periodic basis.
Acquisition organization management regularly reviews evaluation activities.
Strength
Verifying implementation 2 Evaluation activities are reviewed by the project
manager on both a periodic and an event- driven basis.
Project reviews conducted by the project manager cover evaluation
activities. Strength
Chapter 5: HUD?s Software Evaluation Processes Are Not Repeatable
Page 61 GAO- 01- 962 HUD Information Systems
Table 20: Software Evaluation Findings for the Resident Assessment Subsystem
Resident Assessment Subsystem Common feature Key practice Finding Rating
Commitment to perform 1 The acquisition organization has a written policy
for managing the evaluation of acquired software products and services.
HUD has no written policy for managing the evaluation of acquired software
products and services. The project is following the evaluation policies in
HUD?s software development methodology.
Observation Commitment to perform 2 Responsibility has been designated
for evaluation activities. Responsibility for evaluation activities has been
assigned to the IT project lead. Strength
Ability to perform 1 A group is responsible for planning, managing, and
performing evaluation activities.
The IT project lead and support staff are responsible for evaluation
activities. Strength
Ability to perform 2 Adequate resources are provided for evaluation
activities. There is no mechanism to identify resource
needs and ensure that they are provided to the project. According to the
project manager, the project team has adequate resources for conducting
evaluation activities.
Strength Ability to perform 3 Individuals performing evaluation
activities have experience or receive training.
Project team members performing evaluation activities have experience and
have received training.
Strength Ability to perform 4 Members of the project team and
groups supporting the software acquisition receive orientation on the
objectives of the evaluation approach.
Members of the project team or others supporting the software acquisition
did not receive orientation on the objectives of the evaluation approach.
Weakness Activity performed 1 The project team performs its
activities in accordance with its documented evaluation plans.
The project team performs its evaluation activities in accordance with its
documented evaluation plan.
Strength Activity performed 2 The project?s evaluation requirements
are developed in conjunction with the development of the system or software
technical requirements.
Project evaluation requirements are developed in conjunction with software
technical requirements.
Strength Activity performed 3 The project?s evaluation activities are
planned to minimize duplication and take advantage of all evaluation
results, where appropriate.
There appears to be no duplication of evaluation activities. Strength
Activity performed 4 The project team appraises the contractor team?s
performance over the total period of the contract for compliance with
requirements.
The project team appraises the contractor team?s performance over the total
period of the contract for compliance with requirements.
Strength Activity performed 5 Planned evaluations are performed
on the evolving software products and services before acceptance for
operational use.
Planned evaluation activities are performed on the evolving software
products and services before they are accepted for operational use.
Strength Activity performed 6 Results of the evaluations are
analyzed and compared to the contract?s requirements to establish an
objective basis to support the decision to accept the product or services or
to take further action.
Results of the evaluations are analyzed and compared to the contract?s
requirements to establish an objective basis supporting the decision to
accept the software products or to take further action.
Strength
Chapter 5: HUD?s Software Evaluation Processes Are Not Repeatable
Page 62 GAO- 01- 962 HUD Information Systems
Resident Assessment Subsystem Common feature Key practice Finding Rating
Measurement and analysis 1 Measurements are made and used to determine the
status of the evaluation activities and resultant products.
No measurements are made of HUD?s evaluation activities and resultant
products. Weakness
Verifying implementation 1 Evaluation activities are reviewed by acquisition
organization management on a periodic basis.
Acquisition organization management does not review evaluation activities on
a regular basis.
Weakness Verifying implementation 2 Evaluation activities are reviewed by
the project manager on both a periodic and an event- driven basis.
The project manager stated that while he meets with his staff daily to
discuss the status of the project, meeting results and action items are not
documented.
Weakness
Chapter 5: HUD?s Software Evaluation Processes Are Not Repeatable
Page 63 GAO- 01- 962 HUD Information Systems
Table 21: Software Evaluation Findings for HUD?s Central Accounting and
Program System HUD?s Central Accounting and Program System Common feature
Key practice Finding Rating
Commitment to perform 1 The acquisition organization has a written policy
for managing the evaluation of acquired software products and services.
HUD has no written policy for managing the evaluation of acquired software
products and services. The project is following the evaluation policy in
HUD?s software development methodology.
Observation Commitment to perform 2 Responsibility has been designated for
evaluation activities. Responsibility for evaluation activities has been
assigned to the project lead. Strength
Ability to perform 1 A group is responsible for planning, managing, and
performing evaluation activities.
The project lead and support staff are responsible for evaluation
activities. Strength
Ability to perform 2 Adequate resources are provided for evaluation
activities. There is no mechanism to identify needed
resources and ensure that they are provided to the project. The project
manager stated that the team does not have adequate resources for performing
evaluation activities.
Weakness Ability to perform 3 Individuals performing evaluation
activities have experience or receive training
Project team members performing evaluation activities have experience and
have received training.
Strength Ability to perform 4 Members of the project team and
groups supporting the software acquisition receive orientation on the
objectives of the evaluation approach.
Members of the project team and groups supporting the software acquisition
did not receive orientation on the objectives of the evaluation approach.
Weakness Activity performed 1 The project team performs its activities
in accordance with its documented evaluation plans.
There is no evaluation plan; therefore, the activities are not performed in
accordance with it.
Weakness Activity performed 2 The project?s evaluation requirements
are developed in conjunction with the development of the system or software
technical requirements.
Project evaluation requirements were developed in conjunction with software
technical requirements and are designated in the statement of work.
Strength Activity performed 3 The project?s evaluation activities are
planned to minimize duplication and take advantage of all evaluation
results, where appropriate.
There appears to be no duplication of evaluation activities. Strength
Activity performed 4 The project team appraises the contractor team?s
performance over the total period of the contract for compliance with
requirements.
The project team appraises the contractor team?s performance over the total
period of the contract for compliance with requirements.
Strength Activity performed 5 Planned evaluations are performed on
the evolving software products and services before acceptance for
operational use.
Planned evaluation activities are performed on the evolving software
products and services before they are accepted for operational use.
Strength Activity performed 6 Results of the evaluations are analyzed
and compared to the contract?s requirements to establish an objective basis
to support the decision to accept the product or services or to take further
action.
Results of the evaluations are analyzed and compared to the contract?s
requirements to establish an objective basis to support the decision to
accept the software products or to take further action.
Strength
Chapter 5: HUD?s Software Evaluation Processes Are Not Repeatable
Page 64 GAO- 01- 962 HUD Information Systems
HUD?s Central Accounting and Program System Common feature Key practice
Finding Rating
Measurement and analysis 1 Measurements are made and used to determine the
status of the evaluation activities and resultant products.
No measurements are made of HUD?s evaluation activities or resultant
products. Weakness
Verifying implementation 1 Evaluation activities are reviewed by acquisition
organization management on a periodic basis.
Acquisition organization management does not review evaluation activities on
a regular basis.
Weakness Verifying implementation 2 Evaluation activities are reviewed by
the project manager on both a periodic and an event- driven basis.
The project manager stated that while he discusses the status of the project
with team members, meeting results and action items are not documented.
Weakness
Chapter 5: HUD?s Software Evaluation Processes Are Not Repeatable
Page 65 GAO- 01- 962 HUD Information Systems
Table 22: Software Evaluation Findings for the Empowerment Information
System Empowerment Information System Common feature Key practice Finding
Rating
Commitment to perform 1 The acquisition organization has a written policy
for managing the evaluation of acquired software products and services.
HUD has no written policy for managing the evaluation of acquired software
products and services.
Weakness Commitment to perform 2 Responsibility has been designated for
evaluation activities. Responsibility for evaluation activities has been
assigned to the project lead. Strength
Ability to perform 1 A group is responsible for planning, managing, and
performing evaluation activities.
The project lead and team members are responsible for evaluation activities.
Strength
Ability to perform 2 Adequate resources are provided for evaluation
activities. There is no mechanism to identify needed
resources and ensure that they are provided to the project. The project
manager stated that the team does not have adequate resources for performing
evaluation activities.
Weakness Ability to perform 3 Individuals performing evaluation
activities have experience or receive training.
Project team members performing evaluation activities have experience and
have received training.
Strength Ability to perform 4 Members of the project team and
groups supporting the software acquisition receive orientation on the
objectives of the evaluation approach.
Members of the project team or others supporting the software acquisition
did receive orientation on the objectives of the evaluation approach.
Strength Activity performed 1 The project team performs its activities
in accordance with its documented evaluation plans.
There is no evaluation plan; therefore, the activities were not performed in
accordance with it.
Weakness Activity performed 2 The project?s evaluation requirements
are developed in conjunction with the development of the system or software
technical requirements.
The project?s evaluation requirements were not developed in conjunction with
the development of the software technical requirements.
Weakness Activity performed 3 The project?s evaluation activities are
planned to minimize duplication and take advantage of all evaluation
results, where appropriate.
There is no evidence of evaluation activities being performed. Weakness
Activity performed 4 The project team appraises the contractor team?s
performance over the total period of the contract for compliance with
requirements.
The project team assessed contractor team performance for compliance with
requirements only at the end of the effort.
Weakness Activity performed 5 Planned evaluations are performed on
the evolving software products and services before acceptance for
operational use.
No planned evaluations were performed on the software. A user survey was
conducted, and the software was rejected on the basis of the survey.
Weakness Activity performed 6 Results of the evaluations are
analyzed and compared to the contract?s requirements to establish an
objective basis to support the decision to accept the product or services or
to take further action.
No planned evaluations were conducted. The software was rejected based on
the results of the user survey.
Weakness
Chapter 5: HUD?s Software Evaluation Processes Are Not Repeatable
Page 66 GAO- 01- 962 HUD Information Systems
Empowerment Information System Common feature Key practice Finding Rating
Measurement and analysis 1 Measurements are made and used to determine the
status of the evaluation activities and resultant products.
No measurements are made of HUD?s evaluation activities or resultant
products. The cost of the user survey was reported.
Observation Verifying implementation 1 Evaluation activities are reviewed by
acquisition organization management on a periodic basis.
Acquisition organization management does not review evaluation activities on
a regular basis.
Weakness Verifying implementation 2 Evaluation activities are reviewed by
the project manager on both a periodic and an event- driven basis.
The project manager does not formally review evaluation activities, and the
project manager?s reviews are not documented.
Weakness
hapter 6: Conclusions, Recommendations, and Agency Comments
Page 67 GAO- 01- 962 HUD Information Systems
HUD?s software acquisition processes are immature and are not repeatable on
a project- by- project basis because of the many weaknesses it has in the
key practices. Strong performance of the key practices is essential for
achieving effective, repeatable, and lasting implementation and
institutionalization of the four key process areas we reviewed. Currently,
HUD?s success or failure in acquiring software depends largely on specific
individuals, rather than on well- defined and disciplined software
acquisition management practices. As a result, HUD is exposed to higher
risks that software- intensive acquisition projects will not consistently
meet mission requirements, perform as intended, or be delivered on schedule
and within budget.
HUD acknowledged that it has software acquisition weaknesses and is
committed to improving its software and system acquisition processes. We
agree that HUD should begin a software process improvement effort. However,
for HUD to successfully complete such an effort, its software process
improvement plan must address several important issues. To be comprehensive,
this plan should include the results of our review and include measurable
goals and time frames, estimates of resource requirements, and time frames
to reach the repeatable level. It should also contain steps to address the
systemic weaknesses we found. Finally, the plan should include steps to
address the areas we did not review and to ensure that all ongoing as well
as new software acquisition projects adopt processes and meet the
requirements for the repeatable level of maturity.
To improve HUD?s software acquisition capabilities, GAO recommends that the
Secretary of Housing and Urban Development direct the HUD Chief Information
Officer to develop and implement a comprehensive plan for software
acquisition process improvement that is based on the software capability
results in this report and specifies measurable goals and time frames, sets
priorities for initiatives, estimates resource requirements (for trained
staff and funding), and defines a process improvement management structure.
Also, to address the systemic weaknesses mentioned above, the plan should
contain steps to
develop a comprehensive policy for the acquisition of software,
require plans for specific acquisition activities,
measure and track the status of activities performed by the software
acquisition teams, and
report regularly to management on progress and problems. hapter 6:
Conclusions, Recommendations,
and Agency Comments
hapter 6: Conclusions, Recommendations, and Agency Comments
Page 68 GAO- 01- 962 HUD Information Systems
In addition, to ensure that all aspects of software acquisition are
addressed, we recommend that the Secretary direct the CIO to
assess HUD?s maturity in the three key process areas that could not be
evaluated by GAO and include any needed improvement actions in the
comprehensive plan for software acquisition process improvement;
ensure that all new software acquisition projects in HUD adopt processes
that satisfy at least SA- CMM level 2 requirements; and
ensure that process improvement activities are initiated for all ongoing
software acquisition projects.
In providing written comments on a draft of this report, the Chief
Information Officer (CIO) stated that HUD concurred with our recommendations
to strengthen its software acquisition processes. The CIO acknowledged that
its software acquisition processes have weaknesses and stated that it has an
improvement project under way. This project includes plans to seek
contractor assistance to address the weaknesses we identified and help
improve HUD?s software acquisition processes. Finally, the CIO stated that
the department generally agreed with our assessment of its software
acquisition practices. HUD also provided additional documentation on its
practices for the Resident Assessment Subsystem. We reviewed this
information and made revisions where appropriate and supported by the
additional documentation. The department?s comments are reprinted as
appendix I. Agency Comments
and Our Evaluation
Appendix I: Comments From the Department of Housing and Urban Development
Page 69 GAO- 01- 962 HUD Information Systems
Appendix I: Comments From the Department of Housing and Urban Development
Appendix I: Comments From the Department of Housing and Urban Development
Page 70 GAO- 01- 962 HUD Information Systems
Appendix II: GAO Contact and Staff Acknowledgments Page 71 GAO- 01- 962 HUD
Information Systems
David G. Gill (202) 512- 6250 In addition to those named above, Naba
Barkakati, Suzanne M. Burns, John Christian, Barbara Collier, Tonia Johnson,
Barbara Oliver, and Madhav Panwar made key contributions to this report.
Appendix II: GAO Contact and Staff
Acknowledgments GAO Contact Staff Acknowledgments
(310306)
The first copy of each GAO report is free. Additional copies of reports are
$2 each. A check or money order should be made out to the Superintendent of
Documents. VISA and MasterCard credit cards are also accepted.
Orders for 100 or more copies to be mailed to a single address are
discounted 25 percent.
Orders by mail:
U. S. General Accounting Office P. O. Box 37050 Washington, DC 20013
Orders by visiting:
Room 1100 700 4 th St., NW (corner of 4 th and G Sts. NW) Washington, DC
20013
Orders by phone:
(202) 512- 6000 fax: (202) 512- 6061 TDD (202) 512- 2537
Each day, GAO issues a list of newly available reports and testimony. To
receive facsimile copies of the daily list or any list from the past 30
days, please call (202) 512- 6000 using a touchtone phone. A recorded menu
will provide information on how to obtain these lists.
Orders by Internet
For information on how to access GAO reports on the Internet, send an email
message with ?info? in the body to
Info@ www. gao. gov or visit GAO?s World Wide Web home page at http:// www.
gao. gov
Contact one:
Web site: http:// www. gao. gov/ fraudnet/ fraudnet. htm
E- mail: fraudnet@ gao. gov
1- 800- 424- 5454 (automated answering system) Ordering Information
To Report Fraud, Waste, and Abuse in Federal Programs
*** End of document. ***