District of Columbia: Software Acquisition Processes for a New Financial
Management System (Letter Report, 04/30/98, GAO/AIMD-98-88).
Pursuant to a congressional request, GAO reviewed whether the District
of Columbia had implemented disciplined software acquisition processes
for its new financial management system (FMS).
GAO noted that: (1) while the district has many strengths in its
acquisition processes for FMS, it also has many weaknesses; (2) when
compared to standards established by the Software Engineering Institute,
the District's processes for software acquisitions are not mature; (3)
of the six key process areas (KPA) evaluated for the repeatable level,
the District fully satisfied only one--solicitation; (4) severe
weaknesses were found in other critical key processes, including
requirements development and management and evaluation; (5) for example,
the District does not have a policy for establishing and managing
software-related requirements, does not at present have adequate
resources for requirements development, and has not formally designated
responsibility for requirements development and management; (6)
similarly, the District does not have an effective evaluation process,
and is currently unable to objectively determine if the acquired systems
will satisfy the contract requirement; (7) finally, the District has not
satisfied the one KPA evaluated for the defined level of maturity,
acquisition risk management; and (8) the FMS project does not have a
risk management plan and does not track project risk.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: AIMD-98-88
TITLE: District of Columbia: Software Acquisition Processes for a
New Financial Management System
DATE: 04/30/98
SUBJECT: Computer software
Financial management systems
Requirements definition
Computer software verification and validation
Strategic information systems planning
Municipal governments
Financial management systems
IDENTIFIER: District of Columbia
DC Financial Management System
Software Capability Maturity Model
CMM
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to the Chairman, Subcommittee on the District of Columbia,
Committee on Appropriations, House of Representatives
April 1998
DISTRICT OF COLUMBIA - SOFTWARE
ACQUISITION PROCESSES FOR A NEW
FINANCIAL MANAGEMENT SYSTEM
GAO/AIMD-98-88
D.C. Financial Management System
(916247)
Abbreviations
=============================================================== ABBREV
FMS - financial management system
KPA - key process area
SA-CMM - Software Acquisition Capability Maturity Model
SCE - Software Capability Evaluation
SEI - Software Engineering Institute
Letter
=============================================================== LETTER
B-279266
April 30, 1998
The Honorable Charles H. Taylor
Chairman, Subcommittee on the
District of Columbia
Committee on Appropriations
House of Representatives
Dear Mr. Chairman:
This report responds to your request that we determine whether the
District of Columbia had implemented disciplined software acquisition
processes for its new financial management system (FMS). We found
that the District's software acquisition processes for the FMS
acquisition, while having some strengths, are not mature when
compared to standards established by the Software Engineering
Institute (SEI).\1 Accordingly, we are making recommendations for
strengthening these processes as they relate to the FMS project and
to improve any future software acquisitions.
--------------------
\1 Carnegie Mellon University's Software Engineering Institute (SEI),
recognized for its expertise in software processes, has developed
models and methods that define and determine organizations' software
process maturity.
BACKGROUND
------------------------------------------------------------ Letter :1
In September 1997, the District of Columbia Financial Responsibility
and Management Assistance Authority (Authority) awarded a contract to
acquire a new FMS. The overall objective of the FMS project is to
improve the District's financial systems through faster, more
efficient, and accurate processing providing increased functionality,
flexibility, and reduced cost of operations. According to the Chair
of the Authority, the new FMS is intended to (1) eliminate the
principal problems that exist with the current system and ensure that
all financial management guidelines are adhered to, (2) enable
managers to more effectively and efficiently monitor and control
financial resources, and (3) produce timely, accurate, and reliable
information, providing decisionmakers with the basic financial
information needed to make more informed decisions.
The Authority awarded a contract for the new FMS in September 1997
and committed to an aggressive implementation schedule. The schedule
anticipates (1) pilots in five agencies beginning in February 1998,
(2) the accounting system to be implemented by October 1998, and (3)
District-wide implementation by February 1999.
OBJECTIVE, SCOPE, AND
METHODOLOGY
------------------------------------------------------------ Letter :2
We were asked to review the District's efforts to acquire a new
financial management system. Our objective was to determine whether
the District had implemented disciplined software acquisition
processes for acquiring its new financial management system.
To accomplish this, we applied the Software Engineering Institute's
Software Acquisition Capability Maturity Model (SA-CMM) and its
Software Capability Evaluation (SCE) method. SEI's expertise in, and
methods for, software process assessment are recognized and accepted
throughout the industry. Our evaluators were all SEI-trained
software specialists.
SA-CMM ranks organizational maturity according to five levels (see
figure 1). Maturity levels 2 through 5 require the verifiable
existence and use of certain software acquisition processes, known as
key process areas (KPA). According to SEI, an agency that has these
acquisition processes in place is in a much better position to
successfully acquire software than an organization that does not have
these processes in place. We evaluated the District's software
acquisition processes against six of the seven level 2 KPAs (the
transition to support KPA was not evaluated because the District does
not plan to support FMS in-house) and one level 3 KPA (see table 1).
We selected level 2 because it is the minimum level at which any
assurance exists that software acquisition processes are mature
enough to consistently deliver promised software capabilities on time
and within budget. We included one level 3 KPA--acquisition risk
management--because it is considered by software experts to be a very
important process area.
Figure 1: SA-CMM Levels and
Descriptions
(See figure in printed
edition.)
Table 1
SA-CMM KPAs Used to Assess the FMS
Project Software Acquisition Maturity
SA-CMM level 2 key
process areas Description
---------------------- ----------------------------------------------
Software acquisition Ensuring that reasonable planning for the
planning software acquisition is conducted and that all
elements of the project are included.
Solicitation Ensuring that award is made to the contractor
most capable of satisfying the specified
requirements.
Requirements Establishing a common and unambiguous
development and definition of software acquisition
management requirements understood by the acquisition
team, system user, and the contractor.
Project management Managing the activities of the project office
and supporting contractor(s) to ensure a
timely, efficient, and effective software
acquisition.
Contract tracking and Ensuring that the software activities under
oversight contract are being performed in accordance
with contract requirements, and that products
and services will satisfy contract
requirements.
Evaluation Determining that the acquired software
products and services satisfy contract
requirements prior to acceptance and
transition to support.
SA-CMM level 3 key
process area Description
Acquisition risk Identifying risks as early as possible,
management adjusting acquisition strategy to mitigate
those risks, and developing and implementing a
risk management process as an integral part of
the acquisition process.
----------------------------------------------------------------------
As established by the model, each KPA contains five common attributes
that indicate whether the implementation and institutionalization of
a KPA can be effective, repeatable, and lasting. The five common
attributes are:
Commitment to perform. Commitment to perform describes the actions
that the organization must take to establish the process and ensure
that it can endure. Commitment to perform typically involves
establishing organizational policies and sponsorship. Key practices
under commitment to perform include having written organization
policy for the process area and designating and assigning
responsibility to individuals to perform the activities.
Ability to perform. Ability to perform describes the preconditions
that must exist in the project or organization to implement the
software acquisition process competently. Ability to perform
typically involves resources, organizational structures, and
training. Key practices under ability to perform include having
either experienced or trained personnel and providing adequate
resources to conduct the activities for the process area.
Activities performed. Activities performed describes the roles and
procedures necessary to implement a KPA. Activities performed
typically involve establishing plans and procedures, performing the
work, tracking it, and taking appropriate management actions. Key
practices under activities performed include performing these
activities in accordance with documented plans and overall management
of the project team and process area.
Measurement and analysis. Measurement and analysis describes
activities performed to measure the process and analyze the
measurements. Measurement and analysis typically includes defining
the measurements to be taken and the analyses to be conducted to
determine the status and effectiveness of the activities performed.
Verifying implementation. Verifying implementation describes the
steps to ensure that the activities are performed in compliance with
the process that has been established. There are two key practices
under verifying implementation, including reviews by various levels
of management.
In accordance with SEI's methodology, for each KPA selected, we
evaluated the District's policies and practices. This
project-specific comparison can result in one of four possible
outcomes: (1) project strength--an effective implementation of the
key practice, (2) project weakness--ineffective implementation of a
key practice or failure to implement a key practice, (3) project
observation--key practice evaluated but evidence inconclusive and
cannot be characterized as either a strength or weakness, and (4) not
rated--key practice not currently relevant to project, therefore not
evaluated.
As requested, we applied the SEI methodology to evaluate the
District's management of only one project--the acquisition of a new
financial management system. As a result, we have no information
regarding the process strengths and weaknesses of other acquisitions
the District may have underway.
We performed our work at District of Columbia offices in Washington,
D.C. between September 16, 1997, and January 16, 1998, in accordance
with generally accepted government auditing standards.
RESULTS IN BRIEF
------------------------------------------------------------ Letter :3
While the District has many strengths in its acquisition processes
for FMS, it also has many weaknesses. When compared to standards
established by the SEI, the District's processes for software
acquisition are not mature. Of the six KPAs evaluated for the
repeatable level, the District fully satisfied only
one--solicitation. Severe weaknesses were found in other critical
key processes, including requirements development and management and
evaluation. For example, the District does not have a policy for
establishing and managing software-related requirements, does not, at
present, have adequate resources for requirements development, and
has not formally designated responsibility for requirements
development and management.
Similarly, the District does not have an effective evaluation
process, and is currently unable to objectively determine if the
acquired system will satisfy the contract requirements. Finally, the
District has not satisfied the one key process area evaluated for the
"defined" level of maturity, acquisition risk management. The FMS
project does not have a risk management plan and does not track
project risk. Figure 2 provides a comprehensive listing of the FMS
project's strengths, weaknesses, and observations for the seven KPAs.
The items under features in figure 2 refer to the five common
attributes in each KPA and are explained in detail in the following
sections.
Figure 2: Key Process Area
Results
(See figure in printed
edition.)
SOFTWARE ACQUISITION PLANNING
------------------------------------------------------------ Letter :4
The purpose of software acquisition planning is to ensure that
reasonable planning for the software acquisition is conducted and
that all aspects of the total software acquisition effort are
included in these plans at the proper level of detail. The software
acquisition planning process, among other things, includes (1)
addressing software life-cycle support in acquisition plans, (2)
preparing life-cycle software cost estimates, (3) having a written
software acquisition policy, (4) measuring and reporting on the
status of software acquisition planning activities, and (5) having
guidance on software training and experience requirements for project
personnel.
FMS PROJECT PERFORMS SOME
BUT NOT ALL SOFTWARE
ACQUISITION PLANNING
PRACTICES
---------------------------------------------------------- Letter :4.1
The FMS project had many strengths in this KPA. The District
received pro bono assistance from several companies to help define
the acquisition strategy and conduct the activities for software
acquisition planning. An acquisition strategy was developed and the
acquisition planning team was staffed with personnel with software
and systems experience. The team developed a cost estimate and the
District management was briefed by the team on a periodic basis.
This enabled the District management to be informed on the progress
of the acquisition planning and the various activities through the
solicitation phase.
However, the FMS project also had many weaknesses in this KPA.
Weaknesses observed included a lack of policy on acquisition planning
and no specific assignment of responsibility for acquisition
planning. Furthermore, the FMS project did not always document
significant project decisions or update the planning document to
reflect these decisions. For example, when the District decided to
not pursue a single contract to both acquire FMS and outsource data
center operations, the capability assessment (a software acquisition
planning document) was not updated to reflect this decision.
Decisions should be documented and the planning documents updated to
ensure that large acquisitions such as FMS can be effectively
managed. Table 2 shows the strengths and weaknesses for the software
acquisition planning KPA and the specific findings supporting these
ratings.
Table 2
Software Acquisition Planning Findings
Feature Key practice Finding Rating
------------ --------------------------- --------------------------- --------
Commitment 1 The acquisition There is no written policy Weakness
organization has a written for software acquisition
policy for planning the planning.
software acquisition.
Commitment 2 Responsibility for software Responsibility for software Weakness
acquisition planning acquisition planning
activities is designated. activities was not
designated.
Ability 1 The acquisition The acquisition staff (10- Strength
organization has 15 Authority and D.C.
experienced software government staff, and 10-
acquisition management 15 private sector employees
personnel. working on a pro bono
basis) had requisite
experience.
Ability 2 Adequate resources are Required resources for Weakness
provided for software acquisition planning were
acquisition planning not determined.
activities.
Activity 1 Software acquisition The District government Strength
planning personnel are recruited outside software
involved in system experts to assist with
acquisition planning. system acquisition
activities.
Activity 2 The software acquisition The software acquisition Strength
strategy for the project is strategy is developed and
developed and documented. includes risk
identification and
schedules. It is documented
in the capability
assessment document.
Activity 3 The project's software Program changes regarding Weakness
acquisition planning is outsourcing of the data
documented and the planning center and upgrading the
documentation is maintained current system versus
over the life of the buying off the shelf were
project. made, but the software
acquisition planning
documentation was not
updated.
Activity 4 Life-cycle support of the The software acquisition Weakness
software is included in planning documentation does
software acquisition not address life cycle
planning documentation. support of the software.
Activity 5 Life-cycle cost and Life-cycle cost and Strength
schedule estimates for the schedule estimates were
software products and prepared by a consultant.
services being acquired are They were reviewed by the
prepared and independently District Chief Financial
reviewed. Officer and Authority
Executive Director.
Measurement Measurements are made and Measurements of software Weakness
1 used to determine the acquisition activities were
status of the software not taken.
acquisition planning
activities and resultant
products.
Verification Software acquisition The Authority Executive Strength
1 planning activities are Director was briefed on
reviewed by acquisition planning activities on a
organization management on periodic basis.
a periodic basis.
Verification Software acquisition Software acquisition Strength
2 planning activities are planning activities were
reviewed by the project reviewed by the project
manager on both a periodic manager.
and event-driven basis.
--------------------------------------------------------------------------------
SOLICITATION
------------------------------------------------------------ Letter :5
The purpose of solicitation is to prepare a request for proposal that
delineates a project's software-related requirements and select a
contractor that can most cost-effectively satisfy these requirements
while complying with relevant solicitation laws and regulations.
Specific requirements for a solicitation process include, among other
things (1) having and following a solicitation plan, (2) assigning
responsibility and ensuring sufficient resources for coordinating and
conducting solicitation activities, (3) preparing and reviewing cost
and schedule estimates for the software products and services being
acquired, and (4) periodically measuring solicitation work completed
and effort and funds expended, comparing these measures to plans, and
reporting the results to management.
FMS PROJECT PERFORMING MOST
SOLICITATION PRACTICES
---------------------------------------------------------- Letter :5.1
The FMS project exhibited many process strengths during the
solicitation. The District has a policy on solicitation and the FMS
project followed this policy. The project had experienced personnel
on the source selection team and these personnel briefed the team
members on the objectives of the solicitation. However, the District
did not measure either time or funds expended to conduct the
solicitation. Specifically, no evidence was provided to show that
the FMS project tracked personnel hours or costs during the conduct
of the solicitation. Addressing this weakness would enable the
District to better estimate the resources needed to conduct similar
acquisitions in the future. For example, if these data were
collected and made available to other projects, such as the tax
systems upgrade, the District would be in a better position to
understand its own capability to effectively conduct solicitation, to
estimate how long such a solicitation was likely to take, and to
eliminate problems that may have hampered the FMS solicitation.
Table 3 shows the strengths, weaknesses, and observations for the
solicitation KPA and the specific findings supporting these ratings.
Table 3
Solicitation Findings
Feature Key practice Finding Rating
------------ --------------------------- --------------------------- --------
Commitment 1 The acquisition Authority's procurement Strength
organization has a written regulations provides
policy for the conduct of written policies for
the software portion of the solicitation.
solicitation.
Commitment 2 Responsibility for the Authority's Executive Strength
software portion of the Director has designated
solicitation is designated. specific individuals to
serve as members of a
selection committee.
Commitment 3 A selection official has Authority's Executive Strength
been designated to be Director is designated as
responsible for the the source selection
selection process and the official.
decision.
Ability 1 A group that is responsible A solicitation organization Strength
for coordinating and has been identified in the
conducting solicitation source selection plan.
activities exists.
Ability 2 Adequate resources are Adequate resources Strength
provided for solicitation (including Authority staff
activities. and a team of private
sector experts) were
provided for solicitation
activities.
Ability 3 Individuals performing Experienced individuals Strength
solicitation activities from private contractor
have experience or receive staff and public sectors
training. assisted in the
solicitation activities.
Ability 4 The groups supporting the Participants in source Strength
solicitation (e.g., end selection process were
user, systems engineering, trained on the objectives
software support and procedures of FMS
organization, and solicitation.
application domain experts)
receive orientation on the
solicitation's objectives
and procedures.
Activity 1 The project team performs Project team conducted its Observat
its activities in activities according to the ion
accordance with its source selection plan,
documented solicitation however, that plan does not
plans. cover all solicitation
tasks.
Activity 2 The project team performs A number of subcommittees Strength
its activities in evaluated proposals
accordance with its according to the process
documented proposal documented in the source
evaluation plans. selection plan.
Activity 3 Cost and schedule estimates Capabilities assessment Strength
for the software products report documents the cost
and services being sought and schedule estimates for
are prepared. the new financial
management system. A
preliminary economic
analysis of FMS was also
performed.
Activity 4 Software cost and schedule Capabilities assessment Strength
estimates are independently report and the economic
reviewed for analysis of FMS were
comprehensiveness and reviewed by the D.C. Office
realism. of the Inspector General
and a report was issued.
Activity 5 The project team takes The project team ensured Strength
action to ensure the mutual mutual understanding of
understanding of software software requirements by
requirements and plans holding question and answer
prior to contract award. sessions with potential FMS
vendors.
Measurement Measurements are made and No measurements were taken Weakness
1 used to determine the to determine the status of
status of the solicitation solicitation activities.
activities and resultant
products.
Verification Solicitation activities are Authority Executive Strength
1 reviewed by the designated Director and the District
selection official or Chief Financial Officer are
acquisition organization briefed on status of all
management on a periodic FMS activities (including
basis. solicitation) on a periodic
basis.
Verification Solicitation activities are During the solicitation, Strength
2 reviewed by the project the entire team reported to
manager on both a periodic the Executive Director who
and event-driven basis. served as the project
manager. The Executive
Director was briefed
periodically and in
response to significant
events.
--------------------------------------------------------------------------------
REQUIREMENTS DEVELOPMENT AND
MANAGEMENT
------------------------------------------------------------ Letter :6
The purpose of requirements development and management is to
establish and maintain a common and unambiguous definition of
software requirements among the acquisition team, system users, and
software development contractor. This KPA involves two subprocesses:
(1) developing a baseline set of software-related contractual
requirements and (2) managing these requirements and changes to these
requirements for the duration of the acquisition.
A number of requirements development and management practices are
necessary to satisfy this key process area. These include (1) having
a written organizational policy for establishing and managing
requirements allocated to software, (2) documenting plans for the
development and management of requirements, (3) having documented
processes for requirements development, including elicitation,
analysis, and verification, (4) measuring and reporting on the status
of requirements development and management activities to management,
(5) appraising the impact on software of system-level requirements
changes and (6) having a mechanism to ensure that
contractor-delivered work products meet specified requirements.
FMS PROJECT PERFORMING SOME
BUT NOT MOST REQUIREMENTS
DEVELOPMENT AND MANAGEMENT
PRACTICES
---------------------------------------------------------- Letter :6.1
The FMS project has some process strengths in the conduct of
requirements development and management. The project team is
performing requirements management activities in accordance with its
documented plan and software-related contractual requirements have
been baselined. In addition, District management periodically
reviews the status of requirements development and management
activities with the project team. However, in acquiring FMS, the
District did not perform many of the requirements development and
management practices necessary to satisfy this KPA. For example, the
District does not have an organizational policy for establishing and
managing software-related requirements, there is no clear assignment
of responsibility for requirements development and management and no
documented evidence exists to show either resource requirements or
resources expended for requirements development activities.
Currently, the FMS project has begun to hold "requirements
confirmation meetings" with the users to validate the requirements
already specified in the FMS contract. Although requirements should
be validated, this should have been done prior to releasing the
request for proposal to ensure that the proposal accurately reflects
the District's requirements. Changing requirements after contract
award may adversely impact project cost, schedule, and/or
performance. Table 4 shows the strengths, weaknesses, and
observations for the requirements development and management KPA and
the specific findings supporting these ratings.
Table 4
Requirements Development and Management
Findings
Feature Key practice Finding Rating
------------ --------------------------- --------------------------- --------
Commitment 1 The acquisition There is no organizational Weakness
organization has a written policy for establishing and
policy for establishing and managing software-related
managing the software- requirements.
related contractual
requirements.
Commitment 2 Responsibility for There is no formal Weakness
requirements development documentation assigning
and management is anyone responsibility for
designated. requirements development
and management.
Ability 1 A group that is responsible There is no documentation Weakness
for performing requirements assigning any group
development and management responsibility for
activities exists. performing requirements
development and management
activities.
Ability 2 Adequate resources are No documented evidence Weakness
provided for requirements exists to show either
development and management resource requirements or
activities. resources used for
requirements development
and management activities.
Ability 3 Individuals performing Individuals currently Observat
requirements development performing requirements ion
and management activities management have appropriate
have experience or receive experience. However, no
training. documented evidence was
provided to show the
experience of individuals
who developed the
requirements.
Activity 1 The project team performs The requirements management Strength
its activities in activities are incorporated
accordance with its in the project management
documented requirements plan, volumes I and II.
development and management This project team is
plans. performing its activities
in accordance with this
plan.
Activity 2 The project team develops The project team baselined Strength
and baselines the software- software-related
related contractual contractual requirements in
requirements and places the statement of work prior
them under change control to the release of the
early in the project, but solicitation. The
not later than release of requirements are under
the solicitation package. change control.
Activity 3 The project team appraises There have been no Observat
system requirements change requirements changes to ion
requests for their impact date to appraise. However,
on the software being a process to appraise
acquired. changes exists and is
documented in the program
management plan.
Activity 4 The project team appraises The scope change process Observat
all changes to the documented in the project ion
software-related management plan requires
contractual requirements the appraisal of cost and
for their impact on schedule impact. However,
performance, architecture, no evidence provided of any
supportability, system changes to the contract to
resource utilization, and date.
contract schedule and cost.
Activity 5 Bi-directional traceability There is no evidence to Weakness
between the software- show traceability between
related contractual contractual requirements
requirements and the and the contractor's work
contractor's software work products.
products and services is
maintained throughout the
effort.
Measurement Measurements are made and No measurements made to Weakness
1 used to determine the determine status of the
status of the requirements requirements development
development and management and management activities.
activities and resultant
products.
Verification Requirements development Project manager Strength
1 and management activities periodically reviews
are reviewed by acquisition requirements development
organization management and management activities.
(and the contractor) on a
periodic basis.
Verification Requirements development Periodic and event-driven Strength
2 and management activities meetings are held to
are reviewed by the project discuss requirements
manager on both a periodic development and management
and event-driven basis. activities.
--------------------------------------------------------------------------------
PROJECT MANAGEMENT
------------------------------------------------------------ Letter :7
The purpose of project management is to manage the activities of the
project office and supporting contractors to ensure a timely,
efficient, and effective software acquisition. Effective project
management requires, among other things, that project teams (1) be
organized to accomplish the project's objective, (2) have a written
policy for the management of the software project, (3) document their
plans for the activities of the project team, (4) have the authority
to alter either the project's performance, cost, or schedule baseline
while maintaining the other two, and (5) periodically brief
management on the status of project management activities.
FMS PROJECT PERFORMING MOST
PROJECT MANAGEMENT PRACTICES
---------------------------------------------------------- Letter :7.1
The FMS project has many process strengths in project management.
For example, a team was assigned responsibility for managing the
project and staffed with experienced individuals whose roles and
responsibilities were defined. The program management plan was
written and a corrective action system to track issues and problems
was implemented. However, the District has no written policy for the
execution of the software project. As a result, the District has no
assurance that FMS or any other software acquisition project it
undertakes will be conducted in a disciplined manner. Table 5 shows
the strengths, weaknesses, and observations for the project
management KPA and the specific findings supporting these ratings.
Table 5
Project Management Findings
Feature Key practice Finding Rating
------------ --------------------------- --------------------------- --------
Commitment 1 The acquisition There is no documented Weakness
organization has a written policy for the execution of
policy for execution of the the software project.
software project.
Commitment 2 Responsibility for project Responsibility for project Strength
management is designated. management has been
designated to the financial
management system project
team/program manager.
Ability 1 A team that is responsible A team that is responsible Strength
for performing the for performing software
project's software acquisition management
acquisition management activities exists.
activities exists.
Ability 2 Adequate resources for the Adequate personnel Observat
project team and matrix resources for project ion
support persons are management were provided.
provided for the duration At the time of the audit,
of the software acquisition there were no telephones,
project. copiers, or supplies.
Ability 3 When project trade-offs are The project manager is not Weakness
necessary, the project permitted to independently
manager is permitted to alter either the
alter either the performance, cost, or
performance, cost, or schedule.
schedule software
acquisition baseline.
Ability 4 The project team and matrix Project team and matrix Strength
support individual(s) have support individuals have
experience or receive experience in acquisition
training in project management activities.
software acquisition
management activities.
Activity 1 The project team performs Project team activities are Strength
its activities in performed in accordance
accordance with its with program management
documented software plan.
acquisition management
plans.
Activity 2 The organization of the The organization of the Strength
project provides for the project provides for the
management of all project management of all project
functions. functions.
Activity 3 The software acquisition The roles and Strength
management activities of responsibilities of team
the project team are members are defined and
directed to accomplish the serve to accomplish the
project's objectives. project's objectives.
Activity 4 The software acquisition The software acquisition Strength
management activities of management activities of
the project team are the team are controlled.
controlled.
Activity 5 The project team implements A corrective action system, Strength
a corrective action system called the issue process,
for the identification, has been implemented.
recording, tracking, and
correction of problems
discovered during the
software acquisition.
Activity 6 The project team tracks The project team tracks the Strength
project status, execution, project's status and
funding, and expenditures execution. Funding and
and takes action. expenditures are tracked by
the Office of the Chief
Financial Officer.
Measurement Measurements are made and No evidence that Weakness
1 used to determine the measurements are taken to
status of the project determine the status of
management activities and project management
resultant products. activities.
Verification Project management Monthly meetings are held Strength
1 activities are reviewed by where all levels of
acquisition organization management are briefed on
management on a periodic the project status.
basis.
Verification Project management Monthly meetings are held Strength
2 activities are reviewed by where all levels of
the project manager on both management are briefed on
a periodic and event- the project status.
driven basis.
--------------------------------------------------------------------------------
CONTRACT TRACKING AND OVERSIGHT
------------------------------------------------------------ Letter :8
The purpose of contract tracking and oversight is to ensure that (1)
the software development contractor performs according to the terms
of the contract, (2) needed contract changes are identified,
negotiated, and incorporated into the contract, and (3) contractor
performance issues are identified early, when they are easier and
less costly to address. An effective contract tracking and oversight
process, among other things, includes (1) having a written
organizational policy for contract tracking and oversight, (2) having
a documented plan for contract tracking and oversight, (3) conducting
tracking and oversight activities in accordance with the plan, and
(4) ensuring that individuals performing contract tracking and
oversight are suitably experienced or trained.
FMS PROJECT PERFORMING MANY
BUT NOT ALL CONTRACT
TRACKING AND OVERSIGHT
PRACTICES
---------------------------------------------------------- Letter :8.1
The FMS project had many strengths in this KPA. The project has a
designated project manager, a group is responsible for managing
contract tracking and oversight activities, and the team is meeting
periodically with the contractor and tracking issues in a corrective
action system. However, at the time of our review, there was no
contracting specialist supporting the team in the execution of the
contract. In addition, the District has no documented policy for
contract tracking and oversight activities. Table 6 shows the
strengths, weaknesses, and observations for the contract tracking and
oversight KPA and the specific findings supporting these ratings.
Table 6
Contract Tracking and Oversight Findings
Feature Key practice Finding Rating
------------ --------------------------- --------------------------- --------
Commitment 1 The acquisition There is no written policy Weakness
organization has a written for contract tracking and
policy for contract oversight activities for
tracking and oversight of the financial management
the contracted software system project.
effort.
Commitment 2 Responsibility for contract The project manager and the Strength
tracking and oversight program management team are
activities is designated. responsible for contract
tracking and oversight.
Commitment 3 The project team is The project team is not Weakness
supported by contracting supported by contracting
specialists in the specialists.
execution of the contract.
Ability 1 A group that is responsible Project manager and program Strength
for managing contract management staff are
tracking and oversight responsible for managing
activities exists. contract tracking and
oversight activities.
Ability 2 Adequate resources are Staff and funding resources Observat
provided for contract are adequate. However, at ion
tracking and oversight the time of the audit, the
activities. project team was lacking
tools such as telephones,
fax, copiers, and office
supplies.
Ability 3 Individuals performing Staff members have Strength
contract tracking and experience in contract
oversight activities have tracking and oversight.
experience or receive
training.
Activity 1 The project team performs The project team performs Strength
its activities in its activities in
accordance with its accordance with the program
documented contract management plan.
tracking and oversight
plans.
Activity 2 The project team reviews The project team did not Weakness
required contractor review any of the
software planning documents contractor's planning
which, when satisfactory, documents (e.g., project
are used to oversee the management plan, software
contractor's software risk management plan,
engineering effort. software engineering plan,
configuration management
plan).
Activity 3 The project team conducts The team conducts periodic Strength
periodic reviews and reviews with the
interchanges with the contractor.
contractor.
Activity 4 The project team reviews The District does not plan Not
and tracks the development to support FMS in-house. rated
of the software engineering
environment required to
provide life-cycle support
for the acquired software.
Activity 5 Any problems or issues The issue tracking system Strength
found by the project team is used to track and
during contract tracking provide oversight of
and oversight are recorded problems or issues found
in the appropriate during the contract. Issues
corrective action system are tracked to closure.
and tracked to closure.
Activity 6 The project team maintains No one assumed Weakness
the integrity of the responsibility for
contract throughout the maintaining the integrity
contract performance of the contract.
period.
Measurement Measurements are made and No measurements are taken Weakness
1 used to determine the to determine the status of
status of the contract contract tracking and
tracking and oversight oversight activities.
activities and resultant
products.
Verification Contract tracking and Periodic and event-driven Strength
1 oversight activities are meetings are held with the
reviewed by acquisition Authority to review
organization management on contract tracking and
a periodic basis. oversight activities.
Verification Contract tracking and Periodic and event-driven Strength
2 oversight activities are meetings are held with the
reviewed by the project program manager to review
manager on both a periodic contract tracking and
and event-driven basis. oversight activities.
--------------------------------------------------------------------------------
EVALUATION
------------------------------------------------------------ Letter :9
The purpose of evaluation (testing) is to determine that the acquired
software products and services satisfy contract requirements prior to
acceptance. The evaluation process includes (1) documenting
evaluation plans and conducting evaluation activities in accordance
with the plan, (2) developing and managing evaluation requirements in
conjunction with developing software technical requirements, (3)
incorporating evaluation requirements into the solicitation and the
resulting contract, (4) tracking contractor performance of evaluation
activities for compliance with the contract, (5) ensuring that
adequate resources are provided for evaluation activities, and (6)
measuring and reporting on the status of evaluation activities to
management.
FMS PROJECT PERFORMING SOME
BUT NOT MOST EVALUATION
PRACTICES
---------------------------------------------------------- Letter :9.1
The FMS project has some process strengths in this KPA. For example,
responsibility for evaluation activities has been designated to the
project manager, individuals designated to perform evaluation
activities have experience, and members of the evaluation team
received briefings on the objectives of the evaluation. However,
there is no documented evaluation policy or plan, no evidence that
evaluation requirements have been developed, and neither the
Authority nor the project manager reviews the status of evaluation
activities. Table 7 shows the strengths, weaknesses, and
observations for the evaluation KPA and the specific findings
supporting these ratings.
Table 7
Evaluation Findings
Feature Key practice Finding Rating
------------ --------------------------- --------------------------- --------
Commitment 1 The acquisition No written policy exists Weakness
organization has a written for managing the evaluation
policy for managing the of acquired software
evaluation of the acquired products and services.
software products and
services.
Commitment 2 Responsibility for Responsibilities for Strength
evaluation activities is evaluation activities has
clearly designated. been designated to the
program manager.
Ability 1 A group that is responsible The joint program Strength
for planning, managing, and management and contractor
performing evaluation team, along with the users,
activities for the project is responsible for
exists. planning, managing, and
performing evaluation
activities.
Ability 2 Adequate resources are While personnel resource Observat
provided for evaluation requirements for evaluation ion
activities. are identified, other
resources such as
traceability and data
collection tools have not
yet been identified.
Ability 3 Individuals performing Individuals performing Strength
evaluation activities have evaluation have experience
experience or receive in evaluation.
training.
Ability 4 Members of the project team Members of the project team Strength
and groups supporting the receive orientation on the
software acquisition objectives of the
receive orientation on the evaluation.
objectives of the
evaluation approach.
Activity 1 The project team performs There is no documented Weakness
its activities in evaluation plan.
accordance with its
documented evaluation
plans.
Activity 2 The project's evaluation There is no evidence that Weakness
requirements are developed evaluation requirements
in conjunction with the were developed in
development of the system conjunction with system
or software technical requirements.
requirements.
Activity 3 The evaluation requirements Requirements for the Strength
are incorporated into the contractor to support the
solicitation package and evaluation are in the
resulting contract. solicitation and contract.
Activity 4 The project team assesses The project team did not Weakness
the contractor's assess the contractor's
performance for compliance performance for compliance
with evaluation with evaluation
requirements. requirements.
Activity 5 Planned evaluations are Since no documented Weakness
performed on the acquired evaluation plan exists,
software products and there is no plan to follow.
services prior to
acceptance for operational
use.
Activity 6 Results of the evaluations The FMS project has not yet Observat
are analyzed and compared reached the stage where ion
to the contract's evaluation activities are
requirements to establish analyzed and compared to
an objective basis to the contract.
support the decision to
accept the products and
services or to take further
action.
Measurement Measurements are made and No measurements are made to Weakness
1 used to determine the determine the status of
status of the evaluation evaluation activities.
activities and resultant
products.
Verification Evaluation activities are The Authority does not Weakness
1 reviewed by acquisition review the status of
organization management on evaluation activities.
a periodic basis.
Verification Evaluation activities are The project manager does Weakness
2 reviewed by the project not review the status of
manager on both a periodic evaluation activities.
and event-driven basis.
--------------------------------------------------------------------------------
ACQUISITION RISK MANAGEMENT
----------------------------------------------------------- Letter :10
SEI defines risk as the possibility of suffering a loss. The purpose
of acquisition risk management is to formally identify risks as early
as possible and adjust the acquisition to mitigate those risks. An
effective risk management process, among other things, includes (1)
having a written policy on acquisition risk management, (2)
developing a software acquisition risk management plan, (3)
conducting software risk management activities in accordance with the
plan (e.g., identifying risks, taking mitigation actions, and
tracking risk mitigation actions to completion), and (4) measuring
and reporting on the status of acquisition risk management activities
to management.
FMS RISK MANAGEMENT
PROCESSES ARE INEFFECTIVE
--------------------------------------------------------- Letter :10.1
The FMS project had one strength for this KPA. The project has
designated responsibility for risk management to the project
management team. However, the District is not performing any of the
other practices to satisfy this KPA. For example, there is no
written policy or plan for acquisition risk management, resource
requirements for risk management have not been identified, and at the
time of this audit, neither the Authority nor the project manager
were reviewing the activities for risk management. Table 8 shows the
strengths, weaknesses, and observations for the acquisition risk
management KPA and the specific findings supporting these ratings.
Table 8
Acquisition Risk Management Findings
Feature Key practice Finding Rating
------------ --------------------------- --------------------------- --------
Commitment 1 The acquisition There is no written policy Weakness
organization has a written for software acquisition
policy for the management risk management.
of software acquisition
risk.
Commitment 2 Responsibility for software Responsibility for software Strength
acquisition risk management acquisition risk management
activities is designated. was designated to the
acquisition management
contractor and the program
office.
Ability 1 A group that is responsible No group is responsible for Weakness
for coordinating software coordinating software
acquisition risk management acquisition risk management
activities exists. activities.
Ability 2 Adequate resources are Resource requirements for Weakness
provided for software acquisition risk management
acquisition risk management have not yet been defined.
activities.
Ability 3 Individuals performing Individuals designated to Weakness
software acquisition risk perform software
management activities have acquisition risk management
experience or receive do not have experience and
required training. have not yet received
training.
Activity 1 Software acquisition risk Software acquisition risk Weakness
management activities are management activities were
integrated into software not integrated into
acquisition planning. software acquisition
planning.
Activity 2 The software acquisition A software acquisition risk Weakness
risk management plan is management plan has not
developed in accordance been developed in
with the project's defined accordance with a defined
software acquisition software acquisition
process. process.
Activity 3 The project team performs There is no documented Weakness
its software acquisition acquisition risk management
risk management activities plan.
in accordance with its
documented plans.
Activity 4 Risk management is Risk management is not Weakness
conducted as an integral conducted as an integral
part of the solicitation, part of the solicitation,
project performance project performance
management, and contract management, and contract
performance management performance management
processes. processes.
Activity 5 Software acquisition risk Software acquisition risk Weakness
handling actions are handling actions are not
tracked and controlled tracked and controlled
until the risks are until the risks are
mitigated. mitigated.
Measurement Measurements are made and Measurements are not made Weakness
1 used to determine the for risk management.
status of the acquisition
risk management activities
and resultant products.
Verification Acquisition risk management Risk management activities Weakness
1 activities are reviewed by are not reviewed by the
acquisition organization Authority.
management on a periodic
basis.
Verification Acquisition risk management Risk management activities Weakness
2 activities are reviewed by are not reviewed by the
the project manager on both project manager.
a periodic and event-
driven basis.
--------------------------------------------------------------------------------
CONCLUSIONS
----------------------------------------------------------- Letter :11
Leading software acquisition organizations rely on defined and
disciplined software acquisition processes to deliver promised
software capabilities on time and within budget, first on a
project-by-project basis, and later, as the organization's processes
become more mature, consistently across the institution. While the
District has many strengths in its acquisition processes for FMS, it
also has many weaknesses that, overall, make its processes
undisciplined and immature. As a result, the District's success or
failure in acquiring FMS depends largely on specific individuals
rather than on well-defined software acquisition management
practices. This greatly reduces the probability that the system will
consistently perform as intended and be delivered on schedule and
within budget.
To satisfy the intent of all the software acquisition key process
areas and thereby have a reasonable assurance that acquisition
efforts are effectively planned, managed, evaluated, and tracked, the
District must address the many weaknesses identified in this report.
This would entail the District formulating and implementing a written
policy for software acquisition planning, requirements development
and management, project management, contract tracking and oversight,
evaluation, and acquisition risk management. In addition, it is
important for the District to track the various activities for each
KPA to ensure that they are being performed and that evaluation and
risk management activities are being planned and effectively
conducted.
RECOMMENDATIONS
----------------------------------------------------------- Letter :12
We recommend that the Chairman of the District of Columbia Financial
Responsibility and Management Assistance Authority direct the
District's Chief Financial Officer to (1) take the following actions
for the six KPAs we reviewed to ensure that the current FMS
acquisition and implementation is satisfactorily completed and (2)
apply these actions to any future software acquisitions.
Software Acquisition Planning:
-- Document decisions and update the planning documents to ensure
that large acquisitions such as FMS can be effectively managed.
-- Designate responsibility for software acquisition planning
activities.
-- Determine required resources for acquisition planning.
-- Ensure that measurements of software acquisition activities are
taken.
-- Ensure that the software acquisition planning documentation is
updated as well as make program changes regarding outsourcing of
the data center and upgrading the current system versus buying
off-the-shelf.
-- Ensure that the software acquisition planning documentation
addresses life-cycle support of the software.
-- Develop a written policy for software acquisition planning.
Requirements Development and Management:
-- Develop an organizational policy for establishing and managing
software-related requirements.
-- Clearly assign responsibility for requirements development and
management.
-- Document either resource requirements or resources expended for
requirements development activities.
-- Develop the capability to trace between contractual requirements
and the contractor's work products.
-- Develop measurements to determine the status of the requirements
development and management activities.
Project Management:
-- Develop a written policy for the execution of the software
project.
-- Authorize the project manager to independently alter either the
performance, cost, or schedule.
-- Require that measurements be taken to determine the status of
project management activities.
Contract Tracking and Oversight:
-- Develop written policy for contract tracking and oversight
activities for the financial management system project.
-- Support the project team with contracting specialists.
-- Require that the project team review the contractor's planning
documents (for example, the project management plan, software
risk management plan, software engineering plan, configuration
management plan).
-- Assign someone responsibility for maintaining the integrity of
the contract.
-- Take measurements to determine the status of contract tracking
and oversight activities.
Evaluation:
-- Develop written policy for managing the evaluation of acquired
software products and services.
-- Develop a documented evaluation plan.
-- Develop evaluation requirements in conjunction with system
requirements.
-- Assess the contractor's performance for compliance with
evaluation requirements.
-- Develop measurements to determine the status of evaluation
activities.
-- Ensure that the Authority and the project manager review the
status of evaluation activities.
Acquisition Risk Management:
-- Develop written policy for software acquisition risk management.
-- Designate a group to be responsible for coordinating software
acquisition risk management activities.
-- Define resource requirements for acquisition risk management.
-- Ensure that individuals designated to perform software
acquisition risk management have adequate experience and
training.
-- Integrate software acquisition risk management activities into
software acquisition planning.
-- Develop a software acquisition risk management plan in
accordance with a defined software acquisition process.
-- Develop a documented acquisition risk management plan and
conduct risk management as an integral part of the solicitation,
project performance management, and contract performance
management processes.
-- Track and control software acquisition risk handling actions
until the risks are mitigated.
-- Ensure that risk management activities are reviewed by the
Authority and the project manager.
AGENCY COMMENTS AND OUR
EVALUATION
----------------------------------------------------------- Letter :13
GAO requested comments on a draft of this report from the Chairman,
District of Columbia Financial Responsibility and Management
Assistance Authority, and the District's Chief Financial Officer.
They provided us with written comments that are reprinted in
appendixes I and II.
In their comments, the District of Columbia Financial Responsibility
and Management Assistance Authority's Executive Director and the
District of Columbia's Chief Financial Officer acknowledged that the
software acquisition project for the new financial management system
was a high risk initiative and that the District's processes were not
sufficiently mature. The District Chief Financial Officer identified
initiatives in each of the key process areas. Both cited ongoing
corrective actions, which, if properly implemented, will address
several of our recommendations. For example, the Chief Financial
Officer stated that the District is developing a Risk Management Plan
and is evaluating various strategies to identify and manage risks,
and that the Chief Technology Officer for the District of Columbia is
developing policies and procedures for information resource
management which will include software acquisition.
However, the District Chief Financial Officer also added that their
efforts to date have achieved a sound acquisition state consistent
with the intent of the SA-CMM. As discussed in the report,
significant improvements would be necessary to achieve the minimally
acceptable level of maturity as defined by the Software Engineering
Institute's Software Acquisition Maturity Model to satisfy the intent
of all the software acquisition key process areas. Accordingly, the
District has not yet achieved a sound acquisition state consistent
with the intent of the SA-CMM. If the District is to instill the
needed discipline into its systems acquisition processes consistent
with the intent of SA-CMM, it will need to effectively implement all
of our recommendations.
We are sending copies of this report to the Ranking Minority Member
of your Subcommittee and to the Chairmen and Ranking Minority Members
of the Subcommittee on Oversight of Government Management,
Restructuring, and the District of Columbia, Senate Committee on
Governmental Affairs, the Subcommittee on the District of Columbia,
Senate Committee on Appropriations, and the Subcommittee on the
District of Columbia, House Committee on Government Reform and
Oversight. We are also sending copies to the Director of the Office
of Management and Budget, the Chairman of the District of Columbia
Financial Responsibility and Management Assistance Authority, and the
Chief Financial Officer of the District of Columbia. Copies will be
made available to others upon request. If you have questions or wish
to discuss the issues in this report, please contact me at (202)
512-6412. Major contributors to this report are listed in appendix
III.
Sincerely yours,
Dr. Rona B. Stillman
Chief Scientist for
Computers and Telecommunications
(See figure in printed edition.)Appendix I
COMMENTS FROM THE CHAIRMAN,
DISTRICT OF COLUMBIA FINANCIAL
RESPONSIBILITY AND MANAGEMENT
ASSISTANCE AUTHORITY
============================================================== Letter
(See figure in printed edition.)
(See figure in printed edition.)Appendix II
COMMENTS FROM THE DISTRICT OF
COLUMBIA CHIEF FINANCIAL OFFICER
============================================================== Letter
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
MAJOR CONTRIBUTORS TO THIS REPORT
========================================================= Appendix III
ACCOUNTING AND INFORMATION
MANAGEMENT DIVISION, WASHINGTON,
D.C.
Keith Rhodes, Technical Director
Nabajyoti Barkakati, Assistant Director
Hodge Herry, Assistant Director
John C. Martin, Assistant Director
Prithviraj Mukherji, Assistant Director
Madhav Panwar, Assistant Director
Norma Samuel, Project Manager
OFFICE OF GENERAL COUNSEL
Richard Cambosos, Senior Attorney
*** End of document. ***