DC Courts: Disciplined Processes Critical to Successful System	 
Acquisition (28-FEB-02, GAO-02-316).				 
                                                                 
The District of Columbia Courts (DC Courts) is acquiring the	 
Integrated Justice Information System (IJIS) to replace many	 
nonintegrated systems. This system is expected to address current
deficiencies and provide the courts with necessary information	 
critical to its mission. DC Courts has not yet implemented the	 
disciplined processes necessary to reduce the risks associated	 
with acquiring and managing the IJIS acquisition effort at	 
acceptable levels within established resources and schedule. Most
of the DC Courts' requirements, developed in the draft request	 
for proposal, lacked the specificity needed to ensure that	 
requirements had been reduced to acceptable levels and the system
would meet users' needs. DC Courts officials want to use the	 
acquisition process to identify the cost, schedule, and 	 
performance gaps associated with their effort. DC Courts	 
officials acknowledge that this approach increases risk; however,
they believe that accelerating the implementation of a badly	 
needed system justifies those risks. As with any effort,	 
alternative approaches need to be analyzed.			 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-02-316 					        
    ACCNO:   A02810						        
  TITLE:     DC Courts: Disciplined Processes Critical to Successful  
System Acquisition						 
     DATE:   02/28/2002 
  SUBJECT:   Cost analysis					 
	     Information resources management			 
	     Information systems				 
	     Performance measures				 
	     Systems design					 
	     DOJ Edward Byrne Memorial State and		 
	     Local Law Enforcement Assistance Formula		 
	     Grant Program					 
                                                                 
	     Integrated Justice Information System		 
	     SEI Software Acquisition Capability		 
	     Maturity Model					 
                                                                 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-02-316
     
United States General Accounting Office

GAO  Report  to  the Chairman,  Subcommittee  on the  District of  Columbia,
Committee

on Appropriations, House of Representatives

February 2002

DC COURTS

Disciplined Processes Critical to Successful System Acquisition

                                      a

GAO-02-316

United States General Accounting Office Washington, DC 20548

February 28, 2002

The Honorable Joe Knollenberg
Chairman
Subcommittee on the District of Columbia
Committee on Appropriations
House of Representatives

Dear Mr. Chairman:

This letter responds to your August 20, 2001, request that we perform an
initial review of the District of Columbia Courts' (DC Courts) effort to
acquire a new information system. Faced with a myriad of nonintegrated
systems that do not provide the necessary information to support its
overall mission, the DC Courts is in the process of acquiring a replacement
system called the Integrated Justice Information System (IJIS). This
system is expected to address the current system's deficiencies and
provide DC Courts with the information it needs to perform its mission.
The District of Columbia Appropriation Act, 2001,1 provides that none of

the funds in the act or in any other act shall be available for the
purchases,
installations, or operation of IJIS until a detailed plan and design has
been
submitted by DC Courts and approved by the House and Senate
committees on appropriations. DC Courts is in the initial stages of the
system acquisition effort-developing a request for proposal (RFP) to
solicit bids from vendors for the design and implementation of the new
system.2 As required, this detailed plan and design, which includes a draft

RFP, was submitted to the appropriations committees on May 17, 2001, for
review.

Specifically, we assessed whether

1. DC Courts has implemented the disciplined processes for this project to
reduce the risk associated with this effort to acceptable levels;

1Public Law No. 106-522, 114 Stat. 2440, 2442 (2000).

2An  RFP is a formal  request for vendors to provide  a solution to a stated
problem, in this case,  a system to handle DC Courts' management information
needs.

Results in Brief

2. DC Courts' requirements to acquire the system, as identified in its draft
RFP, contain the necessary specificity to reduce the risks from requirement
defects to acceptable levels;3 and

3. DC Courts has performed the necessary actions to determine that a
commercial off-the-shelf system (COTS) would meet its needs.

DC Courts has not yet implemented the disciplined processes necessary to
reduce the risks associated with acquiring and managing the IJIS acquisition
effort to acceptable levels. A disciplined software development and
acquisition process maximizes the likelihood of achieving the intended
results (performance) within established resources (costs) on schedule. DC
Courts officials acknowledged that they do not yet have the disciplined
processes in place to reduce the risks from this effort to acceptable
levels. However, they also acknowledged that disciplined processes were
necessary. They further noted that even though sufficient funding to fully
implement disciplined processes would not be available until the system was
approved and funded, they had already begun to implement some elements of a
disciplined process. For example, at the time of our review, DC Courts was
sending several people to be trained and certified in project management
skills.

The majority of the DC Courts' requirements, developed for the draft RFP,
lacked the necessary specificity to ensure that the defects in these
requirements have been reduced to acceptable levels and that the system
would meet its users' needs. In addition, the requirements in the draft RFP
did not directly relate to industry standards and the terms "customization"
and "modification" were not clearly defined in the draft RFP. We also noted
that the system requirements were not logically grouped.

DC Courts officials are electing to use the acquisition process to identify
the cost, schedule, and performance gaps associated with their effort. DC
Courts officials acknowledged that this approach generally increases risk;
however, they concluded that the benefit to be obtained-accelerating the
implementation of a badly needed system-justifies those risks. At this point
in the process, we concur with DC Courts' decision to use the

3Although all projects of this size can be expected to have some
requirements-related defects, the goal is to reduce the number of such
defects so that they do not significantly affect cost, schedule, or
performance.

acquisition process to identify the cost, schedule, and performance gaps in
this case because of the following mitigating factors.

* DC Courts officials concluded, based on a qualitative analysis, that they
do not have unique requirements that would prevent the utilization of a COTS
product. Furthermore, they recognized that a gap analysis must be performed
as part of the vendor selection process to identify the cost, schedule, and
performance impacts of each vendor's product before deciding which product
to acquire.

* DC Courts is still in the early stages of the system acquisition effort
and has not yet reached the critical point of contract award. Therefore, a
gap analysis can still be performed prior to the system acquisition to
ensure that a COTS product can cost effectively meet DC Courts' needs.

* The requirements that had been developed lacked the necessary specificity
to perform a meaningful gap analysis. Therefore, the time spent on
attempting to perform the gap analysis before issuing the RFP may not have
been effective.

As with any effort, alternative approaches need to be analyzed. In this
case, DC Courts officials believe that the benefits associated with the
reduced risks of performing a detailed gap analysis before the RFP is
issued-having a greater assurance that a COTS product would meet their
needs-were outweighed by the costs associated with delaying this effort.

During the course of our work, DC Courts officials stated their commitment
to go forward with this project only after the necessary actions have been
taken to reduce the risks to acceptable levels. We are making
recommendations to help ensure that DC Courts adequately (1) adopts and
implements disciplined processes to help ensure that its systems development
effort is successful, (2) defines the requirements necessary to develop its
new system in its RFP, and (3) improves its ability to assess potential
solutions. In commenting on a draft of our report, DC Courts agreed with our
findings and recommendations and said that it has begun to implement our
recommendations to ensure the successful acquisition of IJIS as outlined in
our report.

The  District   of  Columbia  Court  Reform   and  Criminal  Procedure  Act
of Background  19704  (Court Reform  Act) transferred jurisdiction  over all
local judicial

4Public Law No. 91-358, 84 Stat. 473 (1970).

matters to a unified court system for the District. This entity, known as DC
Courts, includes

* the Superior Court, which is the trial court with general jurisdiction
over virtually all local legal matters, including criminal, civil, juvenile,
domestic relations, probate, and small claims cases, and

* the Court of Appeals, the highest court of the District of Columbia, which
reviews all appeals from the Superior Court, as well as decisions and orders
of D.C. government administrative agencies.

The Court Reform Act provided for the creation of a policy-making body for
DC Courts, the Joint Committee on Judicial Administration. The joint
committee, composed of the two chief judges and three associate judges,
submits DC Courts' annual budget requests and is responsible for DC Courts'
general personnel policies, accounting and auditing, procurement and
disbursement, development and coordination of statistical and management
information systems and reports, and other related administrative matters.
The joint committee appoints the executive officer, who manages the
day-to-day administrative and financial management of the court system on
the committee's behalf.

The National Capital Revitalization and Self-Government Improvement Act

of 19975 (Revitalization Act) changed DC Courts' funding process,

nonjudicial employee compensation, and functional responsibilities. The

Revitalization Act provides for direct federal funding of DC Courts. The

joint committee submits DC Courts' budget request to the Congress

through the director of the Office of Management and Budget (OMB). In

addition, some DC Courts activities were transferred to the federal

government.

DC Courts' Effort to Acquire New Information System

In October 1998, the Superior Court of the District of Columbia launched the
IJIS project to upgrade and enhance its information management capabilities
and establish a unified, fully integrated computer system that would support
data collection and exchange for all types of cases processed within the
Superior Court. IJIS is a multiyear initiative with a two-fold purpose:
first, to improve data collection and exchange within and across DC Courts'
multiple divisions, which process different types of cases and provide
essential support services, such as research and development and information
technology, and second, to improve

5Public Law No. 105-33, Title XI, 111 Stat. 251, 712 (1997).

interagency data collection and exchange among the District's criminal
justice agencies. Currently, DC Courts information management resources are
divided among 18 different computer systems that have evolved over the past
20 years in response to the differing needs of particular court divisions.
DC Courts' IJIS initiative is part of a District-wide effort to improve the
data collections systems and infrastructure of District criminal justice
agencies and enhance data exchange among those agencies.

The District of Columbia Appropriations Act, 2000,6 provided that, of the
amounts available for operations of DC Courts, an amount not to exceed $2.5
million would be used for the design of IJIS. Due to the time needed

to prepare a detailed requirement analysis to guide the system design, no
contract was awarded or funds spent during fiscal year 2000. As a result, DC
Courts officials requested the $2.5 million to design the new system in

the fiscal year 2001 capital budget.

DC Courts has financed an IJIS requirements analysis through a $350,000
grant from the U.S. Department of Justice's Edward Byrne Memorial State and
Local Law Enforcement Assistance Formula Grant Program through a subgrant of
the D.C. Office of Justice Grants Administration. DC Courts subsequently
awarded a contract to conduct the IJIS requirements analysis, with the
following objectives: (1) assess the DC Courts' existing technology
infrastructure and systems and core business processes supported by these
systems, (2) determine critical information management needed, and (3)
recommend technical and business process solutions that would cost
effectively meet these needs. In September 2000, the contractor issued the
final report on the requirements analysis, which was conducted from January
through August 2000. The IJIS plan and design prepared by DC Courts is based
on the requirements analysis conducted by the contractor.

DC Courts provided its detailed plan and design for the IJIS to the chairmen
of the Senate and House appropriations committees on May 17, 2001. On August
20, 2001, we were asked to review the DC Courts' draft RFP for the design
and implementation of the new system, in conjunction with the subcommittee's
review of the plan and design, before DC Courts began the formal
solicitation process.

6Public Law No. 106-113, 113 Stat. 1501, 1502 (1999).

Scope and Methodology

To carry out our objectives, we reviewed and analyzed

* DC Courts' plan and design for the IJIS, including the draft RFP for the
design and implementation of the new system;

* DC Courts' annual report;

* industry automation standards published by the National Consortium for
Court Functional Standards;7 and

* reports produced by the contractor related to (1) DC Courts' Integrated
Justice Information System Requirements Analysis, (2) Business Process
Descriptions, Data Flow Diagrams and Entity Relationship Diagrams, (3)
Inventory of Data Elements, and (4) Executive Summary (September 2000).

We discussed the IJIS project with the following:

* the chief judge, Superior Court of the District of Columbia;

* a member of the Technology and Automation Committee;

* the executive director, DC Courts;

* the clerk of the court;

* the director of Information Technology, Information Technology (IT)
Division, Court System;

* the information system administrator, IT Division, Court System; * other
DC Courts personnel; and

* representatives from the contractor who prepared the requirements.

We reviewed and analyzed the draft detailed requirements prepared by the

contractor and DC Courts personnel and compared them to industry

standards on selected court activities, such as domestic relations and civil

cases. We also reviewed the Clinger-Cohen Act of 1996;8 federal policy

governing acquisition efforts, including OMB guidance; and guidance and

7The consortium is a subgroup of the Conference of State Court
Administrators and the National Association for Court Management Joint
Technology Committee. Its goal is to develop functional standards for case
management information systems for civil, domestic relations, criminal,
juvenile, probate, and traffic cases.

8Public Law 104-106. The Clinger-Cohen Act requires agencies to analyze
their missions and, based on the analysis, revise mission-related and
administrative processes, as appropriate, before making significant
investments in information technology used to support those missions.

best practice literature9 that the Software Engineering Institute (SEI),10
the Project Management Institute (PMI),11 and the Institute of Electrical
and Electronics Engineers (IEEE)12 have issued on evaluating information
technology investment.

We did not independently verify or audit the cost data we obtained from DC
Courts officials.

Our work was conducted from October 2001 through November 2001 in accordance
with U.S. generally accepted government auditing standards. We requested
comments on a draft of this report from the Joint Committee on Judicial
Administration in the District of Columbia. The joint committee provided us
with written comments that are discussed in the "DC Courts Comments" section
and are reprinted in appendix I.

9U.S. General Accounting Office, Assessing Risks and Returns: A Guide for
Evaluating Federal Agencies' IT Investment Decision-Making, GAO/AIMD-10.1.13
(Washington, D.C., 1997); U.S. General Accounting Office, Executive Guide:
Creating Value Through World-class Financial Management, GAO/AIMD-00-134
(Washington, D.C.: 2000); and U.S. General Accounting Office, Information
Technology: An Audit Guide for Assessing Acquisition Risks, GAO/IMTEC-8.1.4
(Washington, D.C.: 1992).

10SEI is recognized for its experience in software development and
acquisition processes. It has also developed methods and models that can be
used to define disciplined processes and determine whether an organization
has implemented them. SEI's stated mission is to provide leadership in
advancing the state of the practice of software engineering and to improve
the quality of systems that depend on software.

11PMI provides global leadership in the development of standards for the
practice of the project management profession throughout the world. PMI's
standards document, A Guide to the Project Management Body of Knowledge
(PMBOKï¿½ Guide), is a globally recognized standard for managing projects in
today's marketplace. The PMBOKï¿½ Guide is approved as an American National
Standard by the American National Standards Institute.

12Institute of Electrical and Electronics Engineers, Transactions on
Software Engineering (IEEE Transactions on Software Engineering, volume 14,
number 10 1988).

DC Courts Has Not Implemented the Disciplined Processes Necessary to Reduce
Risks to Acceptable Levels

DC Courts has not implemented the disciplined processes necessary to reduce
risks associated with managing its system acquisition to acceptable levels.
However, DC Courts recognizes the importance of these processes and plans to
implement them once funding for the project is available.

Disciplined processes have been shown to reduce the risks associated with
software development and acquisition efforts to acceptable levels and are
fundamental to successful systems acquisition. A disciplined software
development and acquisition process is needed to maximize the likelihood of
achieving the intended results (performance) within established resources
(costs) on schedule. Although a "standard cookbook" of practices that will
guarantee success does not exist, several organizations, such as SEI, PMI,
and IEEE, and individual experts have identified and developed the types of
policies, procedures, and practices that have been demonstrated to reduce
development time and enhance effectiveness.

SEI and others have documented the positive effects of disciplined processes
and have developed methodologies that can be used to determine whether an
organization has such processes. For example, SEI first developed the
Capability Maturity Model (CMM) to determine an organization's ability to
develop software and has also developed a CMM to assess an organization's
ability to acquire software. These methodologies are designed to determine
whether an organization has implemented the types of disciplined processes
that can lead to lower defect rates and help avoid the adverse impacts
associated with common mistakes. Organizations that have focused on
improving their processes have found, over time, that they are able to
reduce their time to market by about one-half and reduce the costs from
defects by factors of 3 to 10.13

The key to having a disciplined system development effort is to have
disciplined processes in multiple areas. For projects such as the one being
undertaken by the DC Courts, these include, at this stage in the acquisition
process, project planning, requirements management, project management,
configuration management, risk management, and testing. Effective processes
should be implemented in each of these areas throughout the project life
cycle since constant changes occur. Table 1

13 Steve  McConnell,   Rapid Development:  Taming  Wild  Software Schedules
(Redmond, Wash.: Microsoft Press, 1996).

  provides a brief description of some of the areas that appear critical to
                                    the

14

DC Courts' effort.

                 Table 1: Examples of Disciplined Processes

Discipline Description

Project planning Project planning is the process used to establish
reasonable plans for performing and managing the software project. This
includes (1) developing estimates of the resources needed for the work to be
performed, (2) establishing the necessary commitments, and (3) defining the
plan necessary to perform the work. Effective planning is needed to identify
and resolve problems as soon as possible when it is the cheapest to fix
them.

a

According to one author, the average system design and implementation
project includes about 80 percent of its time as unplanned rework-fixing
mistakes that were made earlier in the project.

Risk management Risk management is a set of continual activities for
identifying, analyzing, planning, tracking, and controlling risks. Risk
management starts with identifying the risks before they become problems. If
this step is not performed well, the entire risk management process may
become a useless exercise since a process cannot be managed without
information about that process. As with the other disciplined processes,
risk management is designed to eliminate the effects of undesirable events
at the earliest possible stage to avoid the costly consequences of rework.

After the risks are identified, they need to be analyzed so that they can be
better understood and decisions can be made about what actions, if any, will
be taken to address the risks. Basically, this step includes such activities
as evaluating the impact on the project if a risk does occur, determining
the probability of the event occurring, grouping like risks together, and
prioritizing the risk against the other risks. Once the risks are analyzed,
a risk management plan is developed that outlines the information known
about the risks and the actions, if any, that will be taken to mitigate
those risks. Risk management is a continual process because the risks and
actions planned to address those risks need to be monitored to ensure that
the risks are being properly controlled and that new risks are identified as
early as possible. If the actions envisioned in the plan are not adequate,
then additional controls are needed to correct the deficiencies identified.

Testing Testing is the process of executing a program with the intent of
finding errors.b Disciplined system development activities recognize that
testing will not find all defects even though well-designed and executed
testing programs are used. For example, testing performed through the system
testing phase often catches less than

c

60 percent of a program's defects. Consequently, testing alone cannot be
relied on to identify all defects.

aSteve McConnell, Software Project Survival Guide (Redmond, Wash.: Microsoft
Press, 1998).

bGlendford J. Myers, The Art of Software Testing (New York, N.Y.: John Wiley
and Sons, Inc., 1979).

cSee footnote 13.

DC Courts officials agreed that they had not yet fully implemented the
disciplined processes necessary to reduce the risks associated with this
project to acceptable levels. They said that they recognized that the
implementation of the disciplined processes was needed not only for this
project but for other projects as well, and that effective implementation
would take time, management commitment, and funding. They also noted that
they had begun the process of identifying the needed funding and

14A list of other processes necessary to acquire or develop systems can be
obtained from SEI at www.sei.cmu.edu.

DC Courts System Requirements as Specified in RFP Are Inadequate

staffing levels and that they were sending several of the project managers
to training so that they could become certified project managers, which will
facilitate their knowledge of disciplined processes.

Requirements represent the blueprint that system developers and program
managers use to design, develop, and acquire a system. Requirements should
be consistent with one another, verifiable, and directly traceable to
higher-level business or functional requirements. It is critical that
requirements be carefully defined and that they flow directly from the
organization's concept of operations (how the organization's day-to-day
operations are or will be carried out to meet mission needs). Improperly
defined or incomplete requirements have been commonly identified as a root
cause of system failure and systems that do not meet their costs, schedules,
or performance goals. Without adequately defined requirements that have been
properly reviewed and validated, significant risk exists that the system
will need extensive and costly changes before it will meet the DC Courts'
needs.

DC Court system requirements set forth in the draft RFP lacked the necessary
specificity to ensure that the defects in these requirements have been
reduced to acceptable levels and that the system would meet its users'
needs. In addition, the terms "customization" and "modification," though
used in the RFP, were not clearly defined. Also, industry standards that
appeared to be related to the DC Courts were either not used or were
summarized so that the detail provided in the standard was omitted from the
DC Courts' requirements. We also noted that the requirements were not
logically organized-related requirements were not grouped together. Better
grouping of requirements would help the DC Courts and potential contractors
in assessing whether the RFP's requirements adequately describe the
functionality necessary to conduct court business. Because DC Courts'
requirements were not specific, were poorly organized, and did not directly
relate to industry standards, a significant potential exists that the DC
Courts' proposed system may not meet its business needs or may not be
delivered on schedule and within budget if corrective action is not taken.

Requirements Management: A Key Process for Quantifying a System's Purpose
and Function

Requirements management is a process that establishes a common

understanding between the customer and the software project manager

regarding the customer's business needs that will be addressed by a

project.15 A critical part of this process is to ensure that the
requirements

development portion of the effort documents, at a sufficient level of
detail,

the problems that need to be solved and the objectives that need to be

achieved.

Good requirements have several common characteristics:16

* They fully describe the software functionality to be delivered.
Functionality is a defined objective or characteristic action of a system or
component. For example, a system may have inventory control as its primary
functionality.17

* They provide the source of the requirement. For instance, the citation of
the statute, regulation, or industry standard should be shown so that others
can evaluate the applicability of the requirement and better understand the
impacts of changes in the requirement.

* They state the requirement in clear terms that allow for quantitative
evaluation. Specifically, all readers of a requirement should arrive at a
single, consistent interpretation of it.

Studies have shown that problems associated with requirements definition are
key factors in software projects that do not meet their cost, schedule, and
performance goals. For example, see the following:

* A study found that getting a requirement right in the first place costs 50
to 200 times less than waiting until after the system is implemented to get
it right.18

15Carnegie Mellon University Software Engineering Institute, The Capability
Maturity Model: Guidelines for Improving the Software Process (Reading,
Mass.: Addison Wesley Longman, Inc., 1995).

16Karl E. Wiegers, Software Requirement: Practical techniques for gathering
and managing requirements throughout the product development cycle (Redmond,
Wash.: Microsoft Press, 1999).

17IEEE 610.12-1990.

18Barry W. Boehm and Philip N. Papaccio, Understanding and Controlling
Software Costs (IEEE Transactions on Software Engineering, volume 14, number
10 1988).

* A survey of more than 8,000 software projects found that the top three
reasons that projects were delivered late, over budget, and with less
functionality than desired all had to do with requirements management.19

* A study found that the average project experiences about a 25-percent
increase in requirements over its lifetime, which translates into at least a
25-percent increase in the schedule.20

* A study noted that between 40 and 60 percent of all defects found in a

Requirements Were Not Specific

software project could be traced back to errors made during the requirements
development stage.21

The majority of the requirements in the draft RFP lacked the necessary
specificity to ensure that the defects in the requirements had been reduced
to acceptable levels. Acceptable levels refer to the fact that any systems
acquisition effort, such as that being undertaken by the DC Courts, will
have some requirements-related defects. However, the goal is to reduce the
risks and prevent significant requirements defects in order to limit the
negative impact of these defects on the cost, timeliness, and performance of
the project. The lack of specificity of the requirements contained in the
draft RFP indicate that a significant number of requirements-related defects
are present in this document. These requirements-related defects
significantly increase the likelihood that the resulting system will not
meet DC Courts' cost, schedule, and performance goals. In addition, the risk
exists that the DC Courts may have failed to include critical requirements
because of the lack of specificity. Omission of critical requirements will
also adversely affect cost, schedule, or performance goals. As noted
elsewhere, DC Courts officials have agreed with our assessment and have
started taking actions to address these weaknesses.

DC Courts' requirements lacked the specific information necessary to
understand the required functionality that a vendor should provide and how
to determine quantitatively, through testing or other analysis, whether a
vendor's product would adequately address the DC Courts' needs. For
examples, see the following.

19The Standish Group, Charting the Seas of Information Technology (Dennis,
Mass.: The Standish Group, 1994).

20Capers Jones, Assessment and Control of Software Risks (Englewood Cliffs,
N.J.: Yourdon Press, 1994).

21Dean Leffingwell, Calculating the Return on Investment from More Effective
Requirements Management (American Programmer, 1997).

* One requirement stated that the "[s]ystem must accept e-filing documents
and case management data in XML22 standard format and then update the case
management system with the case filing data, case filing fees, and store the
document." Deficiencies in this requirement include (1) the XML "standard"
format is not defined, (2) the business rules that should be used to compute
the filing fees are not referenced, and (3) the response time and capacity
(for example, number and size of documents) was not specified.

* Another requirement stated that the system must have the "[a]bility to
link

[a] case with other related cases having the same case participants or
family unit, because the overall concept in the family court is one family,
one judge." However, the document did not specify such items as (1) the
individuals that should be considered part of the family unit (for example,
mother, father, stepmother, aunt, sister, nephew, guardian, etc.) or (2)
what are considered related cases (for example, criminal, civil, probate,
etc.).

* A user requirement stated that the system must have the "[a]bility to move
quickly between screens. Must have the ability to change screen navigation
per each individual user requirement." However, the term "quickly" was not
defined nor was there a description of how users should define the screen
navigation.

* Another requirement stated that the system must have the "[a]bility to
electronically notify users based on user defined rules when [a] user
defined event has or has not occurred in a timely manner by either FAX or
e-mail." Ambiguities in this requirement include (1) the rules that users
can define, (2) the functional method that a user applies to define a rule,
(3) whether the rules are established by each user or whether "corporate"
rules will be defined, (4) what is considered "timely," and (5) who are
considered users (that is, the reason why DC Courts personnel receive faxes
instead of e-mail from their own system).

Another important aspect of requirements definition includes defining key
terms. In its RFP, the DC Courts did not clearly define the terms
"customization" and "modification," nor did it require that the vendor
communicate the cost, schedule, and performance impacts of implementing a
customization or modification associated with a given requirement.

22The Extensible Markup Language (XML) is a flexible, nonproprietary set of
rules for tagging information so that it can be transmitted using Internet
protocols and processed by disparate computer systems.

The terms "customization" and "modification" may be confusing since a basic
premise is that COTS solutions should not be customized or modified. The
terms "customization" and "modification," as used in this report, are
differentiated from the basic installation processes that are required for
systems such as IJIS. Those processes are commonly referred to as
installation and configuration. For purposes of this report, we are defining
customization and modification as follows:

* Customization: The process of setting parameters within the application to
make it operate in accordance with the entity's business rules.
Customizations are normally supported by the vendor in subsequent upgrades
as part of the normal upgrade process.

* Modification: The process of writing or changing code. Modifications are
not generally supported by the vendor in subsequent upgrades as part of the
normal upgrade process.

A customization in one package may be considered a modification in another
package. For example, the architecture of one product may allow the entity
to develop a report tailored to its needs in such a manner that, when the
system is upgraded, the report will still be available without additional
development work. However, the same report in another system may need to be
rewritten when the system is upgraded. Defining these terms is important to
ensure that the benefits associated with any customizations or modifications
are cost-effective and that alternatives are not available.

Unclear Relationship to Industry Standards

In a number of cases the requirements in the DC Courts' draft RFP did not
directly relate to industry standards published by the consortium. The
consortium has been tasked with developing guidelines that will help state
courts more effectively use their financial and staffing resources to obtain
state-of-the-art computer systems-either through in-house development or
through procurement from software developers. In doing so, the consortium
has focused on ways in which the state courts can

* reduce the time needed to obtain a new computer system,

* improve work processes, and

* reduce staffing requirements.

Recognizing the state courts' need for functional standards in computer
systems and the staffing limitations that exist in most state courts, the
consortium has developed a set of functional standards for developing new
computer systems. Courts nationwide would use these standards to define
functional requirements for in-house systems development and

RFPs for vendor-supplied systems. Each court must customize the standards
and add terminology, details, and specificity based on local and state
procedures, policies, and customs.

DC Courts officials and the contractor told us that the requirements in the
DC Courts' draft RFP were based on the published standards developed by or
currently under development by the consortium. However, for many of the
requirements in the draft RFP, there was no direct link, or
cross-referencing, to the standards. For example, one standard calls for
generating a receipt and assigning a case number when a case file is
received, but the draft RFP requirement does not track back to the industry
standard and only speaks vaguely of notifying users and generating receipts.
Table 2 compares one of the standards to some of the draft RFP's
requirements.

Table 2: Comparison of Industry Standard and RFP's Requirement

Industry standard Draft RFP requirement

Generate receipt for or notify appropriate parties that case filing was
received and accepted, and give them assigned case number (notice, including
electronic acknowledgment, would apply primarily when a case is transferred
from another jurisdiction or filed electronically).

Ability to electronically notify users based on user-defined rules when user
defined event has or has not occurred in a timely manner by either FAX or
e-mail.

Ability to automatically generate a receipt upon the filing of a notice of
appeal stating the case number, date, and time of filing. System should not
allow a case to proceed until filing fee is collected.

Ability to issue a sequentially numbered receipt for the payment of funds on
a specific case.

We were also unable to identify in the draft RFP items contained in the
industry standards that applied to the DC Courts. For example, one standard
required the system to "[g]enerate locally defined case title or style (
i.e., a short phrase that identifies case and includes plaintiff and
defendant names) from party names and other information." We were not able
to readily identify a related requirement in the draft RFP.

Regardless of whether the draft RFP contained all the industry standards
that were applicable, the real key is that the draft RFP did not provide
adequate information so that the prospective vendors and others could
readily map systems built upon these standards to the needs of the DC
Courts. This lack of vital information could lead to adverse impacts as

vendors attempt to rework or develop functions that were not clearly
understood initially. This, in turn, could lead to cost increases, schedule
delays, and reduced performance.

Poor Organization Prevents Requirements Validation

Requirements should be organized logically to facilitate understanding and
requirements validation efforts. The organization of the requirements in the
draft RFP hampers the DC Courts' and potential contractors' ability to
understand and validate the requirements since it is very difficult to
determine where all the requirements related to a given functionality are
documented. Because the requirements were not logically organized, future
validation efforts to identify missing or duplicate requirements may not be
effective.

Requirements validation reduces requirements-related defects by first
assessing whether the requirement document actually satisfies the project's
top-level requirements. The verification process includes ensuring that the

* requirements document describes the intended system behaviors and
characteristics;

* requirements are correctly derived from requirements obtained from other
sources, for example, if the requirements document states that a given
statute requires a certain function, then an effort is undertaken to ensure
that the requirement correctly represents the specified statute;

* requirements are complete, consistent, and of high quality; and

* requirements provide an adequate basis to proceed to the next stage of
system development.

Requirements validation is intended to help ensure that the requirements are
complete, correct, feasible, necessary, prioritized, unambiguous, and
verifiable.23 The organization of the requirements in the draft RFP will
likely hinder such a process. For example, see the following:

* One section, entitled "Case Financial Activity Requirements," contained
many of the financial-related requirements such as preparing a monthly trial
balance and balance sheet. However, another section, entitled "Reporting,"
contained the requirement to produce a Statement of Revenue and Expenses.
Generally, all requirements related to financial statement reporting would
be contained in the same section.

23Karl E. Wiegers.

* The requirements contained in the "Case Financial Activity Requirements"
section were not organized in a functional manner. Disbursement requirements
were followed by adjusting entry requirements, which were followed by
additional disbursement requirements, which were followed by receipt
requirements, which were followed by additional disbursement requirements.
In an appropriately organized requirements document, all related
requirements would be grouped together.

* Bar coding requirements for court documents were contained in at least
three different sections rather than all in the same section.

DC Courts Has a The DC Courts intends to use the acquisition process to
identify any potential gaps between its system requirements and the standard
features Reasonable Basis to available in a given vendor's product instead
of performing a detailed gap

analysis before the RFP is issued. Although this approach can increaseAssume
a Commercial risk, we believe the DC Courts' decision to use the acquisition
process to Product Will Meet Its identify the cost, schedule, and
performance gaps is acceptable in this Needs case because of the following
mitigating factors.

* DC Courts officials concluded, based on a qualitative analysis, that they
do not have unique requirements that would prevent the utilization of a COTS
product. Furthermore, they recognized that a gap analysis must be performed
as part of the vendor selection process to identify the cost, schedule, and
performance impacts of each vendor's product before deciding which product
to acquire.

* DC Courts is still in the early stages of the system acquisition effort
and has not yet reached the critical point of contract award. Therefore, a
gap analysis can still be performed prior to the system acquisition to
ensure that a COTS product can cost effectively meet DC Courts' needs.

* As noted previously, the requirements that had been developed lacked the
necessary specificity to perform a meaningful gap analysis. Therefore, the
time spent on attempting to perform the gap analysis before issuing the RFP
may not have been effective.

As with any effort, alternative approaches need to be analyzed. In this
case, DC Courts officials believe that the benefits associated with the
reduced risks of performing a detailed gap analysis before the RFP is
issued-having a greater assurance that a COTS product would meet their
needs-were outweighed by the costs associated with delaying this effort.

A critical step in acquiring a new system is determining whether the system
requirements can be met by using commercially available systems, commonly
referred to as COTS systems. If COTS systems cannot meet the

majority of the requirements, then the acquirer will need to undertake a
system development effort rather than using a COTS product. To make this
determination, the agency must perform a gap analysis that systematically
and quantitatively compares and contrasts the vendors' products against the
agency's requirements based on functional, technical, and cost differences.

Two different approaches can be taken to perform this gap analysis. One
approach is to perform a gap analysis on the detailed requirements to
determine whether they need to be modified before issuing the RFP for
acquiring a system. A second approach is to structure the acquisition
process so that the vendor identifies the gaps as well as the cost,
schedule, and performance impacts of addressing those gaps. Each approach
has its advantages and disadvantages. For example, if a detailed gap
analysis is performed before the acquisition process begins, and it is later
determined that a COTS product could have cost effectively met the agency's
needs, then time and money have been wasted performing an analysis that, in
effect, will be repeated during the acquisition process. On the other hand,
if the second approach is taken and the gap analysis discloses that a COTS
product cannot cost effectively meet the agency's needs, then the
acquisition process will need to be restructured to support a system
development effort, which translates into schedule delays and additional
costs.

DC Courts officials selected the second approach-using the acquisition
process to identify their project's cost, schedule, and performance gaps.
According to these officials, they selected this approach because they had

* reviewed a number of vendors' products and believed they had a good
understanding of the general capabilities of the marketplace;

* hired a contractor that was knowledgeable of the court environment to help
them understand whether a COTS product would meet their needs;

* determined that, based on interactions with other courts of similar size
and workload, the DC Courts can successfully use comparable practices; and

* already committed to modifying their business processes to reflect the
selected COTS product's capabilities unless a significant reason, such as a
legal requirement, dictated otherwise.

In discussing this issue with DC Courts officials, they acknowledged that
the approach they were taking increased risks; however, they believe that
the benefit obtained-accelerating the implementation of a badly needed
system-justifies those risks.

Actions Taken by DC Courts to Address Our Concerns

*

*

Conclusions

Recommendations

In late October 2001, we discussed our findings on the above three areas
with DC Courts officials. They generally agreed with our findings and
restated their commitment to only go forward with this project when the
necessary actions had been taken to reduce the risks to acceptable levels.
They specifically agreed to perform the following actions:

Delay the procurement of the new system until the requirements can be
revised to provide assurance that defects related to the requirements are
kept to acceptable levels. They have begun the process of selecting a
contractor that will be responsible for developing a new requirements
document. They also stated that the contractor would be responsible for
documenting the requirements so that each requirement (1) fully describes
the system functionality to be delivered, (2) includes the source of the
requirement, and (3) is stated in unambiguous terms that allow for
quantitative evaluation. Develop a plan that can be used to guide DC Courts'
effort to implement the necessary disciplined processes. This includes
identifying the needed skills, determining the best approach to acquiring
the needed skills through training of DC Courts' staff or through
contractors, and securing adequate funding.

DC Courts' planned actions and those taken thus far to address our findings
are a positive step forward and, if effectively implemented, will help
reduce the risks associated with this effort to acceptable levels. We are
reaffirming these actions in our recommendations. Although the DC Courts has
a reasonable basis for believing that a COTS product will meet its needs, it
has not yet defined the requirements adequately or implemented the
disciplined processes necessary to reduce the risks to the project to
acceptable levels-that is, to reduce the risks associated with disciplined
processes and prevent significant requirements-related defects in order to
limit the negative impact on the cost, timeliness, and performance of the
project. DC Courts has stated its commitment to correcting the deficiencies
in its requirements as well as performing a gap analysis during the
preliminary phases of its acquisition project before it commits large
amounts of resources. If properly implemented, these steps should serve to
reduce overall project risk and reduce the likelihood that extensive and
costly corrections will be needed later.

To help ensure that the DC Courts reduces risks associated with its systems
development and implementation and increase the chances of a successful
effort, we recommend that the Joint Committee on Judicial Administration in
the District of Columbia take the following actions:

* Develop a plan on how it will implement the disciplined processes
necessary to reduce the risks associated with this effort to acceptable
levels. This plan should include the processes, such as those identified by
SEI and IEEE, that will be implemented and the resources, such as staffing
and funding, needed to implement the necessary processes effectively.

* Develop requirements that contain the necessary specificity to reduce
requirements-related defects to acceptable levels and add them to the draft
RFP. The requirements management process used to develop and document the
requirements should be adequate to ensure that each requirement (1) fully
describes the functionality to be delivered, (2) includes the source of the
requirement, and (3) is stated in unambiguous terms that allow for
quantitative evaluation.

* Organize the requirements document to facilitate the requirements
validation process used by disciplined organizations.

* Ensure that the acquisition process is adequate to (1) clearly define the
terms customization and modification and (2) ensure that vendor responses
clearly communicate the cost, schedule, and performance impacts of
implementing a customization or modification associated with a given
requirement.

* Evaluate the cost, schedule, and performance impacts of the

DC Courts Comments

customizations and modifications identified during the system acquisition
process and ensure that the benefits are cost-effective and that
alternatives to customization or modification are not available.

In commenting on a draft of our report, DC Courts agreed with our findings
and recommendations and said that it has begun to implement our
recommendations to ensure the successful acquisition of IJIS as outlined in
our report. Specifically, DC Courts said that it is implementing the
disciplined processes critical to successful systems acquisition. DC Courts
also noted that it has a project under way to institute the necessary
disciplined processes for the entire system development life cycle (SDLC).
DC Courts further noted that it is contracting with subject matter experts
in both the SDLC disciplines and court operations to increase the
specificity in the RFP and to reduce the risks from requirement-related
defects to acceptable levels. DC Courts provided additional details on the
actions taken to address the findings and recommendations included in our
report. If these actions are successfully implemented, they will address our
concerns and help ensure the success of the DC Courts' IJIS acquisition. The
DC Courts' comments are reprinted in appendix I.

We are sending copies of this report to the ranking minority member of your
subcommittee and to other interested congressional committees. We

are also sending copies to the Joint Committee on Judicial Administration
in the District of Columbia, the chief judge of the Superior Court of the
District of Columbia, and the executive director of the District of
Columbia Courts. Copies of this report will also be made available to
others upon request.

Please contact me at (202) 512-9406 or by e-mail at [email protected] if you
or your staff have any questions concerning this report. Key contributors
to this report were Linda Elmore, John C. Martin, Meg Mills, and Norma
Samuel.

Sincerely yours,

Jeanette M. Franzel
Acting Director, Financial Management and Assurance

                        Page 22 GAO-02-316 DC Courts

                        Page 23 GAO-02-316 DC Courts

                        Page 24 GAO-02-316 DC Courts

GAO's Mission

Obtaining Copies of GAO Reports and Testimony

The General Accounting Office, the investigative arm of Congress, exists to
support Congress in meeting its constitutional responsibilities and to help
improve the performance and accountability of the federal government for the
American people. GAO examines the use of public funds; evaluates federal
programs and policies; and provides analyses, recommendations, and other
assistance to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents is through the
Internet. GAO's Web site (www.gao.gov) contains abstracts and full-text
files of current reports and testimony and an expanding archive of older
products. The Web site features a search engine to help you locate documents
using key words and phrases. You can print these documents in their
entirety, including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its Web
site daily. The list contains links to the full-text document files. To have
GAO e-mail this list to you every afternoon, go to www.gao.gov and select
"Subscribe to daily e-mail alert for newly released products" under the GAO
Reports heading.

Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out to
the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:

U.S. General Accounting Office P.O. Box 37050 Washington, D.C. 20013

To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061

Visit GAO's Document GAO Building

Distribution  Center  Room 1100,  700 4th  Street, NW  (corner of 4th  and G
Streets, NW) Washington, D.C. 20013

To Report Fraud, Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm,

Waste, and Abuse in E-mail: [email protected], or

Federal Programs 1-800-424-5454 or (202) 512-7470 (automated answering
system).

Jeff   Nelligan,  Managing   Director,   [email protected]   (202)  512-4800

Public Affairs U.S.  General Accounting Office, 441 G. Street NW, Room 7149,
Washington, D.C. 20548
*** End of document. ***