Customs Service Modernization: Ineffective Software Development Processes
Increase Customs System Development Risks (Chapter Report, 02/11/99,
GAO/AIMD-99-35).
Pursuant to a congressional request, GAO reviewed the Customs Service's
software development maturity and improvement activities, focusing on:
(1) the maturity of Customs' software development processes; and (2)
whether Customs has an effective software process improvement program.
GAO noted that: (1) because of the number and severity of Customs'
software development process weaknesses, Customs did not fully satisfy
any of the key process areas (KPA) necessary to achieve the repeatable
level of process maturity; (2) as a result, its processes for developing
software, a complex and expensive component of Customs' systems, are ad
hoc, sometimes chaotic, and not repeatable across projects; (3) Customs
had some practice strengths in all but one of the five KPAs evaluated
(i.e., requirements management, software project planning, software
project tracking and oversight, software quality assurance, and software
configuration management); however, GAO also found extensive and
significant weaknesses in each of these KPAs; (4) some of these
weaknesses were systemic, recurring in each of the KPAs; (5) for
example, Customs had no written policy for managing or implementing any
of the KPAs; (6) none of the projects had: (a) an approved quality
assurance plan; (b) documented procedures for determining the project
cost, schedule, or effort; or (c) any outside group reviewing or
reporting on the project's compliance with defined processes; (7) these
weaknesses are some of the reasons for Customs' limited success, for
example, in delivering promised Automated Commercial Environment (ACE)
capabilities on time; (8) Customs does not have a software development
process improvement program, and it has not taken the basic steps to
initiate one; (9) these steps, many of which are described in Software
Engineering Institute's initiating, diagnosing, establishing, acting,
and leveraging model for process improvement, include assigning
responsibility and authority for process improvement, establishing a
process improvement management structure, defining a plan of action, and
committing needed resources; and (10) until Customs establishes an
effective process improvement program, its software processes will
remain poorly defined and undisciplined, and its software projects are
likely to suffer cost, schedule, and performance shortfalls.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: AIMD-99-35
TITLE: Customs Service Modernization: Ineffective Software
Development Processes Increase Customs System Development
Risks
DATE: 02/11/99
SUBJECT: Computer software
Information resources management
Customs administration
Computer software verification and validation
Information systems
IDENTIFIER: Customs Service National Customs Automation Program
Customs Service Automated Export System
Customs Service Automated Commercial Environment System
CMM
Software Capability Maturity Model
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to Congressional Requesters
February 1999
CUSTOMS SERVICE MODERNIZATION -
INEFFECTIVE SOFTWARE DEVELOPMENT
PROCESSES INCREASE CUSTOMS SYSTEM
DEVELOPMENT RISKS
GAO/AIMD-99-35
Customs Service Modernization
(511115)
Abbreviations
=============================================================== ABBREV
ACE - Automated Commercial Environment
ACS - Automated Commercial System
AES - Automated Export System
CMM0 - Capability Maturity Model
GAO - General Accounting Office
IDEAL - Initiating, Diagnosing, Establishing, Acting, Leveraging
KPA - key process area
NAFTA - North American Free Trade Agreement
NCAP - National Customs Automation Program
NCAP/P - National Customs Automation Program Prototype
RM - requirements management
SCE - Software Capability Evaluation
SCM - software configuration management
SDLC - Software Development Life Cycle
SEI - Software Engineering Institute
SEPG - Software Engineering Process Group
SPP - software project planning
SPTO - software project tracking and oversight
SQA - software quality assurance
SW-CMM - Software Capability Maturity Model
Letter
=============================================================== LETTER
B-280417
February 11, 1999
The Honorable Ben Nighthorse Campbell
Chairman, Subcommittee on Treasury,
General Government, and Civil Service
Committee on Appropriations
United States Senate
The Honorable Jim Kolbe
Chairman, Subcommittee on Treasury,
Postal, and General Government
Committee on Appropriations
House of Representatives
This report responds to your requests that we review the U. S.
Customs Service's software development maturity and improvement
activities. Customs plans to spend hundreds of millions of dollars
developing and maintaining software-intensive systems. We found that
Customs' software development processes are immature and are making
recommendations to the Commissioner of Customs for strengthening
them.
We are sending copies of this report to the Secretary of the
Treasury; Commissioner of Customs; Director of the Office of
Management and Budget; Ranking Minority Members of the Subcommittee
on Treasury and General Government of the Senate Committee on
Appropriations and the Subcommittee on Treasury, Postal Service, and
General Government of the House Committee on Appropriations; and
other congressional committees. We will also make copies available
to other interested parties upon request. If you have questions or
wish to discuss the issues in this report, please contact me at (202)
512-6240. Major contributors to this report are listed in appendix
II.
Randolph C. Hite
Associate Director, Governmentwide
and Defense Information Systems
EXECUTIVE SUMMARY
============================================================ Chapter 0
PURPOSE
---------------------------------------------------------- Chapter 0:1
The Customs Service relies extensively on software intensive systems
to perform its core missions of enforcing laws governing the flow of
goods and persons across U.S. borders, and assessing and collecting
billions of dollars annually in duties, taxes, and fees on imported
merchandise. Because software is a complex and expensive component
of these systems, Customs must use defined and disciplined processes
if it expects to develop and maintain software effectively.
Recognizing software's importance to Customs' mission effectiveness,
the Chairman, Subcommittee on Treasury and General Government, Senate
Committee on Appropriations, and the Chairman, Subcommittee on
Treasury, Postal Service and General Government, House Committee on
Appropriations, asked GAO to determine (1) the maturity of Customs'
software development processes, and (2) whether Customs has an
effective software process improvement program.
BACKGROUND
---------------------------------------------------------- Chapter 0:2
During fiscal year 1997, Customs collected $22.1 billion in revenue
at more than 300 ports of entry, and it processed nearly 450 million
passengers who entered the U.S. during the year. Each year Customs
also provides trade statistics used in developing trade policy and
negotiating trade agreements with various countries. Customs expects
its workload to burgeon. Accordingly, Customs plans to spend
hundreds of millions of dollars over the next 5 years developing
software for new and existing information systems.
Software quality is governed largely by the quality of the processes
involved in developing or acquiring, and maintaining it. Carnegie
Mellon University's Software Engineering Institute (SEI), recognized
for its expertise in software processes, has developed models and
methods that define and determine organizations' software process
maturity. Together, they provide a logical framework for baselining
an organization's current process capabilities (i.e., strengths and
weaknesses) and providing a structured plan for incremental process
improvement.
Using SEI's Software Capability Maturity Model\SM (SW-CMM0)\1 and the
SEI software capability evaluation method\2 , GAO staff trained at
SEI evaluated Customs' software development/maintenance maturity in
five of the six key process areas (KPA) that are necessary to attain
a "repeatable" level of process maturity.\3 GAO did not evaluate
Customs in the sixth repeatable level KPA, software subcontract
management, because Customs did not use subcontractors on any of the
projects that GAO evaluated.\4 An organization at the repeatable
level of process maturity has the necessary process discipline in
place to repeat earlier successes on projects in similar
applications. The repeatable level of process maturity is the second
level on SEI's five-level scale. Organizations that do not satisfy
the requirements for the "repeatable" level are by default judged to
be at the "initial" level of maturity, meaning that their processes
are ad hoc, sometimes even chaotic, with few of the processes defined
and success dependent mainly on the heroic efforts of individuals.
To aid such organizations in maturing their processes, SEI has also
published a software process improvement model called IDEAL\S \M \5
that defines a systematic, five-phase process improvement approach.\6
As part of its evaluation, GAO examined three software projects.
These were (1) development of the first software release of the
National Customs Automation Program (NCAP 0.1), which is the first
phase of the Automated Commercial Environment (ACE); (2) software
maintenance on the Automated Export System (AES); and (3) software
maintenance on the Administrative Security System.\7
--------------------
\1 Capability Maturity Model\SM is the service mark of Carnegie
Mellon University, and CMM0 is registered in the U.S. Patent and
Trademark Office.
\2 We used the latest version (version 1.1) of the software
development CMM for our evaluation.
\3 The five KPAs are requirements management, software project
planning, software project tracking and oversight, software quality
assurance, and software configuration management. According to the
SW-CMM, requirements management is the process for establishing a
common understanding between the customer and the software developer
of the customer's requirements; software project planning is the
process for establishing reasonable plans for engineering the
software and managing the software project; software project tracking
and oversight is the process of providing adequate visibility into
the software project's progress to permit effective action when
deviations from plans occur; software quality assurance is the
process of verifying for management that software project plans,
standards, and procedures are being followed; and software
configuration management is the process of establishing and
maintaining the integrity of the software products throughout the
project's life cycle.
\4 According to the SW-CMM, software subcontract management is the
process of selecting a software subcontractor, establishing
commitments with the subcontractor, and tracking and reviewing the
subcontractor's performance and results.
\5 IDEAL is a service mark of Carnegie Mellon University.
\6 IDEAL: A User's Guide for Software Process Improvement
(CMU/SEI-96-HB-001).
\7 The Subcommittee Chairmen requested that we evaluate ACE, which is
the largest and most important system that Customs is developing.
The other two projects were selected by Customs on the basis of the
following GAO specified criteria: each project should be managed by
a different software team, at least one project should involve a
legacy system, at least one project should involve Year 2000 software
conversion, and each project should be relatively large and important
to accomplishing Customs' mission.
RESULTS IN BRIEF
---------------------------------------------------------- Chapter 0:3
Because of the number and severity of Customs' software development
process weaknesses, Customs did not fully satisfy any of the key
process areas (KPAs) necessary to achieve the "repeatable" level of
process maturity. As a result, its processes for developing
software, a complex and expensive component of Customs' systems, are
ad hoc, sometimes chaotic, and not repeatable across projects.
Customs had some practice strengths in all but one of the five KPAs
evaluated (i.e., requirements management, software project planning,
software project tracking and oversight, software quality assurance,
and software configuration management); however, GAO also found
extensive and significant weaknesses in each of these KPAs. Some of
these weaknesses were systemic, recurring in each of the KPAs. For
example, Customs had no written policy for managing or implementing
any of the KPAs; and none of the projects had (1) an approved quality
assurance plan; (2) documented procedures for determining the project
cost, schedule, or effort; or (3) any outside group reviewing or
reporting on the project's compliance with defined processes. These
weaknesses are some of the reasons for Customs' limited success, for
example, in delivering promised ACE capabilities on time.
Currently, Customs does not have a software development process
improvement program, and it has not taken the basic steps to initiate
one. These steps, many of which are described in SEI's IDEAL\8 model
for process improvement, include assigning responsibility and
authority for process improvement, establishing a process improvement
management structure, defining a plan of action, and committing
needed resources. Until Customs establishes an effective process
improvement program, its software processes will remain poorly
defined and undisciplined, and its software projects are likely to
suffer cost, schedule, and performance shortfalls.
--------------------
\8 IDEAL stands for the five phases of the model--Initiating,
Diagnosing, Establishing, Acting, and Leveraging.
PRINCIPAL FINDINGS
---------------------------------------------------------- Chapter 0:4
CUSTOMS SOFTWARE DEVELOPMENT
PROCESSES ARE IMMATURE
-------------------------------------------------------- Chapter 0:4.1
GAO evaluated how effectively each of the three Customs software
projects implemented five of the six level 2 KPAs: requirements
management, software project planning, software project tracking and
oversight, software quality assurance, and software configuration
management. To attain a level 2 or repeatable maturity rating,
Customs would have to effectively implement all of the key practices
for all five relevant KPAs.
GAO found that while Customs had some strengths (i.e., practices that
are effectively implemented), it had too many weaknesses (i.e.,
practices that are ineffectively implemented) to satisfy any of the
level 2 KPAs. The strengths and weaknesses for all three projects
are tallied in table 1. In summary, of the total number of KPA
practices rated, 35 percent constituted strengths, 61 percent were
weaknesses, and 4 percent were observations. An observation
indicates that the evidence was inconclusive and did not clearly
support a determination of either strength or weakness. To reach the
repeatable level of maturity, Customs must eliminate the key practice
weaknesses identified in this report.\9
Table 1
Collective Number of KPA Strengths,
Weaknesses, and Observations on the
Three Projects
Number Number
of of Number of
strength weakness observati
Key process area s es ons
--------------------------------------- -------- -------- ---------
Requirements management 23 13 0
Software project planning 32 38 5
Software project tracking and oversight 28 40 4
Software quality assurance 3 46 2
Software configuration management 17 46 0
======================================================================
Total 103 183 11
----------------------------------------------------------------------
Also, GAO found that while the three projects varied as to the number
of key practice strengths, weaknesses, and observations under each of
the five "common features" or practice groupings (commitment to
perform, ability to perform, activities performed, measurement and
analysis, and verification of implementation), the NCAP 0.1 project
displayed a better strengths to weaknesses ratio across all KPAs
(about 1:1) than either AES or the Administrative Security System
(about 1:2 and 1:4, respectively). By increasing its software
maturity, Customs will reduce both the number of weaknesses on
individual projects and the variability in process discipline among
projects.
--------------------
\9 SEI groups each of its KPA practices into one of five "common
features" or practice attributes. These are "commitment to perform,
ability to perform, activities performed, measurement and analysis,
and verifying implementation."
CUSTOMS DOES NOT HAVE A
SOFTWARE PROCESS IMPROVEMENT
PROGRAM
-------------------------------------------------------- Chapter 0:4.2
SEI's IDEAL model for software process improvement defines five
sequential phases. The phases are (1) initiating the program,
including assigning organizational roles and responsibility,
establishing a program management structure, developing an action
plan, and allocating needed resources; (2) assessing the
organization's current level of process maturity; (3) establishing a
plan for addressing the identified process weaknesses; (4) developing
and implementing new or improved processes; and (5) learning from
process improvement experiences to strengthen the program.
In 1996, Customs initiated efforts to improve its software processes.
As part of this effort, Customs had a contractor assess its software
development capabilities and develop a process improvement plan.
Customs then established process improvement teams for strengthening
two level-2 KPAs (software project planning and project tracking and
oversight).
In 1997, Customs discontinued its process improvement program,
deciding at that time to redirect its process improvement resources
to Year 2000 conversion. As a result, Customs' software development
capability, which is fundamental to its ability to effectively
develop new systems like ACE, and maintain existing systems like AES,
remains ad hoc and undisciplined. Customs officials stated their
commitment to using the results of GAO's software capability
evaluation to baseline its strengths and weaknesses and address its
weaknesses, but Customs has not yet established a software process
improvement program. In particular, it has not assigned
organizational responsibility and authority for process improvement,
established a program management structure, defined a plan of action,
or committed the necessary resources (trained staff and funding) to
execute the plan.
RECOMMENDATIONS
---------------------------------------------------------- Chapter 0:5
GAO recommends that, after ensuring that its mission-critical systems
are Year 2000 compliant, but before investing in major software
development efforts like ACE, the Commissioner of Customs direct the
Customs Chief Information Officer to:
-- assign responsibility and authority for software process
improvement;
-- develop and implement a formal plan for software development
process improvement that is based on the software capability
evaluation results contained in this report and specifies
measurable goals and time frames, prioritizes initiatives,
estimates resource requirements (trained staff and funding), and
defines a process improvement management structure;
-- ensure that every new software development effort in Customs
adopts processes that satisfy at least SW-CMM level 2
requirements; and
-- ensure that process improvement activities are initiated for all
ongoing essential software maintenance projects.
AGENCY COMMENTS AND GAO'S
EVALUATION
---------------------------------------------------------- Chapter 0:6
In its written comments on a draft of this report, Customs
acknowledged the importance of software process improvement and
maturity. Also, it agreed with GAO's overall findings, including
that Customs' software development processes have not attained SW-CMM
level 2 maturity.
Customs stated that it has taken the first step toward implementing
GAO's recommendations by assigning responsibility and authority for
software process improvement as part of a reorganization of the
Office of Information and Technology that it plans to implement in
early 1999. Customs commented that once the reorganization is
implemented, a formal software process improvement program will be
established, including definition of an action plan, commitment of
resources, and specification of goals for achieving CMM levels 2 and
3. According to Customs, these improvement activities are in their
early stages. When they are successfully implemented, they should
address many of GAO's recommendations.
Customs also stated that, because its legacy systems are aging and
need to be enhanced and replaced, software process improvement must
occur in parallel with continued software development investments.
History has shown that attempting to modernize without first
instituting disciplined software processes has been a characteristic
of failed modernization programs.\10 Until it implements disciplined
software processes (at least level 2 process maturity), Customs
cannot prudently manage major software investments, such as ACE with
an estimated life cycle cost exceeding $1 billion.
In its comments, Customs also asked to meet with GAO to discuss
system-specific KPA practice strength and weakness determinations.
GAO met with Customs prior to requesting comments on a draft of this
report to discuss system-specific determinations, and then again
after receiving Customs' comments to ensure that all findings were
clear. GAO is prepared to continue assisting Customs as it improves
its software processes.
Customs provided other comments on a draft of this report. Each of
Customs' comments, along with GAO's responses, is discussed in detail
in chapter 8 and appendix I of this report.
--------------------
\10 Tax Systems Modernization: Management and Technical Weaknesses
Must Be Corrected If Modernization Is To Succeed (GAO/AIMD-95-156,
July 26, 1995), Tax Systems Modernization: Actions Underway But IRS
Has Not Yet Corrected Management and Technical Weaknesses
(GAO/AIMD-96-106, June 7, 1996), and Air Traffic Control: Immature
Software Acquisition Processes Increase FAA System Acquisition Risks
(GAO/AIMD-97-47, March 21, 1997).
INTRODUCTION
============================================================ Chapter 1
The mission of the Customs Service is to ensure that all goods and
persons entering and exiting the United States do so in compliance
with all U.S. laws and regulations. It does this by (1) enforcing
the laws governing the flow of goods and persons across the borders
of the United States and (2) assessing and collecting duties, taxes,
and fees on imported merchandise. During fiscal year 1997, Customs
collected $22.1 billion in revenue\1 at more than 300 ports of entry
and reported that it processed nearly 450 million passengers who
entered the United States during the year.
To accomplish its mission, Customs is organized into six lines of
business--trade compliance, outbound, passenger, finance, human
resources, and investigations. Each business area is described
below.
-- Trade compliance includes enforcement of laws and regulations
associated with the importation of goods into the United States.
To do so, Customs (1) works with the trade community to promote
understanding of applicable laws and regulations, (2)
selectively examines cargo to ensure that only eligible goods
enter the country, (3) reviews documentation associated with
cargo entries to ensure that they are properly valued and
classified, (4) collects billions of dollars annually in duties,
taxes, and fees associated with imported cargo, (5) assesses
fines and penalties for noncompliance with trade laws and
regulations, (6) seizes and accounts for illegal cargo, and (7)
manages the collection of these moneys to ensure that all
trade-related debts due to Customs are paid and properly
accounted for.
-- Outbound includes Customs enforcement of laws and regulations
associated with the movement of merchandise and conveyances from
the United States. To do so, Customs (1) selectively inspects
cargo at U.S. ports to guard against the exportation of illegal
goods, such as protected technologies, stolen vehicles, and
illegal currency, (2) collects, disseminates, and uses
intelligence to identify high-risk cargo and passengers, (3)
seizes and accounts for illegal cargo, (4) assesses and collects
fines and penalties associated with the exportation of illegal
cargo, and (5) physically examines baggage and cargo at airport
facilities for explosive and nuclear materials. In addition,
the outbound business includes collecting and disseminating
trade data within the federal government. Accurate trade data
are crucial to establishing accurate trade statistics on which
to base trade policy decisions and negotiate trade agreements
with other countries. By the year 2000, Customs estimates that
exports will be valued at $1.2 trillion, compared to a reported
$696 million in 1994.
-- Passenger includes processing all passengers and crew of
arriving and departing (1) air and sea conveyances and (2) land
vehicles and pedestrians. In fiscal year 1997, Customs reported
it processed nearly 450 million travelers and, by the year 2000,
expects almost 500 million passengers to arrive in the United
States annually. Many of Customs' passenger activities focus on
illegal immigration and drug smuggling and are coordinated with
other federal agencies, such as the Immigration and
Naturalization Service and the Department of Agriculture's
Animal and Plant Health Inspection Service. Activities include
targeting high-risk passengers, which requires timely and
accurate information, and physically inspecting selected
passengers, baggage, and vehicles to determine compliance with
laws and regulations.
-- Finance includes asset and revenue management activities. Asset
management consists of activities to (1) formulate Customs'
budget, (2) properly allocate and distribute funds, and (3)
acquire, manage, and account for personnel, goods, and services.
Revenue management encompasses all Customs activities to
identify and establish amounts owed Customs, collect these
amounts, and accurately report the status of revenue from all
sources. Sources of revenue include duties, fees, taxes, other
user fees, and forfeited currency and property. The revenue
management activities interrelate closely with the revenue
collection activities in the trade compliance, outbound, and
passenger business areas.
-- Human resources is responsible for filling positions, providing
employee benefits and services, training employees, facilitating
workforce effectiveness, and processing personnel actions for
Customs' 18,000 employees and managers.
-- Investigations includes activities to detect and eliminate
narcotics and money laundering operations. Customs works with
other agencies and foreign governments to reduce drug-related
activity by interdicting (seizing and destroying) narcotics,
investigating organizations involved in drug smuggling, and
deterring smuggling efforts through various other methods.
Customs also develops and provides information to the trade and
carrier communities to assist them in their efforts to prevent
smuggling organizations from using cargo containers and
commercial conveyances to introduce narcotics into the United
States.
To carry out its responsibilities, Customs relies on information
systems and processes to assist its staff in (1) documenting,
inspecting, and accounting for the movement and disposition of
imported goods and (2) collecting and accounting for the related
revenues. Customs expects its reliance on information systems to
increase as a result of its burgeoning workload. For 1995 through
2001, Customs estimates that the annual volume of import trade
between the United States and other countries will increase from $761
billion to $1.1 trillion. This will result in Customs processing an
estimated increase of 7.5 million commercial entries--from 13.1
million to 20.6 million annually--during the same period. Recent
trade agreements, such as the North American Free Trade Agreement
(NAFTA), have also increased the number and complexity of trade
provisions that Customs must enforce.
Customs recognizes that its ability to process the growing volume of
imports while improving compliance with trade laws depends heavily on
successfully modernizing its trade compliance process and its
supporting automated systems. To speed the processing of imports and
improve compliance with trade laws, the Congress enacted
legislation\2 that eliminated certain legislatively mandated paper
requirements and required Customs to establish the National Customs
Automation Program (NCAP). The legislation also specified certain
functions that NCAP must provide, including giving members of the
trade community the capability to electronically file import entries
at remote locations and enabling Customs to electronically process
"drawback" claims.\3 In response to the legislation, Customs began in
1994 to modernize the information systems that support operations.
--------------------
\1 Includes tariff duty, user fees, Internal Revenue Service excise
taxes, and other assessments.
\2 North American Free Trade Agreement Implementation Act, Public Law
103-182, 19 U.S.C. 1411 et seq.
\3 Drawbacks are refunds of duties and taxes paid on imported goods
that are subsequently exported or destroyed.
CURRENT PROJECTS: A BRIEF
DESCRIPTION
---------------------------------------------------------- Chapter 1:1
Customs has several projects underway to develop and acquire new
software and evolve (i.e., maintain) existing software to support its
six business areas. Customs' fiscal year 1998 budget for information
management and technology activities was about $147 million.
Customs' major information technology effort is its Automated
Commercial Environment (ACE) system. In 1994, Customs began to
develop ACE to replace its existing automated import system, the
Automated Commercial System (ACS). ACE is intended to provide an
integrated, automated information system for collecting,
disseminating, and analyzing import-related data and ensuring the
proper collection and allocation of revenues, totaling about $19
billion annually. According to Customs, ACE is planned to automate
critical functions that the Congress specified when it established
NCAP.
Customs reported that it spent $47.8 million on ACE as of the end of
fiscal year 1997. In November 1997, Customs estimated it would cost
$1.05 billion to develop, operate, and maintain ACE over the 15 years
from fiscal year 1994 through fiscal year 2008. Customs plans to
deploy ACE to more than 300 ports that handle commercial cargo
imports.
Customs plans to develop and deploy ACE in multiple phases.
According to Customs, the first phase, known as NCAP, is an ACE
prototype. Customs currently plans to deploy NCAP in four releases.
The first release was deployed for field evaluation at three
locations in May 1998,\4 and the fourth is scheduled for 1999.
Customs, however, has not adhered to previous NCAP deployment
schedules. Specifically, implementation of the NCAP prototype
slipped from January 1997 to August 1997 and then again to a series
of four releases beginning in October 1997, with the fourth release
starting in June 1998.
Customs also has several other projects underway to modify or enhance
existing systems that support its six business areas. For example,
in fiscal year 1998, Customs planned to spend about $3.7 million to
enhance its Automated Export System (AES), which supports the
outbound business area and is designed to improve Customs' collection
and reporting of export statistics and to enforce export regulations.
In addition, Customs planned to spend another $4.6 million to
maintain its administrative systems supporting its finance and human
resource business areas.
--------------------
\4 The first release of the NCAP prototype was deployed to Detroit,
Michigan; Laredo, Texas; and Port Huron, Michigan.
OBJECTIVES, SCOPE, AND
METHODOLOGY
---------------------------------------------------------- Chapter 1:2
The Chairman, Subcommittee on Treasury and General Government, Senate
Committee on Appropriations, and the Chairman, Subcommittee on
Treasury, Postal Service and General Government, House Committee on
Appropriations, requested that we review Customs' ability to develop
software for its computer systems. Our objectives were to determine
(1) the maturity of Customs' software development processes and (2)
the effectiveness of Customs' software process improvement program.
To determine Customs' software development process maturity, we
applied the Software Engineering Institute's (SEI) Software
Capability Maturity Model\SM (SW-CMM0)\5 and its Software Capability
Evaluation (SCE) method. SEI's expertise in software process
maturity as well as its capability maturity models and evaluation
methods are widely accepted throughout the software industry. All
our specialists were SEI-trained.
The SW-CMM ranks organizational maturity according to five levels.
(See figure 1.1.) Maturity levels 2 through 5 require the verifiable
existence and use of certain software development processes, known as
key process areas (KPA). According to SEI, an organization that has
these processes in place is in a much better position to successfully
develop software than an organization that does not have these
processes in place. We evaluated Customs' software development
processes against five of the six level 2 KPAs.
Figure 1.1: SW-CMM Levels and
Descriptions
(See figure in printed
edition.)
The sixth level 2 KPA, software subcontract management, was not
evaluated because Customs did not use subcontractors on any of the
projects that we evaluated. (See table 1.1.)
Table 1.1
SW-CMM KPAs Used to Assess Customs'
Software Development Maturity
CMM Level 2 KPAs Summary description
---------------------- ----------------------------------------------
Requirements Defining, validating, and prioritizing
management requirements, such as functions, performance,
and delivery dates.
Software project Developing estimates for the work to be
planning performed, establishing the necessary
commitments, and defining the plan to perform
the work.
Software project Tracking and reviewing software
tracking and oversight accomplishments and results against documented
estimates, commitments, and plans and
adjusting these based on the actual
accomplishments and results.
Software quality Reviewing and auditing the software products
assurance and activities to ensure that they comply with
the applicable processes, standards, and
procedures and providing the staff and
managers with the results of their reviews and
audits.
Software configuration Selecting project baseline items, such as
management specifications; systematically controlling
these items and changes to them; and recording
and reporting status and change activity for
these items.
----------------------------------------------------------------------
As established by the model, each KPA contains five common attributes
that indicate whether the implementation and institutionalization of
a KPA can be effective, repeatable, and lasting. The five common
features are:
-- Commitment to perform: The actions that the organization must
take to establish the process and ensure that it can endure.
Commitment to perform typically involves establishing
organizational policies and senior management sponsorship.
-- Ability to perform: The preconditions that must exist in the
project or organization to implement the software development
process competently. Ability to perform typically involves
resources, organizational structures, and training.
-- Activities performed: The roles and procedures necessary to
implement a KPA. Activities performed typically involve
establishing plans and procedures, performing the work, tracking
it, and taking appropriate management actions.
-- Measurement and analysis: Activities performed to measure the
process and analyze the measurements. Measurement and analysis
typically includes defining the measurements to be taken and the
analyses to be conducted to determine the status and
effectiveness of the activities performed.
-- Verifying implementation: The steps to ensure that the
activities are performed in compliance with the process that has
been established. Verification typically encompasses reviews by
management.
In accordance with SEI's SCE method and, for five of the six KPAs in
level 2, we evaluated Customs' institutional policies and practices
and compared project-specific guidance and practices against the five
common attributes. This project-specific comparison can result in
one of four possible outcomes: (1) project strengthan effective
implementation of the key practice, (2) project weaknessineffective
implementation of a key practice or failure to implement a key
practice, (3) project observationkey practice evaluated but evidence
inconclusive and cannot be characterized as either strength or
weakness, and (4) not ratedkey practice not currently relevant to
project, therefore, not evaluated.
We performed the project-specific evaluations on three ongoing
Customs software development projects, each of which is described
below. As requested by the Subcommittee Chairmen, one of the
projects evaluated was ACE, which is the largest and most important
system that Customs is developing. The other two projects were
selected by Customs on the basis of the following GAO specified
criteria: (1) each project should be managed by a different software
team, (2) at least one project should involve a legacy system, (3) at
least one project should involve Year 2000 software conversion, and
(4) each project should be relatively large and important to
accomplishing Customs' mission. The projects we evaluated are:
-- National Customs Automation Program (NCAP 0.1): NCAP 0.1 was
the first component of the National Customs Automation Program
Prototype (NCAP/P). NCAP/P, in turn, is the first phase of the
Automated Commercial Environment (ACE). Customs began
developing ACE in 1994 to address the new import processing
requirements established by the National Customs Automation
Program. ACE is also intended to replace the agency's legacy
automated import system, the Automated Commercial System (ACS).
NCAP 0.1 was installed at three field locations in May 1998.
-- Automated Export System (AES): AES is an export information
gathering and processing system, developed through cooperative
efforts by Customs, the Bureau of Census, other federal agencies
with export missions, and the export trade community. AES is
designed to improve the collection of trade statistics; assist
in the creation of a paperless export environment; facilitate
the release of exports subject to licensing requirements; and
consolidate export data required by several government agencies,
easing the data filing burden for exporters while streamlining
the federal data collection process.
Customs installed AES in all U.S. vessel ports in October 1996,
and currently it is operational in all ports, including air,
rail, and truck transit ports. Customs and Census officials
estimate that they spent approximately $12.9 million to develop
and implement AES from fiscal year 1992 to 1997. These costs
included, among other things, expenses for contractors, travel,
and training. According to Customs' and Census' figures, both
agencies estimate that together they will spend an additional
$32.2 million through fiscal year 2002 on AES implementation and
maintenance.
-- Administrative Security System: The Administrative Security
System assists users in requesting access to administrative
systems. Users' requests are electronically submitted to the
appropriate official for approvals. In addition, other portions
of the Administrative Security System provide functionality to
allow the System Administrators the ability to prepare and
maintain user profiles, request logs, and electronic approval
and disapproval reports.
To assess the effectiveness of Customs' software process improvement
program, we interviewed the Director, Technical Architecture Group,
Office of Information and Technology, to determine: (1) process
improvements that are planned and underway, (2) the rationale for
each initiative, (3) the relative priority of each, (4) progress made
on each initiative, and (5) obstacles, if any, impeding progress. We
also reviewed past process improvement plans, meeting minutes, and
related documentation. Further, we reviewed SEI's model for software
process improvement, known as IDEAL\SM .\6 IDEAL defines five
sequential phases of software process improvement that can be used to
develop a long range, integrated plan for initiating and managing a
software process improvement program.\7
Customs provided written comments on a draft of this report. These
comments are presented and evaluated in chapter 8, and are reprinted
in appendix I. We performed our work at Customs' Newington,
Virginia, Data Center from February 1998 through November 1998, in
accordance with generally accepted government auditing standards.
--------------------
\5 Capability Maturity Model\SM is the service mark of Carnegie
Mellon University, and CMM0 is registered in the U.S. Patent and
Trademark Office.
\6 IDEAL is a service mark of Carnegie Mellon University.
\7 IDEAL: A User's Guide for Software Process Improvement
(CMU/SEI-96-HB-001).
REQUIREMENTS MANAGEMENT
============================================================ Chapter 2
The purpose of requirements management is to establish agreement
between the customer and the software developers of the customer's
requirements that will be implemented by the software developers.
This agreement typically is referred to as the "system requirements
allocated to the software." The agreement covers both technical and
nontechnical (e.g., delivery dates) requirements. The agreement
forms the basis for estimating, planning, performing, and tracking
the software developer's activities throughout the software life
cycle.
According to the SW-CMM, a repeatable requirements management
process, among other things, includes (1) documenting the system
requirements allocated to software, (2) providing adequate resources
and funding for managing the allocated requirements, (3) following a
written organizational policy for requirements management, (4) having
a quality assurance group that reviews the activities and work
products for managing allocated requirements and reports the results,
(5) using the allocated requirements as the basis for software plans,
work products, and activities, and (6) training members of the
software engineering group to perform their requirements management
activities.
CUSTOMS' REQUIREMENTS
MANAGEMENT PROCESS DOES NOT
SATISFY SEI'S CRITERIA
---------------------------------------------------------- Chapter 2:1
All three projects had practice strengths in this KPA. For example,
each project documented the system requirements allocated to software
and ensured that adequate resources and funding for managing the
allocated requirements were provided. One of the projects, NCAP 0.1,
had strengths in all but two practices under this KPA; however, each
practice weakness is significant.
Collectively, the projects had many weaknesses in this KPA, and thus
Customs' requirements management processes do not meet "repeatable"
maturity level criteria. For example, none of the projects had a
written organizational policy governing requirements management, and
none had a quality assurance group for reviewing and reporting on the
activities and work products associated with managing the allocated
requirements. In the absence of these two practices, management is
missing two means for ensuring that software requirements are managed
in a prescribed manner. Also, two of the projects did not use the
allocated software requirements as the basis for software plans, work
products, and activities, which increases the risk that the software
developed will not fully satisfy requirements. Further, members of
two projects' software engineering groups were not trained to perform
requirements management activities, thus increasing the chances of
mismanagement.
Table 2.1 provides a comprehensive list of the three projects'
strengths and weaknesses for the requirements management KPA. The
specific findings supporting the practice ratings cited in table 2.1
are in tables 2.2 through 2.4.
Table 2.1
Requirements Management
Administrati
Key practice NCAP 0.1 AES ve
------------ ---------------------------- ---------- ---------- ------------
Commitment 1 The project follows a Weakness Weakness Weakness
written organizational
policy for managing the
system requirements
allocated to software.
Ability 1 For each project, Strength Strength Strength
responsibility is
established for analyzing
the system requirements and
allocating them to hardware,
software, and other system
components.
Ability 2 The allocated requirements Strength Strength Strength
are documented.
Ability 3 Adequate resources and Strength Strength Strength
funding are provided for
managing the allocated
requirements.
Ability 4 Members of the software Strength Weakness Weakness
engineering group and other
software-related groups are
trained to perform their
requirements management
activities.
Activity 1 The software engineering Strength Strength Weakness
group reviews the allocated
requirements before they are
incorporated into the
software project.
Activity 2 The software engineering Strength Weakness Weakness
group uses the allocated
requirements as the basis
for software plans, work
products, and activities.
Activity 3 Changes to the allocated Strength Strength Weakness
requirements are reviewed
and incorporated into the
software project.
Measurement Measurements are made and Strength Strength Strength
1 used to determine the status
of the activities for
managing the allocated
requirements.
Verification The activities for managing Strength Weakness Strength
1 the allocated requirements
are reviewed with senior
management on a periodic
basis.
Verification The activities for managing Strength Strength Strength
2 the allocated requirements
are reviewed with the
project manager on both a
periodic and event-driven
basis.
Verification The software quality Weakness Weakness Weakness
3 assurance group reviews and/
or audits the activities and
work products for managing
the allocated requirements
and reports the results.
--------------------------------------------------------------------------------
Table 2.2
Requirements Management Findings for
NCAP 0.1
National Customs Automation Program 0.1
------------------------------------------------------------------
Key practice Finding Rating
------------ -------------------------- -------------------------- ----------
Commitment 1 The project follows a There is no written Weakness
written organizational organizational policy for
policy for managing the managing the system
system requirements requirements allocated to
allocated to software. software.
Ability 1 For each project, The Process Analysis and Strength
responsibility is Requirements Team is
established for analyzing responsible for analyzing
the system requirements system requirements and
and allocating them to allocating them to
hardware, software, and hardware, software, and
other system components. other system components.
Ability 2 The allocated requirements Allocated requirements are Strength
are documented. documented.
Ability 3 Adequate resources and Adequate resources and Strength
funding are provided for funding are provided for
managing the allocated managing the allocated
requirements. requirements.
Ability 4 Members of the software Members of the software Strength
engineering group and engineering group and
other software-related other software-related
groups are trained to groups are trained to
perform their requirements perform their requirements
management activities. management activities.
Activity 1 The software engineering The software engineering Strength
group reviews the group reviews the
allocated requirements allocated requirements
before they are before they are
incorporated into the incorporated into the
software project. software project.
Activity 2 The software engineering The software engineering Strength
group uses the allocated group uses the allocated
requirements as the basis requirements as the basis
for software plans, work for software plans, work
products, and activities. products, and activities.
Activity 3 Changes to the allocated Changes to the allocated Strength
requirements are reviewed requirements are reviewed
and incorporated into the and incorporated into the
software project. software project.
Measurement Measurements are made and Measurements are made and Strength
1 used to determine the used to determine the
status of the activities status of the activities
for managing the allocated for managing the allocated
requirements. requirements.
Verification The activities for Periodic meetings with Strength
1 managing the allocated senior management include
requirements are reviewed reviews of allocated
with senior management on requirements.
a periodic basis.
Verification The activities for The activities for Strength
2 managing the allocated managing the allocated
requirements are reviewed requirements are reviewed
with the project manager with the project manager
on both a periodic and on both a periodic and
event-driven basis. event-driven basis.
Verification The software quality There is no software Weakness
3 assurance group reviews quality assurance group,
and/or audits the therefore, no reviews and/
activities and work or audits are done.
products for managing the
allocated requirements and
reports the results.
--------------------------------------------------------------------------------
Table 2.3
Requirements Management Findings for AES
Automated Export System
------------------------------------------------------------------
Key practice Finding Rating
------------ -------------------------- -------------------------- ----------
Commitment 1 The project follows a There is no written Weakness
written organizational organizational policy for
policy for managing the managing systems
system requirements requirements allocated to
allocated to software. software.
Ability 1 For each project, The project team Strength
responsibility is established responsibility
established for analyzing for analyzing the system
the system requirements requirements and
and allocating them to allocating them to
hardware, software, and hardware, software, and
other system components. other system components.
Ability 2 The allocated requirements The allocated requirements Strength
are documented. are documented in the
system functional
requirements document and
in the change requests
document.
Ability 3 Adequate resources and Adequate resources and Strength
funding are provided for funding are provided for
managing the allocated managing allocated
requirements. requirements.
Ability 4 Members of the software One member of the software Weakness
engineering group and engineering group has been
other software-related trained to perform
groups are trained to requirements management
perform their requirements activities, but others
management activities. have not.
Activity 1 The software engineering The software engineering Strength
group reviews the group reviews the
allocated requirements allocated requirements
before they are before they are
incorporated into the incorporated into the
software project. software project.
Activity 2 The software engineering The software engineering Weakness
group uses the allocated group does not use the
requirements as the basis allocated requirements as
for software plans, work the basis for software
products, and activities. plans, work products, and
activities.
Activity 3 Changes to the allocated The Change Request Board Strength
requirements are reviewed reviews and approves all
and incorporated into the changes made to allocated
software project. requirements and
incorporates them into the
software project.
Measurement Measurements are made and Measurements are made and Strength
1 used to determine the used to track changes to
status of the activities requirements and hours
for managing the allocated spent on requirements
requirements. management. A performance
measurement plan is
available.
Verification The activities for The activities for Weakness
1 managing the allocated managing the allocated
requirements are reviewed requirements are not
with senior management on reviewed with senior
a periodic basis. management on a periodic
basis.
Verification The activities for The activities for Strength
2 managing the allocated managing the allocated
requirements are reviewed requirements are reviewed
with the project manager with the project manager
on both a periodic and on both a periodic and
event-driven basis. event-driven basis.
Verification The software quality The individual responsible Weakness
3 assurance group reviews for software quality
and/or audits the assurance did not review
activities and work and/or audit the
products for managing the activities and work
allocated requirements and products for managing the
reports the results. allocated requirements.
--------------------------------------------------------------------------------
Table 2.4
Requirements Management Findings for
Administrative Security System
Administrative Security System
------------------------------------------------------------------
Key practice Finding Rating
------------ -------------------------- -------------------------- ----------
Commitment 1 The project follows a There is no written Weakness
written organizational organizational policy on
policy for managing the requirements management.
system requirements
allocated to software.
Ability 1 For each project, Project leaders are Strength
responsibility is responsible for analyzing
established for analyzing systems requirements and
the system requirements allocating them to
and allocating them to software, hardware or
hardware, software, and other components.
other system components.
Ability 2 The allocated requirements Allocated requirements for Strength
are documented. the system are documented.
Ability 3 Adequate resources and Adequate resources and Strength
funding are provided for funding are available for
managing the allocated managing the allocated
requirements. requirements.
Ability 4 Members of the software There was no evidence to Weakness
engineering group and show that members of the
other software-related software engineering group
groups are trained to received training in
perform their requirements requirements management.
management activities.
Activity 1 The software engineering The software engineering Weakness
group reviews the group did not review the
allocated requirements allocated requirements
before they are before they were
incorporated into the incorporated into the
software project. software project.
Activity 2 The software engineering Members of the software Weakness
group uses the allocated engineering group did not
requirements as the basis use the allocated
for software plans, work requirements as the basis
products, and activities. for software plans, work
products, and activities.
Activity 3 Changes to the allocated Changes to the allocated Weakness
requirements are reviewed requirements are not
and incorporated into the reviewed by the software
software project. engineering group and
incorporated into the
software project.
Measurement Measurements are made and Measurements are made and Strength
1 used to determine the used to determine the
status of the activities status of the activities
for managing the allocated for managing the allocated
requirements. requirements.
Verification The activities for Requirements management Strength
1 managing the allocated activities are discussed
requirements are reviewed at periodic meetings with
with senior management on senior management.
a periodic basis.
Verification The activities for The activities for Strength
2 managing the allocated managing the allocated
requirements are reviewed requirements are reviewed
with the project manager with the project manager
on both a periodic and on both a periodic and
event-driven basis. event-driven basis.
Verification The software quality There is no software Weakness
3 assurance group reviews quality assurance (SQA)
and/or audits the group.
activities and work
products for managing the
allocated requirements and
reports the results.
--------------------------------------------------------------------------------
CONCLUSIONS
---------------------------------------------------------- Chapter 2:2
While Customs' projects had several practice strengths in this KPA,
the number and significance of their practice weaknesses mean that
Customs' ability to manage software requirements is not repeatable.
As a result, Customs is at risk of producing systems that fail to
provide promised capabilities, and cost more and take longer than
necessary.
SOFTWARE PROJECT PLANNING
============================================================ Chapter 3
The purpose of software project planning is to establish reasonable
plans for performing the software engineering and for managing the
software project. According to the SW-CMM, a repeatable software
project planning process, among other things, includes (1)
documenting the software project plan, and preparing plans for
software engineering facilities and support tools, (2) identifying
the work products needed to establish and maintain control of the
software project, (3) following a written organizational policy for
planning a software project, (4) having a quality assurance group
that reviews the activities and work products for software project
planning and reports the results, (5) estimating the software
project's efforts and costs, and estimating its critical computer
resources according to a documented procedure, (6) making and using
measurements to determine the status of planning activities, and (7)
training personnel in software project planning and estimating.
CUSTOMS IS NOT PERFORMING
SOFTWARE PROJECT PLANNING
EFFECTIVELY
---------------------------------------------------------- Chapter 3:1
All of the projects that we evaluated had key practice strengths in
this KPA. For example, all had strengths in (1) documenting a
software project plan and preparing plans for the software
engineering facilities and support tools needed to develop the
software and (2) identifying the work products needed to control the
software project. NCAP 0.1, in particular, had many additional
practice strengths.
However, many significant practice weaknesses were found in all three
projects. None of the projects followed an organizational software
project planning policy, and none had a quality assurance group
conducting reviews and/or audits. As a result, the projects
performed these practices differently and inconsistently, and
controls were unreliable. For example, while the NCAP 0.1 project
followed a documented procedure for estimating the size of software
work products (or changes to the size of work products), and made and
used measurements to determine the status of software planning
activities, neither of the other two projects performed these
practices and none of the projects had personnel trained in software
project planning and estimating. Such project planning weaknesses
mean that management has no assurance that it will get the
consistent, complete, and reliable information about the projects'
expected costs and schedules needed to make expeditious and informed
investment decisions.
Table 3.1 provides a comprehensive list of the three projects'
strengths, weaknesses, and observations for the software project
planning KPA. The specific findings supporting the practice ratings
cited in table 3.1 are in tables 3.2 through 3.4.
Table 3.1
Software Project Planning
Administrati
Key practice NCAP 0.1 AES ve
------------ ---------------------------- ---------- ---------- ------------
Commitment 1 A project software manager Strength Strength Weakness
is designated to be
responsible for negotiating
commitments and developing
the project's software
development plan.
Commitment 2 The project follows a Weakness Weakness Weakness
written organizational
policy for planning a
software project.
Ability 1 A documented and approved Strength Observatio Observation
statement of work exists for n
the software project.
Ability 2 Responsibilities for Strength Observatio Strength
developing the software n
development plan are
assigned.
Ability 3 Adequate resources and Strength Weakness Strength
funding are provided for
planning the software
project.
Ability 4 The software managers, Weakness Weakness Weakness
software engineers, and
other individuals involved
in the software project
planning are trained in the
software estimating and
planning procedures
applicable to their areas of
responsibility.
Activity 1 The software engineering Strength Weakness Weakness
group participates on the
project proposal team.
Activity 2 Software project planning is Strength Weakness Strength
initiated in the early
stages of, and in parallel
with, the overall project
planning.
Activity 3 The software engineering Strength Weakness Observation
group participates with
other affected groups in the
overall project planning
throughout the project's
life.
Activity 4 Software project commitments Weakness Observatio Weakness
made to individuals and n
groups external to the
organization are reviewed
with senior management
according to a documented
procedure.
Activity 5 A software life cycle with Weakness Strength Strength
predefined stages of
manageable size is
identified or defined.
Activity 6 The project's software Strength Strength Weakness
development plan is
developed according to a
documented procedure.
Activity 7 The plan for the software Strength Strength Strength
project is documented.
Activity 8 Software work products that Strength Strength Strength
are needed to establish and
maintain control of the
software project are
identified.
Activity 9 Estimates for the size of Strength Weakness Weakness
the software work products
(or changes to the size of
software work products) are
derived according to a
documented procedure.
Activity 10 Estimates for the software Weakness Weakness Weakness
project's effort and costs
are derived according to a
documented procedure.
Activity 11 Estimates for the project's Weakness Weakness Weakness
critical computer resources
are derived according to a
documented procedure.
Activity 12 The project's software Weakness Weakness Weakness
schedule is derived
according to a documented
procedure.
Activity 13 The software risks Strength Strength Weakness
associated with the cost,
resource, schedule, and
technical aspects of the
project are identified,
assessed, and documented.
Activity 14 Plans for the project's Strength Strength Strength
software engineering
facilities and support tools
are prepared.
Activity 15 Software planning data are Strength Weakness Weakness
recorded.
Measurement Measurements are made and Strength Weakness Weakness
1 used to determine the status
of the software planning
activities.
Verification The activities for software Strength Weakness Weakness
1 project planning are
reviewed with senior
management on a periodic
basis.
Verification The activities for software Strength Strength Weakness
2 project planning are
reviewed with the project
manager on both a periodic
and event-driven basis.
Verification The software quality Weakness Weakness Weakness
3 assurance group reviews and/
or audits the activities and
work products for software
project planning and reports
the results.
--------------------------------------------------------------------------------
Table 3.2
Software Project Planning Findings for
NCAP 0.1
National Customs Automation Program 0.1
------------------------------------------------------------------
Key practice Finding Rating
------------ -------------------------- -------------------------- ----------
Commitment 1 A project software manager The NCAP project has a Strength
is designated to be software manager
responsible for designated to be
negotiating commitments responsible for
and developing the negotiating commitments
project's software and developing the
development plan. project's software
development plan.
Commitment 2 The project follows a The project does not Weakness
written organizational follow a written
policy for planning a organizational policy for
software project. planning a software
project.
Ability 1 A documented and approved The approved project plan Strength
statement of work exists meets the requirements for
for the software project. a statement of work for
the project.
Ability 2 Responsibilities for The project manager has Strength
developing the software been assigned
development plan are responsibility for
assigned. developing the software
development plan.
Ability 3 Adequate resources and Adequate resources and Strength
funding are provided for funding have been provided
planning the software for planning the software
project. project.
Ability 4 The software managers, Project personnel are not Weakness
software engineers, and trained in software
other individuals involved project planning and
in the software project estimating procedures.
planning are trained in
the software estimating
and planning procedures
applicable to their areas
of responsibility.
Activity 1 The software engineering The software engineering Strength
group participates on the group participates on the
project proposal team. project proposal team.
Activity 2 Software project planning Software project planning Strength
is initiated in the early is initiated in the early
stages of, and in parallel stages of, and in parallel
with, the overall project with, the overall project
planning. planning.
Activity 3 The software engineering The software engineering Strength
group participates with group participates with
other affected groups in other affected groups in
the overall project the overall project
planning throughout the planning throughout the
project's life. project's life.
Activity 4 Software project Software project Weakness
commitments made to commitments made to
individuals and groups individuals and groups
external to the external to the
organization are reviewed organization are not
with senior management reviewed with senior
according to a documented management and there is no
procedure. documented procedure for
such reviews.
Activity 5 A software life cycle with There is no documented Weakness
predefined stages of evidence that a software
manageable size is life cycle was selected
identified or defined. for the project.
Activity 6 The project's software The project has a software Strength
development plan is development plan that is
developed according to a developed according to a
documented procedure. documented procedure.
Activity 7 The plan for the software The plan for the software Strength
project is documented. project is documented in
the project plan.
Activity 8 Software work products Software work products Strength
that are needed to that are needed to
establish and maintain establish and maintain
control of the software control of the software
project are identified. project are identified in
the project plan.
Activity 9 Estimates for the size of Estimates for the size of Strength
the software work products the software work products
(or changes to the size of (or changes to the size of
software work products) software work products)
are derived according to a are derived according to a
documented procedure. documented procedure.
Activity 10 Estimates for the software Estimates for the software Weakness
project's effort and costs project's effort and costs
are derived according to a are not derived according
documented procedure. to a documented procedure.
Activity 11 Estimates for the Estimates for the Weakness
project's critical project's critical
computer resources are computer resources are not
derived according to a derived according to a
documented procedure. documented procedure.
Activity 12 The project's software There is no documented Weakness
schedule is derived procedure for deriving the
according to a documented software schedule.
procedure.
Activity 13 The software risks The software risks Strength
associated with the cost, associated with the cost,
resource, schedule, and resource, schedule, and
technical aspects of the technical aspects of the
project are identified, project are identified,
assessed, and documented. assessed, and documented.
Activity 14 Plans for the project's Plans for the project's Strength
software engineering software engineering
facilities and support facilities and support
tools are prepared. tools are prepared.
Activity 15 Software planning data are Software planning data are Strength
recorded. recorded.
Measurement Measurements are made and Measurements are made and Strength
1 used to determine the used to determine the
status of the software status of the software
planning activities. planning activities.
Verification The activities for The activities for Strength
1 software project planning software project planning
are reviewed with senior are reviewed with senior
management on a periodic management, including
basis. reports to Treasury and
Customs Investment Review
Boards (IRB), on a
periodic basis.
Verification The activities for The activities for Strength
2 software project planning software project planning
are reviewed with the are reviewed with the
project manager on both a project manager on both a
periodic and event-driven periodic and event-driven
basis. basis through weekly
status reports and
meetings.
Verification The software quality There is no SQA group. Weakness
3 assurance group reviews
and/or audits the
activities and work
products for software
project planning and
reports the results.
--------------------------------------------------------------------------------
Table 3.3
Software Project Planning Findings for
AES
Automated Export System
------------------------------------------------------------------
Key practice Finding Rating
------------ -------------------------- -------------------------- ----------
Commitment 1 A project software manager A project software manager Strength
is designated to be is designated to be
responsible for responsible for
negotiating commitments negotiating commitments
and developing the and developing the
project's software project's software
development plan. development plan.
Commitment 2 The project follows a The project does not Weakness
written organizational follow a written
policy for planning a organizational policy for
software project. planning a software
project.
Ability 1 A documented and approved The AES System and Observatio
statement of work exists Functional Requirements n
for the software project. Table documents the
statement of work for this
project. However, this
document is not approved.
Ability 2 Responsibilities for Responsibilities for Observatio
developing the software developing the software n
development plan are development plan have not
assigned. been assigned; however, a
plan has been developed.
Ability 3 Adequate resources and Adequate resources and Weakness
funding are provided for funding are not provided
planning the software for planning the software
project. project.
Ability 4 The software managers, Individuals involved in Weakness
software engineers, and the software project
other individuals involved planning are not trained
in the software project in the software estimating
planning are trained in and planning procedures.
the software estimating
and planning procedures
applicable to their areas
of responsibility.
Activity 1 The software engineering The software engineering Weakness
group participates on the group does not participate
project proposal team. in project proposal
preparation.
Activity 2 Software project planning Software project planning Weakness
is initiated in the early is not initiated in the
stages of, and in parallel early stages of, and in
with, the overall project parallel with, the overall
planning. project planning.
Activity 3 The software engineering The software engineering Weakness
group participates with group does not participate
other affected groups in in the overall project
the overall project planning throughout the
planning throughout the project's life cycle.
project's life.
Activity 4 Software project Software project Observatio
commitments made to commitments made to n
individuals and groups individuals and groups
external to the external to the
organization are reviewed organization are reviewed
with senior management with senior management;
according to a documented however, there is no
procedure. documented procedure for
these reviews.
Activity 5 A software life cycle with The project uses the Strength
predefined stages of waterfall model identified
manageable size is in the Customs software
identified or defined. development life cycle
document.
Activity 6 The project's software The project's software Strength
development plan is development plan was
developed according to a developed according to a
documented procedure. documented procedure in
the SDLC.
Activity 7 The plan for the software The plan for the software Strength
project is documented. project is documented.
Activity 8 Software work products Software work products Strength
that are needed to that are needed to
establish and maintain establish and maintain
control of the software control of the software
project are identified. project have been
identified.
Activity 9 Estimates for the size of There is no documented Weakness
the software work products procedure for estimating
(or changes to the size of the size of software work
software work products) products (or changes to
are derived according to a the size of software work
documented procedure. products).
Activity 10 Estimates for the software There is no documented Weakness
project's effort and costs procedure for estimating
are derived according to a project effort and cost.
documented procedure.
Activity 11 Estimates for the There is no documented Weakness
project's critical procedure for estimating
computer resources are the project's computer
derived according to a resources.
documented procedure.
Activity 12 The project's software There is no documented Weakness
schedule is derived procedure for deriving the
according to a documented software schedule.
procedure.
Activity 13 The software risks The software risks Strength
associated with the cost, associated with the cost,
resource, schedule, and resource, schedule, and
technical aspects of the technical aspects of the
project are identified, project are identified,
assessed, and documented. assessed, and documented.
Activity 14 Plans for the project's AES is an ongoing project. Strength
software engineering The software engineering
facilities and support facilities and support
tools are prepared. tools already exist, and
are documented in the
software development plan.
Activity 15 Software planning data are Software planning data, Weakness
recorded. such as estimating the
project's critical
computer resources, are
not recorded.
Measurement Measurements are made and Measurements are not made Weakness
1 used to determine the and used to determine the
status of the software status of the software
planning activities. planning activities.
Verification The activities for The activities for Weakness
1 software project planning software project planning
are reviewed with senior are not reviewed with
management on a periodic senior management on a
basis. periodic basis.
Verification The activities for The activities for Strength
2 software project planning software project planning
are reviewed with the are reviewed with the
project manager on both a project manager on both a
periodic and event-driven periodic and event-driven
basis. basis.
Verification The software quality The individual performing Weakness
3 assurance group reviews software quality assurance
and/or audits the activities does not
activities and work conduct reviews and/or
products for software audits.
project planning and
reports the results.
--------------------------------------------------------------------------------
Table 3.4
Software Project Planning Findings for
Administrative Security System
Administrative Security System
------------------------------------------------------------------
Key practice Finding Rating
------------ -------------------------- -------------------------- ----------
Commitment 1 A project software manager A project software manager Weakness
is designated to be has not been designated to
responsible for be responsible for
negotiating commitments negotiating commitments
and developing the and developing the
project's software project's software
development plan. development plan. The
software engineering group
was not sure who was
responsible for
negotiating commitments.
Commitment 2 The project follows a The project does not Weakness
written organizational follow a written
policy for planning a organizational policy for
software project. planning a software
project.
Ability 1 A documented and approved A documented statement of Observatio
statement of work exists work exists for the n
for the software project. software project, but has
not been approved.
Ability 2 Responsibilities for Responsibilities for Strength
developing the software developing the software
development plan are development plan were
assigned. assigned to a team of
seven whose names appear
on the document.
Ability 3 Adequate resources and Adequate resources and Strength
funding are provided for funding are available for
planning the software planning the software
project. project.
Ability 4 The software managers, The software managers, Weakness
software engineers, and software engineers, and
other individuals involved other individuals involved
in the software project in the software project
planning are trained in planning are not trained
the software estimating in the software estimating
and planning procedures and planning procedures
applicable to their areas applicable to their areas
of responsibility. of responsibility.
Activity 1 The software engineering The software engineering Weakness
group participates on the group did not participate
project proposal team. on the project proposal
team.
Activity 2 Software project planning Software project planning Strength
is initiated in the early is initiated in the early
stages of, and in parallel stages of, and in parallel
with, the overall project with, the overall project
planning. planning.
Activity 3 The software engineering The software engineering Observatio
group participates with group occasionally n
other affected groups in participates with other
the overall project affected groups in the
planning throughout the overall project planning
project's life. throughout the project's
life.
Activity 4 Software project There is no documented Weakness
commitments made to procedure to ensure that
individuals and groups software project
external to the commitments made to
organization are reviewed individuals and groups
with senior management external to the
according to a documented organization are reviewed
procedure. with senior management.
Activity 5 A software life cycle with A software life cycle with Strength
predefined stages of predefined stages of
manageable size is manageable size has been
identified or defined. identified in the SDLC.
Activity 6 The project's software The project has a software Weakness
development plan is development plan; but it
developed according to a was not developed
documented procedure. according to a documented
procedure.
Activity 7 The plan for the software The plan for the software Strength
project is documented. project is documented.
Activity 8 Software work products Software work products Strength
that are needed to that are needed to
establish and maintain establish and maintain
control of the software control of the software
project are identified. project are identified.
Activity 9 Estimates for the size of No documented procedure is Weakness
the software work products used for estimating the
(or changes to the size of size of software work
software work products) products (or changes to
are derived according to a the size of software work
documented procedure. products).
Activity 10 Estimates for the software No documented procedure is Weakness
project's effort and costs used for estimating the
are derived according to a project's effort and cost.
documented procedure.
Activity 11 Estimates for the No documented procedure is Weakness
project's critical used for estimating the
computer resources are project's computer
derived according to a resources.
documented procedure.
Activity 12 The project's software No documented procedure is Weakness
schedule is derived used for deriving the
according to a documented project schedule.
procedure.
Activity 13 The software risks The software risks Weakness
associated with the cost, associated with the cost,
resource, schedule, and resource, schedule, and
technical aspects of the technical aspects of the
project are identified, project have been
assessed, and documented. identified and documented,
but have not been assessed
for impact and mitigation
options.
Activity 14 Plans for the project's Existing tools and the Strength
software engineering current software
facilities and support engineering environment
tools are prepared. have been identified for
use.
Activity 15 Software planning data are Software planning data are Weakness
recorded. not recorded.
Measurement Measurements are made and No measurements are made Weakness
1 used to determine the to determine the status of
status of the software software planning
planning activities. activities.
Verification The activities for While periodic meetings Weakness
1 software project planning are held with senior
are reviewed with senior management, there was no
management on a periodic evidence that activities
basis. for software project
planning are addressed.
Verification The activities for While periodic meetings Weakness
2 software project planning are held with the project
are reviewed with the manager, there was no
project manager on both a evidence that activities
periodic and event-driven for software project
basis. planning are addressed.
Verification The software quality There is no software Weakness
3 assurance group reviews quality assurance group.
and/or audits the
activities and work
products for software
project planning and
reports the results.
--------------------------------------------------------------------------------
CONCLUSIONS
---------------------------------------------------------- Chapter 3:2
Effective planning is the cornerstone of successful software
development project management. While Customs showed some strengths
in this KPA, its many weaknesses render its software project planning
processes unrepeatable. Therefore, Customs has no assurance that the
projects are effectively establishing plans, including reliable
projections of costs and schedules, and effectively measuring and
monitoring progress and taking needed corrective actions
expeditiously.
SOFTWARE PROJECT TRACKING AND
OVERSIGHT
============================================================ Chapter 4
The purpose of software project tracking and oversight is to provide
adequate visibility into the progress of the software development so
that management can act effectively when the software project's
performance deviates significantly from the software plans. Software
project tracking and oversight involves tracking and reviewing the
software accomplishments and results against documented estimates,
commitments, and plans, and adjusting these plans based on the actual
accomplishments and results.
According to the SW-CMM, effective software project tracking and
oversight, among other things, includes (1) designating a project
software manager to be responsible for the project's software
activities and results, (2) having a documented software development
plan for tracking software activities and communicating status, (3)
following a written organizational policy for managing the project,
(4) conducting periodic internal reviews to track technical progress,
plans, performance, and issues against the software development plan,
(5) tracking the software risks associated with the cost, resource,
schedule, and technical aspects of the project, (6) explicitly
assigning responsibility for software work products and activities,
(7) tracking the sizes of the software work products (or sizes of the
changes to the software work products) and taking corrective actions
as necessary, and (8) periodically reviewing the activities for
software project tracking and oversight with senior management.
CUSTOMS' SOFTWARE PROJECT
TRACKING AND OVERSIGHT PROCESS
IS IMMATURE
---------------------------------------------------------- Chapter 4:1
The projects evaluated exhibited some software project tracking and
oversight practice strengths. For example, all three of the projects
had a project software manager designated to be responsible for the
project's software activities and results, and all had a documented
software development plan for tracking software activities and
communicating status. Also, NCAP 0.1 had strengths in all but five
of this KPA's 24 key practices.
However, the three projects collectively had many weaknesses, and
these weaknesses, including the five for NCAP 0.1, were significant
and thus preclude Customs from meeting SEI's repeatable maturity
level criteria. For example, none of the projects followed a written
organizational policy for managing the software project. With no
established policy, Customs increases the risk that key tracking and
oversight activities will not be performed effectively. For example,
for two of the three projects, the project managers did not (1)
conduct periodic internal reviews to track technical progress, plans,
performance, and issues against the software development plan, (2)
track software risks associated with cost, resource, schedule, and
technical aspects of the project, (3) explicitly assign
responsibility to individuals for software work products and
activities, (4) track the sizes of the software work products (or
sizes of the changes to the software work products) and take
corrective actions, or (5) periodically review software project
tracking and oversight activities with senior management.
Table 4.1 provides a comprehensive list of the three projects'
strengths, weaknesses, and observations for the software project
tracking and oversight KPA. The specific findings supporting the
practice ratings cited in table 4.1 are in tables 4.2 through 4.4.
Table 4.1
Software Project Tracking and Oversight
Summary
Administrati
Key practice NCAP 0.1 AES ve
------------ ---------------------------- ---------- ---------- ------------
Commitment 1 A project software manager Strength Strength Strength
is designated to be
responsible for the
project's software
activities and results.
Commitment 2 The project follows a Weakness Weakness Weakness
written organizational
policy for managing the
software project.
Ability 1 A software development plan Strength Observatio Observation
for the software project is n
documented and approved.
Ability 2 The project software manager Strength Weakness Weakness
explicitly assigns
responsibility for software
work products and
activities.
Ability 3 Adequate resources and Strength Weakness Weakness
funding are provided for
tracking the software
project.
Ability 4 The software managers are Weakness Strength Weakness
trained in managing the
technical and personnel
aspects of the software
project.
Ability 5 First-line software managers Strength Strength Weakness
receive orientation in the
technical aspects of the
software project.
Activity 1 A documented software Strength Strength Strength
development plan is used for
tracking the software
activities and communicating
status.
Activity 2 The project's software Weakness Weakness Weakness
development plan is revised
according to a documented
procedure.
Activity 3 Software project commitments Weakness Observatio Observation
and changes to commitments n
made to individuals and
groups external to the
organization are reviewed
with senior management
according to a documented
procedure.
Activity 4 Approved changes to Strength Strength Weakness
commitments that affect the
software project are
communicated to the members
of the software engineering
group and other software-
related groups.
Activity 5 The sizes of the software Strength Weakness Weakness
work products (or sizes of
the changes to the software
work products) are tracked,
and corrective actions are
taken as necessary.
Activity 6 The project's software Strength Strength Weakness
effort and costs are
tracked, and corrective
actions are taken as
necessary.
Activity 7 The project's critical Strength Weakness Weakness
computer resources are
tracked, and corrective
actions are taken as
necessary.
Activity 8 The project's software Strength Weakness Strength
schedule is tracked, and
corrective actions are taken
as necessary.
Activity 9 Software engineering Strength Weakness Weakness
technical activities are
tracked, and corrective
actions are taken as
necessary.
Activity 10 The software risks Strength Weakness Weakness
associated with cost,
resource, schedule, and
technical aspects of the
project are tracked.
Activity 11 Actual measurement data and Strength Weakness Weakness
replanning data for the
software project are
recorded.
Activity 12 The software engineering Strength Weakness Weakness
group conducts periodic
internal reviews to track
technical progress, plans,
performance, and issues
against the software
development plan.
Activity 13 Formal reviews to address Strength Weakness Weakness
the accomplishments and
results of the software
project are conducted at
selected project milestones
according to a documented
procedure.
Measurement Measurements are made and Strength Weakness Weakness
1 used to determine the status
of the software tracking and
oversight activities.
Verification The activities for software Strength Weakness Weakness
1 project tracking and
oversight are reviewed with
senior management on a
periodic basis.
Verification The activities for software Strength Weakness Weakness
2 project tracking and
oversight are reviewed with
the project manager on both
a periodic and event-driven
basis.
Verification The software quality Weakness Weakness Weakness
3 assurance group reviews and/
or audits the activities and
work products for software
project tracking and
oversight and reports the
results.
--------------------------------------------------------------------------------
Table 4.2
Software Project Tracking and Oversight
Findings for NCAP 0.1
National Customs Automation Program 0.1
------------------------------------------------------------------
Key practice Finding Rating
------------ -------------------------- -------------------------- ----------
Commitment 1 A project software manager The project software Strength
is designated to be manager is designated to
responsible for the be responsible for the
project's software project's software
activities and results. activities and results.
Commitment 2 The project follows a There is no written Weakness
written organizational organizational policy for
policy for managing the managing the software
software project. project.
Ability 1 A software development An approved and documented Strength
plan for the software software development plan
project is documented and is contained in the
approved. project plan.
Ability 2 The project software The project software Strength
manager explicitly assigns manager explicitly assigns
responsibility for responsibility for
software work products and software work products and
activities. activities.
Ability 3 Adequate resources and Adequate resources and Strength
funding are provided for funding are provided for
tracking the software tracking the software
project. project.
Ability 4 The software managers are The software managers are Weakness
trained in managing the not trained in managing
technical and personnel the technical and
aspects of the software personnel aspects of the
project. software project.
Ability 5 First-line software First-line software Strength
managers receive managers receive
orientation in the orientation in the
technical aspects of the technical aspects of the
software project. software project
Activity 1 A documented software A documented software Strength
development plan is used development plan is used
for tracking the software for tracking the software
activities and activities and
communicating status. communicating status.
Activity 2 The project's software No documented procedure Weakness
development plan is exists for revising the
revised according to a software development plan.
documented procedure.
Activity 3 Software project Software project Weakness
commitments and changes to commitments and changes to
commitments made to commitments made to
individuals and groups individuals and groups
external to the external to the
organization are reviewed organization are not
with senior management reviewed with senior
according to a documented management. Also, there is
procedure. no documented procedure
for such reviews.
Activity 4 Approved changes to Approved changes to Strength
commitments that affect commitments that affect
the software project are the software project are
communicated to the communicated to the
members of the software members of the software
engineering group and engineering group and
other software-related other software-related
groups. groups through weekly
staff meetings.
Activity 5 The sizes of the software The sizes of the software Strength
work products (or sizes of work products (or sizes of
the changes to the the changes to the
software work products) software work products)
are tracked, and are tracked; however, at
corrective actions are the time of our
taken as necessary. evaluation, no corrective
actions were needed.
Activity 6 The project's software The project's software Strength
effort and costs are effort and costs are
tracked, and corrective tracked; however, at the
actions are taken as time of our evaluation, no
necessary. corrective actions were
needed.
Activity 7 The project's critical The project's critical Strength
computer resources are computer resources are
tracked, and corrective tracked; however, at the
actions are taken as time of our evaluation, no
necessary. corrective actions were
needed.
Activity 8 The project's software The project's software Strength
schedule is tracked, and schedule is tracked;
corrective actions are however, at the time of
taken as necessary. our evaluation, no
corrective actions were
needed.
Activity 9 Software engineering Software engineering Strength
technical activities are technical activities are
tracked, and corrective tracked, and corrective
actions are taken as actions are taken as
necessary. necessary.
Activity 10 The software risks The software risks Strength
associated with cost, associated with cost,
resource, schedule, and resource, schedule, and
technical aspects of the technical aspects of the
project are tracked. project are tracked.
Activity 11 Actual measurement data Actual measurement data Strength
and replanning data for and replanning data for
the software project are the software project are
recorded. recorded.
Activity 12 The software engineering The software engineering Strength
group conducts periodic group conducts periodic
internal reviews to track internal reviews to track
technical progress, plans, technical progress, plans,
performance, and issues performance, and issues
against the software against the software
development plan. development plan.
Activity 13 Formal reviews to address Formal reviews to address Strength
the accomplishments and the accomplishments and
results of the software results of the software
project are conducted at project are conducted at
selected project selected project
milestones according to a milestones according to a
documented procedure. documented procedure.
Measurement Measurements are made and Measurements are made and Strength
1 used to determine the used to determine the
status of the software status of the software
tracking and oversight tracking and oversight
activities. activities.
Verification The activities for The activities for Strength
1 software project tracking software project tracking
and oversight are reviewed and oversight are reviewed
with senior management on with senior management on
a periodic basis. a weekly basis.
Verification The activities for The activities for Strength
2 software project tracking software project tracking
and oversight are reviewed and oversight are reviewed
with the project manager with the project manager
on both a periodic and on both a periodic and
event-driven basis. event-driven basis.
Verification The software quality No software quality Weakness
3 assurance group reviews assurance group exists;
and/or audits the therefore, there are no
activities and work reviews and/or audits of
products for software the activities and work
project tracking and products for software
oversight and reports the project tracking and
results. oversight.
--------------------------------------------------------------------------------
Table 4.3
Software Project Tracking and Oversight
Findings for AES
Automated Export System
------------------------------------------------------------------
Key practice Finding Rating
------------ -------------------------- -------------------------- ----------
Commitment 1 A project software manager A project software manager Strength
is designated to be is designated to be
responsible for the responsible for the
project's software project's software
activities and results. activities and results.
Commitment 2 The project follows a There is no written Weakness
written organizational organizational policy for
policy for managing the managing the software
software project. project.
Ability 1 A software development A software development Observatio
plan for the software plan for the project n
project is documented and exists. However, this plan
approved. has not been approved.
Ability 2 The project software The project software Weakness
manager explicitly assigns manager does not
responsibility for explicitly assign
software work products and responsibility for
activities. software work products and
activities.
Ability 3 Adequate resources and Adequate resources and Weakness
funding are provided for funding are not provided
tracking the software for tracking the software
project. project.
Ability 4 The software managers are The software managers are Strength
trained in managing the trained in managing the
technical and personnel technical and personnel
aspects of the software aspects of the software
project. projects through training
given at off-site retreats
and guidance provided in
the AES users' guide.
Ability 5 First-line software First-line software Strength
managers receive managers receive
orientation in the orientation in the
technical aspects of the technical aspects of the
software project. software project.
Activity 1 A documented software A documented software Strength
development plan is used development plan is used
for tracking the software for tracking the software
activities and activities and
communicating status. communicating status.
Activity 2 The project's software No documented procedure Weakness
development plan is exists for revising the
revised according to a software development plan.
documented procedure.
Activity 3 Software project Software project Observatio
commitments and changes to commitments and changes to n
commitments made to commitments made to
individuals and groups individuals and groups
external to the external to the
organization are reviewed organization are reviewed
with senior management in periodic meetings.
according to a documented However, there is no
procedure. documented procedure for
these reviews.
Activity 4 Approved changes to Approved changes to Strength
commitments that affect commitments that affect
the software project are the software project are
communicated to the communicated to the
members of the software members of the software
engineering group and engineering group and
other software-related other software-related
groups. groups through e-mail and
weekly status meetings.
Activity 5 The sizes of the software The sizes of the software Weakness
work products (or sizes of work products (or sizes of
the changes to the the changes to the
software work products) software work products)
are tracked, and are not tracked.
corrective actions are
taken as necessary.
Activity 6 The project's software The project's software Strength
effort and costs are effort and costs are
tracked, and corrective tracked, and corrective
actions are taken as action is taken as
necessary. necessary.
Activity 7 The project's critical The project's critical Weakness
computer resources are computer resources are not
tracked, and corrective tracked.
actions are taken as
necessary.
Activity 8 The project's software The project's software Weakness
schedule is tracked, and schedule is not tracked.
corrective actions are
taken as necessary.
Activity 9 Software engineering Software engineering Weakness
technical activities are technical activities are
tracked, and corrective not tracked.
actions are taken as
necessary.
Activity 10 The software risks The software risks Weakness
associated with cost, associated with cost,
resource, schedule, and resource, schedule, and
technical aspects of the technical aspects of the
project are tracked. project are not tracked.
Activity 11 Actual measurement data Actual measurement data Weakness
and replanning data for and replanning data for
the software project are the software project are
recorded. not recorded.
Activity 12 The software engineering The software engineering Weakness
group conducts periodic group does not conduct
internal reviews to track periodic internal reviews
technical progress, plans, to track technical
performance, and issues progress, plans,
against the software performance, and issues
development plan. against the software
development plan.
Activity 13 Formal reviews to address No documented procedure Weakness
the accomplishments and exists for conducting
results of the software formal reviews to address
project are conducted at the accomplishments and
selected project results of the software
milestones according to a project at selected
documented procedure. project milestones.
Measurement Measurements are made and Measurements are not made Weakness
1 used to determine the and used to determine the
status of the software status of software
tracking and oversight tracking and oversight
activities. activities.
Verification The activities for The activities for Weakness
1 software project tracking software project tracking
and oversight are reviewed and oversight are not
with senior management on reviewed with senior
a periodic basis. management on a periodic
basis.
Verification The activities for The activities for Weakness
2 software project tracking software project tracking
and oversight are reviewed and oversight are not
with the project manager reviewed with the project
on both a periodic and manager.
event-driven basis.
Verification The software quality The software quality Weakness
3 assurance group reviews assurance group does not
and/or audits the review and/or audit the
activities and work activities and work
products for software products for software
project tracking and project tracking and
oversight and reports the oversight and report the
results. results.
--------------------------------------------------------------------------------
Table 4.4
Software Project Tracking and Oversight
Findings for Administrative Security
System
Administrative Security System
------------------------------------------------------------------
Key practice Finding Rating
------------ -------------------------- -------------------------- ----------
Commitment 1 A project software manager A project software manager Strength
is designated to be is designated to be
responsible for the responsible for the
project's software project's software
activities and results. activities and results.
Commitment 2 The project follows a There is no organizational Weakness
written organizational policy for managing the
policy for managing the software project.
software project.
Ability 1 A software development The project plan contains Observatio
plan for the software the software development n
project is documented and plan; however, this plan
approved. has not been approved.
Ability 2 The project software The project software Weakness
manager explicitly assigns manager does not
responsibility for explicitly assign
software work products and responsibility for
activities. software work products and
activities.
Ability 3 Adequate resources and Adequate resources and Weakness
funding are provided for funding are not provided
tracking the software for tracking the software
project. project.
Ability 4 The software managers are The software managers are Weakness
trained in managing the not trained in managing
technical and personnel the technical and
aspects of the software personnel aspects of the
project. software project.
Ability 5 First-line software First-line software Weakness
managers receive managers did not receive
orientation in the orientation on the
technical aspects of the technical aspects of the
software project. software project.
Activity 1 A documented software A documented software Strength
development plan is used development plan is used
for tracking the software for tracking the software
activities and activities and
communicating status. communicating status.
Activity 2 The project's software No documented procedure Weakness
development plan is exists for revising the
revised according to a software development plan.
documented procedure.
Activity 3 Software project Software project Observatio
commitments and changes to commitments and changes to n
commitments made to commitments made to
individuals and groups individuals and groups
external to the external to the
organization are reviewed organization are reviewed
with senior management in periodic meetings.
according to a documented However, there is no
procedure. documented procedure for
these reviews.
Activity 4 Approved changes to There is no evidence to Weakness
commitments that affect show that changes to
the software project are commitments are either
communicated to the approved by project
members of the software managers or communicated
engineering group and to the software engineers
other software-related and other software-
groups. related groups.
Activity 5 The sizes of the software The sizes of the software Weakness
work products (or sizes of work products (or sizes of
the changes to the the changes to the
software work products) software work products)
are tracked, and are not tracked.
corrective actions are
taken as necessary.
Activity 6 The project's software The project's software Weakness
effort and costs are effort and costs are not
tracked, and corrective tracked.
actions are taken as
necessary.
Activity 7 The project's critical The project's critical Weakness
computer resources are computer resources are not
tracked, and corrective tracked.
actions are taken as
necessary.
Activity 8 The project's software The project's software Strength
schedule is tracked, and schedule is tracked, and
corrective actions are corrective actions are
taken as necessary. taken as necessary.
Activity 9 Software engineering Software engineering Weakness
technical activities are technical activities are
tracked, and corrective not tracked.
actions are taken as
necessary.
Activity 10 The software risks The software risks Weakness
associated with cost, associated with cost,
resource, schedule, and resource, schedule, and
technical aspects of the technical aspects of the
project are tracked. project are not tracked.
Activity 11 Actual measurement data Actual measurement data Weakness
and replanning data for and replanning data for
the software project are the software project are
recorded. not recorded.
Activity 12 The software engineering The software engineering Weakness
group conducts periodic group does notconduct
internal reviews to track periodic internal reviews
technical progress, plans, to track technical
performance, and issues progress, plans,
against the software performance, and issues
development plan. against the software
development plan.
Activity 13 Formal reviews to address No documented procedure Weakness
the accomplishments and exists for conducting
results of the software formal reviews to address
project are conducted at the accomplishments and
selected project results of the software
milestones according to a project at selected
documented procedure. project milestones.
Measurement Measurements are made and Measurements are not made Weakness
1 used to determine the and used to determine the
status of the software status of software
tracking and oversight tracking and oversight
activities. activities.
Verification The activities for The activities for Weakness
1 software project tracking software project tracking
and oversight are reviewed and oversight are not
with senior management on reviewed with senior
a periodic basis. management on a periodic
basis.
Verification The activities for The activities for Weakness
2 software project tracking software project tracking
and oversight are reviewed and oversight are not
with the project manager reviewed with the project
on both a periodic and manager.
event-driven basis.
Verification The software quality No software quality Weakness
3 assurance group reviews assurance group exists;
and/or audits the therefore, there are no
activities and work reviews and/or audits of
products for software the activities and work
project tracking and products for software
oversight and reports the project tracking and
results. oversight.
--------------------------------------------------------------------------------
CONCLUSIONS
---------------------------------------------------------- Chapter 4:2
Despite several practice strengths in this KPA, the number and
significance of the practice weaknesses that we found mean that
Customs' current process for tracking and overseeing its projects is
not repeatable, thereby increasing the chances of its software
projects being late, costing more than expected, and not performing
as intended.
SOFTWARE QUALITY ASSURANCE
============================================================ Chapter 5
The purpose of software quality assurance is to independently review
and audit the software products and activities to verify that they
comply with the applicable procedures and standards and to provide
the software project and higher-level managers with the results of
these independent reviews and audits.
According to the SW-CMM, a repeatable software quality assurance
process, among other things, includes (1) preparing a software
quality assurance plan for the project according to a documented
procedure, (2) having a written organizational policy for
implementing software quality assurance, (3) conducting audits of
designated work processes and products to verify compliance, (4)
documenting deviations identified in the software activities and
software work products and handling them according to a documented
procedure, and (5) having experts independent of the software quality
assurance group periodically review the activities and work products
of the project's software quality assurance group.
CUSTOMS LACKS A SOFTWARE
QUALITY ASSURANCE PROCESS
---------------------------------------------------------- Chapter 5:1
All of the projects evaluated had extensive and significant software
quality assurance practice weaknesses. For example, two of the
projects did not have a software quality assurance plan; and none of
the projects (1) had a written organizational policy for implementing
software quality assurance, (2) conducted audits of designated work
products to verify compliance, (3) documented deviations identified
in the software activities and software work products and handled
them according to a documented procedure, or (4) had experts
independent of the software quality assurance group periodically
review the group's work products. In fact, only one of the projects,
AES, had any software quality assurance practice strengths, and these
strengths were limited to only a few practices. In this case, the
project had assigned responsibility for software quality assurance to
a single individual and, for example, a software quality assurance
plan had been drafted, although not according to a documented
procedure. This virtual absence of software quality assurance on
Customs' software projects increases greatly the risk of software
process and product standards not being met, which in turn increases
the risk of software not performing as intended, and costing more and
taking longer to develop than necessary.
Table 5.1 provides a comprehensive list of the three projects'
strengths, weaknesses, and observations for the software quality
assurance KPA. The specific findings supporting the practice ratings
cited in table 5.1 are in tables 5.2 through 5.4.
Table 5.1
Software Quality Assurance
Administrati
Key practice NCAP 0.1 AES ve
------------ ---------------------------- ---------- ---------- ------------
Commitment 1 The project follows a Weakness Weakness Weakness
written organizational
policy for implementing
software quality assurance
(SQA).
Ability 1 A group that is responsible Observatio Strength Weakness
for coordinating and n
implementing SQA for the
project (i.e., the SQA
group) exists.
Ability 2 Adequate resources and Weakness Weakness Weakness
funding are provided for
performing the SQA
activities.
Ability 3 Members of the SQA group are Weakness Strength Weakness
trained to perform their SQA
activities.
Ability 4 The members of the software Weakness Strength Weakness
project receive orientation
on the role,
responsibilities, authority,
and value of the SQA group.
Activity 1 A SQA plan is prepared for Weakness Observatio Weakness
the software project n
according to a documented
procedure.
Activity 2 The SQA group's activities Weakness Weakness Weakness
are performed in accordance
with the SQA plan.
Activity 3 The SQA group participates Weakness Weakness Weakness
in the preparation and
review of the project's
software development plan,
standards, and procedures.
Activity 4 The SQA group reviews the Weakness Weakness Weakness
software engineering
activities to verify
compliance.
Activity 5 The SQA group audits Weakness Weakness Weakness
designated software work
products to verify
compliance.
Activity 6 The SQA group periodically Weakness Weakness Weakness
reports the results of its
activities to the software
engineering group.
Activity 7 Deviations identified in the Weakness Weakness Weakness
software activities and
software work products are
documented and handled
according to a documented
procedure.
Activity 8 The SQA group conducts Weakness Weakness Weakness
periodic reviews of its
activities and findings with
the customer's SQA
personnel, as appropriate.
Measurement Measurements are made and Weakness Weakness Weakness
1 used to determine the cost
and schedule status of the
SQA activities.
Verification The SQA activities are Weakness Weakness Weakness
1 reviewed with senior
management on a periodic
basis.
Verification The SQA activities are Weakness Weakness Weakness
2 reviewed with the project
manager on both a periodic
and event-driven basis.
Verification Experts independent of the Weakness Weakness Weakness
3 SQA group periodically
review the activities and
software work products of
the project's SQA group.
--------------------------------------------------------------------------------
Table 5.2
Software Quality Assurance Findings for
NCAP 0.1
National Customs Automation Program 0.1
------------------------------------------------------------------
Key practice Finding Rating
------------ -------------------------- -------------------------- ----------
Commitment 1 The project follows a There is no written Weakness
written organizational organizational policy for
policy for implementing implementing SQA.
software quality assurance
(SQA).
Ability 1 A group that is Although there is no group Observatio
responsible for responsible for n
coordinating and coordinating and
implementing SQA for the implementing SQA for the
project (i.e., the SQA project, there are plans
group) exists. to establish one and
assign responsibility.
Ability 2 Adequate resources and There is no SQA group, and Weakness
funding are provided for no resources and funding
performing the SQA are provided for
activities. performing the SQA
activities.
Ability 3 Members of the SQA group There is no SQA group or Weakness
are trained to perform plan to provide SQA
their SQA activities. training.
Ability 4 The members of the Project staff do not Weakness
software project receive receive orientation on the
orientation on the role, role, responsibilities,
responsibilities, authority, and value of
authority, and value of the SQA group.
the SQA group.
Activity 1 The SQA plan is prepared There is no SQA plan or Weakness
for the software project documented procedure for
according to a documented preparing one.
procedure.
Activity 2 The SQA group's activities There is no SQA group. Weakness
are performed in
accordance with the SQA
plan.
Activity 3 The SQA group participates There is no SQA group. Weakness
in the preparation and
review of the project's
software development plan,
standards, and procedures.
Activity 4 The SQA group reviews the There is no SQA group. Weakness
software engineering
activities to verify
compliance.
Activity 5 The SQA group audits There is no SQA group. Weakness
designated software work
products to verify
compliance.
Activity 6 The SQA group periodically There is no SQA group. Weakness
reports the results of its
activities to the software
engineering group.
Activity 7 Deviations identified in Procedures for handling Weakness
the software activities deviations in testing
and software work products activities are documented.
are documented and handled However, procedures for
according to a documented handling deviations in
procedure. other software development
activities (such as
compliance with
organizational policy and
standards and adherence to
software development plan)
are not documented.
Activity 8 The SQA group conducts There is no SQA group. Weakness
periodic reviews of its
activities and findings
with the customer's SQA
personnel, as appropriate.
Measurement Measurements are made and There is no SQA group. Weakness
1 used to determine the cost
and schedule status of the
SQA activities.
Verification The SQA activities are There is no SQA group. Weakness
1 reviewed with senior
management on a periodic
basis.
Verification The SQA activities are There is no SQA group. Weakness
2 reviewed with the project
manager on both a periodic
and event-driven basis.
Verification Experts independent of the There is no SQA group. Weakness
3 SQA group periodically
review the activities and
software work products of
the project's SQA group.
--------------------------------------------------------------------------------
Table 5.3
Software Quality Assurance Findings for
AES
Automated Export System
------------------------------------------------------------------
Key practice Finding Rating
------------ -------------------------- -------------------------- ----------
Commitment 1 The project follows a There is no written Weakness
written organizational organizational policy for
policy for implementing implementing SQA.
software quality assurance
(SQA).
Ability 1 A group that is A single individual is Strength
responsible for responsible for
coordinating and coordinating and
implementing SQA for the implementing SQA for the
project (i.e., the SQA project.
group) exists.
Ability 2 Adequate resources and Adequate resources and Weakness
funding are provided for funding are not provided
performing the SQA for performing the SQA
activities. activities.
Ability 3 Members of the SQA group Training has been provided Strength
are trained to perform to the single individual
their SQA activities. responsible for SQA to
prepare him to perform his
activities.
Ability 4 The members of the Members of the software Strength
software project receive project are oriented on
orientation on the role, the role,
responsibilities, responsibilities,
authority, and value of authority, and value of
the SQA group. SQA. Briefings are held
during retreats.
Activity 1 A SQA plan is prepared for The SQA plan is not Observatio
the software project prepared according to a n
according to a documented documented procedure;
procedure. however, there is a draft
plan.
Activity 2 The SQA group's activities The SQA group's activities Weakness
are performed in are not performed in
accordance with the SQA accordance with the SQA
plan. plan.
Activity 3 The SQA group participates The individual responsible Weakness
in the preparation and for SQA does not
review of the project's participate in the
software development plan, preparation and review of
standards, and procedures. the project's software
development plan,
standards, and procedures.
Activity 4 The SQA group reviews the The individual responsible Weakness
software engineering for SQA does not review
activities to verify the software engineering
compliance. activities to verify
compliance.
Activity 5 The SQA group audits The individual responsible Weakness
designated software work for SQA does not audit
products to verify software work products to
compliance. verify compliance.
Activity 6 The SQA group periodically The individual responsible Weakness
reports the results of its for SQA periodically
activities to the software reports the results of
engineering group. testing activities to the
software engineering
group. Other activities,
such as the results of
reviews of work products
and audits, are not
reported.
Activity 7 Deviations identified in Procedures for handling Weakness
the software activities deviations in testing
and software work products activities are documented.
are documented and handled However, procedures for
according to a documented handling deviations in
procedure. other software development
activities (such as
compliance with
organizational policy and
standards and adherence to
the software development
plan) are not documented.
Activity 8 The SQA group conducts The SQA group does not Weakness
periodic reviews of its conduct periodic reviews
activities and findings of its activities and
with the customer's SQA findings with the
personnel, as appropriate. customer's SQA personnel,
as appropriate.
Measurement Measurements are made and Measurements are not made Weakness
1 used to determine the cost to determine the cost and
and schedule status of the schedule status of the SQA
SQA activities. activities.
Verification The SQA activities are Senior management is not Weakness
1 reviewed with senior made aware of the SQA
management on a periodic activities.
basis.
Verification The SQA activities are The project manager and Weakness
2 reviewed with the project team members are not made
manager on both a periodic aware of SQA activities.
and event-driven basis.
Verification Experts independent of the SQA's activities and Weakness
3 SQA group periodically software work products are
review the activities and not reviewed by experts
software work products of independent of the SQA
the project's SQA group. group.
--------------------------------------------------------------------------------
Table 5.4
Software Quality Assurance Findings for
Administrative Security System
Administrative Security System
------------------------------------------------------------------
Key practice Finding Rating
------------ -------------------------- -------------------------- ----------
Commitment 1 The project follows a There is no written Weakness
written organizational organizational policy for
policy for implementing implementing SQA.
software quality assurance
(SQA).
Ability 1 A group that is No group is responsible Weakness
responsible for for coordinating and
coordinating and implementing SQA for the
implementing SQA for the project.
project (i.e., the SQA
group) exists.
Ability 2 Adequate resources and Adequate resources and Weakness
funding are provided for funding are not provided
performing the SQA for performing the SQA
activities. activities.
Ability 3 Members of the SQA group There is no SQA group. Weakness
are trained to perform
their SQA activities.
Ability 4 The members of the Project staff do not Weakness
software project receive receive orientation on the
orientation on the role, role, responsibilities,
responsibilities, authority, and value of
authority, and value of the SQA group.
the SQA group.
Activity 1 A SQA plan is prepared for There is no documented Weakness
the software project procedure for preparing a
according to a documented SQA plan.
procedure.
Activity 2 The SQA group's activities There is no SQA group. Weakness
are performed in
accordance with the SQA
plan.
Activity 3 The SQA group participates There is no SQA group. Weakness
in the preparation and
review of the project's
software development plan,
standards, and procedures.
Activity 4 The SQA group reviews the There is no SQA group. Weakness
software engineering
activities to verify
compliance.
Activity 5 The SQA group audits There is no SQA group. Weakness
designated software work
products to verify
compliance.
Activity 6 The SQA group periodically There is no SQA group. Weakness
reports the results of its
activities to the software
engineering group.
Activity 7 Deviations identified in Procedures for handling Weakness
the software activities deviations in testing
and software work products activities are documented.
are documented and handled However, procedures for
according to a documented handling deviations in
procedure. other software development
activities (such as
compliance with
organizational policy and
standards and adherence to
the software development
plan) are not documented.
Activity 8 The SQA group conducts There is no SQA group. Weakness
periodic reviews of its
activities and findings
with the customer's SQA
personnel, as appropriate.
Measurement Measurements are made and There is no SQA group. Weakness
1 used to determine the cost
and schedule status of the
SQA activities.
Verification The SQA activities are The SQA activities are not Weakness
1 reviewed with senior reviewed with senior
management on a periodic management on a periodic
basis. basis.
Verification The SQA activities are The SQA activities are not Weakness
2 reviewed with the project reviewed with the project
manager on both a periodic manager on both a periodic
and event-driven basis. and event-driven basis.
Verification Experts independent of the There is no SQA group. Weakness
3 SQA group periodically
review the activities and
software work products of
the project's SQA group.
--------------------------------------------------------------------------------
CONCLUSIONS
---------------------------------------------------------- Chapter 5:2
Customs' software quality assurance process has many weaknesses and
is, therefore, undefined and undisciplined. As a result, Customs
cannot provide management with independent information about
adherence to software process and product standards. To develop and
maintain software effectively, Customs must adopt a structured and
rigorous approach to software quality assurance.
SOFTWARE CONFIGURATION MANAGEMENT
============================================================ Chapter 6
The purpose of software configuration management is to establish and
maintain the integrity of the products of the software project
throughout the project's software life- cycle. Software
configuration management involves establishing product baselines and
systematically controlling changes to them.
According to the SW-CMM, a repeatable software configuration
management process, among other things, includes (1) preparing a
software configuration management plan according to a documented
procedure, (2) establishing a configuration management library system
as a repository for the software baselines, (3) identifying software
work products to be placed under configuration management, (4)
controlling the release of products from the software baseline
library according to a documented procedure, (5) following a written
organizational policy for implementing software configuration
management, (6) recording the status of configuration items/units
according to a documented procedure, (7) making and using
measurements to determine the status of the software configuration
management activities, and (8) reviewing software configuration
management activities with senior management on a periodic basis.
CUSTOMS' SOFTWARE CONFIGURATION
MANAGEMENT PROCESS IS IMMATURE
---------------------------------------------------------- Chapter 6:1
Customs' processes for software configuration management show
strengths in several activities. For example, all three projects had
developed software configuration management plans according to a
documented procedure. Also, two of the projects (NCAP 0.1 and AES)
established configuration management library systems as repositories
for the software baselines, identified software work products to be
placed under configuration management, and controlled the release of
products from the software baseline library according to a documented
procedure.
However, the projects had many practice weakness that collectively
jeopardize Customs' ability to maintain the integrity of the
projects' software products. For example, none of the projects had a
written organizational policy for implementing software configuration
management, and none had documented procedures for recording the
status of configuration items (e.g., code, documents). Moreover,
none of the projects made or used measurements to determine the
status of the software configuration management activities, or
reviewed software configuration management activities with senior
management on a periodic basis.
Table 6.1 provides a comprehensive list of the three projects'
strengths and weaknesses for the software configuration management
KPA. The specific findings supporting the practice ratings cited in
table 6.1 are in tables 6.2 through 6.4.
Figure 6.1
Software Configuration Management
Administrati
Key practice NCAP 0.1 AES ve
------------ ---------------------------- ---------- ---------- ------------
Commitment 1 The project follows a Weakness Weakness Weakness
written organizational
policy for implementing
software configuration
management (SCM).
Ability 1 A board having the authority Weakness Strength Weakness
for managing the project's
software baselines (i.e., a
software configuration
control board) exists or is
established.
Ability 2 A group that is responsible Weakness Strength Strength
for coordinating and
implementing SCM for the
project (i.e., the SCM
group) exists.
Ability 3 Adequate resources and Weakness Weakness Weakness
funding are provided for
performing the SCM
activities.
Ability 4 Members of the SCM group are Weakness Strength Weakness
trained in the objectives,
procedures, and methods for
performing their SCM
activities.
Ability 5 Members of the software Weakness Strength Weakness
engineering group and other
software-related groups are
trained to perform their SCM
activities.
Activity 1 A SCM plan is prepared for Strength Strength Strength
each software project
according to a documented
procedure.
Activity 2 A documented and approved Weakness Weakness Weakness
SCM plan is used as the
basis for performing the SCM
activities.
Activity 3 A configuration management Strength Strength Weakness
library system is
established as a repository
for the software baselines.
Activity 4 The software work products Strength Strength Weakness
to be placed under
configuration management are
identified.
Activity 5 Change requests and problem Strength Weakness Weakness
reports for all
configuration items/risks
are initiated, recorded,
reviewed, approved, and
tracked according to a
documented procedure.
Activity 6 Changes to baselines are Strength Weakness Weakness
controlled according to a
documented procedure.
Activity 7 Products from the software Strength Strength Weakness
baseline library are created
and their release is
controlled according to a
documented procedure.
Activity 8 The status of configuration Weakness Weakness Weakness
items/units is recorded
according to a documented
procedure.
Activity 9 Standard reports documenting Weakness Weakness Weakness
the SCM activities and the
contents of the software
baseline are developed and
made available to affected
groups and individuals.
Activity 10 Software baseline audits are Weakness Weakness Weakness
conducted according to a
documented procedure.
Measurement Measurements are made and Weakness Weakness Weakness
1 used to determine the status
of the SCM activities.
Verification The SCM activities are Weakness Weakness Weakness
1 reviewed with senior
management on a periodic
basis.
Verification The SCM activities are Weakness Strength Weakness
2 reviewed with the project
manager on both a periodic
and event-driven basis.
Verification The SCM group periodically Weakness Weakness Weakness
3 audits software baselines to
verify that they conform to
the documentation that
defines them.
Verification The software quality Weakness Weakness Weakness
4 assurance group reviews and/
or audits the activities and
work products for SCM and
reports the results.
--------------------------------------------------------------------------------
Table 6.2
Software Configuration Management
Findings for NCAP 0.1
National Customs Automation Program 0.1
------------------------------------------------------------------
Key practice Finding Rating
------------ -------------------------- -------------------------- ----------
Commitment 1 The project follows a The project has no Weakness
written organizational organizational policy for
policy for implementing implementing SCM.
software configuration
management (SCM).
Ability 1 A board having the The project does not have Weakness
authority for managing the a SCM board with authority
project's software for managing software
baselines (i.e., a baselines.
software configuration
control board) exists or
is established.
Ability 2 A group that is The project does not have Weakness
responsible for a group that is
coordinating and responsible for
implementing SCM for the coordinating and
project (i.e., the SCM implementing SCM
group) exists. functions.
Ability 3 Adequate resources and Adequate resources and Weakness
funding are provided for funding are not provided
performing the SCM for performing the SCM
activities. activities.
Ability 4 Members of the SCM group There is no SCM group. Weakness
are trained in the
objectives, procedures,
and methods for performing
their SCM activities.
Ability 5 Members of the software Members of the software Weakness
engineering group and engineering group and
other software-related other software-related
groups are trained to groups are not trained to
perform their SCM perform their SCM
activities. activities
Activity 1 A SCM plan is prepared for A SCM plan was prepared Strength
each software project according to procedures
according to a documented documented in the October
procedure. 1996 SDLC.
Activity 2 A documented and approved The documented and Weakness
SCM plan is used as the approved SCM plan is used
basis for performing the as the basis for
SCM activities. performing code control;
however, the plan is not
used as a basis for doing
SCM on software
documentation and other
software engineering
products such as cost
estimates and schedules.
Activity 3 A configuration management A configuration management Strength
library system is library system is
established as a established as a
repository for the repository for the
software baselines. software baselines.
Activity 4 The software work products The software work products Strength
to be placed under to be placed under
configuration management configuration management
are identified. are identified.
Activity 5 Change requests and Change requests and Strength
problem reports for all problem items/units are
configuration items/risks initiated, recorded,
are initiated, recorded, reviewed, approved, and
reviewed, approved, and tracked according to a
tracked according to a procedure documented in
documented procedure. the SCM plan.
Activity 6 Changes to baselines are Changes to baselines are Strength
controlled according to a controlled according to a
documented procedure. documented procedure in
the SCM plan.
Activity 7 Products from the software Products from the software Strength
baseline library are baseline library are
created and their release created and their release
is controlled according to is controlled according to
a documented procedure. the SCM plan.
Activity 8 The status of The status of SCM items/ Weakness
configuration items/units units are not
is recorded according to a recorded according to a
documented procedure. documented procedure.
Activity 9 Standard reports Standard reports Weakness
documenting the SCM documenting the SCM
activities and the activities and contents of
contents of the software the software baseline are
baseline are developed and not developed.
made available to affected
groups and individuals.
Activity 10 Software baseline audits Software baseline audits Weakness
are conducted according to are not conducted.
a documented procedure.
Measurement Measurements are made and No measurements are taken Weakness
1 used to determine the to determine the status of
status of the SCM SCM activities.
activities.
Verification The SCM activities are The SCM activities are not Weakness
1 reviewed with senior reviewed with senior
management on a periodic management on a periodic
basis. basis.
Verification The SCM activities are The SCM activities are not Weakness
2 reviewed with the project reviewed with the project
manager on both a periodic manager on both a periodic
and event-driven basis. and event-driven basis.
Verification The SCM group periodically No SCM audits are done to Weakness
3 audits software baselines verify that software
to verify that they baselines conform to the
conform to the documentation that defines
documentation that defines them.
them.
Verification The software quality There is no quality Weakness
4 assurance group reviews assurance group;
and/or audits the therefore, no one reviews
activities and work and/or audits the
products for SCM and activities and work
reports the results. products for SCM
activities performed.
--------------------------------------------------------------------------------
Table 6.3
Software Configuration Management
Findings for AES
Automated Export System
------------------------------------------------------------------
Key practice Finding Rating
------------ ------------------------- ------------------------- ------------
Commitment 1 The project follows a The project does not have Weakness
written organizational a written organizational
policy for implementing policy for SCM.
software configuration
management (SCM).
Ability 1 A board having the AES has a Configuration Strength
authority for managing Control Board. This board
the project's software has the authority for
baselines (i.e., a managing the project's
software configuration software baselines. The
control board) exists or board consists of program
is established. support and user
personnel.
Ability 2 A group that is A group that is Strength
responsible for responsible for
coordinating and coordinating and
implementing SCM for the implementing SCM for the
project (i.e., the SCM project (i.e., the SCM
group) exists. group) exists.
Ability 3 Adequate resources and Adequate resources and Weakness
funding are provided for funding are not provided
performing the SCM for performing the SCM
activities. activities.
Ability 4 Members of the SCM group Members of the SCM group Strength
are trained in the have extensive experience
objectives, procedures, in the objectives,
and methods for procedures, and methods
performing their SCM for performing their SCM
activities. activities and receive
on-the-job training.
Ability 5 Members of the software Members of the software Strength
engineering group and engineering group and
other software-related other software-related
groups are trained to groups have been briefed
perform their SCM on performing their SCM
activities. duties at periodic
retreat meetings.
Activity 1 A SCM plan is prepared A SCM plan was prepared Strength
for each software project according to procedures
according to a documented documented in the October
procedure. 1996 SDLC.
Activity 2 A documented and approved The documented and Weakness
SCM plan is used as the approved SCM plan is used
basis for performing the as the basis for
SCM activities. performing code control;
however, the plan is not
used as a basis for doing
SCM on software
documentation and other
software engineering
products such as cost
estimates and schedules.
Activity 3 A configuration The AES library is used Strength
management library system as a repository for
is established as a software baselines.
repository for the
software baselines.
Activity 4 The software work Software work products to Strength
products to be placed be placed under SCM are
under configuration identified in the SCM
management are plan.
identified.
Activity 5 Change requests and Change requests and Weakness
problem reports for all problem reports for all
configuration items/ configuration items are
units are initiated, not initiated, recorded,
recorded, reviewed, reviewed, approved, and
approved, and tracked tracked according to a
according to a documented documented procedure.
procedure.
Activity 6 Changes to baselines are The procedures for Weakness
controlled according to a controlling changes to
documented procedure. baselines are documented
in the SCM and SQA plans.
However, there was no
evidence provided to show
that these procedures
were being followed.
Activity 7 Products from the Products from the Strength
software baseline library baseline library are
are created and their created, and releases are
release is controlled controlled according to
according to a documented procedures in the SCM
procedure. plan.
Activity 8 The status of The status of Weakness
configuration items/ configuration items/
units is recorded units is not recorded
according to a documented according to a documented
procedure. procedure.
Activity 9 Standard reports No reports documenting Weakness
documenting the SCM SCM activities and
activities and the contents of the software
contents of the software baseline are made
baseline are developed available to affected
and made available to groups and individuals.
affected groups and
individuals.
Activity 10 Software baseline audits Software baseline audits Weakness
are conducted according are not conducted.
to a documented
procedure.
Measurement Measurements are made and Measurements are not made Weakness
1 used to determine the and used to determine the
status of the SCM status of the SCM
activities. activities.
Verification The SCM activities are The SCM activities are Weakness
1 reviewed with senior not reviewed with senior
management on a periodic management on a periodic
basis. basis.
Verification The SCM activities are Project management is Strength
2 reviewed with the project updated on SCM activities
manager on both a on a weekly basis and bi-
periodic and event- monthly at the retreat
driven basis. meetings, and as events
warrant.
Verification The SCM group The SCM group does not Weakness
3 periodically audits verify that the software
software baselines to baseline conforms to the
verify that they conform documentation that
to the documentation that defines it.
defines them.
Verification The software quality The SQA group does not Weakness
4 assurance group reviews perform reviews and/or
and/or audits the audits of the activities
activities and work and work products for SCM
products for SCM and and does not report the
reports the results. results.
--------------------------------------------------------------------------------
Table 6.4
Software Configuration Management
Findings for Administrative Security
System
Administrative Security System
------------------------------------------------------------------
Key practice Finding Rating
------------ -------------------------- -------------------------- ----------
Commitment 1 The project follows a The project does not have Weakness
written organizational a written organizational
policy for implementing policy for implementing
software configuration software configuration
management (SCM). management.
Ability 1 A board having the The project does not have Weakness
authority for managing the a board (software
project's software configuration control
baselines (i.e., a board) with the authority
software configuration for managing the software
control board) exists or baselines.
is established.
Ability 2 A group that is The project manager is Strength
responsible for responsible for
coordinating and coordinating and
implementing SCM for the implementing SCM
project (i.e., the SCM functions.
group) exists.
Ability 3 Adequate resources and Adequate resources and Weakness
funding are provided for funding are not provided
performing the SCM for performing the SCM
activities. activities.
Ability 4 Members of the SCM group There is no evidence that Weakness
are trained in the the personnel assigned SCM
objectives, procedures, functions on the project
and methods for performing are trained in objectives,
their SCM activities. procedures, and methods
for doing SCM.
Ability 5 Members of the software Members of the software Weakness
engineering group and engineering group are not
other software-related trained to perform their
groups are trained to SCM functions.
perform their SCM
activities.
Activity 1 A SCM plan is prepared for A SCM plan was prepared Strength
each software project according to procedures
according to a documented documented in the October
procedure. 1996 SDLC.
Activity 2 A documented and approved Although there is a SCM Weakness
SCM plan is used as the plan for the project,
basis for performing the there is no evidence that
SCM activities. it is used to perform SCM
functions.
Activity 3 A configuration management A configuration management Weakness
library system is library system is
established as a established and used as a
repository for the repository for software
software baselines. baselines for source code.
However, other items in
the software baseline
(such as plans and
schedules) are not
identified or controlled.
Activity 4 The software work products Software work products to Weakness
to be placed under be placed under
configuration management configuration management
are identified. (other than source code)
are not identified.
Activity 5 Change requests and The Automated Request For Weakness
problem reports for all Service system handles
configuration items/risks change requests and
are initiated, recorded, problem reports. However,
reviewed, approved, and there is no documented
tracked according to a procedure for its use and
documented procedure. there was no evidence that
the system was being used
to initiate, record,
review, approve, or track
change requests and
problem reports.
Activity 6 Changes to baselines are There is no documented Weakness
controlled according to a procedure to control
documented procedure. changes to baselines.
Activity 7 Products from the software There is no documented Weakness
baseline library are procedure that defines how
created and their release products from the software
is controlled according to baseline library should be
a documented procedure. created and released.
Activity 8 The status of The status of SCM items/ Weakness
configuration items/units units is not recorded
is recorded according to a according to a documented
documented procedure. procedure.
Activity 9 Standard reports Affected groups and Weakness
documenting the SCM individuals are not
activities and the notified of the SCM
contents of the software activities or informed of
baseline are developed and the contents of software
made available to affected baselines.
groups and individuals.
Activity 10 Software baseline audits Software baseline audits Weakness
are conducted according to are not conducted.
a documented procedure.
Measurement Measurements are made and No measurements are made Weakness
1 used to determine the to determine the status of
status of the SCM the SCM activities.
activities.
Verification The SCM activities are The SCM activities are not Weakness
1 reviewed with senior reviewed with senior
management on a periodic management on a periodic
basis. basis.
Verification The SCM activities are Project managers are not Weakness
2 reviewed with the project updated on SCM activities
manager on both a periodic on a periodic basis.
and event-driven basis.
Verification The SCM group periodically The SCM does not Weakness
3 audits software baselines periodically audit the
to verify that they software baselines.
conform to the
documentation that defines
them.
Verification The software quality There is no quality Weakness
4 assurance group reviews assurance group.
and/or audits the
activities and work
products for SCM and
reports the results.
--------------------------------------------------------------------------------
CONCLUSIONS
---------------------------------------------------------- Chapter 6:2
Customs has many configuration management process weaknesses, and
thus its capability to establish and maintain the integrity of the
wide range of software products is nonrepeatable and ineffective.
Without a mature configuration management process, Customs can lose
control of the current software product baseline, potentially
producing and using inconsistent product versions and creating
operational problems.
CUSTOMS LACKS A SOFTWARE PROCESS
IMPROVEMENT PROGRAM
============================================================ Chapter 7
To consistently develop software with specified functionality on time
and within budget, Customs must improve its software development
processes. According to SEI, an effective process improvement
program includes (1) establishing a process improvement management
structure, (2) developing a process improvement plan, (3) determining
the organization's baseline capability and using this as a basis for
targeting process initiatives, and (4) dedicating adequate resources
for implementing the plan.
Although it has attempted in the past to initiate and sustain process
improvement activities, these activities were terminated without
having improved Customs processes. Currently, Customs has no
software process improvement program.
SEI HAS DEFINED A FIVE-PHASE
MODEL FOR SOFTWARE PROCESS
IMPROVEMENT
---------------------------------------------------------- Chapter 7:1
In 1996, SEI published a software process improvement model, called
IDEAL.\1 This model has five phases: Initiating, Diagnosing,
Establishing, Acting, and Leveraging-- IDEAL. Each of the phases is
summarized below.
-- Initiating phase: During this phase, an organization
establishes the management structure of the process improvement
program, defines and assigns roles and responsibilities,
allocates initial resources, develops a plan to guide the
organization through the first three phases of the program, and
obtains management approval and funding for the program. Two
key organizational components of the program management
structure established during this phase are a management
steering group and a software engineering process group (SEPG).
Responsibility for this phase rests with senior management.
-- Diagnosing phase: During this phase, the SEPG appraises the
current level of software process maturity to establish a
baseline of the organization's process capability, and
identifies any ongoing process improvement initiatives. The
SEPG then uses the baseline to identify weaknesses and target
process improvement activities. It also compares these targeted
activities with any ongoing process improvement activities and
reconciles any differences. Responsibility for this phase rests
primarily with line managers and practitioners.
-- Establishing phase: During this phase, the SEPG prioritizes the
software process improvement activities and develops strategies
for pursuing them. It then develops a process improvement
action plan that details the activities and strategies and
includes measurable goals for the activities and metrics for
monitoring progress against the goals. Also during this phase,
the resources needed to implement the plan are committed and
training is provided for SEPG's technical working groups, who
will be responsible for developing and testing new or improved
processes. Responsibility for this phase resides primarily with
line managers and practitioners.
-- Acting phase: In this phase, the work groups create and
evaluate new and improved processes. Evaluation of the
processes is based on pilot tests that are formally planned and
executed. If the pilots are successful, the work groups develop
plans for organization-wide adoption and institutionalization,
and once approved, execute them. Responsibility for this phase
resides primarily with line managers and practitioners.
-- Leveraging phase: During this phase, results and lessons
learned from earlier phases are assessed and applied, as
appropriate, to enhance the process improvement program's
structure and plans. Responsibility for this phase rests
primarily with senior management.
--------------------
\1 IDEAL: A User's Guide for Software Process Improvement
(CMU/SEI-96-HB-001).
CUSTOMS DOES NOT CURRENTLY HAVE
A SOFTWARE PROCESS IMPROVEMENT
PROGRAM
---------------------------------------------------------- Chapter 7:2
In 1996, Customs initiated some limited software process improvement
activities. Specifically, it hired a contractor to develop a process
improvement plan, which was completed in September 1996. According
to the plan, Customs was to reach CMM level 2 process maturity (the
repeatable level) by 1998 and CMM level 3 (the defined level) by
2002. Customs began limited implementation of the plan in May of
1997, when it established process improvement teams for two
KPAs--software project planning and project tracking and oversight.
Generally, the teams were tasked with defining, implementing, and
maintaining CMM-based processes for their respective KPAs. Customs
did not staff or fund any other KPA improvement activities at this
time. In August 1997, Customs discontinued all process improvement
activities. Customs officials stated that this decision was based on
the need to focus staff and resources on the agency's Year 2000
conversion program.
Currently, Customs does not have a software development process
improvement program, and it has not taken steps to initiate one.
Although it has assigned two people part-time to process improvement,
it has not assigned organizational responsibility and authority,
established a program management structure, developed a plan of
action, and committed resources needed (trained staff and funding) to
execute the plan.
CONCLUSIONS
---------------------------------------------------------- Chapter 7:3
Customs does not have an effective software development process
improvement program. As a result, it cannot expect to improve its
immature software development processes.
OVERALL CONCLUSIONS,
RECOMMENDATIONS, AND AGENCY
COMMENTS
============================================================ Chapter 8
Customs develops and maintains software for systems that are critical
to its ability to fulfill its mission. However, its software
development processes are ad hoc and sometimes chaotic, and are not
repeatable even on a project-by-project basis. As a result, Customs'
success or failure in developing software depends largely on specific
individuals, rather than on well-defined and disciplined software
management practices. This greatly reduces the probability that its
software projects, whether new developments or maintenance of
existing software, will consistently perform as intended and be
delivered on schedule and within budget. For Customs software
projects to mature beyond this initial level, the agency must
implement basic management controls and instill self-discipline in
its software projects.
Customs acknowledges the importance of software process maturity and
the need to improve its software development processes. However, it
does not have a program for improving its software development
processes and has not begun to establish one. Until it does, Customs
has no assurance that its large investment in software development
and maintenance will produce systems that perform needed functions,
on time, and within budget.
RECOMMENDATIONS
---------------------------------------------------------- Chapter 8:1
We recommend that, after ensuring that its mission-critical systems
are Year 2000 compliant but before investing in major software
development efforts like ACE, the Commissioner of Customs direct the
Chief Information Officer to
-- assign responsibility and authority for software development
process improvement;
-- develop and implement a formal plan for software development
process improvement that is based on the software capability
evaluation results contained in this report and specifies
measurable goals and time frames, prioritizes initiatives,
estimates resource requirements (trained staff and funding) and
defines a process improvement management structure;
-- ensure that every new software development effort in Customs
adopts processes that satisfy at least SW-CMM level 2
requirements; and
-- ensure that process improvement activities are initiated for all
ongoing essential software maintenance projects.
AGENCY COMMENTS AND OUR
EVALUATION
---------------------------------------------------------- Chapter 8:2
In its written comments on a draft of this report, Customs
acknowledged the importance of software process improvement and
maturity. Also, it agreed with GAO's overall findings, including
that Customs' software development processes have not attained SW-CMM
level 2 maturity.
To address these weaknesses, Customs stated that it has taken the
first step toward implementing our recommendations by assigning
responsibility and authority for software process improvement as part
of a reorganization of its Office of Information and Technology,
which Customs stated will be implemented in early 1999. Customs
further stated that once the reorganization is implemented, a formal
software process improvement program will be established, and that
this program will include definition of an action plan, commitment of
resources, and specification of goals for achieving CMM levels 2 and
3. According to Customs, these improvement activities are in their
early stages. When they are successfully implemented, they should
address many of our recommendations.
Customs also stated that because its legacy systems are aging and
need to be enhanced and replaced, software process improvement must
occur in parallel with continued software development investments.
History has shown that attempting to modernize without first
instituting disciplined software processes has been a characteristic
of failed modernization programs.\1 Until it implements disciplined
software processes (i.e., at least level 2 process maturity), Customs
cannot prudently manage major system investments, such as ACE with an
estimated life cycle cost exceeding $1 billion.
Customs' comments also included a request to meet with us to discuss
system-specific KPA practice strength and weakness determinations.
We met prior to requesting comments on a draft of this report and
then again on January 12, 1999, to discuss SEI's SW-CMM requirements
and the basis for our determinations. We are prepared to continue
assisting Customs as it improves its software processes.
Appendix I provides the full text of Customs' comments and our
responses to additional Customs comments not discussed above.
(See figure in printed edition.)Appendix I
--------------------
\1 Tax Systems Modernization: Management and Technical Weaknesses
Must Be Corrected If Modernization Is To Succeed (GAO/AIMD-95-156,
July 26, 1995), Tax Systems Modernization: Actions Underway But IRS
Has Not Yet Corrected Management and Technical Weaknesses
(GAO/AIMD-96-106, June 7, 1996), and Air Traffic Control: Immature
Software Acquisition Processes Increase FAA System Acquisition Risks
(GAO/AIMD-97-47, March 21, 1997).
COMMENTS FROM THE DEPARTMENT OF
THE TREASURY
============================================================ Chapter 8
(See figure in printed edition.)
The following are responses to additional comments in Customs' letter
dated December 16, 1998.
GAO COMMENTS
1. The report has been modified to reflect Customs' comment.
2. Customs provided us a copy of its new Systems Development Life
Cycle (SDLC) document after it provided us comments on a draft of
this report, and we have not yet evaluated it. To develop systems
effectively, Customs will have to ensure that its SDLC addresses the
software development KPAs discussed in this report and that it
effectively implements the SDLC.
3. The statement that "Customs has a history of performing poorly
when developing software-intensive systems" has been removed from the
report. Regarding the statement in the report referring to "Customs'
limited success in delivering promised system capabilities on time
and within budget," there is clear evidence that Customs has not been
successful in developing ACE software on time. Specifically, the
first release of ACE, which is Customs' largest software development
effort, was delivered over 8 months late. Also, because Customs does
not track actual ACE costs by release against original estimates, it
does not know the cost performance history of ACE. With the respect
to the three systems that Customs cites as having been successfully
delivered, the first (the Year 2000 program) is still incomplete and
thus its success cannot yet be assessed. Because we have not
previously reviewed the other two (AES and Tinman), we cannot comment
on either's success in delivering promised capabilities on time and
within budget. The report has been clarified to reflect these
points.
4. The report has been clarified to reflect Customs' comment.
MAJOR CONTRIBUTORS TO THIS REPORT
========================================================== Appendix II
ACCOUNTING AND INFORMATION
MANAGEMENT DIVISION
Dr. Rona B. Stillman, Chief Scientist for Computers and
Telecommunications
Jack L. Brock, Director, Governmentwide and Defense Information
Systems
Madhav S. Panwar, Technical Assistant Director
Dr. Nabajyoti Barkakati, Technical Assistant Director
Prithviraj Mukherji, Assistant Director
Carl M. Urie, Assistant Director
Bernard R. Anderson, Senior Information Systems Analyst
Suzanne M. Burns, Senior Information Systems Analyst
ATLANTA FIELD OFFICE
Teresa F. Tucker, Senior Information Systems Analyst
*** End of document. ***