Customs Service Modernization: Ineffective Software Development Processes
Increase Customs System Development Risks (Chapter Report, 02/11/99,
GAO/AIMD-99-35).

Pursuant to a congressional request, GAO reviewed the Customs Service's
software development maturity and improvement activities, focusing on:
(1) the maturity of Customs' software development processes; and (2)
whether Customs has an effective software process improvement program.

GAO noted that: (1) because of the number and severity of Customs'
software development process weaknesses, Customs did not fully satisfy
any of the key process areas (KPA) necessary to achieve the repeatable
level of process maturity; (2) as a result, its processes for developing
software, a complex and expensive component of Customs' systems, are ad
hoc, sometimes chaotic, and not repeatable across projects; (3) Customs
had some practice strengths in all but one of the five KPAs evaluated
(i.e., requirements management, software project planning, software
project tracking and oversight, software quality assurance, and software
configuration management); however, GAO also found extensive and
significant weaknesses in each of these KPAs; (4) some of these
weaknesses were systemic, recurring in each of the KPAs; (5) for
example, Customs had no written policy for managing or implementing any
of the KPAs; (6) none of the projects had: (a) an approved quality
assurance plan; (b) documented procedures for determining the project
cost, schedule, or effort; or (c) any outside group reviewing or
reporting on the project's compliance with defined processes; (7) these
weaknesses are some of the reasons for Customs' limited success, for
example, in delivering promised Automated Commercial Environment (ACE)
capabilities on time; (8) Customs does not have a software development
process improvement program, and it has not taken the basic steps to
initiate one; (9) these steps, many of which are described in Software
Engineering Institute's initiating, diagnosing, establishing, acting,
and leveraging model for process improvement, include assigning
responsibility and authority for process improvement, establishing a
process improvement management structure, defining a plan of action, and
committing needed resources; and (10) until Customs establishes an
effective process improvement program, its software processes will
remain poorly defined and undisciplined, and its software projects are
likely to suffer cost, schedule, and performance shortfalls.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  AIMD-99-35
     TITLE:  Customs Service Modernization: Ineffective Software 
             Development Processes Increase Customs System Development
             Risks
      DATE:  02/11/99
   SUBJECT:  Computer software
             Information resources management
             Customs administration
             Computer software verification and validation
             Information systems
IDENTIFIER:  Customs Service National Customs Automation Program
             Customs Service Automated Export System
             Customs Service Automated Commercial Environment System
             CMM
             Software Capability Maturity Model
             
******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved.  Major          **
** divisions and subdivisions of the text, such as Chapters,    **
** Sections, and Appendixes, are identified by double and       **
** single lines.  The numbers on the right end of these lines   **
** indicate the position of each of the subsections in the      **
** document outline.  These numbers do NOT correspond with the  **
** page numbers of the printed product.                         **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                                            **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************


Cover
================================================================ COVER


Report to Congressional Requesters

February 1999

CUSTOMS SERVICE MODERNIZATION -
INEFFECTIVE SOFTWARE DEVELOPMENT
PROCESSES INCREASE CUSTOMS SYSTEM
DEVELOPMENT RISKS

GAO/AIMD-99-35

Customs Service Modernization

(511115)


Abbreviations
=============================================================== ABBREV

  ACE - Automated Commercial Environment
  ACS - Automated Commercial System
  AES - Automated Export System
  CMM0 - Capability Maturity Model
  GAO - General Accounting Office
  IDEAL - Initiating, Diagnosing, Establishing, Acting, Leveraging
  KPA - key process area
  NAFTA - North American Free Trade Agreement
  NCAP - National Customs Automation Program
  NCAP/P - National Customs Automation Program Prototype
  RM - requirements management
  SCE - Software Capability Evaluation
  SCM - software configuration management
  SDLC - Software Development Life Cycle
  SEI - Software Engineering Institute
  SEPG - Software Engineering Process Group
  SPP - software project planning
  SPTO - software project tracking and oversight
  SQA - software quality assurance
  SW-CMM - Software Capability Maturity Model

Letter
=============================================================== LETTER


B-280417

February 11, 1999

The Honorable Ben Nighthorse Campbell
Chairman, Subcommittee on Treasury,
 General Government, and Civil Service
Committee on Appropriations
United States Senate

The Honorable Jim Kolbe
Chairman, Subcommittee on Treasury,
 Postal, and General Government
Committee on Appropriations
House of Representatives

This report responds to your requests that we review the U.  S. 
Customs Service's software development maturity and improvement
activities.  Customs plans to spend hundreds of millions of dollars
developing and maintaining software-intensive systems.  We found that
Customs' software development processes are immature and are making
recommendations to the Commissioner of Customs for strengthening
them. 

We are sending copies of this report to the Secretary of the
Treasury; Commissioner of Customs; Director of the Office of
Management and Budget; Ranking Minority Members of the Subcommittee
on Treasury and General Government of the Senate Committee on
Appropriations and the Subcommittee on Treasury, Postal Service, and
General Government of the House Committee on Appropriations; and
other congressional committees.  We will also make copies available
to other interested parties upon request.  If you have questions or
wish to discuss the issues in this report, please contact me at (202)
512-6240.  Major contributors to this report are listed in appendix
II. 

Randolph C.  Hite
Associate Director, Governmentwide
and Defense Information Systems


EXECUTIVE SUMMARY
============================================================ Chapter 0


   PURPOSE
---------------------------------------------------------- Chapter 0:1

The Customs Service relies extensively on software intensive systems
to perform its core missions of enforcing laws governing the flow of
goods and persons across U.S.  borders, and assessing and collecting
billions of dollars annually in duties, taxes, and fees on imported
merchandise.  Because software is a complex and expensive component
of these systems, Customs must use defined and disciplined processes
if it expects to develop and maintain software effectively. 

Recognizing software's importance to Customs' mission effectiveness,
the Chairman, Subcommittee on Treasury and General Government, Senate
Committee on Appropriations, and the Chairman, Subcommittee on
Treasury, Postal Service and General Government, House Committee on
Appropriations, asked GAO to determine (1) the maturity of Customs'
software development processes, and (2) whether Customs has an
effective software process improvement program. 


   BACKGROUND
---------------------------------------------------------- Chapter 0:2

During fiscal year 1997, Customs collected $22.1 billion in revenue
at more than 300 ports of entry, and it processed nearly 450 million
passengers who entered the U.S.  during the year.  Each year Customs
also provides trade statistics used in developing trade policy and
negotiating trade agreements with various countries.  Customs expects
its workload to burgeon.  Accordingly, Customs plans to spend
hundreds of millions of dollars over the next 5 years developing
software for new and existing information systems. 

Software quality is governed largely by the quality of the processes
involved in developing or acquiring, and maintaining it.  Carnegie
Mellon University's Software Engineering Institute (SEI), recognized
for its expertise in software processes, has developed models and
methods that define and determine organizations' software process
maturity.  Together, they provide a logical framework for baselining
an organization's current process capabilities (i.e., strengths and
weaknesses) and providing a structured plan for incremental process
improvement. 

Using SEI's Software Capability Maturity Model\SM (SW-CMM0)\1 and the
SEI software capability evaluation method\2 , GAO staff trained at
SEI evaluated Customs' software development/maintenance maturity in
five of the six key process areas (KPA) that are necessary to attain
a "repeatable" level of process maturity.\3 GAO did not evaluate
Customs in the sixth repeatable level KPA, software subcontract
management, because Customs did not use subcontractors on any of the
projects that GAO evaluated.\4 An organization at the repeatable
level of process maturity has the necessary process discipline in
place to repeat earlier successes on projects in similar
applications.  The repeatable level of process maturity is the second
level on SEI's five-level scale.  Organizations that do not satisfy
the requirements for the "repeatable" level are by default judged to
be at the "initial" level of maturity, meaning that their processes
are ad hoc, sometimes even chaotic, with few of the processes defined
and success dependent mainly on the heroic efforts of individuals. 
To aid such organizations in maturing their processes, SEI has also
published a software process improvement model called IDEAL\S \M \5
that defines a systematic, five-phase process improvement approach.\6

As part of its evaluation, GAO examined three software projects. 
These were (1) development of the first software release of the
National Customs Automation Program (NCAP 0.1), which is the first
phase of the Automated Commercial Environment (ACE); (2) software
maintenance on the Automated Export System (AES); and (3) software
maintenance on the Administrative Security System.\7


--------------------
\1 Capability Maturity Model\SM is the service mark of Carnegie
Mellon University, and CMM0 is registered in the U.S.  Patent and
Trademark Office. 

\2 We used the latest version (version 1.1) of the software
development CMM for our evaluation. 

\3 The five KPAs are requirements management, software project
planning, software project tracking and oversight, software quality
assurance, and software configuration management.  According to the
SW-CMM, requirements management is the process for establishing a
common understanding between the customer and the software developer
of the customer's requirements; software project planning is the
process for establishing reasonable plans for engineering the
software and managing the software project; software project tracking
and oversight is the process of providing adequate visibility into
the software project's progress to permit effective action when
deviations from plans occur; software quality assurance is the
process of verifying for management that software project plans,
standards, and procedures are being followed; and software
configuration management is the process of establishing and
maintaining the integrity of the software products throughout the
project's life cycle. 

\4 According to the SW-CMM, software subcontract management is the
process of selecting a software subcontractor, establishing
commitments with the subcontractor, and tracking and reviewing the
subcontractor's performance and results. 

\5 IDEAL is a service mark of Carnegie Mellon University. 

\6 IDEAL:  A User's Guide for Software Process Improvement
(CMU/SEI-96-HB-001). 

\7 The Subcommittee Chairmen requested that we evaluate ACE, which is
the largest and most important system that Customs is developing. 
The other two projects were selected by Customs on the basis of the
following GAO specified criteria:  each project should be managed by
a different software team, at least one project should involve a
legacy system, at least one project should involve Year 2000 software
conversion, and each project should be relatively large and important
to accomplishing Customs' mission.




   RESULTS IN BRIEF
---------------------------------------------------------- Chapter 0:3

Because of the number and severity of Customs' software development
process weaknesses, Customs did not fully satisfy any of the key
process areas (KPAs) necessary to achieve the "repeatable" level of
process maturity.  As a result, its processes for developing
software, a complex and expensive component of Customs' systems, are
ad hoc, sometimes chaotic, and not repeatable across projects. 

Customs had some practice strengths in all but one of the five KPAs
evaluated (i.e., requirements management, software project planning,
software project tracking and oversight, software quality assurance,
and software configuration management); however, GAO also found
extensive and significant weaknesses in each of these KPAs.  Some of
these weaknesses were systemic, recurring in each of the KPAs.  For
example, Customs had no written policy for managing or implementing
any of the KPAs; and none of the projects had (1) an approved quality
assurance plan; (2) documented procedures for determining the project
cost, schedule, or effort; or (3) any outside group reviewing or
reporting on the project's compliance with defined processes.  These
weaknesses are some of the reasons for Customs' limited success, for
example, in delivering promised ACE capabilities on time. 

Currently, Customs does not have a software development process
improvement program, and it has not taken the basic steps to initiate
one.  These steps, many of which are described in SEI's IDEAL\8 model
for process improvement, include assigning responsibility and
authority for process improvement, establishing a process improvement
management structure, defining a plan of action, and committing
needed resources.  Until Customs establishes an effective process
improvement program, its software processes will remain poorly
defined and undisciplined, and its software projects are likely to
suffer cost, schedule, and performance shortfalls. 


--------------------
\8 IDEAL stands for the five phases of the model--Initiating,
Diagnosing, Establishing, Acting, and Leveraging. 


   PRINCIPAL FINDINGS
---------------------------------------------------------- Chapter 0:4


      CUSTOMS SOFTWARE DEVELOPMENT
      PROCESSES ARE IMMATURE
-------------------------------------------------------- Chapter 0:4.1

GAO evaluated how effectively each of the three Customs software
projects implemented five of the six level 2 KPAs:  requirements
management, software project planning, software project tracking and
oversight, software quality assurance, and software configuration
management.  To attain a level 2 or repeatable maturity rating,
Customs would have to effectively implement all of the key practices
for all five relevant KPAs. 

GAO found that while Customs had some strengths (i.e., practices that
are effectively implemented), it had too many weaknesses (i.e.,
practices that are ineffectively implemented) to satisfy any of the
level 2 KPAs.  The strengths and weaknesses for all three projects
are tallied in table 1.  In summary, of the total number of KPA
practices rated, 35 percent constituted strengths, 61 percent were
weaknesses, and 4 percent were observations.  An observation
indicates that the evidence was inconclusive and did not clearly
support a determination of either strength or weakness.  To reach the
repeatable level of maturity, Customs must eliminate the key practice
weaknesses identified in this report.\9



                                Table 1
                
                  Collective Number of KPA Strengths,
                  Weaknesses, and Observations on the
                             Three Projects

                                           Number    Number
                                               of        of  Number of
                                         strength  weakness  observati
Key process area                                s        es        ons
---------------------------------------  --------  --------  ---------
Requirements management                        23        13          0
Software project planning                      32        38          5
Software project tracking and oversight        28        40          4
Software quality assurance                      3        46          2
Software configuration management              17        46          0
======================================================================
Total                                         103       183         11
----------------------------------------------------------------------
Also, GAO found that while the three projects varied as to the number
of key practice strengths, weaknesses, and observations under each of
the five "common features" or practice groupings (commitment to
perform, ability to perform, activities performed, measurement and
analysis, and verification of implementation), the NCAP 0.1 project
displayed a better strengths to weaknesses ratio across all KPAs
(about 1:1) than either AES or the Administrative Security System
(about 1:2 and 1:4, respectively).  By increasing its software
maturity, Customs will reduce both the number of weaknesses on
individual projects and the variability in process discipline among
projects. 


--------------------
\9 SEI groups each of its KPA practices into one of five "common
features" or practice attributes.  These are "commitment to perform,
ability to perform, activities performed, measurement and analysis,
and verifying implementation."


      CUSTOMS DOES NOT HAVE A
      SOFTWARE PROCESS IMPROVEMENT
      PROGRAM
-------------------------------------------------------- Chapter 0:4.2

SEI's IDEAL model for software process improvement defines five
sequential phases.  The phases are (1) initiating the program,
including assigning organizational roles and responsibility,
establishing a program management structure, developing an action
plan, and allocating needed resources; (2) assessing the
organization's current level of process maturity; (3) establishing a
plan for addressing the identified process weaknesses; (4) developing
and implementing new or improved processes; and (5) learning from
process improvement experiences to strengthen the program. 

In 1996, Customs initiated efforts to improve its software processes. 
As part of this effort, Customs had a contractor assess its software
development capabilities and develop a process improvement plan. 
Customs then established process improvement teams for strengthening
two level-2 KPAs (software project planning and project tracking and
oversight). 

In 1997, Customs discontinued its process improvement program,
deciding at that time to redirect its process improvement resources
to Year 2000 conversion.  As a result, Customs' software development
capability, which is fundamental to its ability to effectively
develop new systems like ACE, and maintain existing systems like AES,
remains ad hoc and undisciplined.  Customs officials stated their
commitment to using the results of GAO's software capability
evaluation to baseline its strengths and weaknesses and address its
weaknesses, but Customs has not yet established a software process
improvement program.  In particular, it has not assigned
organizational responsibility and authority for process improvement,
established a program management structure, defined a plan of action,
or committed the necessary resources (trained staff and funding) to
execute the plan. 


   RECOMMENDATIONS
---------------------------------------------------------- Chapter 0:5

GAO recommends that, after ensuring that its mission-critical systems
are Year 2000 compliant, but before investing in major software
development efforts like ACE, the Commissioner of Customs direct the
Customs Chief Information Officer to: 

  -- assign responsibility and authority for software process
     improvement;

  -- develop and implement a formal plan for software development
     process improvement that is based on the software capability
     evaluation results contained in this report and specifies
     measurable goals and time frames, prioritizes initiatives,
     estimates resource requirements (trained staff and funding), and
     defines a process improvement management structure;

  -- ensure that every new software development effort in Customs
     adopts processes that satisfy at least SW-CMM level 2
     requirements; and

  -- ensure that process improvement activities are initiated for all
     ongoing essential software maintenance projects. 


   AGENCY COMMENTS AND GAO'S
   EVALUATION
---------------------------------------------------------- Chapter 0:6

In its written comments on a draft of this report, Customs
acknowledged the importance of software process improvement and
maturity.  Also, it agreed with GAO's overall findings, including
that Customs' software development processes have not attained SW-CMM
level 2 maturity. 

Customs stated that it has taken the first step toward implementing
GAO's recommendations by assigning responsibility and authority for
software process improvement as part of a reorganization of the
Office of Information and Technology that it plans to implement in
early 1999.  Customs commented that once the reorganization is
implemented, a formal software process improvement program will be
established, including definition of an action plan, commitment of
resources, and specification of goals for achieving CMM levels 2 and
3.  According to Customs, these improvement activities are in their
early stages.  When they are successfully implemented, they should
address many of GAO's recommendations. 

Customs also stated that, because its legacy systems are aging and
need to be enhanced and replaced, software process improvement must
occur in parallel with continued software development investments. 
History has shown that attempting to modernize without first
instituting disciplined software processes has been a characteristic
of failed modernization programs.\10 Until it implements disciplined
software processes (at least level 2 process maturity), Customs
cannot prudently manage major software investments, such as ACE with
an estimated life cycle cost exceeding $1 billion. 

In its comments, Customs also asked to meet with GAO to discuss
system-specific KPA practice strength and weakness determinations. 
GAO met with Customs prior to requesting comments on a draft of this
report to discuss system-specific determinations, and then again
after receiving Customs' comments to ensure that all findings were
clear.  GAO is prepared to continue assisting Customs as it improves
its software processes. 

Customs provided other comments on a draft of this report.  Each of
Customs' comments, along with GAO's responses, is discussed in detail
in chapter 8 and appendix I of this report. 


--------------------
\10 Tax Systems Modernization:  Management and Technical Weaknesses
Must Be Corrected If Modernization Is To Succeed (GAO/AIMD-95-156,
July 26, 1995), Tax Systems Modernization:  Actions Underway But IRS
Has Not Yet Corrected Management and Technical Weaknesses
(GAO/AIMD-96-106, June 7, 1996), and Air Traffic Control:  Immature
Software Acquisition Processes Increase FAA System Acquisition Risks
(GAO/AIMD-97-47, March 21, 1997). 


INTRODUCTION
============================================================ Chapter 1

The mission of the Customs Service is to ensure that all goods and
persons entering and exiting the United States do so in compliance
with all U.S.  laws and regulations.  It does this by (1) enforcing
the laws governing the flow of goods and persons across the borders
of the United States and (2) assessing and collecting duties, taxes,
and fees on imported merchandise.  During fiscal year 1997, Customs
collected $22.1 billion in revenue\1 at more than 300 ports of entry
and reported that it processed nearly 450 million passengers who
entered the United States during the year. 

To accomplish its mission, Customs is organized into six lines of
business--trade compliance, outbound, passenger, finance, human
resources, and investigations.  Each business area is described
below. 

  -- Trade compliance includes enforcement of laws and regulations
     associated with the importation of goods into the United States. 
     To do so, Customs (1) works with the trade community to promote
     understanding of applicable laws and regulations, (2)
     selectively examines cargo to ensure that only eligible goods
     enter the country, (3) reviews documentation associated with
     cargo entries to ensure that they are properly valued and
     classified, (4) collects billions of dollars annually in duties,
     taxes, and fees associated with imported cargo, (5) assesses
     fines and penalties for noncompliance with trade laws and
     regulations, (6) seizes and accounts for illegal cargo, and (7)
     manages the collection of these moneys to ensure that all
     trade-related debts due to Customs are paid and properly
     accounted for. 

  -- Outbound includes Customs enforcement of laws and regulations
     associated with the movement of merchandise and conveyances from
     the United States.  To do so, Customs (1) selectively inspects
     cargo at U.S.  ports to guard against the exportation of illegal
     goods, such as protected technologies, stolen vehicles, and
     illegal currency, (2) collects, disseminates, and uses
     intelligence to identify high-risk cargo and passengers, (3)
     seizes and accounts for illegal cargo, (4) assesses and collects
     fines and penalties associated with the exportation of illegal
     cargo, and (5) physically examines baggage and cargo at airport
     facilities for explosive and nuclear materials.  In addition,
     the outbound business includes collecting and disseminating
     trade data within the federal government.  Accurate trade data
     are crucial to establishing accurate trade statistics on which
     to base trade policy decisions and negotiate trade agreements
     with other countries.  By the year 2000, Customs estimates that
     exports will be valued at $1.2 trillion, compared to a reported
     $696 million in 1994. 

  -- Passenger includes processing all passengers and crew of
     arriving and departing (1) air and sea conveyances and (2) land
     vehicles and pedestrians.  In fiscal year 1997, Customs reported
     it processed nearly 450 million travelers and, by the year 2000,
     expects almost 500 million passengers to arrive in the United
     States annually.  Many of Customs' passenger activities focus on
     illegal immigration and drug smuggling and are coordinated with
     other federal agencies, such as the Immigration and
     Naturalization Service and the Department of Agriculture's
     Animal and Plant Health Inspection Service.  Activities include
     targeting high-risk passengers, which requires timely and
     accurate information, and physically inspecting selected
     passengers, baggage, and vehicles to determine compliance with
     laws and regulations. 

  -- Finance includes asset and revenue management activities.  Asset
     management consists of activities to (1) formulate Customs'
     budget, (2) properly allocate and distribute funds, and (3)
     acquire, manage, and account for personnel, goods, and services. 
     Revenue management encompasses all Customs activities to
     identify and establish amounts owed Customs, collect these
     amounts, and accurately report the status of revenue from all
     sources.  Sources of revenue include duties, fees, taxes, other
     user fees, and forfeited currency and property.  The revenue
     management activities interrelate closely with the revenue
     collection activities in the trade compliance, outbound, and
     passenger business areas. 

  -- Human resources is responsible for filling positions, providing
     employee benefits and services, training employees, facilitating
     workforce effectiveness, and processing personnel actions for
     Customs' 18,000 employees and managers. 

  -- Investigations includes activities to detect and eliminate
     narcotics and money laundering operations.  Customs works with
     other agencies and foreign governments to reduce drug-related
     activity by interdicting (seizing and destroying) narcotics,
     investigating organizations involved in drug smuggling, and
     deterring smuggling efforts through various other methods. 
     Customs also develops and provides information to the trade and
     carrier communities to assist them in their efforts to prevent
     smuggling organizations from using cargo containers and
     commercial conveyances to introduce narcotics into the United
     States. 

To carry out its responsibilities, Customs relies on information
systems and processes to assist its staff in (1) documenting,
inspecting, and accounting for the movement and disposition of
imported goods and (2) collecting and accounting for the related
revenues.  Customs expects its reliance on information systems to
increase as a result of its burgeoning workload.  For 1995 through
2001, Customs estimates that the annual volume of import trade
between the United States and other countries will increase from $761
billion to $1.1 trillion.  This will result in Customs processing an
estimated increase of 7.5 million commercial entries--from 13.1
million to 20.6 million annually--during the same period.  Recent
trade agreements, such as the North American Free Trade Agreement
(NAFTA), have also increased the number and complexity of trade
provisions that Customs must enforce. 

Customs recognizes that its ability to process the growing volume of
imports while improving compliance with trade laws depends heavily on
successfully modernizing its trade compliance process and its
supporting automated systems.  To speed the processing of imports and
improve compliance with trade laws, the Congress enacted
legislation\2 that eliminated certain legislatively mandated paper
requirements and required Customs to establish the National Customs
Automation Program (NCAP).  The legislation also specified certain
functions that NCAP must provide, including giving members of the
trade community the capability to electronically file import entries
at remote locations and enabling Customs to electronically process
"drawback" claims.\3 In response to the legislation, Customs began in
1994 to modernize the information systems that support operations. 


--------------------
\1 Includes tariff duty, user fees, Internal Revenue Service excise
taxes, and other assessments. 

\2 North American Free Trade Agreement Implementation Act, Public Law
103-182, 19 U.S.C.  1411 et seq. 

\3 Drawbacks are refunds of duties and taxes paid on imported goods
that are subsequently exported or destroyed. 


   CURRENT PROJECTS:  A BRIEF
   DESCRIPTION
---------------------------------------------------------- Chapter 1:1

Customs has several projects underway to develop and acquire new
software and evolve (i.e., maintain) existing software to support its
six business areas.  Customs' fiscal year 1998 budget for information
management and technology activities was about $147 million. 

Customs' major information technology effort is its Automated
Commercial Environment (ACE) system.  In 1994, Customs began to
develop ACE to replace its existing automated import system, the
Automated Commercial System (ACS).  ACE is intended to provide an
integrated, automated information system for collecting,
disseminating, and analyzing import-related data and ensuring the
proper collection and allocation of revenues, totaling about $19
billion annually.  According to Customs, ACE is planned to automate
critical functions that the Congress specified when it established
NCAP. 

Customs reported that it spent $47.8 million on ACE as of the end of
fiscal year 1997.  In November 1997, Customs estimated it would cost
$1.05 billion to develop, operate, and maintain ACE over the 15 years
from fiscal year 1994 through fiscal year 2008.  Customs plans to
deploy ACE to more than 300 ports that handle commercial cargo
imports. 

Customs plans to develop and deploy ACE in multiple phases. 
According to Customs, the first phase, known as NCAP, is an ACE
prototype.  Customs currently plans to deploy NCAP in four releases. 
The first release was deployed for field evaluation at three
locations in May 1998,\4 and the fourth is scheduled for 1999. 
Customs, however, has not adhered to previous NCAP deployment
schedules.  Specifically, implementation of the NCAP prototype
slipped from January 1997 to August 1997 and then again to a series
of four releases beginning in October 1997, with the fourth release
starting in June 1998. 

Customs also has several other projects underway to modify or enhance
existing systems that support its six business areas.  For example,
in fiscal year 1998, Customs planned to spend about $3.7 million to
enhance its Automated Export System (AES), which supports the
outbound business area and is designed to improve Customs' collection
and reporting of export statistics and to enforce export regulations. 
In addition, Customs planned to spend another $4.6 million to
maintain its administrative systems supporting its finance and human
resource business areas. 


--------------------
\4 The first release of the NCAP prototype was deployed to Detroit,
Michigan; Laredo, Texas; and Port Huron, Michigan. 


   OBJECTIVES, SCOPE, AND
   METHODOLOGY
---------------------------------------------------------- Chapter 1:2

The Chairman, Subcommittee on Treasury and General Government, Senate
Committee on Appropriations, and the Chairman, Subcommittee on
Treasury, Postal Service and General Government, House Committee on
Appropriations, requested that we review Customs' ability to develop
software for its computer systems.  Our objectives were to determine
(1) the maturity of Customs' software development processes and (2)
the effectiveness of Customs' software process improvement program. 

To determine Customs' software development process maturity, we
applied the Software Engineering Institute's (SEI) Software
Capability Maturity Model\SM (SW-CMM0)\5 and its Software Capability
Evaluation (SCE) method.  SEI's expertise in software process
maturity as well as its capability maturity models and evaluation
methods are widely accepted throughout the software industry.  All
our specialists were SEI-trained. 

The SW-CMM ranks organizational maturity according to five levels. 
(See figure 1.1.) Maturity levels 2 through 5 require the verifiable
existence and use of certain software development processes, known as
key process areas (KPA).  According to SEI, an organization that has
these processes in place is in a much better position to successfully
develop software than an organization that does not have these
processes in place.  We evaluated Customs' software development
processes against five of the six level 2 KPAs. 

   Figure 1.1:  SW-CMM Levels and
   Descriptions

   (See figure in printed
   edition.)

The sixth level 2 KPA, software subcontract management, was not
evaluated because Customs did not use subcontractors on any of the
projects that we evaluated.  (See table 1.1.)



                               Table 1.1
                
                  SW-CMM KPAs Used to Assess Customs'
                     Software Development Maturity

CMM Level 2 KPAs        Summary description
----------------------  ----------------------------------------------
Requirements            Defining, validating, and prioritizing
management              requirements, such as functions, performance,
                        and delivery dates.

Software project        Developing estimates for the work to be
planning                performed, establishing the necessary
                        commitments, and defining the plan to perform
                        the work.

Software project        Tracking and reviewing software
tracking and oversight  accomplishments and results against documented
                        estimates, commitments, and plans and
                        adjusting these based on the actual
                        accomplishments and results.

Software quality        Reviewing and auditing the software products
assurance               and activities to ensure that they comply with
                        the applicable processes, standards, and
                        procedures and providing the staff and
                        managers with the results of their reviews and
                        audits.

Software configuration  Selecting project baseline items, such as
management              specifications; systematically controlling
                        these items and changes to them; and recording
                        and reporting status and change activity for
                        these items.
----------------------------------------------------------------------
As established by the model, each KPA contains five common attributes
that indicate whether the implementation and institutionalization of
a KPA can be effective, repeatable, and lasting.  The five common
features are: 

  -- Commitment to perform:  The actions that the organization must
     take to establish the process and ensure that it can endure. 
     Commitment to perform typically involves establishing
     organizational policies and senior management sponsorship. 

  -- Ability to perform:  The preconditions that must exist in the
     project or organization to implement the software development
     process competently.  Ability to perform typically involves
     resources, organizational structures, and training. 

  -- Activities performed:  The roles and procedures necessary to
     implement a KPA.  Activities performed typically involve
     establishing plans and procedures, performing the work, tracking
     it, and taking appropriate management actions. 

  -- Measurement and analysis:  Activities performed to measure the
     process and analyze the measurements.  Measurement and analysis
     typically includes defining the measurements to be taken and the
     analyses to be conducted to determine the status and
     effectiveness of the activities performed. 

  -- Verifying implementation:  The steps to ensure that the
     activities are performed in compliance with the process that has
     been established.  Verification typically encompasses reviews by
     management. 

In accordance with SEI's SCE method and, for five of the six KPAs in
level 2, we evaluated Customs' institutional policies and practices
and compared project-specific guidance and practices against the five
common attributes.  This project-specific comparison can result in
one of four possible outcomes:  (1) project strengthan effective
implementation of the key practice, (2) project weaknessineffective
implementation of a key practice or failure to implement a key
practice, (3) project observationkey practice evaluated but evidence
inconclusive and cannot be characterized as either strength or
weakness, and (4) not ratedkey practice not currently relevant to
project, therefore, not evaluated. 

We performed the project-specific evaluations on three ongoing
Customs software development projects, each of which is described
below.  As requested by the Subcommittee Chairmen, one of the
projects evaluated was ACE, which is the largest and most important
system that Customs is developing.  The other two projects were
selected by Customs on the basis of the following GAO specified
criteria:  (1) each project should be managed by a different software
team, (2) at least one project should involve a legacy system, (3) at
least one project should involve Year 2000 software conversion, and
(4) each project should be relatively large and important to
accomplishing Customs' mission.  The projects we evaluated are: 

  -- National Customs Automation Program (NCAP 0.1):  NCAP 0.1 was
     the first component of the National Customs Automation Program
     Prototype (NCAP/P).  NCAP/P, in turn, is the first phase of the
     Automated Commercial Environment (ACE).  Customs began
     developing ACE in 1994 to address the new import processing
     requirements established by the National Customs Automation
     Program.  ACE is also intended to replace the agency's legacy
     automated import system, the Automated Commercial System (ACS). 
     NCAP 0.1 was installed at three field locations in May 1998. 

  -- Automated Export System (AES):  AES is an export information
     gathering and processing system, developed through cooperative
     efforts by Customs, the Bureau of Census, other federal agencies
     with export missions, and the export trade community.  AES is
     designed to improve the collection of trade statistics; assist
     in the creation of a paperless export environment; facilitate
     the release of exports subject to licensing requirements; and
     consolidate export data required by several government agencies,
     easing the data filing burden for exporters while streamlining
     the federal data collection process.

     Customs installed AES in all U.S.  vessel ports in October 1996,
     and currently it is operational in all ports, including air,
     rail, and truck transit ports.  Customs and Census officials
     estimate that they spent approximately $12.9 million to develop
     and implement AES from fiscal year 1992 to 1997.  These costs
     included, among other things, expenses for contractors, travel,
     and training.  According to Customs' and Census' figures, both
     agencies estimate that together they will spend an additional
     $32.2 million through fiscal year 2002 on AES implementation and
     maintenance. 

  -- Administrative Security System:  The Administrative Security
     System assists users in requesting access to administrative
     systems.  Users' requests are electronically submitted to the
     appropriate official for approvals.  In addition, other portions
     of the Administrative Security System provide functionality to
     allow the System Administrators the ability to prepare and
     maintain user profiles, request logs, and electronic approval
     and disapproval reports. 

To assess the effectiveness of Customs' software process improvement
program, we interviewed the Director, Technical Architecture Group,
Office of Information and Technology, to determine:  (1) process
improvements that are planned and underway, (2) the rationale for
each initiative, (3) the relative priority of each, (4) progress made
on each initiative, and (5) obstacles, if any, impeding progress.  We
also reviewed past process improvement plans, meeting minutes, and
related documentation.  Further, we reviewed SEI's model for software
process improvement, known as IDEAL\SM .\6 IDEAL defines five
sequential phases of software process improvement that can be used to
develop a long range, integrated plan for initiating and managing a
software process improvement program.\7

Customs provided written comments on a draft of this report.  These
comments are presented and evaluated in chapter 8, and are reprinted
in appendix I.  We performed our work at Customs' Newington,
Virginia, Data Center from February 1998 through November 1998, in
accordance with generally accepted government auditing standards. 


--------------------
\5 Capability Maturity Model\SM is the service mark of Carnegie
Mellon University, and CMM0 is registered in the U.S.  Patent and
Trademark Office. 

\6 IDEAL is a service mark of Carnegie Mellon University. 

\7 IDEAL:  A User's Guide for Software Process Improvement
(CMU/SEI-96-HB-001). 


REQUIREMENTS MANAGEMENT
============================================================ Chapter 2

The purpose of requirements management is to establish agreement
between the customer and the software developers of the customer's
requirements that will be implemented by the software developers. 
This agreement typically is referred to as the "system requirements
allocated to the software." The agreement covers both technical and
nontechnical (e.g., delivery dates) requirements.  The agreement
forms the basis for estimating, planning, performing, and tracking
the software developer's activities throughout the software life
cycle. 

According to the SW-CMM, a repeatable requirements management
process, among other things, includes (1) documenting the system
requirements allocated to software, (2) providing adequate resources
and funding for managing the allocated requirements, (3) following a
written organizational policy for requirements management, (4) having
a quality assurance group that reviews the activities and work
products for managing allocated requirements and reports the results,
(5) using the allocated requirements as the basis for software plans,
work products, and activities, and (6) training members of the
software engineering group to perform their requirements management
activities. 


   CUSTOMS' REQUIREMENTS
   MANAGEMENT PROCESS DOES NOT
   SATISFY SEI'S CRITERIA
---------------------------------------------------------- Chapter 2:1

All three projects had practice strengths in this KPA.  For example,
each project documented the system requirements allocated to software
and ensured that adequate resources and funding for managing the
allocated requirements were provided.  One of the projects, NCAP 0.1,
had strengths in all but two practices under this KPA; however, each
practice weakness is significant. 

Collectively, the projects had many weaknesses in this KPA, and thus
Customs' requirements management processes do not meet "repeatable"
maturity level criteria.  For example, none of the projects had a
written organizational policy governing requirements management, and
none had a quality assurance group for reviewing and reporting on the
activities and work products associated with managing the allocated
requirements.  In the absence of these two practices, management is
missing two means for ensuring that software requirements are managed
in a prescribed manner.  Also, two of the projects did not use the
allocated software requirements as the basis for software plans, work
products, and activities, which increases the risk that the software
developed will not fully satisfy requirements.  Further, members of
two projects' software engineering groups were not trained to perform
requirements management activities, thus increasing the chances of
mismanagement. 

Table 2.1 provides a comprehensive list of the three projects'
strengths and weaknesses for the requirements management KPA.  The
specific findings supporting the practice ratings cited in table 2.1
are in tables 2.2 through 2.4. 



                                    Table 2.1
                     
                             Requirements Management

                                                                    Administrati
              Key practice                  NCAP 0.1    AES         ve
------------  ----------------------------  ----------  ----------  ------------
Commitment 1  The project follows a         Weakness    Weakness    Weakness
              written organizational
              policy for managing the
              system requirements
              allocated to software.

Ability 1     For each project,             Strength    Strength    Strength
              responsibility is
              established for analyzing
              the system requirements and
              allocating them to hardware,
              software, and other system
              components.

Ability 2     The allocated requirements    Strength    Strength    Strength
              are documented.

Ability 3     Adequate resources and        Strength    Strength    Strength
              funding are provided for
              managing the allocated
              requirements.

Ability 4     Members of the software       Strength    Weakness    Weakness
              engineering group and other
              software-related groups are
              trained to perform their
              requirements management
              activities.

Activity 1    The software engineering      Strength    Strength    Weakness
              group reviews the allocated
              requirements before they are
              incorporated into the
              software project.

Activity 2    The software engineering      Strength    Weakness    Weakness
              group uses the allocated
              requirements as the basis
              for software plans, work
              products, and activities.

Activity 3    Changes to the allocated      Strength    Strength    Weakness
              requirements are reviewed
              and incorporated into the
              software project.

Measurement   Measurements are made and     Strength    Strength    Strength
1             used to determine the status
              of the activities for
              managing the allocated
              requirements.

Verification  The activities for managing   Strength    Weakness    Strength
1             the allocated requirements
              are reviewed with senior
              management on a periodic
              basis.

Verification  The activities for managing   Strength    Strength    Strength
2             the allocated requirements
              are reviewed with the
              project manager on both a
              periodic and event-driven
              basis.

Verification  The software quality          Weakness    Weakness    Weakness
3             assurance group reviews and/
              or audits the activities and
              work products for managing
              the allocated requirements
              and reports the results.
--------------------------------------------------------------------------------


                                    Table 2.2
                     
                       Requirements Management Findings for
                                     NCAP 0.1

                           National Customs Automation Program 0.1
              ------------------------------------------------------------------
              Key practice                Finding                     Rating
------------  --------------------------  --------------------------  ----------
Commitment 1  The project follows a       There is no written         Weakness
              written organizational      organizational policy for
              policy for managing the     managing the system
              system requirements         requirements allocated to
              allocated to software.      software.

Ability 1     For each project,           The Process Analysis and    Strength
              responsibility is           Requirements Team is
              established for analyzing   responsible for analyzing
              the system requirements     system requirements and
              and allocating them to      allocating them to
              hardware, software, and     hardware, software, and
              other system components.    other system components.

Ability 2     The allocated requirements  Allocated requirements are  Strength
              are documented.             documented.

Ability 3     Adequate resources and      Adequate resources and      Strength
              funding are provided for    funding are provided for
              managing the allocated      managing the allocated
              requirements.               requirements.

Ability 4     Members of the software     Members of the software     Strength
              engineering group and       engineering group and
              other software-related      other software-related
              groups are trained to       groups are trained to
              perform their requirements  perform their requirements
              management activities.      management activities.

Activity 1    The software engineering    The software engineering    Strength
              group reviews the           group reviews the
              allocated requirements      allocated requirements
              before they are             before they are
              incorporated into the       incorporated into the
              software project.           software project.

Activity 2    The software engineering    The software engineering    Strength
              group uses the allocated    group uses the allocated
              requirements as the basis   requirements as the basis
              for software plans, work    for software plans, work
              products, and activities.   products, and activities.

Activity 3    Changes to the allocated    Changes to the allocated    Strength
              requirements are reviewed   requirements are reviewed
              and incorporated into the   and incorporated into the
              software project.           software project.

Measurement   Measurements are made and   Measurements are made and   Strength
1             used to determine the       used to determine the
              status of the activities    status of the activities
              for managing the allocated  for managing the allocated
              requirements.               requirements.

Verification  The activities for          Periodic meetings with      Strength
1             managing the allocated      senior management include
              requirements are reviewed   reviews of allocated
              with senior management on   requirements.
              a periodic basis.

Verification  The activities for          The activities for          Strength
2             managing the allocated      managing the allocated
              requirements are reviewed   requirements are reviewed
              with the project manager    with the project manager
              on both a periodic and      on both a periodic and
              event-driven basis.         event-driven basis.

Verification  The software quality        There is no software        Weakness
3             assurance group reviews     quality assurance group,
              and/or audits the           therefore, no reviews and/
              activities and work         or audits are done.
              products for managing the
              allocated requirements and
              reports the results.
--------------------------------------------------------------------------------


                                    Table 2.3
                     
                     Requirements Management Findings for AES

                                   Automated Export System
              ------------------------------------------------------------------
              Key practice                Finding                     Rating
------------  --------------------------  --------------------------  ----------
Commitment 1  The project follows a       There is no written         Weakness
              written organizational      organizational policy for
              policy for managing the     managing systems
              system requirements         requirements allocated to
              allocated to software.      software.

Ability 1     For each project,           The project team            Strength
              responsibility is           established responsibility
              established for analyzing   for analyzing the system
              the system requirements     requirements and
              and allocating them to      allocating them to
              hardware, software, and     hardware, software, and
              other system components.    other system components.

Ability 2     The allocated requirements  The allocated requirements  Strength
              are documented.             are documented in the
                                          system functional
                                          requirements document and
                                          in the change requests
                                          document.

Ability 3     Adequate resources and      Adequate resources and      Strength
              funding are provided for    funding are provided for
              managing the allocated      managing allocated
              requirements.               requirements.

Ability 4     Members of the software     One member of the software  Weakness
              engineering group and       engineering group has been
              other software-related      trained to perform
              groups are trained to       requirements management
              perform their requirements  activities, but others
              management activities.      have not.

Activity 1    The software engineering    The software engineering    Strength
              group reviews the           group reviews the
              allocated requirements      allocated requirements
              before they are             before they are
              incorporated into the       incorporated into the
              software project.           software project.

Activity 2    The software engineering    The software engineering    Weakness
              group uses the allocated    group does not use the
              requirements as the basis   allocated requirements as
              for software plans, work    the basis for software
              products, and activities.   plans, work products, and
                                          activities.

Activity 3    Changes to the allocated    The Change Request Board    Strength
              requirements are reviewed   reviews and approves all
              and incorporated into the   changes made to allocated
              software project.           requirements and
                                          incorporates them into the
                                          software project.

Measurement   Measurements are made and   Measurements are made and   Strength
1             used to determine the       used to track changes to
              status of the activities    requirements and hours
              for managing the allocated  spent on requirements
              requirements.               management. A performance
                                          measurement plan is
                                          available.

Verification  The activities for          The activities for          Weakness
1             managing the allocated      managing the allocated
              requirements are reviewed   requirements are not
              with senior management on   reviewed with senior
              a periodic basis.           management on a periodic
                                          basis.

Verification  The activities for          The activities for          Strength
2             managing the allocated      managing the allocated
              requirements are reviewed   requirements are reviewed
              with the project manager    with the project manager
              on both a periodic and      on both a periodic and
              event-driven basis.         event-driven basis.

Verification  The software quality        The individual responsible  Weakness
3             assurance group reviews     for software quality
              and/or audits the           assurance did not review
              activities and work         and/or audit the
              products for managing the   activities and work
              allocated requirements and  products for managing the
              reports the results.        allocated requirements.
--------------------------------------------------------------------------------


                                    Table 2.4
                     
                       Requirements Management Findings for
                          Administrative Security System

                                Administrative Security System
              ------------------------------------------------------------------
              Key practice                Finding                     Rating
------------  --------------------------  --------------------------  ----------
Commitment 1  The project follows a       There is no written         Weakness
              written organizational      organizational policy on
              policy for managing the     requirements management.
              system requirements
              allocated to software.

Ability 1     For each project,           Project leaders are         Strength
              responsibility is           responsible for analyzing
              established for analyzing   systems requirements and
              the system requirements     allocating them to
              and allocating them to      software, hardware or
              hardware, software, and     other components.
              other system components.

Ability 2     The allocated requirements  Allocated requirements for  Strength
              are documented.             the system are documented.

Ability 3     Adequate resources and      Adequate resources and      Strength
              funding are provided for    funding are available for
              managing the allocated      managing the allocated
              requirements.               requirements.

Ability 4     Members of the software     There was no evidence to    Weakness
              engineering group and       show that members of the
              other software-related      software engineering group
              groups are trained to       received training in
              perform their requirements  requirements management.
              management activities.

Activity 1    The software engineering    The software engineering    Weakness
              group reviews the           group did not review the
              allocated requirements      allocated requirements
              before they are             before they were
              incorporated into the       incorporated into the
              software project.           software project.

Activity 2    The software engineering    Members of the software     Weakness
              group uses the allocated    engineering group did not
              requirements as the basis   use the allocated
              for software plans, work    requirements as the basis
              products, and activities.   for software plans, work
                                          products, and activities.

Activity 3    Changes to the allocated    Changes to the allocated    Weakness
              requirements are reviewed   requirements are not
              and incorporated into the   reviewed by the software
              software project.           engineering group and
                                          incorporated into the
                                          software project.

Measurement   Measurements are made and   Measurements are made and   Strength
1             used to determine the       used to determine the
              status of the activities    status of the activities
              for managing the allocated  for managing the allocated
              requirements.               requirements.

Verification  The activities for          Requirements management     Strength
1             managing the allocated      activities are discussed
              requirements are reviewed   at periodic meetings with
              with senior management on   senior management.
              a periodic basis.

Verification  The activities for          The activities for          Strength
2             managing the allocated      managing the allocated
              requirements are reviewed   requirements are reviewed
              with the project manager    with the project manager
              on both a periodic and      on both a periodic and
              event-driven basis.         event-driven basis.


Verification  The software quality        There is no software        Weakness
3             assurance group reviews     quality assurance (SQA)
              and/or audits the           group.
              activities and work
              products for managing the
              allocated requirements and
              reports the results.
--------------------------------------------------------------------------------


   CONCLUSIONS
---------------------------------------------------------- Chapter 2:2

While Customs' projects had several practice strengths in this KPA,
the number and significance of their practice weaknesses mean that
Customs' ability to manage software requirements is not repeatable. 
As a result, Customs is at risk of producing systems that fail to
provide promised capabilities, and cost more and take longer than
necessary. 


SOFTWARE PROJECT PLANNING
============================================================ Chapter 3

The purpose of software project planning is to establish reasonable
plans for performing the software engineering and for managing the
software project.  According to the SW-CMM, a repeatable software
project planning process, among other things, includes (1)
documenting the software project plan, and preparing plans for
software engineering facilities and support tools, (2) identifying
the work products needed to establish and maintain control of the
software project, (3) following a written organizational policy for
planning a software project, (4) having a quality assurance group
that reviews the activities and work products for software project
planning and reports the results, (5) estimating the software
project's efforts and costs, and estimating its critical computer
resources according to a documented procedure, (6) making and using
measurements to determine the status of planning activities, and (7)
training personnel in software project planning and estimating. 


   CUSTOMS IS NOT PERFORMING
   SOFTWARE PROJECT PLANNING
   EFFECTIVELY
---------------------------------------------------------- Chapter 3:1

All of the projects that we evaluated had key practice strengths in
this KPA.  For example, all had strengths in (1) documenting a
software project plan and preparing plans for the software
engineering facilities and support tools needed to develop the
software and (2) identifying the work products needed to control the
software project.  NCAP 0.1, in particular, had many additional
practice strengths. 

However, many significant practice weaknesses were found in all three
projects.  None of the projects followed an organizational software
project planning policy, and none had a quality assurance group
conducting reviews and/or audits.  As a result, the projects
performed these practices differently and inconsistently, and
controls were unreliable.  For example, while the NCAP 0.1 project
followed a documented procedure for estimating the size of software
work products (or changes to the size of work products), and made and
used measurements to determine the status of software planning
activities, neither of the other two projects performed these
practices and none of the projects had personnel trained in software
project planning and estimating.  Such project planning weaknesses
mean that management has no assurance that it will get the
consistent, complete, and reliable information about the projects'
expected costs and schedules needed to make expeditious and informed
investment decisions. 

Table 3.1 provides a comprehensive list of the three projects'
strengths, weaknesses, and observations for the software project
planning KPA.  The specific findings supporting the practice ratings
cited in table 3.1 are in tables 3.2 through 3.4. 



                                    Table 3.1
                     
                            Software Project Planning

                                                                    Administrati
              Key practice                  NCAP 0.1    AES         ve
------------  ----------------------------  ----------  ----------  ------------
Commitment 1  A project software manager    Strength    Strength    Weakness
              is designated to be
              responsible for negotiating
              commitments and developing
              the project's software
              development plan.

Commitment 2  The project follows a         Weakness    Weakness    Weakness
              written organizational
              policy for planning a
              software project.

Ability 1     A documented and approved     Strength    Observatio  Observation
              statement of work exists for              n
              the software project.

Ability 2     Responsibilities for          Strength    Observatio  Strength
              developing the software                   n
              development plan are
              assigned.

Ability 3     Adequate resources and        Strength    Weakness    Strength
              funding are provided for
              planning the software
              project.

Ability 4     The software managers,        Weakness    Weakness    Weakness
              software engineers, and
              other individuals involved
              in the software project
              planning are trained in the
              software estimating and
              planning procedures
              applicable to their areas of
              responsibility.

Activity 1    The software engineering      Strength    Weakness    Weakness
              group participates on the
              project proposal team.

Activity 2    Software project planning is  Strength    Weakness    Strength
              initiated in the early
              stages of, and in parallel
              with, the overall project
              planning.

Activity 3    The software engineering      Strength    Weakness    Observation
              group participates with
              other affected groups in the
              overall project planning
              throughout the project's
              life.

Activity 4    Software project commitments  Weakness    Observatio  Weakness
              made to individuals and                   n
              groups external to the
              organization are reviewed
              with senior management
              according to a documented
              procedure.

Activity 5    A software life cycle with    Weakness    Strength    Strength
              predefined stages of
              manageable size is
              identified or defined.

Activity 6    The project's software        Strength    Strength    Weakness
              development plan is
              developed according to a
              documented procedure.

Activity 7    The plan for the software     Strength    Strength    Strength
              project is documented.

Activity 8    Software work products that   Strength    Strength    Strength
              are needed to establish and
              maintain control of the
              software project are
              identified.

Activity 9    Estimates for the size of     Strength    Weakness    Weakness
              the software work products
              (or changes to the size of
              software work products) are
              derived according to a
              documented procedure.

Activity 10   Estimates for the software    Weakness    Weakness    Weakness
              project's effort and costs
              are derived according to a
              documented procedure.

Activity 11   Estimates for the project's   Weakness    Weakness    Weakness
              critical computer resources
              are derived according to a
              documented procedure.

Activity 12   The project's software        Weakness    Weakness    Weakness
              schedule is derived
              according to a documented
              procedure.

Activity 13   The software risks            Strength    Strength    Weakness
              associated with the cost,
              resource, schedule, and
              technical aspects of the
              project are identified,
              assessed, and documented.

Activity 14   Plans for the project's       Strength    Strength    Strength
              software engineering
              facilities and support tools
              are prepared.

Activity 15   Software planning data are    Strength    Weakness    Weakness
              recorded.

Measurement   Measurements are made and     Strength    Weakness    Weakness
1             used to determine the status
              of the software planning
              activities.

Verification  The activities for software   Strength    Weakness    Weakness
1             project planning are
              reviewed with senior
              management on a periodic
              basis.

Verification  The activities for software   Strength    Strength    Weakness
2             project planning are
              reviewed with the project
              manager on both a periodic
              and event-driven basis.

Verification  The software quality          Weakness    Weakness    Weakness
3             assurance group reviews and/
              or audits the activities and
              work products for software
              project planning and reports
              the results.
--------------------------------------------------------------------------------



                                    Table 3.2
                     
                      Software Project Planning Findings for
                                     NCAP 0.1

                           National Customs Automation Program 0.1
              ------------------------------------------------------------------
              Key practice                Finding                     Rating
------------  --------------------------  --------------------------  ----------
Commitment 1  A project software manager  The NCAP project has a      Strength
              is designated to be         software manager
              responsible for             designated to be
              negotiating commitments     responsible for
              and developing the          negotiating commitments
              project's software          and developing the
              development plan.           project's software
                                          development plan.

Commitment 2  The project follows a       The project does not        Weakness
              written organizational      follow a written
              policy for planning a       organizational policy for
              software project.           planning a software
                                          project.

Ability 1     A documented and approved   The approved project plan   Strength
              statement of work exists    meets the requirements for
              for the software project.   a statement of work for
                                          the project.

Ability 2     Responsibilities for        The project manager has     Strength
              developing the software     been assigned
              development plan are        responsibility for
              assigned.                   developing the software
                                          development plan.

Ability 3     Adequate resources and      Adequate resources and      Strength
              funding are provided for    funding have been provided
              planning the software       for planning the software
              project.                    project.

Ability 4     The software managers,      Project personnel are not   Weakness
              software engineers, and     trained in software
              other individuals involved  project planning and
              in the software project     estimating procedures.
              planning are trained in
              the software estimating
              and planning procedures
              applicable to their areas
              of responsibility.

Activity 1    The software engineering    The software engineering    Strength
              group participates on the   group participates on the
              project proposal team.      project proposal team.

Activity 2    Software project planning   Software project planning   Strength
              is initiated in the early   is initiated in the early
              stages of, and in parallel  stages of, and in parallel
              with, the overall project   with, the overall project
              planning.                   planning.

Activity 3    The software engineering    The software engineering    Strength
              group participates with     group participates with
              other affected groups in    other affected groups in
              the overall project         the overall project
              planning throughout the     planning throughout the
              project's life.             project's life.

Activity 4    Software project            Software project            Weakness
              commitments made to         commitments made to
              individuals and groups      individuals and groups
              external to the             external to the
              organization are reviewed   organization are not
              with senior management      reviewed with senior
              according to a documented   management and there is no
              procedure.                  documented procedure for
                                          such reviews.

Activity 5    A software life cycle with  There is no documented      Weakness
              predefined stages of        evidence that a software
              manageable size is          life cycle was selected
              identified or defined.      for the project.

Activity 6    The project's software      The project has a software  Strength
              development plan is         development plan that is
              developed according to a    developed according to a
              documented procedure.       documented procedure.

Activity 7    The plan for the software   The plan for the software   Strength
              project is documented.      project is documented in
                                          the project plan.

Activity 8    Software work products      Software work products      Strength
              that are needed to          that are needed to
              establish and maintain      establish and maintain
              control of the software     control of the software
              project are identified.     project are identified in
                                          the project plan.

Activity 9    Estimates for the size of   Estimates for the size of   Strength
              the software work products  the software work products
              (or changes to the size of  (or changes to the size of
              software work products)     software work products)
              are derived according to a  are derived according to a
              documented procedure.       documented procedure.

Activity 10   Estimates for the software  Estimates for the software  Weakness
              project's effort and costs  project's effort and costs
              are derived according to a  are not derived according
              documented procedure.       to a documented procedure.

Activity 11   Estimates for the           Estimates for the           Weakness
              project's critical          project's critical
              computer resources are      computer resources are not
              derived according to a      derived according to a
              documented procedure.       documented procedure.

Activity 12   The project's software      There is no documented      Weakness
              schedule is derived         procedure for deriving the
              according to a documented   software schedule.
              procedure.

Activity 13   The software risks          The software risks          Strength
              associated with the cost,   associated with the cost,
              resource, schedule, and     resource, schedule, and
              technical aspects of the    technical aspects of the
              project are identified,     project are identified,
              assessed, and documented.   assessed, and documented.

Activity 14   Plans for the project's     Plans for the project's     Strength
              software engineering        software engineering
              facilities and support      facilities and support
              tools are prepared.         tools are prepared.

Activity 15   Software planning data are  Software planning data are  Strength
              recorded.                   recorded.

Measurement   Measurements are made and   Measurements are made and   Strength
1             used to determine the       used to determine the
              status of the software      status of the software
              planning activities.        planning activities.

Verification  The activities for          The activities for          Strength
1             software project planning   software project planning
              are reviewed with senior    are reviewed with senior
              management on a periodic    management, including
              basis.                      reports to Treasury and
                                          Customs Investment Review
                                          Boards (IRB), on a
                                          periodic basis.

Verification  The activities for          The activities for          Strength
2             software project planning   software project planning
              are reviewed with the       are reviewed with the
              project manager on both a   project manager on both a
              periodic and event-driven   periodic and event-driven
              basis.                      basis through weekly
                                          status reports and
                                          meetings.

Verification  The software quality        There is no SQA group.      Weakness
3             assurance group reviews
              and/or audits the
              activities and work
              products for software
              project planning and
              reports the results.
--------------------------------------------------------------------------------



                                    Table 3.3
                     
                      Software Project Planning Findings for
                                       AES

                                   Automated Export System
              ------------------------------------------------------------------
              Key practice                Finding                     Rating
------------  --------------------------  --------------------------  ----------
Commitment 1  A project software manager  A project software manager  Strength
              is designated to be         is designated to be
              responsible for             responsible for
              negotiating commitments     negotiating commitments
              and developing the          and developing the
              project's software          project's software
              development plan.           development plan.

Commitment 2  The project follows a       The project does not        Weakness
              written organizational      follow a written
              policy for planning a       organizational policy for
              software project.           planning a software
                                          project.

Ability 1     A documented and approved   The AES System and          Observatio
              statement of work exists    Functional Requirements     n
              for the software project.   Table documents the
                                          statement of work for this
                                          project. However, this
                                          document is not approved.

Ability 2     Responsibilities for        Responsibilities for        Observatio
              developing the software     developing the software     n
              development plan are        development plan have not
              assigned.                   been assigned; however, a
                                          plan has been developed.

Ability 3     Adequate resources and      Adequate resources and      Weakness
              funding are provided for    funding are not provided
              planning the software       for planning the software
              project.                    project.

Ability 4     The software managers,      Individuals involved in     Weakness
              software engineers, and     the software project
              other individuals involved  planning are not trained
              in the software project     in the software estimating
              planning are trained in     and planning procedures.
              the software estimating
              and planning procedures
              applicable to their areas
              of responsibility.

Activity 1    The software engineering    The software engineering    Weakness
              group participates on the   group does not participate
              project proposal team.      in project proposal
                                          preparation.

Activity 2    Software project planning   Software project planning   Weakness
              is initiated in the early   is not initiated in the
              stages of, and in parallel  early stages of, and in
              with, the overall project   parallel with, the overall
              planning.                   project planning.

Activity 3    The software engineering    The software engineering    Weakness
              group participates with     group does not participate
              other affected groups in    in the overall project
              the overall project         planning throughout the
              planning throughout the     project's life cycle.
              project's life.

Activity 4    Software project            Software project            Observatio
              commitments made to         commitments made to         n
              individuals and groups      individuals and groups
              external to the             external to the
              organization are reviewed   organization are reviewed
              with senior management      with senior management;
              according to a documented   however, there is no
              procedure.                  documented procedure for
                                          these reviews.

Activity 5    A software life cycle with  The project uses the        Strength
              predefined stages of        waterfall model identified
              manageable size is          in the Customs software
              identified or defined.      development life cycle
                                          document.

Activity 6    The project's software      The project's software      Strength
              development plan is         development plan was
              developed according to a    developed according to a
              documented procedure.       documented procedure in
                                          the SDLC.

Activity 7    The plan for the software   The plan for the software   Strength
              project is documented.      project is documented.

Activity 8    Software work products      Software work products      Strength
              that are needed to          that are needed to
              establish and maintain      establish and maintain
              control of the software     control of the software
              project are identified.     project have been
                                          identified.

Activity 9    Estimates for the size of   There is no documented      Weakness
              the software work products  procedure for estimating
              (or changes to the size of  the size of software work
              software work products)     products (or changes to
              are derived according to a  the size of software work
              documented procedure.       products).

Activity 10   Estimates for the software  There is no documented      Weakness
              project's effort and costs  procedure for estimating
              are derived according to a  project effort and cost.
              documented procedure.

Activity 11   Estimates for the           There is no documented      Weakness
              project's critical          procedure for estimating
              computer resources are      the project's computer
              derived according to a      resources.
              documented procedure.

Activity 12   The project's software      There is no documented      Weakness
              schedule is derived         procedure for deriving the
              according to a documented   software schedule.
              procedure.

Activity 13   The software risks          The software risks          Strength
              associated with the cost,   associated with the cost,
              resource, schedule, and     resource, schedule, and
              technical aspects of the    technical aspects of the
              project are identified,     project are identified,
              assessed, and documented.   assessed, and documented.

Activity 14   Plans for the project's     AES is an ongoing project.  Strength
              software engineering        The software engineering
              facilities and support      facilities and support
              tools are prepared.         tools already exist, and
                                          are documented in the
                                          software development plan.

Activity 15   Software planning data are  Software planning data,     Weakness
              recorded.                   such as estimating the
                                          project's critical
                                          computer resources, are
                                          not recorded.

Measurement   Measurements are made and   Measurements are not made   Weakness
1             used to determine the       and used to determine the
              status of the software      status of the software
              planning activities.        planning activities.

Verification  The activities for          The activities for          Weakness
1             software project planning   software project planning
              are reviewed with senior    are not reviewed with
              management on a periodic    senior management on a
              basis.                      periodic basis.

Verification  The activities for          The activities for          Strength
2             software project planning   software project planning
              are reviewed with the       are reviewed with the
              project manager on both a   project manager on both a
              periodic and event-driven   periodic and event-driven
              basis.                      basis.

Verification  The software quality        The individual performing   Weakness
3             assurance group reviews     software quality assurance
              and/or audits the           activities does not
              activities and work         conduct reviews and/or
              products for software       audits.
              project planning and
              reports the results.
--------------------------------------------------------------------------------



                                    Table 3.4
                     
                      Software Project Planning Findings for
                          Administrative Security System

                                Administrative Security System
              ------------------------------------------------------------------
              Key practice                Finding                     Rating
------------  --------------------------  --------------------------  ----------
Commitment 1  A project software manager  A project software manager  Weakness
              is designated to be         has not been designated to
              responsible for             be responsible for
              negotiating commitments     negotiating commitments
              and developing the          and developing the
              project's software          project's software
              development plan.           development plan. The
                                          software engineering group
                                          was not sure who was
                                          responsible for
                                          negotiating commitments.

Commitment 2  The project follows a       The project does not        Weakness
              written organizational      follow a written
              policy for planning a       organizational policy for
              software project.           planning a software
                                          project.

Ability 1     A documented and approved   A documented statement of   Observatio
              statement of work exists    work exists for the         n
              for the software project.   software project, but has
                                          not been approved.

Ability 2     Responsibilities for        Responsibilities for        Strength
              developing the software     developing the software
              development plan are        development plan were
              assigned.                   assigned to a team of
                                          seven whose names appear
                                          on the document.

Ability 3     Adequate resources and      Adequate resources and      Strength
              funding are provided for    funding are available for
              planning the software       planning the software
              project.                    project.


Ability 4     The software managers,      The software managers,      Weakness
              software engineers, and     software engineers, and
              other individuals involved  other individuals involved
              in the software project     in the software project
              planning are trained in     planning are not trained
              the software estimating     in the software estimating
              and planning procedures     and planning procedures
              applicable to their areas   applicable to their areas
              of responsibility.          of responsibility.

Activity 1    The software engineering    The software engineering    Weakness
              group participates on the   group did not participate
              project proposal team.      on the project proposal
                                          team.

Activity 2    Software project planning   Software project planning   Strength
              is initiated in the early   is initiated in the early
              stages of, and in parallel  stages of, and in parallel
              with, the overall project   with, the overall project
              planning.                   planning.

Activity 3    The software engineering    The software engineering    Observatio
              group participates with     group occasionally          n
              other affected groups in    participates with other
              the overall project         affected groups in the
              planning throughout the     overall project planning
              project's life.             throughout the project's
                                          life.

Activity 4    Software project            There is no documented      Weakness
              commitments made to         procedure to ensure that
              individuals and groups      software project
              external to the             commitments made to
              organization are reviewed   individuals and groups
              with senior management      external to the
              according to a documented   organization are reviewed
              procedure.                  with senior management.

Activity 5    A software life cycle with  A software life cycle with  Strength
              predefined stages of        predefined stages of
              manageable size is          manageable size has been
              identified or defined.      identified in the SDLC.

Activity 6    The project's software      The project has a software  Weakness
              development plan is         development plan; but it
              developed according to a    was not developed
              documented procedure.       according to a documented
                                          procedure.

Activity 7    The plan for the software   The plan for the software   Strength
              project is documented.      project is documented.

Activity 8    Software work products      Software work products      Strength
              that are needed to          that are needed to
              establish and maintain      establish and maintain
              control of the software     control of the software
              project are identified.     project are identified.

Activity 9    Estimates for the size of   No documented procedure is  Weakness
              the software work products  used for estimating the
              (or changes to the size of  size of software work
              software work products)     products (or changes to
              are derived according to a  the size of software work
              documented procedure.       products).

Activity 10   Estimates for the software  No documented procedure is  Weakness
              project's effort and costs  used for estimating the
              are derived according to a  project's effort and cost.
              documented procedure.

Activity 11   Estimates for the           No documented procedure is  Weakness
              project's critical          used for estimating the
              computer resources are      project's computer
              derived according to a      resources.
              documented procedure.

Activity 12   The project's software      No documented procedure is  Weakness
              schedule is derived         used for deriving the
              according to a documented   project schedule.
              procedure.

Activity 13   The software risks          The software risks          Weakness
              associated with the cost,   associated with the cost,
              resource, schedule, and     resource, schedule, and
              technical aspects of the    technical aspects of the
              project are identified,     project have been
              assessed, and documented.   identified and documented,
                                          but have not been assessed
                                          for impact and mitigation
                                          options.

Activity 14   Plans for the project's     Existing tools and the      Strength
              software engineering        current software
              facilities and support      engineering environment
              tools are prepared.         have been identified for
                                          use.

Activity 15   Software planning data are  Software planning data are  Weakness
              recorded.                   not recorded.

Measurement   Measurements are made and   No measurements are made    Weakness
1             used to determine the       to determine the status of
              status of the software      software planning
              planning activities.        activities.

Verification  The activities for          While periodic meetings     Weakness
1             software project planning   are held with senior
              are reviewed with senior    management, there was no
              management on a periodic    evidence that activities
              basis.                      for software project
                                          planning are addressed.

Verification  The activities for          While periodic meetings     Weakness
2             software project planning   are held with the project
              are reviewed with the       manager, there was no
              project manager on both a   evidence that activities
              periodic and event-driven   for software project
              basis.                      planning are addressed.

Verification  The software quality        There is no software        Weakness
3             assurance group reviews     quality assurance group.
              and/or audits the
              activities and work
              products for software
              project planning and
              reports the results.
--------------------------------------------------------------------------------


   CONCLUSIONS
---------------------------------------------------------- Chapter 3:2

Effective planning is the cornerstone of successful software
development project management.  While Customs showed some strengths
in this KPA, its many weaknesses render its software project planning
processes unrepeatable.  Therefore, Customs has no assurance that the
projects are effectively establishing plans, including reliable
projections of costs and schedules, and effectively measuring and
monitoring progress and taking needed corrective actions
expeditiously. 


SOFTWARE PROJECT TRACKING AND
OVERSIGHT
============================================================ Chapter 4

The purpose of software project tracking and oversight is to provide
adequate visibility into the progress of the software development so
that management can act effectively when the software project's
performance deviates significantly from the software plans.  Software
project tracking and oversight involves tracking and reviewing the
software accomplishments and results against documented estimates,
commitments, and plans, and adjusting these plans based on the actual
accomplishments and results. 

According to the SW-CMM, effective software project tracking and
oversight, among other things, includes (1) designating a project
software manager to be responsible for the project's software
activities and results, (2) having a documented software development
plan for tracking software activities and communicating status, (3)
following a written organizational policy for managing the project,
(4) conducting periodic internal reviews to track technical progress,
plans, performance, and issues against the software development plan,
(5) tracking the software risks associated with the cost, resource,
schedule, and technical aspects of the project, (6) explicitly
assigning responsibility for software work products and activities,
(7) tracking the sizes of the software work products (or sizes of the
changes to the software work products) and taking corrective actions
as necessary, and (8) periodically reviewing the activities for
software project tracking and oversight with senior management. 


   CUSTOMS' SOFTWARE PROJECT
   TRACKING AND OVERSIGHT PROCESS
   IS IMMATURE
---------------------------------------------------------- Chapter 4:1

The projects evaluated exhibited some software project tracking and
oversight practice strengths.  For example, all three of the projects
had a project software manager designated to be responsible for the
project's software activities and results, and all had a documented
software development plan for tracking software activities and
communicating status.  Also, NCAP 0.1 had strengths in all but five
of this KPA's 24 key practices. 

However, the three projects collectively had many weaknesses, and
these weaknesses, including the five for NCAP 0.1, were significant
and thus preclude Customs from meeting SEI's repeatable maturity
level criteria.  For example, none of the projects followed a written
organizational policy for managing the software project.  With no
established policy, Customs increases the risk that key tracking and
oversight activities will not be performed effectively.  For example,
for two of the three projects, the project managers did not (1)
conduct periodic internal reviews to track technical progress, plans,
performance, and issues against the software development plan, (2)
track software risks associated with cost, resource, schedule, and
technical aspects of the project, (3) explicitly assign
responsibility to individuals for software work products and
activities, (4) track the sizes of the software work products (or
sizes of the changes to the software work products) and take
corrective actions, or (5) periodically review software project
tracking and oversight activities with senior management. 

Table 4.1 provides a comprehensive list of the three projects'
strengths, weaknesses, and observations for the software project
tracking and oversight KPA.  The specific findings supporting the
practice ratings cited in table 4.1 are in tables 4.2 through 4.4. 



                                    Table 4.1
                     
                     Software Project Tracking and Oversight
                                     Summary

                                                                    Administrati
              Key practice                  NCAP 0.1    AES         ve
------------  ----------------------------  ----------  ----------  ------------
Commitment 1  A project software manager    Strength    Strength    Strength
              is designated to be
              responsible for the
              project's software
              activities and results.

Commitment 2  The project follows a         Weakness    Weakness    Weakness
              written organizational
              policy for managing the
              software project.

Ability 1     A software development plan   Strength    Observatio  Observation
              for the software project is               n
              documented and approved.

Ability 2     The project software manager  Strength    Weakness    Weakness
              explicitly assigns
              responsibility for software
              work products and
              activities.

Ability 3     Adequate resources and        Strength    Weakness    Weakness
              funding are provided for
              tracking the software
              project.

Ability 4     The software managers are     Weakness    Strength    Weakness
              trained in managing the
              technical and personnel
              aspects of the software
              project.

Ability 5     First-line software managers  Strength    Strength    Weakness
              receive orientation in the
              technical aspects of the
              software project.

Activity 1    A documented software         Strength    Strength    Strength
              development plan is used for
              tracking the software
              activities and communicating
              status.

Activity 2    The project's software        Weakness    Weakness    Weakness
              development plan is revised
              according to a documented
              procedure.

Activity 3    Software project commitments  Weakness    Observatio  Observation
              and changes to commitments                n
              made to individuals and
              groups external to the
              organization are reviewed
              with senior management
              according to a documented
              procedure.

Activity 4    Approved changes to           Strength    Strength    Weakness
              commitments that affect the
              software project are
              communicated to the members
              of the software engineering
              group and other software-
              related groups.

Activity 5    The sizes of the software     Strength    Weakness    Weakness
              work products (or sizes of
              the changes to the software
              work products) are tracked,
              and corrective actions are
              taken as necessary.

Activity 6    The project's software        Strength    Strength    Weakness
              effort and costs are
              tracked, and corrective
              actions are taken as
              necessary.

Activity 7    The project's critical        Strength    Weakness    Weakness
              computer resources are
              tracked, and corrective
              actions are taken as
              necessary.

Activity 8    The project's software        Strength    Weakness    Strength
              schedule is tracked, and
              corrective actions are taken
              as necessary.

Activity 9    Software engineering          Strength    Weakness    Weakness
              technical activities are
              tracked, and corrective
              actions are taken as
              necessary.

Activity 10   The software risks            Strength    Weakness    Weakness
              associated with cost,
              resource, schedule, and
              technical aspects of the
              project are tracked.

Activity 11   Actual measurement data and   Strength    Weakness    Weakness
              replanning data for the
              software project are
              recorded.

Activity 12   The software engineering      Strength    Weakness    Weakness
              group conducts periodic
              internal reviews to track
              technical progress, plans,
              performance, and issues
              against the software
              development plan.

Activity 13   Formal reviews to address     Strength    Weakness    Weakness
              the accomplishments and
              results of the software
              project are conducted at
              selected project milestones
              according to a documented
              procedure.

Measurement   Measurements are made and     Strength    Weakness    Weakness
1             used to determine the status
              of the software tracking and
              oversight activities.

Verification  The activities for software   Strength    Weakness    Weakness
1             project tracking and
              oversight are reviewed with
              senior management on a
              periodic basis.

Verification  The activities for software   Strength    Weakness    Weakness
2             project tracking and
              oversight are reviewed with
              the project manager on both
              a periodic and event-driven
              basis.

Verification  The software quality          Weakness    Weakness    Weakness
3             assurance group reviews and/
              or audits the activities and
              work products for software
              project tracking and
              oversight and reports the
              results.
--------------------------------------------------------------------------------


                                    Table 4.2
                     
                     Software Project Tracking and Oversight
                              Findings for NCAP 0.1

                           National Customs Automation Program 0.1
              ------------------------------------------------------------------
              Key practice                Finding                     Rating
------------  --------------------------  --------------------------  ----------
Commitment 1  A project software manager  The project software        Strength
              is designated to be         manager is designated to
              responsible for the         be responsible for the
              project's software          project's software
              activities and results.     activities and results.

Commitment 2  The project follows a       There is no written         Weakness
              written organizational      organizational policy for
              policy for managing the     managing the software
              software project.           project.

Ability 1     A software development      An approved and documented  Strength
              plan for the software       software development plan
              project is documented and   is contained in the
              approved.                   project plan.

Ability 2     The project software        The project software        Strength
              manager explicitly assigns  manager explicitly assigns
              responsibility for          responsibility for
              software work products and  software work products and
              activities.                 activities.

Ability 3     Adequate resources and      Adequate resources and      Strength
              funding are provided for    funding are provided for
              tracking the software       tracking the software
              project.                    project.

Ability 4     The software managers are   The software managers are   Weakness
              trained in managing the     not trained in managing
              technical and personnel     the technical and
              aspects of the software     personnel aspects of the
              project.                    software project.

Ability 5     First-line software         First-line software         Strength
              managers receive            managers receive
              orientation in the          orientation in the
              technical aspects of the    technical aspects of the
              software project.           software project

Activity 1    A documented software       A documented software       Strength
              development plan is used    development plan is used
              for tracking the software   for tracking the software
              activities and              activities and
              communicating status.       communicating status.

Activity 2    The project's software      No documented procedure     Weakness
              development plan is         exists for revising the
              revised according to a      software development plan.
              documented procedure.

Activity 3    Software project            Software project            Weakness
              commitments and changes to  commitments and changes to
              commitments made to         commitments made to
              individuals and groups      individuals and groups
              external to the             external to the
              organization are reviewed   organization are not
              with senior management      reviewed with senior
              according to a documented   management. Also, there is
              procedure.                  no documented procedure
                                          for such reviews.

Activity 4    Approved changes to         Approved changes to         Strength
              commitments that affect     commitments that affect
              the software project are    the software project are
              communicated to the         communicated to the
              members of the software     members of the software
              engineering group and       engineering group and
              other software-related      other software-related
              groups.                     groups through weekly
                                          staff meetings.

Activity 5    The sizes of the software   The sizes of the software   Strength
              work products (or sizes of  work products (or sizes of
              the changes to the          the changes to the
              software work products)     software work products)
              are tracked, and            are tracked; however, at
              corrective actions are      the time of our
              taken as necessary.         evaluation, no corrective
                                          actions were needed.

Activity 6    The project's software      The project's software      Strength
              effort and costs are        effort and costs are
              tracked, and corrective     tracked; however, at the
              actions are taken as        time of our evaluation, no
              necessary.                  corrective actions were
                                          needed.

Activity 7    The project's critical      The project's critical      Strength
              computer resources are      computer resources are
              tracked, and corrective     tracked; however, at the
              actions are taken as        time of our evaluation, no
              necessary.                  corrective actions were
                                          needed.

Activity 8    The project's software      The project's software      Strength
              schedule is tracked, and    schedule is tracked;
              corrective actions are      however, at the time of
              taken as necessary.         our evaluation, no
                                          corrective actions were
                                          needed.

Activity 9    Software engineering        Software engineering        Strength
              technical activities are    technical activities are
              tracked, and corrective     tracked, and corrective
              actions are taken as        actions are taken as
              necessary.                  necessary.

Activity 10   The software risks          The software risks          Strength
              associated with cost,       associated with cost,
              resource, schedule, and     resource, schedule, and
              technical aspects of the    technical aspects of the
              project are tracked.        project are tracked.

Activity 11   Actual measurement data     Actual measurement data     Strength
              and replanning data for     and replanning data for
              the software project are    the software project are
              recorded.                   recorded.

Activity 12   The software engineering    The software engineering    Strength
              group conducts periodic     group conducts periodic
              internal reviews to track   internal reviews to track
              technical progress, plans,  technical progress, plans,
              performance, and issues     performance, and issues
              against the software        against the software
              development plan.           development plan.

Activity 13   Formal reviews to address   Formal reviews to address   Strength
              the accomplishments and     the accomplishments and
              results of the software     results of the software
              project are conducted at    project are conducted at
              selected project            selected project
              milestones according to a   milestones according to a
              documented procedure.       documented procedure.

Measurement   Measurements are made and   Measurements are made and   Strength
1             used to determine the       used to determine the
              status of the software      status of the software
              tracking and oversight      tracking and oversight
              activities.                 activities.

Verification  The activities for          The activities for          Strength
1             software project tracking   software project tracking
              and oversight are reviewed  and oversight are reviewed
              with senior management on   with senior management on
              a periodic basis.           a weekly basis.

Verification  The activities for          The activities for          Strength
2             software project tracking   software project tracking
              and oversight are reviewed  and oversight are reviewed
              with the project manager    with the project manager
              on both a periodic and      on both a periodic and
              event-driven basis.         event-driven basis.

Verification  The software quality        No software quality         Weakness
3             assurance group reviews     assurance group exists;
              and/or audits the           therefore, there are no
              activities and work         reviews and/or audits of
              products for software       the activities and work
              project tracking and        products for software
              oversight and reports the   project tracking and
              results.                    oversight.
--------------------------------------------------------------------------------


                                    Table 4.3
                     
                     Software Project Tracking and Oversight
                                 Findings for AES

                                   Automated Export System
              ------------------------------------------------------------------
              Key practice                Finding                     Rating
------------  --------------------------  --------------------------  ----------
Commitment 1  A project software manager  A project software manager  Strength
              is designated to be         is designated to be
              responsible for the         responsible for the
              project's software          project's software
              activities and results.     activities and results.

Commitment 2  The project follows a       There is no written         Weakness
              written organizational      organizational policy for
              policy for managing the     managing the software
              software project.           project.

Ability 1     A software development      A software development      Observatio
              plan for the software       plan for the project        n
              project is documented and   exists. However, this plan
              approved.                   has not been approved.

Ability 2     The project software        The project software        Weakness
              manager explicitly assigns  manager does not
              responsibility for          explicitly assign
              software work products and  responsibility for
              activities.                 software work products and
                                          activities.

Ability 3     Adequate resources and      Adequate resources and      Weakness
              funding are provided for    funding are not provided
              tracking the software       for tracking the software
              project.                    project.

Ability 4     The software managers are   The software managers are   Strength
              trained in managing the     trained in managing the
              technical and personnel     technical and personnel
              aspects of the software     aspects of the software
              project.                    projects through training
                                          given at off-site retreats
                                          and guidance provided in
                                          the AES users' guide.

Ability 5     First-line software         First-line software         Strength
              managers receive            managers receive
              orientation in the          orientation in the
              technical aspects of the    technical aspects of the
              software project.           software project.

Activity 1    A documented software       A documented software       Strength
              development plan is used    development plan is used
              for tracking the software   for tracking the software
              activities and              activities and
              communicating status.       communicating status.

Activity 2    The project's software      No documented procedure     Weakness
              development plan is         exists for revising the
              revised according to a      software development plan.
              documented procedure.

Activity 3    Software project            Software project            Observatio
              commitments and changes to  commitments and changes to  n
              commitments made to         commitments made to
              individuals and groups      individuals and groups
              external to the             external to the
              organization are reviewed   organization are reviewed
              with senior management      in periodic meetings.
              according to a documented   However, there is no
              procedure.                  documented procedure for
                                          these reviews.

Activity 4    Approved changes to         Approved changes to         Strength
              commitments that affect     commitments that affect
              the software project are    the software project are
              communicated to the         communicated to the
              members of the software     members of the software
              engineering group and       engineering group and
              other software-related      other software-related
              groups.                     groups through e-mail and
                                          weekly status meetings.

Activity 5    The sizes of the software   The sizes of the software   Weakness
              work products (or sizes of  work products (or sizes of
              the changes to the          the changes to the
              software work products)     software work products)
              are tracked, and            are not tracked.
              corrective actions are
              taken as necessary.

Activity 6    The project's software      The project's software      Strength
              effort and costs are        effort and costs are
              tracked, and corrective     tracked, and corrective
              actions are taken as        action is taken as
              necessary.                  necessary.

Activity 7    The project's critical      The project's critical      Weakness
              computer resources are      computer resources are not
              tracked, and corrective     tracked.
              actions are taken as
              necessary.

Activity 8    The project's software      The project's software      Weakness
              schedule is tracked, and    schedule is not tracked.
              corrective actions are
              taken as necessary.

Activity 9    Software engineering        Software engineering        Weakness
              technical activities are    technical activities are
              tracked, and corrective     not tracked.
              actions are taken as
              necessary.

Activity 10   The software risks          The software risks          Weakness
              associated with cost,       associated with cost,
              resource, schedule, and     resource, schedule, and
              technical aspects of the    technical aspects of the
              project are tracked.        project are not tracked.

Activity 11   Actual measurement data     Actual measurement data     Weakness
              and replanning data for     and replanning data for
              the software project are    the software project are
              recorded.                   not recorded.

Activity 12   The software engineering    The software engineering    Weakness
              group conducts periodic     group does not conduct
              internal reviews to track   periodic internal reviews
              technical progress, plans,  to track technical
              performance, and issues     progress, plans,
              against the software        performance, and issues
              development plan.           against the software
                                          development plan.

Activity 13   Formal reviews to address   No documented procedure     Weakness
              the accomplishments and     exists for conducting
              results of the software     formal reviews to address
              project are conducted at    the accomplishments and
              selected project            results of the software
              milestones according to a   project at selected
              documented procedure.       project milestones.

Measurement   Measurements are made and   Measurements are not made   Weakness
1             used to determine the       and used to determine the
              status of the software      status of software
              tracking and oversight      tracking and oversight
              activities.                 activities.

Verification  The activities for          The activities for          Weakness
1             software project tracking   software project tracking
              and oversight are reviewed  and oversight are not
              with senior management on   reviewed with senior
              a periodic basis.           management on a periodic
                                          basis.

Verification  The activities for          The activities for          Weakness
2             software project tracking   software project tracking
              and oversight are reviewed  and oversight are not
              with the project manager    reviewed with the project
              on both a periodic and      manager.
              event-driven basis.

Verification  The software quality        The software quality        Weakness
3             assurance group reviews     assurance group does not
              and/or audits the           review and/or audit the
              activities and work         activities and work
              products for software       products for software
              project tracking and        project tracking and
              oversight and reports the   oversight and report the
              results.                    results.
--------------------------------------------------------------------------------


                                    Table 4.4
                     
                     Software Project Tracking and Oversight
                       Findings for Administrative Security
                                      System

                                Administrative Security System
              ------------------------------------------------------------------
              Key practice                Finding                     Rating
------------  --------------------------  --------------------------  ----------
Commitment 1  A project software manager  A project software manager  Strength
              is designated to be         is designated to be
              responsible for the         responsible for the
              project's software          project's software
              activities and results.     activities and results.

Commitment 2  The project follows a       There is no organizational  Weakness
              written organizational      policy for managing the
              policy for managing the     software project.
              software project.

Ability 1     A software development      The project plan contains   Observatio
              plan for the software       the software development    n
              project is documented and   plan; however, this plan
              approved.                   has not been approved.

Ability 2     The project software        The project software        Weakness
              manager explicitly assigns  manager does not
              responsibility for          explicitly assign
              software work products and  responsibility for
              activities.                 software work products and
                                          activities.

Ability 3     Adequate resources and      Adequate resources and      Weakness
              funding are provided for    funding are not provided
              tracking the software       for tracking the software
              project.                    project.


Ability 4     The software managers are   The software managers are   Weakness
              trained in managing the     not trained in managing
              technical and personnel     the technical and
              aspects of the software     personnel aspects of the
              project.                    software project.

Ability 5     First-line software         First-line software         Weakness
              managers receive            managers did not receive
              orientation in the          orientation on the
              technical aspects of the    technical aspects of the
              software project.           software project.

Activity 1    A documented software       A documented software       Strength
              development plan is used    development plan is used
              for tracking the software   for tracking the software
              activities and              activities and
              communicating status.       communicating status.

Activity 2    The project's software      No documented procedure     Weakness
              development plan is         exists for revising the
              revised according to a      software development plan.
              documented procedure.

Activity 3    Software project            Software project            Observatio
              commitments and changes to  commitments and changes to  n
              commitments made to         commitments made to
              individuals and groups      individuals and groups
              external to the             external to the
              organization are reviewed   organization are reviewed
              with senior management      in periodic meetings.
              according to a documented   However, there is no
              procedure.                  documented procedure for
                                          these reviews.

Activity 4    Approved changes to         There is no evidence to     Weakness
              commitments that affect     show that changes to
              the software project are    commitments are either
              communicated to the         approved by project
              members of the software     managers or communicated
              engineering group and       to the software engineers
              other software-related      and other software-
              groups.                     related groups.

Activity 5    The sizes of the software   The sizes of the software   Weakness
              work products (or sizes of  work products (or sizes of
              the changes to the          the changes to the
              software work products)     software work products)
              are tracked, and            are not tracked.
              corrective actions are
              taken as necessary.

Activity 6    The project's software      The project's software      Weakness
              effort and costs are        effort and costs are not
              tracked, and corrective     tracked.
              actions are taken as
              necessary.

Activity 7    The project's critical      The project's critical      Weakness
              computer resources are      computer resources are not
              tracked, and corrective     tracked.
              actions are taken as
              necessary.

Activity 8    The project's software      The project's software      Strength
              schedule is tracked, and    schedule is tracked, and
              corrective actions are      corrective actions are
              taken as necessary.         taken as necessary.

Activity 9    Software engineering        Software engineering        Weakness
              technical activities are    technical activities are
              tracked, and corrective     not tracked.
              actions are taken as
              necessary.

Activity 10   The software risks          The software risks          Weakness
              associated with cost,       associated with cost,
              resource, schedule, and     resource, schedule, and
              technical aspects of the    technical aspects of the
              project are tracked.        project are not tracked.

Activity 11   Actual measurement data     Actual measurement data     Weakness
              and replanning data for     and replanning data for
              the software project are    the software project are
              recorded.                   not recorded.

Activity 12   The software engineering    The software engineering    Weakness
              group conducts periodic     group does notconduct
              internal reviews to track   periodic internal reviews
              technical progress, plans,  to track technical
              performance, and issues     progress, plans,
              against the software        performance, and issues
              development plan.           against the software
                                          development plan.

Activity 13   Formal reviews to address   No documented procedure     Weakness
              the accomplishments and     exists for conducting
              results of the software     formal reviews to address
              project are conducted at    the accomplishments and
              selected project            results of the software
              milestones according to a   project at selected
              documented procedure.       project milestones.

Measurement   Measurements are made and   Measurements are not made   Weakness
1             used to determine the       and used to determine the
              status of the software      status of software
              tracking and oversight      tracking and oversight
              activities.                 activities.

Verification  The activities for          The activities for          Weakness
1             software project tracking   software project tracking
              and oversight are reviewed  and oversight are not
              with senior management on   reviewed with senior
              a periodic basis.           management on a periodic
                                          basis.

Verification  The activities for          The activities for          Weakness
2             software project tracking   software project tracking
              and oversight are reviewed  and oversight are not
              with the project manager    reviewed with the project
              on both a periodic and      manager.
              event-driven basis.

Verification  The software quality        No software quality         Weakness
3             assurance group reviews     assurance group exists;
              and/or audits the           therefore, there are no
              activities and work         reviews and/or audits of
              products for software       the activities and work
              project tracking and        products for software
              oversight and reports the   project tracking and
              results.                    oversight.
--------------------------------------------------------------------------------

   CONCLUSIONS
---------------------------------------------------------- Chapter 4:2

Despite several practice strengths in this KPA, the number and
significance of the practice weaknesses that we found mean that
Customs' current process for tracking and overseeing its projects is
not repeatable, thereby increasing the chances of its software
projects being late, costing more than expected, and not performing
as intended. 


SOFTWARE QUALITY ASSURANCE
============================================================ Chapter 5

The purpose of software quality assurance is to independently review
and audit the software products and activities to verify that they
comply with the applicable procedures and standards and to provide
the software project and higher-level managers with the results of
these independent reviews and audits. 

According to the SW-CMM, a repeatable software quality assurance
process, among other things, includes (1) preparing a software
quality assurance plan for the project according to a documented
procedure, (2) having a written organizational policy for
implementing software quality assurance, (3) conducting audits of
designated work processes and products to verify compliance, (4)
documenting deviations identified in the software activities and
software work products and handling them according to a documented
procedure, and (5) having experts independent of the software quality
assurance group periodically review the activities and work products
of the project's software quality assurance group. 


   CUSTOMS LACKS A SOFTWARE
   QUALITY ASSURANCE PROCESS
---------------------------------------------------------- Chapter 5:1

All of the projects evaluated had extensive and significant software
quality assurance practice weaknesses.  For example, two of the
projects did not have a software quality assurance plan; and none of
the projects (1) had a written organizational policy for implementing
software quality assurance, (2) conducted audits of designated work
products to verify compliance, (3) documented deviations identified
in the software activities and software work products and handled
them according to a documented procedure, or (4) had experts
independent of the software quality assurance group periodically
review the group's work products.  In fact, only one of the projects,
AES, had any software quality assurance practice strengths, and these
strengths were limited to only a few practices.  In this case, the
project had assigned responsibility for software quality assurance to
a single individual and, for example, a software quality assurance
plan had been drafted, although not according to a documented
procedure.  This virtual absence of software quality assurance on
Customs' software projects increases greatly the risk of software
process and product standards not being met, which in turn increases
the risk of software not performing as intended, and costing more and
taking longer to develop than necessary. 

Table 5.1 provides a comprehensive list of the three projects'
strengths, weaknesses, and observations for the software quality
assurance KPA.  The specific findings supporting the practice ratings
cited in table 5.1 are in tables 5.2 through 5.4. 



                                    Table 5.1
                     
                            Software Quality Assurance

                                                                    Administrati
              Key practice                  NCAP 0.1    AES         ve
------------  ----------------------------  ----------  ----------  ------------
Commitment 1  The project follows a         Weakness    Weakness    Weakness
              written organizational
              policy for implementing
              software quality assurance
              (SQA).

Ability 1     A group that is responsible   Observatio  Strength    Weakness
              for coordinating and          n
              implementing SQA for the
              project (i.e., the SQA
              group) exists.

Ability 2     Adequate resources and        Weakness    Weakness    Weakness
              funding are provided for
              performing the SQA
              activities.

Ability 3     Members of the SQA group are  Weakness    Strength    Weakness
              trained to perform their SQA
              activities.

Ability 4     The members of the software   Weakness    Strength    Weakness
              project receive orientation
              on the role,
              responsibilities, authority,
              and value of the SQA group.

Activity 1    A SQA plan is prepared for    Weakness    Observatio  Weakness
              the software project                      n
              according to a documented
              procedure.

Activity 2    The SQA group's activities    Weakness    Weakness    Weakness
              are performed in accordance
              with the SQA plan.

Activity 3    The SQA group participates    Weakness    Weakness    Weakness
              in the preparation and
              review of the project's
              software development plan,
              standards, and procedures.

Activity 4    The SQA group reviews the     Weakness    Weakness    Weakness
              software engineering
              activities to verify
              compliance.

Activity 5    The SQA group audits          Weakness    Weakness    Weakness
              designated software work
              products to verify
              compliance.

Activity 6    The SQA group periodically    Weakness    Weakness    Weakness
              reports the results of its
              activities to the software
              engineering group.

Activity 7    Deviations identified in the  Weakness    Weakness    Weakness
              software activities and
              software work products are
              documented and handled
              according to a documented
              procedure.

Activity 8    The SQA group conducts        Weakness    Weakness    Weakness
              periodic reviews of its
              activities and findings with
              the customer's SQA
              personnel, as appropriate.

Measurement   Measurements are made and     Weakness    Weakness    Weakness
1             used to determine the cost
              and schedule status of the
              SQA activities.

Verification  The SQA activities are        Weakness    Weakness    Weakness
1             reviewed with senior
              management on a periodic
              basis.

Verification  The SQA activities are        Weakness    Weakness    Weakness
2             reviewed with the project
              manager on both a periodic
              and event-driven basis.

Verification  Experts independent of the    Weakness    Weakness    Weakness
3             SQA group periodically
              review the activities and
              software work products of
              the project's SQA group.
--------------------------------------------------------------------------------


                                    Table 5.2
                     
                     Software Quality Assurance Findings for
                                     NCAP 0.1

                           National Customs Automation Program 0.1
              ------------------------------------------------------------------
              Key practice                Finding                     Rating
------------  --------------------------  --------------------------  ----------
Commitment 1  The project follows a       There is no written         Weakness
              written organizational      organizational policy for
              policy for implementing     implementing SQA.
              software quality assurance
              (SQA).

Ability 1     A group that is             Although there is no group  Observatio
              responsible for             responsible for             n
              coordinating and            coordinating and
              implementing SQA for the    implementing SQA for the
              project (i.e., the SQA      project, there are plans
              group) exists.              to establish one and
                                          assign responsibility.

Ability 2     Adequate resources and      There is no SQA group, and  Weakness
              funding are provided for    no resources and funding
              performing the SQA          are provided for
              activities.                 performing the SQA
                                          activities.

Ability 3     Members of the SQA group    There is no SQA group or    Weakness
              are trained to perform      plan to provide SQA
              their SQA activities.       training.

Ability 4     The members of the          Project staff do not        Weakness
              software project receive    receive orientation on the
              orientation on the role,    role, responsibilities,
              responsibilities,           authority, and value of
              authority, and value of     the SQA group.
              the SQA group.

Activity 1    The SQA plan is prepared    There is no SQA plan or     Weakness
              for the software project    documented procedure for
              according to a documented   preparing one.
              procedure.

Activity 2    The SQA group's activities  There is no SQA group.      Weakness
              are performed in
              accordance with the SQA
              plan.

Activity 3    The SQA group participates  There is no SQA group.      Weakness
              in the preparation and
              review of the project's
              software development plan,
              standards, and procedures.

Activity 4    The SQA group reviews the   There is no SQA group.      Weakness
              software engineering
              activities to verify
              compliance.

Activity 5    The SQA group audits        There is no SQA group.      Weakness
              designated software work
              products to verify
              compliance.

Activity 6    The SQA group periodically  There is no SQA group.      Weakness
              reports the results of its
              activities to the software
              engineering group.

Activity 7    Deviations identified in    Procedures for handling     Weakness
              the software activities     deviations in testing
              and software work products  activities are documented.
              are documented and handled  However, procedures for
              according to a documented   handling deviations in
              procedure.                  other software development
                                          activities (such as
                                          compliance with
                                          organizational policy and
                                          standards and adherence to
                                          software development plan)
                                          are not documented.

Activity 8    The SQA group conducts      There is no SQA group.      Weakness
              periodic reviews of its
              activities and findings
              with the customer's SQA
              personnel, as appropriate.

Measurement   Measurements are made and   There is no SQA group.      Weakness
1             used to determine the cost
              and schedule status of the
              SQA activities.

Verification  The SQA activities are      There is no SQA group.      Weakness
1             reviewed with senior
              management on a periodic
              basis.

Verification  The SQA activities are      There is no SQA group.      Weakness
2             reviewed with the project
              manager on both a periodic
              and event-driven basis.

Verification  Experts independent of the  There is no SQA group.      Weakness
3             SQA group periodically
              review the activities and
              software work products of
              the project's SQA group.
--------------------------------------------------------------------------------



                                    Table 5.3
                     
                     Software Quality Assurance Findings for
                                       AES

                                   Automated Export System
              ------------------------------------------------------------------
              Key practice                Finding                     Rating
------------  --------------------------  --------------------------  ----------
Commitment 1  The project follows a       There is no written         Weakness
              written organizational      organizational policy for
              policy for implementing     implementing SQA.
              software quality assurance
              (SQA).

Ability 1     A group that is             A single individual is      Strength
              responsible for             responsible for
              coordinating and            coordinating and
              implementing SQA for the    implementing SQA for the
              project (i.e., the SQA      project.
              group) exists.

Ability 2     Adequate resources and      Adequate resources and      Weakness
              funding are provided for    funding are not provided
              performing the SQA          for performing the SQA
              activities.                 activities.

Ability 3     Members of the SQA group    Training has been provided  Strength
              are trained to perform      to the single individual
              their SQA activities.       responsible for SQA to
                                          prepare him to perform his
                                          activities.

Ability 4     The members of the          Members of the software     Strength
              software project receive    project are oriented on
              orientation on the role,    the role,
              responsibilities,           responsibilities,
              authority, and value of     authority, and value of
              the SQA group.              SQA. Briefings are held
                                          during retreats.

Activity 1    A SQA plan is prepared for  The SQA plan is not         Observatio
              the software project        prepared according to a     n
              according to a documented   documented procedure;
              procedure.                  however, there is a draft
                                          plan.

Activity 2    The SQA group's activities  The SQA group's activities  Weakness
              are performed in            are not performed in
              accordance with the SQA     accordance with the SQA
              plan.                       plan.

Activity 3    The SQA group participates  The individual responsible  Weakness
              in the preparation and      for SQA does not
              review of the project's     participate in the
              software development plan,  preparation and review of
              standards, and procedures.  the project's software
                                          development plan,
                                          standards, and procedures.

Activity 4    The SQA group reviews the   The individual responsible  Weakness
              software engineering        for SQA does not review
              activities to verify        the software engineering
              compliance.                 activities to verify
                                          compliance.

Activity 5    The SQA group audits        The individual responsible  Weakness
              designated software work    for SQA does not audit
              products to verify          software work products to
              compliance.                 verify compliance.

Activity 6    The SQA group periodically  The individual responsible  Weakness
              reports the results of its  for SQA periodically
              activities to the software  reports the results of
              engineering group.          testing activities to the
                                          software engineering
                                          group. Other activities,
                                          such as the results of
                                          reviews of work products
                                          and audits, are not
                                          reported.

Activity 7    Deviations identified in    Procedures for handling     Weakness
              the software activities     deviations in testing
              and software work products  activities are documented.
              are documented and handled  However, procedures for
              according to a documented   handling deviations in
              procedure.                  other software development
                                          activities (such as
                                          compliance with
                                          organizational policy and
                                          standards and adherence to
                                          the software development
                                          plan) are not documented.

Activity 8    The SQA group conducts      The SQA group does not      Weakness
              periodic reviews of its     conduct periodic reviews
              activities and findings     of its activities and
              with the customer's SQA     findings with the
              personnel, as appropriate.  customer's SQA personnel,
                                          as appropriate.

Measurement   Measurements are made and   Measurements are not made   Weakness
1             used to determine the cost  to determine the cost and
              and schedule status of the  schedule status of the SQA
              SQA activities.             activities.

Verification  The SQA activities are      Senior management is not    Weakness
1             reviewed with senior        made aware of the SQA
              management on a periodic    activities.
              basis.

Verification  The SQA activities are      The project manager and     Weakness
2             reviewed with the project   team members are not made
              manager on both a periodic  aware of SQA activities.
              and event-driven basis.

Verification  Experts independent of the  SQA's activities and        Weakness
3             SQA group periodically      software work products are
              review the activities and   not reviewed by experts
              software work products of   independent of the SQA
              the project's SQA group.    group.
--------------------------------------------------------------------------------



                                    Table 5.4
                     
                     Software Quality Assurance Findings for
                          Administrative Security System

                                Administrative Security System
              ------------------------------------------------------------------
              Key practice                Finding                     Rating
------------  --------------------------  --------------------------  ----------
Commitment 1  The project follows a       There is no written         Weakness
              written organizational      organizational policy for
              policy for implementing     implementing SQA.
              software quality assurance
              (SQA).

Ability 1     A group that is             No group is responsible     Weakness
              responsible for             for coordinating and
              coordinating and            implementing SQA for the
              implementing SQA for the    project.
              project (i.e., the SQA
              group) exists.

Ability 2     Adequate resources and      Adequate resources and      Weakness
              funding are provided for    funding are not provided
              performing the SQA          for performing the SQA
              activities.                 activities.

Ability 3     Members of the SQA group    There is no SQA group.      Weakness
              are trained to perform
              their SQA activities.

Ability 4     The members of the          Project staff do not        Weakness
              software project receive    receive orientation on the
              orientation on the role,    role, responsibilities,
              responsibilities,           authority, and value of
              authority, and value of     the SQA group.
              the SQA group.

Activity 1    A SQA plan is prepared for  There is no documented      Weakness
              the software project        procedure for preparing a
              according to a documented   SQA plan.
              procedure.

Activity 2    The SQA group's activities  There is no SQA group.      Weakness
              are performed in
              accordance with the SQA
              plan.

Activity 3    The SQA group participates  There is no SQA group.      Weakness
              in the preparation and
              review of the project's
              software development plan,
              standards, and procedures.

Activity 4    The SQA group reviews the   There is no SQA group.      Weakness
              software engineering
              activities to verify
              compliance.

Activity 5    The SQA group audits        There is no SQA group.      Weakness
              designated software work
              products to verify
              compliance.

Activity 6    The SQA group periodically  There is no SQA group.      Weakness
              reports the results of its
              activities to the software
              engineering group.

Activity 7    Deviations identified in    Procedures for handling     Weakness
              the software activities     deviations in testing
              and software work products  activities are documented.
              are documented and handled  However, procedures for
              according to a documented   handling deviations in
              procedure.                  other software development
                                          activities (such as
                                          compliance with
                                          organizational policy and
                                          standards and adherence to
                                          the software development
                                          plan) are not documented.

Activity 8    The SQA group conducts      There is no SQA group.      Weakness
              periodic reviews of its
              activities and findings
              with the customer's SQA
              personnel, as appropriate.

Measurement   Measurements are made and   There is no SQA group.      Weakness
1             used to determine the cost
              and schedule status of the
              SQA activities.

Verification  The SQA activities are      The SQA activities are not  Weakness
1             reviewed with senior        reviewed with senior
              management on a periodic    management on a periodic
              basis.                      basis.

Verification  The SQA activities are      The SQA activities are not  Weakness
2             reviewed with the project   reviewed with the project
              manager on both a periodic  manager on both a periodic
              and event-driven basis.     and event-driven basis.

Verification  Experts independent of the  There is no SQA group.      Weakness
3             SQA group periodically
              review the activities and
              software work products of
              the project's SQA group.
--------------------------------------------------------------------------------

   CONCLUSIONS
---------------------------------------------------------- Chapter 5:2

Customs' software quality assurance process has many weaknesses and
is, therefore, undefined and undisciplined.  As a result, Customs
cannot provide management with independent information about
adherence to software process and product standards.  To develop and
maintain software effectively, Customs must adopt a structured and
rigorous approach to software quality assurance. 


SOFTWARE CONFIGURATION MANAGEMENT
============================================================ Chapter 6

The purpose of software configuration management is to establish and
maintain the integrity of the products of the software project
throughout the project's software life- cycle.  Software
configuration management involves establishing product baselines and
systematically controlling changes to them. 

According to the SW-CMM, a repeatable software configuration
management process, among other things, includes (1) preparing a
software configuration management plan according to a documented
procedure, (2) establishing a configuration management library system
as a repository for the software baselines, (3) identifying software
work products to be placed under configuration management, (4)
controlling the release of products from the software baseline
library according to a documented procedure, (5) following a written
organizational policy for implementing software configuration
management, (6) recording the status of configuration items/units
according to a documented procedure, (7) making and using
measurements to determine the status of the software configuration
management activities, and (8) reviewing software configuration
management activities with senior management on a periodic basis. 


   CUSTOMS' SOFTWARE CONFIGURATION
   MANAGEMENT PROCESS IS IMMATURE
---------------------------------------------------------- Chapter 6:1

Customs' processes for software configuration management show
strengths in several activities.  For example, all three projects had
developed software configuration management plans according to a
documented procedure.  Also, two of the projects (NCAP 0.1 and AES)
established configuration management library systems as repositories
for the software baselines, identified software work products to be
placed under configuration management, and controlled the release of
products from the software baseline library according to a documented
procedure. 

However, the projects had many practice weakness that collectively
jeopardize Customs' ability to maintain the integrity of the
projects' software products.  For example, none of the projects had a
written organizational policy for implementing software configuration
management, and none had documented procedures for recording the
status of configuration items (e.g., code, documents).  Moreover,
none of the projects made or used measurements to determine the
status of the software configuration management activities, or
reviewed software configuration management activities with senior
management on a periodic basis. 

Table 6.1 provides a comprehensive list of the three projects'
strengths and weaknesses for the software configuration management
KPA.  The specific findings supporting the practice ratings cited in
table 6.1 are in tables 6.2 through 6.4. 



                                    Figure 6.1
                     
                        Software Configuration Management

                                                                    Administrati
              Key practice                  NCAP 0.1    AES         ve
------------  ----------------------------  ----------  ----------  ------------
Commitment 1  The project follows a         Weakness    Weakness    Weakness
              written organizational
              policy for implementing
              software configuration
              management (SCM).

Ability 1     A board having the authority  Weakness    Strength    Weakness
              for managing the project's
              software baselines (i.e., a
              software configuration
              control board) exists or is
              established.

Ability 2     A group that is responsible   Weakness    Strength    Strength
              for coordinating and
              implementing SCM for the
              project (i.e., the SCM
              group) exists.

Ability 3     Adequate resources and        Weakness    Weakness    Weakness
              funding are provided for
              performing the SCM
              activities.

Ability 4     Members of the SCM group are  Weakness    Strength    Weakness
              trained in the objectives,
              procedures, and methods for
              performing their SCM
              activities.

Ability 5     Members of the software       Weakness    Strength    Weakness
              engineering group and other
              software-related groups are
              trained to perform their SCM
              activities.

Activity 1    A SCM plan is prepared for    Strength    Strength    Strength
              each software project
              according to a documented
              procedure.

Activity 2    A documented and approved     Weakness    Weakness    Weakness
              SCM plan is used as the
              basis for performing the SCM
              activities.

Activity 3    A configuration management    Strength    Strength    Weakness
              library system is
              established as a repository
              for the software baselines.

Activity 4    The software work products    Strength    Strength    Weakness
              to be placed under
              configuration management are
              identified.

Activity 5    Change requests and problem   Strength    Weakness    Weakness
              reports for all
              configuration items/risks
              are initiated, recorded,
              reviewed, approved, and
              tracked according to a
              documented procedure.

Activity 6    Changes to baselines are      Strength    Weakness    Weakness
              controlled according to a
              documented procedure.

Activity 7    Products from the software    Strength    Strength    Weakness
              baseline library are created
              and their release is
              controlled according to a
              documented procedure.

Activity 8    The status of configuration   Weakness    Weakness    Weakness
              items/units is recorded
              according to a documented
              procedure.

Activity 9    Standard reports documenting  Weakness    Weakness    Weakness
              the SCM activities and the
              contents of the software
              baseline are developed and
              made available to affected
              groups and individuals.

Activity 10   Software baseline audits are  Weakness    Weakness    Weakness
              conducted according to a
              documented procedure.

Measurement   Measurements are made and     Weakness    Weakness    Weakness
1             used to determine the status
              of the SCM activities.

Verification  The SCM activities are        Weakness    Weakness    Weakness
1             reviewed with senior
              management on a periodic
              basis.

Verification  The SCM activities are        Weakness    Strength    Weakness
2             reviewed with the project
              manager on both a periodic
              and event-driven basis.

Verification  The SCM group periodically    Weakness    Weakness    Weakness
3             audits software baselines to
              verify that they conform to
              the documentation that
              defines them.

Verification  The software quality          Weakness    Weakness    Weakness
4             assurance group reviews and/
              or audits the activities and
              work products for SCM and
              reports the results.
--------------------------------------------------------------------------------



                                    Table 6.2
                     
                        Software Configuration Management
                              Findings for NCAP 0.1

                           National Customs Automation Program 0.1
              ------------------------------------------------------------------
              Key practice                Finding                     Rating
------------  --------------------------  --------------------------  ----------
Commitment 1  The project follows a       The project has no          Weakness
              written organizational      organizational policy for
              policy for implementing     implementing SCM.
              software configuration
              management (SCM).

Ability 1     A board having the          The project does not have   Weakness
              authority for managing the  a SCM board with authority
              project's software          for managing software
              baselines (i.e., a          baselines.
              software configuration
              control board) exists or
              is established.

Ability 2     A group that is             The project does not have   Weakness
              responsible for             a group that is
              coordinating and            responsible for
              implementing SCM for the    coordinating and
              project (i.e., the SCM      implementing SCM
              group) exists.              functions.

Ability 3     Adequate resources and      Adequate resources and      Weakness
              funding are provided for    funding are not provided
              performing the SCM          for performing the SCM
              activities.                 activities.

Ability 4     Members of the SCM group    There is no SCM group.      Weakness
              are trained in the
              objectives, procedures,
              and methods for performing
              their SCM activities.

Ability 5     Members of the software     Members of the software     Weakness
              engineering group and       engineering group and
              other software-related      other software-related
              groups are trained to       groups are not trained to
              perform their SCM           perform their SCM
              activities.                 activities

Activity 1    A SCM plan is prepared for  A SCM plan was prepared     Strength
              each software project       according to procedures
              according to a documented   documented in the October
              procedure.                  1996 SDLC.

Activity 2    A documented and approved   The documented and          Weakness
              SCM plan is used as the     approved SCM plan is used
              basis for performing the    as the basis for
              SCM activities.             performing code control;
                                          however, the plan is not
                                          used as a basis for doing
                                          SCM on software
                                          documentation and other
                                          software engineering
                                          products such as cost
                                          estimates and schedules.

Activity 3    A configuration management  A configuration management  Strength
              library system is           library system is
              established as a            established as a
              repository for the          repository for the
              software baselines.         software baselines.

Activity 4    The software work products  The software work products  Strength
              to be placed under          to be placed under
              configuration management    configuration management
              are identified.             are identified.

Activity 5    Change requests and         Change requests and         Strength
              problem reports for all     problem items/units are
              configuration items/risks   initiated, recorded,
              are initiated, recorded,    reviewed, approved, and
              reviewed, approved, and     tracked according to a
              tracked according to a      procedure documented in
              documented procedure.       the SCM plan.

Activity 6    Changes to baselines are    Changes to baselines are    Strength
              controlled according to a   controlled according to a
              documented procedure.       documented procedure in
                                          the SCM plan.

Activity 7    Products from the software  Products from the software  Strength
              baseline library are        baseline library are
              created and their release   created and their release
              is controlled according to  is controlled according to
              a documented procedure.     the SCM plan.

Activity 8    The status of               The status of SCM items/    Weakness
              configuration items/units   units are not
              is recorded according to a  recorded according to a
              documented procedure.       documented procedure.

Activity 9    Standard reports            Standard reports            Weakness
              documenting the SCM         documenting the SCM
              activities and the          activities and contents of
              contents of the software    the software baseline are
              baseline are developed and  not developed.
              made available to affected
              groups and individuals.

Activity 10   Software baseline audits    Software baseline audits    Weakness
              are conducted according to  are not conducted.
              a documented procedure.

Measurement   Measurements are made and   No measurements are taken   Weakness
1             used to determine the       to determine the status of
              status of the SCM           SCM activities.
              activities.

Verification  The SCM activities are      The SCM activities are not  Weakness
1             reviewed with senior        reviewed with senior
              management on a periodic    management on a periodic
              basis.                      basis.

Verification  The SCM activities are      The SCM activities are not  Weakness
2             reviewed with the project   reviewed with the project
              manager on both a periodic  manager on both a periodic
              and event-driven basis.     and event-driven basis.

Verification  The SCM group periodically  No SCM audits are done to   Weakness
3             audits software baselines   verify that software
              to verify that they         baselines conform to the
              conform to the              documentation that defines
              documentation that defines  them.
              them.

Verification  The software quality        There is no quality         Weakness
4             assurance group reviews     assurance group;
              and/or audits the           therefore, no one reviews
              activities and work         and/or audits the
              products for SCM and        activities and work
              reports the results.        products for SCM
                                          activities performed.
--------------------------------------------------------------------------------



                                    Table 6.3
                     
                        Software Configuration Management
                                 Findings for AES

                                   Automated Export System
              ------------------------------------------------------------------
              Key practice               Finding                    Rating
------------  -------------------------  -------------------------  ------------
Commitment 1  The project follows a      The project does not have  Weakness
              written organizational     a written organizational
              policy for implementing    policy for SCM.
              software configuration
              management (SCM).

Ability 1     A board having the         AES has a Configuration    Strength
              authority for managing     Control Board. This board
              the project's software     has the authority for
              baselines (i.e., a         managing the project's
              software configuration     software baselines. The
              control board) exists or   board consists of program
              is established.            support and user
                                         personnel.

Ability 2     A group that is            A group that is            Strength
              responsible for            responsible for
              coordinating and           coordinating and
              implementing SCM for the   implementing SCM for the
              project (i.e., the SCM     project (i.e., the SCM
              group) exists.             group) exists.

Ability 3     Adequate resources and     Adequate resources and     Weakness
              funding are provided for   funding are not provided
              performing the SCM         for performing the SCM
              activities.                activities.

Ability 4     Members of the SCM group   Members of the SCM group   Strength
              are trained in the         have extensive experience
              objectives, procedures,    in the objectives,
              and methods for            procedures, and methods
              performing their SCM       for performing their SCM
              activities.                activities and receive
                                         on-the-job training.

Ability 5     Members of the software    Members of the software    Strength
              engineering group and      engineering group and
              other software-related     other software-related
              groups are trained to      groups have been briefed
              perform their SCM          on performing their SCM
              activities.                duties at periodic
                                         retreat meetings.

Activity 1    A SCM plan is prepared     A SCM plan was prepared    Strength
              for each software project  according to procedures
              according to a documented  documented in the October
              procedure.                 1996 SDLC.

Activity 2    A documented and approved  The documented and         Weakness
              SCM plan is used as the    approved SCM plan is used
              basis for performing the   as the basis for
              SCM activities.            performing code control;
                                         however, the plan is not
                                         used as a basis for doing
                                         SCM on software
                                         documentation and other
                                         software engineering
                                         products such as cost
                                         estimates and schedules.

Activity 3    A configuration            The AES library is used    Strength
              management library system  as a repository for
              is established as a        software baselines.
              repository for the
              software baselines.

Activity 4    The software work          Software work products to  Strength
              products to be placed      be placed under SCM are
              under configuration        identified in the SCM
              management are             plan.
              identified.

Activity 5    Change requests and        Change requests and        Weakness
              problem reports for all    problem reports for all
              configuration items/       configuration items are
              units are initiated,       not initiated, recorded,
              recorded, reviewed,        reviewed, approved, and
              approved, and tracked      tracked according to a
              according to a documented  documented procedure.
              procedure.

Activity 6    Changes to baselines are   The procedures for         Weakness
              controlled according to a  controlling changes to
              documented procedure.      baselines are documented
                                         in the SCM and SQA plans.
                                         However, there was no
                                         evidence provided to show
                                         that these procedures
                                         were being followed.

Activity 7    Products from the          Products from the          Strength
              software baseline library  baseline library are
              are created and their      created, and releases are
              release is controlled      controlled according to
              according to a documented  procedures in the SCM
              procedure.                 plan.

Activity 8    The status of              The status of              Weakness
              configuration items/       configuration items/
              units is recorded          units is not recorded
              according to a documented  according to a documented
              procedure.                 procedure.

Activity 9    Standard reports           No reports documenting     Weakness
              documenting the SCM        SCM activities and
              activities and the         contents of the software
              contents of the software   baseline are made
              baseline are developed     available to affected
              and made available to      groups and individuals.
              affected groups and
              individuals.

Activity 10   Software baseline audits   Software baseline audits   Weakness
              are conducted according    are not conducted.
              to a documented
              procedure.

Measurement   Measurements are made and  Measurements are not made  Weakness
1             used to determine the      and used to determine the
              status of the SCM          status of the SCM
              activities.                activities.

Verification  The SCM activities are     The SCM activities are     Weakness
1             reviewed with senior       not reviewed with senior
              management on a periodic   management on a periodic
              basis.                     basis.

Verification  The SCM activities are     Project management is      Strength
2             reviewed with the project  updated on SCM activities
              manager on both a          on a weekly basis and bi-
              periodic and event-        monthly at the retreat
              driven basis.              meetings, and as events
                                         warrant.

Verification  The SCM group              The SCM group does not     Weakness
3             periodically audits        verify that the software
              software baselines to      baseline conforms to the
              verify that they conform   documentation that
              to the documentation that  defines it.
              defines them.

Verification  The software quality       The SQA group does not     Weakness
4             assurance group reviews    perform reviews and/or
              and/or audits the          audits of the activities
              activities and work        and work products for SCM
              products for SCM and       and does not report the
              reports the results.       results.
--------------------------------------------------------------------------------



                                    Table 6.4
                     
                        Software Configuration Management
                       Findings for Administrative Security
                                      System

                                Administrative Security System
              ------------------------------------------------------------------
              Key practice                Finding                     Rating
------------  --------------------------  --------------------------  ----------
Commitment 1  The project follows a       The project does not have   Weakness
              written organizational      a written organizational
              policy for implementing     policy for implementing
              software configuration      software configuration
              management (SCM).           management.

Ability 1     A board having the          The project does not have   Weakness
              authority for managing the  a board (software
              project's software          configuration control
              baselines (i.e., a          board) with the authority
              software configuration      for managing the software
              control board) exists or    baselines.
              is established.

Ability 2     A group that is             The project manager is      Strength
              responsible for             responsible for
              coordinating and            coordinating and
              implementing SCM for the    implementing SCM
              project (i.e., the SCM      functions.
              group) exists.

Ability 3     Adequate resources and      Adequate resources and      Weakness
              funding are provided for    funding are not provided
              performing the SCM          for performing the SCM
              activities.                 activities.

Ability 4     Members of the SCM group    There is no evidence that   Weakness
              are trained in the          the personnel assigned SCM
              objectives, procedures,     functions on the project
              and methods for performing  are trained in objectives,
              their SCM activities.       procedures, and methods
                                          for doing SCM.

Ability 5     Members of the software     Members of the software     Weakness
              engineering group and       engineering group are not
              other software-related      trained to perform their
              groups are trained to       SCM functions.
              perform their SCM
              activities.

Activity 1    A SCM plan is prepared for  A SCM plan was prepared     Strength
              each software project       according to procedures
              according to a documented   documented in the October
              procedure.                  1996 SDLC.

Activity 2    A documented and approved   Although there is a SCM     Weakness
              SCM plan is used as the     plan for the project,
              basis for performing the    there is no evidence that
              SCM activities.             it is used to perform SCM
                                          functions.

Activity 3    A configuration management  A configuration management  Weakness
              library system is           library system is
              established as a            established and used as a
              repository for the          repository for software
              software baselines.         baselines for source code.
                                          However, other items in
                                          the software baseline
                                          (such as plans and
                                          schedules) are not
                                          identified or controlled.

Activity 4    The software work products  Software work products to   Weakness
              to be placed under          be placed under
              configuration management    configuration management
              are identified.             (other than source code)
                                          are not identified.

Activity 5    Change requests and         The Automated Request For   Weakness
              problem reports for all     Service system handles
              configuration items/risks   change requests and
              are initiated, recorded,    problem reports. However,
              reviewed, approved, and     there is no documented
              tracked according to a      procedure for its use and
              documented procedure.       there was no evidence that
                                          the system was being used
                                          to initiate, record,
                                          review, approve, or track
                                          change requests and
                                          problem reports.

Activity 6    Changes to baselines are    There is no documented      Weakness
              controlled according to a   procedure to control
              documented procedure.       changes to baselines.


Activity 7    Products from the software  There is no documented      Weakness
              baseline library are        procedure that defines how
              created and their release   products from the software
              is controlled according to  baseline library should be
              a documented procedure.     created and released.

Activity 8    The status of               The status of SCM items/    Weakness
              configuration items/units   units is not recorded
              is recorded according to a  according to a documented
              documented procedure.       procedure.

Activity 9    Standard reports            Affected groups and         Weakness
              documenting the SCM         individuals are not
              activities and the          notified of the SCM
              contents of the software    activities or informed of
              baseline are developed and  the contents of software
              made available to affected  baselines.
              groups and individuals.

Activity 10   Software baseline audits    Software baseline audits    Weakness
              are conducted according to  are not conducted.
              a documented procedure.

Measurement   Measurements are made and   No measurements are made    Weakness
1             used to determine the       to determine the status of
              status of the SCM           the SCM activities.
              activities.

Verification  The SCM activities are      The SCM activities are not  Weakness
1             reviewed with senior        reviewed with senior
              management on a periodic    management on a periodic
              basis.                      basis.

Verification  The SCM activities are      Project managers are not    Weakness
2             reviewed with the project   updated on SCM activities
              manager on both a periodic  on a periodic basis.
              and event-driven basis.

Verification  The SCM group periodically  The SCM does not            Weakness
3             audits software baselines   periodically audit the
              to verify that they         software baselines.
              conform to the
              documentation that defines
              them.

Verification  The software quality        There is no quality         Weakness
4             assurance group reviews     assurance group.
              and/or audits the
              activities and work
              products for SCM and
              reports the results.
--------------------------------------------------------------------------------

   CONCLUSIONS
---------------------------------------------------------- Chapter 6:2

Customs has many configuration management process weaknesses, and
thus its capability to establish and maintain the integrity of the
wide range of software products is nonrepeatable and ineffective. 
Without a mature configuration management process, Customs can lose
control of the current software product baseline, potentially
producing and using inconsistent product versions and creating
operational problems. 


CUSTOMS LACKS A SOFTWARE PROCESS
IMPROVEMENT PROGRAM
============================================================ Chapter 7

To consistently develop software with specified functionality on time
and within budget, Customs must improve its software development
processes.  According to SEI, an effective process improvement
program includes (1) establishing a process improvement management
structure, (2) developing a process improvement plan, (3) determining
the organization's baseline capability and using this as a basis for
targeting process initiatives, and (4) dedicating adequate resources
for implementing the plan. 

Although it has attempted in the past to initiate and sustain process
improvement activities, these activities were terminated without
having improved Customs processes.  Currently, Customs has no
software process improvement program. 


   SEI HAS DEFINED A FIVE-PHASE
   MODEL FOR SOFTWARE PROCESS
   IMPROVEMENT
---------------------------------------------------------- Chapter 7:1

In 1996, SEI published a software process improvement model, called
IDEAL.\1 This model has five phases:  Initiating, Diagnosing,
Establishing, Acting, and Leveraging-- IDEAL.  Each of the phases is
summarized below. 

  -- Initiating phase:  During this phase, an organization
     establishes the management structure of the process improvement
     program, defines and assigns roles and responsibilities,
     allocates initial resources, develops a plan to guide the
     organization through the first three phases of the program, and
     obtains management approval and funding for the program.  Two
     key organizational components of the program management
     structure established during this phase are a management
     steering group and a software engineering process group (SEPG). 
     Responsibility for this phase rests with senior management. 

  -- Diagnosing phase:  During this phase, the SEPG appraises the
     current level of software process maturity to establish a
     baseline of the organization's process capability, and
     identifies any ongoing process improvement initiatives.  The
     SEPG then uses the baseline to identify weaknesses and target
     process improvement activities.  It also compares these targeted
     activities with any ongoing process improvement activities and
     reconciles any differences.  Responsibility for this phase rests
     primarily with line managers and practitioners. 

  -- Establishing phase:  During this phase, the SEPG prioritizes the
     software process improvement activities and develops strategies
     for pursuing them.  It then develops a process improvement
     action plan that details the activities and strategies and
     includes measurable goals for the activities and metrics for
     monitoring progress against the goals.  Also during this phase,
     the resources needed to implement the plan are committed and
     training is provided for SEPG's technical working groups, who
     will be responsible for developing and testing new or improved
     processes.  Responsibility for this phase resides primarily with
     line managers and practitioners. 

  -- Acting phase:  In this phase, the work groups create and
     evaluate new and improved processes.  Evaluation of the
     processes is based on pilot tests that are formally planned and
     executed.  If the pilots are successful, the work groups develop
     plans for organization-wide adoption and institutionalization,
     and once approved, execute them.  Responsibility for this phase
     resides primarily with line managers and practitioners. 

  -- Leveraging phase:  During this phase, results and lessons
     learned from earlier phases are assessed and applied, as
     appropriate, to enhance the process improvement program's
     structure and plans.  Responsibility for this phase rests
     primarily with senior management. 


--------------------
\1 IDEAL:  A User's Guide for Software Process Improvement
(CMU/SEI-96-HB-001). 


   CUSTOMS DOES NOT CURRENTLY HAVE
   A SOFTWARE PROCESS IMPROVEMENT
   PROGRAM
---------------------------------------------------------- Chapter 7:2

In 1996, Customs initiated some limited software process improvement
activities.  Specifically, it hired a contractor to develop a process
improvement plan, which was completed in September 1996.  According
to the plan, Customs was to reach CMM level 2 process maturity (the
repeatable level) by 1998 and CMM level 3 (the defined level) by
2002.  Customs began limited implementation of the plan in May of
1997, when it established process improvement teams for two
KPAs--software project planning and project tracking and oversight. 
Generally, the teams were tasked with defining, implementing, and
maintaining CMM-based processes for their respective KPAs.  Customs
did not staff or fund any other KPA improvement activities at this
time.  In August 1997, Customs discontinued all process improvement
activities.  Customs officials stated that this decision was based on
the need to focus staff and resources on the agency's Year 2000
conversion program. 

Currently, Customs does not have a software development process
improvement program, and it has not taken steps to initiate one. 
Although it has assigned two people part-time to process improvement,
it has not assigned organizational responsibility and authority,
established a program management structure, developed a plan of
action, and committed resources needed (trained staff and funding) to
execute the plan. 


   CONCLUSIONS
---------------------------------------------------------- Chapter 7:3

Customs does not have an effective software development process
improvement program.  As a result, it cannot expect to improve its
immature software development processes. 


OVERALL CONCLUSIONS,
RECOMMENDATIONS, AND AGENCY
COMMENTS
============================================================ Chapter 8

Customs develops and maintains software for systems that are critical
to its ability to fulfill its mission.  However, its software
development processes are ad hoc and sometimes chaotic, and are not
repeatable even on a project-by-project basis.  As a result, Customs'
success or failure in developing software depends largely on specific
individuals, rather than on well-defined and disciplined software
management practices.  This greatly reduces the probability that its
software projects, whether new developments or maintenance of
existing software, will consistently perform as intended and be
delivered on schedule and within budget.  For Customs software
projects to mature beyond this initial level, the agency must
implement basic management controls and instill self-discipline in
its software projects. 

Customs acknowledges the importance of software process maturity and
the need to improve its software development processes.  However, it
does not have a program for improving its software development
processes and has not begun to establish one.  Until it does, Customs
has no assurance that its large investment in software development
and maintenance will produce systems that perform needed functions,
on time, and within budget. 


   RECOMMENDATIONS
---------------------------------------------------------- Chapter 8:1

We recommend that, after ensuring that its mission-critical systems
are Year 2000 compliant but before investing in major software
development efforts like ACE, the Commissioner of Customs direct the
Chief Information Officer to

  -- assign responsibility and authority for software development
     process improvement;

  -- develop and implement a formal plan for software development
     process improvement that is based on the software capability
     evaluation results contained in this report and specifies
     measurable goals and time frames, prioritizes initiatives,
     estimates resource requirements (trained staff and funding) and
     defines a process improvement management structure;

  -- ensure that every new software development effort in Customs
     adopts processes that satisfy at least SW-CMM level 2
     requirements; and

  -- ensure that process improvement activities are initiated for all
     ongoing essential software maintenance projects. 


   AGENCY COMMENTS AND OUR
   EVALUATION
---------------------------------------------------------- Chapter 8:2

In its written comments on a draft of this report, Customs
acknowledged the importance of software process improvement and
maturity.  Also, it agreed with GAO's overall findings, including
that Customs' software development processes have not attained SW-CMM
level 2 maturity. 

To address these weaknesses, Customs stated that it has taken the
first step toward implementing our recommendations by assigning
responsibility and authority for software process improvement as part
of a reorganization of its Office of Information and Technology,
which Customs stated will be implemented in early 1999.  Customs
further stated that once the reorganization is implemented, a formal
software process improvement program will be established, and that
this program will include definition of an action plan, commitment of
resources, and specification of goals for achieving CMM levels 2 and
3.  According to Customs, these improvement activities are in their
early stages.  When they are successfully implemented, they should
address many of our recommendations. 

Customs also stated that because its legacy systems are aging and
need to be enhanced and replaced, software process improvement must
occur in parallel with continued software development investments. 
History has shown that attempting to modernize without first
instituting disciplined software processes has been a characteristic
of failed modernization programs.\1 Until it implements disciplined
software processes (i.e., at least level 2 process maturity), Customs
cannot prudently manage major system investments, such as ACE with an
estimated life cycle cost exceeding $1 billion. 

Customs' comments also included a request to meet with us to discuss
system-specific KPA practice strength and weakness determinations. 
We met prior to requesting comments on a draft of this report and
then again on January 12, 1999, to discuss SEI's SW-CMM requirements
and the basis for our determinations.  We are prepared to continue
assisting Customs as it improves its software processes. 

Appendix I provides the full text of Customs' comments and our
responses to additional Customs comments not discussed above. 



(See figure in printed edition.)Appendix I

--------------------
\1 Tax Systems Modernization:  Management and Technical Weaknesses
Must Be Corrected If Modernization Is To Succeed (GAO/AIMD-95-156,
July 26, 1995), Tax Systems Modernization:  Actions Underway But IRS
Has Not Yet Corrected Management and Technical Weaknesses
(GAO/AIMD-96-106, June 7, 1996), and Air Traffic Control:  Immature
Software Acquisition Processes Increase FAA System Acquisition Risks
(GAO/AIMD-97-47, March 21, 1997). 


COMMENTS FROM THE DEPARTMENT OF
THE TREASURY
============================================================ Chapter 8



(See figure in printed edition.)


The following are responses to additional comments in Customs' letter
dated December 16, 1998. 

GAO COMMENTS

1.  The report has been modified to reflect Customs' comment. 

2.  Customs provided us a copy of its new Systems Development Life
Cycle (SDLC) document after it provided us comments on a draft of
this report, and we have not yet evaluated it.  To develop systems
effectively, Customs will have to ensure that its SDLC addresses the
software development KPAs discussed in this report and that it
effectively implements the SDLC. 

3.  The statement that "Customs has a history of performing poorly
when developing software-intensive systems" has been removed from the
report.  Regarding the statement in the report referring to "Customs'
limited success in delivering promised system capabilities on time
and within budget," there is clear evidence that Customs has not been
successful in developing ACE software on time.  Specifically, the
first release of ACE, which is Customs' largest software development
effort, was delivered over 8 months late.  Also, because Customs does
not track actual ACE costs by release against original estimates, it
does not know the cost performance history of ACE.  With the respect
to the three systems that Customs cites as having been successfully
delivered, the first (the Year 2000 program) is still incomplete and
thus its success cannot yet be assessed.  Because we have not
previously reviewed the other two (AES and Tinman), we cannot comment
on either's success in delivering promised capabilities on time and
within budget.  The report has been clarified to reflect these
points. 

4.  The report has been clarified to reflect Customs' comment. 


MAJOR CONTRIBUTORS TO THIS REPORT
========================================================== Appendix II

ACCOUNTING AND INFORMATION
MANAGEMENT DIVISION

Dr.  Rona B.  Stillman, Chief Scientist for Computers and
 Telecommunications
Jack L.  Brock, Director, Governmentwide and Defense Information
 Systems
Madhav S.  Panwar, Technical Assistant Director
Dr.  Nabajyoti Barkakati, Technical Assistant Director
Prithviraj Mukherji, Assistant Director
Carl M.  Urie, Assistant Director
Bernard R.  Anderson, Senior Information Systems Analyst
Suzanne M.  Burns, Senior Information Systems Analyst

ATLANTA FIELD OFFICE

Teresa F.  Tucker, Senior Information Systems Analyst


*** End of document. ***