Customs Service Modernization: Serious Management and Technical
Weaknesses Must Be Corrected (Chapter Report, 02/26/99, GAO/AIMD-99-41).

Pursuant to a congressional request, GAO reviewed the Customs Service's
management of the Automated Commercial Environment (ACE), focusing on
whether Customs has adequately justified ACE cost-effectiveness.

GAO noted that: (1) Customs is not managing ACE effectively, and it does
not have a firm basis for concluding that ACE is a cost-effective
solution to modernizing its commercial environment; (2) GAO found
serious weaknesses relating to architectural definition, investment
management, and software development and acquisition that must be
corrected before further investment in ACE is justified; (3) Customs is
not building ACE within the context of a complete and enforced systems
architecture; (4) in May 1998, GAO reported that Customs' architecture
was incomplete because it was not based on a complete understanding of
its enterprisewide functional and information needs; (5) GAO also
reported that Customs had not yet instituted effective procedures for
ensuring compliance with the architecture once it is completed; (6)
until its architecture is completed and effectively enforced, Customs
will not have adequate assurance that information systems like ACE will
optimally support its needs across all business areas; (7) further,
Customs lacks a reliable estimate of what ACE will cost to build,
deploy, and maintain, and Customs has neither adequately justified, nor
is it effectively monitoring, ACE's cost-effectiveness; (8)
specifically, Customs did not use rigorous cost estimating techniques in
preparing its cost estimate, and did not disclose the inherent
imprecision of the estimate; (9) additionally, Customs omitted costs and
inflated benefits in preparing its cost-benefit analysis; (10) moreover,
Customs is not using effective incremental investment management
practices; (11) while Customs plans to develop/acquire ACE in 21
increments, these increments are not individually cost-benefit
justified, and Customs is not determining what benefits each increment,
once operational, actually provides; (12) as a result, Customs will not
know if ACE's expected return-on-investment is actually being realized
until it has already spent hundreds of millions of dollars
developing/acquiring the entire system; and (13) GAO found that Customs
has neither the capability to effectively develop nor acquire ACE and
that its processes for doing both, according to widely accepted and
proven software capability maturity models, are ad hoc, immature, and
ineffective.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  AIMD-99-41
     TITLE:  Customs Service Modernization: Serious Management and 
             Technical Weaknesses Must Be Corrected
      DATE:  02/26/99
   SUBJECT:  ADP procurement
             Customs administration
             Information resources management
             Strategic information systems planning
             Cost effectiveness analysis
             Systems conversions
             Systems design
             Life cycle costs
             Cost analysis
             Computer software verification and validation
IDENTIFIER:  Customs Service Automated Commercial Environment System
             Customs Service National Customs Automation Program
             Software Capability Maturity Model
             Treasury International Trade Data System
             
******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved.  Major          **
** divisions and subdivisions of the text, such as Chapters,    **
** Sections, and Appendixes, are identified by double and       **
** single lines.  The numbers on the right end of these lines   **
** indicate the position of each of the subsections in the      **
** document outline.  These numbers do NOT correspond with the  **
** page numbers of the printed product.                         **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                                            **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************


Cover
================================================================ COVER


Report to Congressional Requesters

February 1999

CUSTOMS SERVICE MODERNIZATION -
SERIOUS MANAGEMENT AND TECHNICAL
WEAKNESSES MUST BE CORRECTED

GAO/AIMD-99-41

Customs Service Modernization

(511114)


Abbreviations
=============================================================== ABBREV

  ACE - Automated Commercial Environment
  ACS - Automated Commercial System
  CIO - chief information officer
  CMM - Capability Maturity Model\SM
  IRB - Investment Review Board
  IT - information technology
  ITDS - International Trade Data System
  KPA - key process area
  NAFTA - North American Free Trade Agreement
  NCAP - National Customs Automation Program
  OMB - Office of Management and Budget
  SA-CMM - Software Acquisition Capability Maturity Model\SM
  SCE - software capability evaluation
  SEI - Software Engineering Institute
  SW-CMM - Software Development Capability Maturity Model\SM
  SW-CMM -
  SA-CMM -

Letter
=============================================================== LETTER


B-280416

February 26, 1999

The Honorable Ben Nighthorse Campbell
Chairman, Subcommittee on Treasury,
 General Government, and Civil Service
Committee on Appropriations
United States Senate

The Honorable Jim Kolbe
Chairman, Subcommittee on Treasury,
 Postal Service, and General Government
Committee on Appropriations
House of Representatives

This report responds to your requests that we review the U.  S. 
Customs Service's management of the Automated Commercial Environment
(ACE), including whether Customs has adequately justified ACE
cost-effectiveness.  Customs plans to spend over $1 billion on ACE,
which is planned to support modernized import processing.  We found
that Customs' is not managing ACE effectively and it does not have a
firm basis for concluding that ACE is cost-effective.  Accordingly,
we are making recommendations to the Commissioner of Customs for
strengthening the management and technical weaknesses we identified. 

We are sending copies of this report to the Secretary of the
Treasury; Commissioner of Customs; Director of the Office of
Management and Budget; and Ranking Minority Members of the
Subcommittee on Treasury, General Government, and Civil Service of
the Senate Committee on Appropriations and the Subcommittee on
Treasury, Postal Service, and General Government of the House
Committee on Appropriations.  We are also sending copies of this
report to the Chairmen and Ranking Minority Members of the Senate
Committee on Finance and the House Committee on Ways and Means and to
other congressional committees.  We will also make copies available
to other interested parties upon request.  If you have questions or
wish to discuss the issues in this report, please contact me at (202)
512-6240.  Major contributors to this report are listed in appendix
V. 

Randolph C.  Hite
Associate Director, Governmentwide
 and Defense Information Systems


EXECUTIVE SUMMARY
============================================================ Chapter 0


   PURPOSE
---------------------------------------------------------- Chapter 0:1

The U.S.  Customs Service plans to spend well over $1 billion to
modernize its systems environment for certain core missions: 
facilitating international trade, enforcing laws governing the flow
of goods across U.S.  borders, and assessing and collecting about $22
billion annually on imported merchandise.  The Clinger-Cohen Act of
1996, and related system and software engineering best practices,
provide federal agencies with a framework for effectively managing
such modernization efforts. 

The Customs modernization effort, known as the Automated Commercial
Environment or ACE, is of longstanding concern to the House
Appropriations Committee, Subcommittee on Treasury, Postal Service,
and General Government, and the Senate Appropriations Committee,
Subcommittee on Treasury, General Government, and Civil Service. 
Accordingly, the Subcommittee Chairmen asked GAO to determine whether
Customs is effectively managing ACE, including whether Customs has
adequately justified ACE cost-effectiveness. 


   BACKGROUND
---------------------------------------------------------- Chapter 0:2

Title VI of the North American Free Trade Agreement Implementation
Act, Public Law 103-182, enabled Customs to speed the processing of
imports and improve compliance with trade laws.  Customs refers to
this legislation as the "Customs Modernization and Informed
Compliance Act" or "Mod" Act.  The primary purpose of the act is to
streamline and automate Customs' commercial operations.  According to
Customs, modernized commercial operations will permit it to more
efficiently handle its burgeoning import workloads and expedite the
movement of merchandise at more than 300 ports of entry.  For 1995
through 2001, Customs estimates that the annual dollar volume of
import trade will increase from
$761 billion to $1.1 trillion, with the number of commercial entries
processed annually increasing from 13.1 million to 20.6 million. 

ACE is Customs' system solution to a modernized commercial
environment.  In November 1997, Customs estimated that it would cost
$1.05 billion to develop, operate, and maintain ACE over the 15 year
period between fiscal year 1994 and 2008.  Customs plans to develop
and deploy ACE in multiple increments.  The first four increments are
known collectively as the National Customs Automation Program (NCAP). 
The first increment, NCAP 0.1, was deployed for field operation and
evaluation in May 1998.  As of the end of fiscal year 1998, Customs
reported that it had spent $62.1 million on ACE. 


   RESULTS IN BRIEF
---------------------------------------------------------- Chapter 0:3

Customs is not managing ACE effectively, and it does not have a firm
basis for concluding that ACE is a cost-effective solution to
modernizing its commercial environment.  GAO found serious weaknesses
relating to architectural definition, investment management, and
software development and acquisition that must be corrected before
further investment in ACE is justified. 

First, Customs is not building ACE within the context of a complete
and enforced systems architecture or "construction plan" that
precludes inconsistent system design and development decisions.  In
May 1998, GAO reported that Customs' architecture was incomplete
because it was not based on a complete understanding of its
enterprisewide functional and information needs.  GAO also reported
that Customs had not yet instituted effective procedures for ensuring
compliance with the architecture once it is completed.  Customs is
attempting to complete its architecture, but it has not yet done so. 
Until its architecture is completed and effectively enforced, Customs
will not have adequate assurance that information systems like ACE
will optimally support its needs across all business areas. 

Further, Customs lacks a reliable estimate of what ACE will cost to
build, deploy, and maintain; and Customs has neither adequately
justified, nor is it effectively monitoring, ACE's
cost-effectiveness.  Specifically, Customs did not use rigorous cost
estimating techniques in preparing its cost estimate, and did not
disclose the inherent imprecision of the estimate.  Additionally,
Customs omitted costs and inflated benefits in preparing its
cost-benefit analysis.  Moreover, Customs is not using effective
incremental investment management practices.  While Customs plans to
develop/acquire ACE in 21 increments, these increments are not
individually cost-benefit justified, and Customs is not determining
what benefits each increment, once operational, actually provides. 
As a result, Customs will not know if ACE's expected
return-on-investment is actually being realized until it has already
spent hundreds of millions of dollars developing/acquiring the entire
system.  This combination of unreliable investment information and
analysis, and a "grand design" approach to justifying and managing
system investments,\1

has failed consistently on other agency modernization efforts over
the past two decades, has been abandoned by successful organizations,
and was a major reason for the information technology investment
management reforms in the Clinger-Cohen Act of 1996. 

These investment risks and uncertainties are compounded by the fact
that Customs is not employing effective software engineering
practices on ACE.  Specifically, Customs developed the first ACE
increment in-house using its own software developers, but because of
cost and schedule delays, it decided to acquire the second ACE
increment from a software development contractor.  GAO found that
Customs has neither the capability to effectively develop nor acquire
ACE and that its processes for doing both, according to widely
accepted and proven software capability maturity models, are ad hoc,
immature, and ineffective. 


--------------------
\1 The "grand design" approach involves investing in a large,
long-term, expensive project based on cost and benefit estimates
prepared at the outset and attempting to deliver the entire project
years later as a single increment. 


   PRINCIPAL FINDINGS
---------------------------------------------------------- Chapter 0:4


      CUSTOMS IS DEVELOPING ACE
      WITHOUT A COMPLETE
      ENTERPRISE SYSTEMS
      ARCHITECTURE
-------------------------------------------------------- Chapter 0:4.1

The Clinger-Cohen Act recognizes the importance of architectures by
requiring agency Chief Information Officers (CIO) to develop,
maintain, and facilitate an integrated systems architecture.  Customs
does not have a complete target systems architecture.  In May 1998,
GAO reported that Customs' target systems architecture was not
effective because it was neither complete nor enforced, and GAO made
several recommendations for needed improvements.\2 For example, GAO
reported that the architecture did not (1) fully describe Customs'
business functions, (2) define the information needs and flows among
these functions, and (3) establish the technical standards, products,
and services that will be used to build systems that support these
defined business functions and needs.  GAO also reported that Customs
did not require that its systems be architecturally compliant or that
architectural deviations be justified and documented. 

Customs officials acknowledged the limitations in the agency's
architecture and its enforcement.  They agreed to define functions,
information needs, and flows across Customs' six business areas and,
in light of this definition, reevaluate the technical characteristics
the architecture specified for its system environment.  Customs
originally planned to have completed its target architecture by
September 1998, but that date has slipped to May 1999.  Since
receiving a draft of this report, Customs has changed its investment
management procedures with the intent of ensuring that its
architecture, once completed, can be enforced effectively.  Until its
architecture is complete, however, Customs risks spending millions of
dollars to develop, acquire, and maintain information systems,
including ACE, that do not effectively and efficiently support the
agency's mission needs. 


--------------------
\2 Customs Service Modernization:  Architecture Must Be Complete and
Enforced to Effectively Build and Maintain Systems (GAO/AIMD-98-70,
May 1998). 


      ACE INVESTMENT MANAGEMENT IS
      NOT EFFECTIVE
-------------------------------------------------------- Chapter 0:4.2

The Clinger-Cohen Act and Office of Management and Budget (OMB)
guidance provide an effective framework for information technology
(IT) investment management.  Together, they establish requirements
for (1) identifying all promising alternative system solutions, (2)
developing reliable estimates of project life-cycle costs and
benefits, and investing in the most cost-beneficial alternative, and
(3) to the maximum extent practical, structuring major projects into
a series of increments to ensure that each increment constitutes a
wise investment. 

Customs did not consider potential alternatives, such as using the
International Trade Data System (ITDS), to perform certain critical
functions, before deciding to invest in ACE.\3 ITDS was initiated in
1995 as a project to implement the National Performance Review
recommendation to develop a coordinated, governmentwide system for
the collection, use, and dissemination of trade data.  At that time,
a multiagency board of directors was established, headed by the
Department of the Treasury with Customs and other major trade-related
agencies represented.  The Department of the Treasury is responsible
for designing and developing ITDS, which is expected to reduce the
burden that federal agencies place on international organizations by
requiring that they respond to duplicative data requests.  Treasury
intends for the system to serve as the single point for collecting,
editing, and validating trade data as well as collecting and
accounting for trade revenue--functions that are also planned for
ACE. 

Further, for the alternative it selected, i.e., ACE, Customs did not
develop a reliable life-cycle cost estimate.  Carnegie Mellon
University's Software Engineering Institute (SEI) has developed
criteria by which the reliability of project cost estimates can be
assessed.  Using SEI's criteria, GAO found that the processes used by
Customs to develop its $1.05 billion ACE life-cycle cost estimate
were neither thorough nor disciplined, and as a result, Customs' ACE
cost estimate is not reliable and does not provide a sound basis for
investment decision-making.  For example, Customs did not use a cost
model, did not account for changes in its approach to building
different ACE increments, did not account for changes to ACE software
and hardware architecture, and did not have the requisite historical
project cost data upon which to compare its ACE estimate. 
Additionally, the $1.05 billion cost estimate omits various relevant
cost elements, such as requirements definition, data warehouse
development, system documentation development, system integration,
training, and hardware/software technology refreshment.  Customs then
exacerbated these problems by representing its ACE cost estimate as a
precise, point estimate, rather than explicitly describing the
estimate's inherent uncertainty to ensure that it would be used
appropriately. 

Moreover, Customs' projections of ACE benefits are overstated by at
least $52.8 million.  The analysis includes $203.5 million in savings
attributable to 10 years of avoided maintenance and support costs on
the Automated Commercial System (ACS)--the system that ACE is to
replace.  However, Customs will not avoid maintenance and support
costs for 10 years.  Because Customs plans to run both systems in
parallel for 4 years, it will expend $52.8 million on continued
maintenance and support of ACS during this period. 

Lastly, although Customs has decided to implement ACE as a series of
21 increments, it is not making its investment decisions
incrementally as required by the Clinger-Cohen Act and OMB. 
Specifically, Customs is not justifying investing in each increment
on the basis of measurable benefits, and once it has deployed an
increment at a pilot site for evaluation, it is not validating the
benefits that the increment actually provides.  Instead, Customs has
estimated costs and benefits for the entire system (i.e., all 21
increments).  Such estimates of many system increments to be
delivered over many years are impossible to make accurately because
later increments are not well understood or defined, and are subject
to change in light of experiences on nearer term increments and
changing business needs.  By using an inaccurate, aggregated estimate
that is not refined as increments are developed, Customs is
committing enormous resources with no assurance that it will achieve
a reasonable return on its investment.  This "grand design" or "big
bang" approach to managing large system modernization projects has
repeatedly proven to be ineffective, resulting in huge sums invested
in systems that do not provide expected benefits. 


--------------------
\3 The U.S.  Department of the Treasury has developed an ITDS project
plan, system specification, cost-benefit analysis, and related
documentation.  It plans to begin developing ITDS in 1999 and to
fully implement it by 2003.  It estimates that ITDS will cost $256
million to develop, deploy, and maintain through 2005. 


      CUSTOMS LACKS THE CAPABILITY
      TO DEVELOP OR ACQUIRE ACE
      SOFTWARE EFFECTIVELY
-------------------------------------------------------- Chapter 0:4.3

The Clinger-Cohen Act requires the establishment of effective IT
management processes, including processes for managing software
development and acquisition.  SEI has developed criteria for
determining organizations' software development and acquisition
effectiveness or maturity.\4

Using SEI criteria for process maturity at the "repeatable" level,
which is the second level on SEI's five-level scale and means that an
organization has the software development/acquisition rigor and
discipline to repeat project successes, GAO evaluated ACE software
processes.  In February 1999, GAO reported that Customs lacked the
capability to develop software effectively on several projects,
including ACE.\5 For example, GAO reported that NCAP 0.1 lacked an
effective software configuration management process, which is
important for establishing and maintaining the integrity of the
software products during development, and NCAP 0.1 did not have any
type of software quality assurance program, which greatly increases
the risk of ACE software not meeting process and product standards. 
GAO also reported that Customs lacked a software process improvement
program to effectively address these and other software process
weaknesses.  Accordingly, GAO made several recommendations for needed
improvements.  Customs agreed with GAO's findings and stated that it
initiated steps to implement GAO's recommendations, including
assigning responsibility for software process improvement. 

Additionally, GAO found that Customs lacks the capability to acquire
ACE software effectively.  For example, Customs did not have an
effective software acquisition planning process, and therefore could
not effectively establish reasonable plans for performing software
engineering and for managing the software project.  Also, Customs did
not have an effective evaluation process, which means that it lacked
the means for assuring that contractor-developed software satisfied
defined requirements. 

Because of these and other serious software process weaknesses,
Customs' ability to either develop or acquire ACE software is
immature, and therefore project success is unlikely. 


--------------------
\4 Software Development Capability Maturity Model\SM (SW-CMM
) and Software Acquisition Capability Maturity Model\SM
(SA-CMM ).  Capability Maturity Model\SM is a service mark
of Carnegie Mellon University, and CMM is registered in
the U.S.  Patent and Trademark Office. 

\5 Customs Service Modernization:  Immature Software Development
Processes Increase Customs System Development Risks (GAO/AIMD-99-35,
February 11, 1999). 


   RECOMMENDATIONS
---------------------------------------------------------- Chapter 0:5

In addition to previous recommendations to improve Customs'
management of information technology,L\6 GAO recommends that Customs
correct the management and technical weaknesses discussed in this
report before building ACE.  To accomplish this, GAO recommends that
the Commissioner of Customs, with the support of Customs' CIO, ensure
that Customs

(1) rigorously analyze alternative approaches to building ACE,
including ITDS as an alternative to developing ACE entirely within
Customs;

(2) make investment decisions incrementally, i.e., for each
increment: 

use disciplined processes to prepare a rigorous life-cycle cost
estimate, including an explicit discussion of its inherent
uncertainty;

prepare realistic and supportable benefit expectations;

require a favorable return-on-investment and compliance with Customs'
architecture before making any investment; and

validate actual costs and benefits once an increment is piloted,
compare these with estimates, use the results in making further
decisions on subsequent increments, and report the results to
Customs' House and Senate appropriations and authorizing committees;
and

(3) strengthen ACE software acquisition management by: 

establishing an effective process improvement program and correcting
the weaknesses in ACE software acquisition processes identified in
this report, thereby bringing ACE processes to at least SEI level 2
and

requiring at least SEI level 2 processes of all ACE software
contractors. 


--------------------
\6 GAO/AIMD-99-35, February 11, 1999 and GAO/AIMD-98-70, May 5, 1998. 


   AGENCY COMMENTS
---------------------------------------------------------- Chapter 0:6

In its written comments on a draft of this report, Customs agreed
with GAO's conclusions and recommendations and stated that it is
committed to addressing the problems discussed in the report.  To
this end, Customs cited a number of actions that it has underway and
planned over the next few years to improve management of information
technology in general, and ACE in specific.  To fully correct the
management and technical weaknesses discussed in this report, Customs
must follow through and effectively implement actions to address all
of GAO's recommendations.  Customs' comments are discussed in greater
detail in chapter 5 and the full text of its comments are reproduced
in appendix I of this report. 


INTRODUCTION
============================================================ Chapter 1

The mission of the Customs Service is to ensure that all goods and
persons entering and exiting the United States do so in compliance
with all U.S.  laws and regulations.  It does this by (1) enforcing
the laws governing the flow of goods and persons across U.S.  borders
and (2) assessing and collecting duties, taxes, and fees on imported
merchandise.  During fiscal year 1997, Customs collected $22.1
billion in revenue\1 at more than 300 ports of entry, and it
processed nearly 450 million passengers entering the United States. 


--------------------
\1 Includes tariff duty, user fees, Internal Revenue Service excise
taxes, and other assessments. 


   CUSTOMS' CORE BUSINESS
   OPERATIONS:  A BRIEF
   DESCRIPTION
---------------------------------------------------------- Chapter 1:1

To accomplish its mission, Customs is organized into the following
six business areas:  trade compliance, outbound, passenger, finance,
human resources, and investigations.  Each business area is described
below. 

  -- Trade compliance includes enforcement of laws and regulations
     associated with the importation of goods into the United States. 
     To do so, Customs (1) works with the trade community to promote
     understanding of applicable laws and regulations, (2)
     selectively examines cargo to ensure that only eligible goods
     enter the country, (3) reviews documentation associated with
     cargo entries to ensure that they are properly valued and
     classified, (4) collects billions of dollars annually in duties,
     taxes, and fees associated with imported cargo, (5) assesses
     fines and penalties for noncompliance with trade laws and
     regulation, (6) seizes and accounts for illegal cargo, and (7)
     manages the collection of these moneys to ensure that all
     trade-related debts due to Customs are paid and properly
     accounted for. 

  -- Outbound includes Customs enforcement of laws and regulations
     associated with the movement of merchandise and conveyances from
     the United States.  To do so, Customs (1) selectively inspects
     cargo at U.S.  ports to guard against the exportation of illegal
     goods, such as protected technologies, stolen vehicles, and
     illegal currency, (2) collects, disseminates, and uses
     intelligence to identify high-risk cargo and passengers, (3)
     assesses and collects fines and penalties associated with the
     exportation of illegal cargo, and (4) physically examines
     baggage and cargo at airport facilities for explosive and
     nuclear materials.  In addition, the outbound business includes
     collecting and disseminating trade data within the federal
     government.  Accurate trade data are crucial to establishing
     accurate trade statistics on which to base trade policy
     decisions and negotiate trade agreements with other countries. 
     By the year 2000, Customs estimates that exports will be valued
     at $1.2 trillion, as compared to a reported $696 billion in
     1994. 

  -- Passenger includes processing all passengers and crew of
     arriving and departing air, sea, and land conveyances and
     pedestrians.  In fiscal year 1997, Customs reported that it
     processed nearly 450 million travelers, and by the year 2000,
     expects almost 500 million passengers to arrive in the United
     States annually.  Many of Customs' passenger activities are
     coordinated with other federal agencies, such as the Immigration
     and Naturalization Service and the Department of Agriculture's
     Animal and Plant Health Inspection Service.  Activities include
     targeting high-risk passengers, which requires timely and
     accurate information, and physically inspecting selected
     passengers, baggage, and vehicles to determine compliance with
     laws and regulations. 

  -- Finance includes asset and revenue management activities.  Asset
     management consists of activities to (1) formulate Customs'
     budget, (2) properly allocate and distribute funds, and (3)
     acquire, manage, and account for personnel, goods, and services. 
     Revenue management encompasses all Customs activities to
     identify and establish amounts owed Customs, collect these
     amounts, and accurately report the status of revenue from all
     sources.  Sources of revenue include duties, fees, taxes, other
     user fees, and forfeited currency and property.  The revenue
     management activities interrelate closely with the revenue
     collection activities in the trade compliance, outbound, and
     passenger business areas. 

  -- Human resources is responsible for filling positions, providing
     employee benefits and services, training employees, facilitating
     workforce effectiveness, and processing personnel actions for
     Customs' 18,000 employees and managers. 

  -- Investigations includes activities to detect and eliminate
     narcotics and money laundering operations.  Customs works with
     other agencies and foreign governments to reduce drug-related
     activity by interdicting (seizing and destroying) narcotics,
     investigating organizations involved in drug smuggling, and
     deterring smuggling efforts through various other methods. 
     Customs also develops and provides information to the trade and
     carrier communities to assist them in their efforts to prevent
     smuggling organizations from using cargo containers and
     commercial conveyances to introduce narcotics into the United
     States. 

To carry out its responsibilities, Customs relies on information
systems and processes to assist its staff in (1) documenting,
inspecting, and accounting for the movement and disposition of
imported goods and (2) collecting and accounting for the related
revenues.  Customs expects its reliance on information systems to
increase as a result of its burgeoning workload.  For 1995 through
2001, Customs estimates that the annual volume of import trade
between the United States and other countries will increase from $761
billion to $1.1 trillion.  This will result in Customs processing an
estimated increase of 7.5 million import entries--from 13.1 million
to 20.6 million annually--during the same period.  Recent trade
agreements, such as the North American Free Trade Agreement (NAFTA),
have also increased the number and complexity of trade provisions
that Customs must enforce. 


   CUSTOMS IS LEGISLATIVELY
   MANDATED TO MODERNIZE
   OPERATIONS AND SYSTEMS
---------------------------------------------------------- Chapter 1:2

Both Customs and the Congress recognize that the ability to process
the growing volume of imports while improving compliance with trade
laws depends heavily on successfully modernizing Customs' trade
compliance process and its supporting automated systems.  To speed
the processing of imports and improve compliance with trade laws, the
Congress enacted Title VI of the North American Free Trade Agreement
Implementation Act in December 1993.\2 Customs refers to this
legislation as the "Customs Modernization and Informed Compliance
Act" or "Mod" Act. 

The primary purpose of the act is to streamline and automate Customs'
commercial operations by eliminating certain legislatively mandated
paper requirements and requiring Customs to establish the National
Customs Automation Program (NCAP).  The legislation also specified
certain functions that NCAP must provide, including giving members of
the trade community the capability to electronically file import
entries at remote locations as well as enabling Customs to
electronically process "drawback" claims, which are refunds of duties
and taxes paid on imported goods that are subsequently exported or
destroyed.  According to Customs, the act provides the legal
framework to automate commercial operations and thereby streamline
the processing, and expedite the movement of merchandise at more than
300 ports of entry. 


--------------------
\2 Public Law 103-182, 19 U.S.C.  1411 et seq. 


   ACE PURPOSE, PLANS, AND STATUS
---------------------------------------------------------- Chapter 1:3

ACE is intended to support the trade compliance business process,
implement Mod Act requirements, and replace the existing import
system, the Automated Commercial System (ACS).  ACS is nearly 15
years old and Customs reports that it is becoming increasingly
difficult and expensive to operate, maintain, and enhance due to the
system's antiquated hardware and software and limited processing
capacity. 

Currently, ACE's architecture is to include both (1) mainframe-based,
centralized processing to support high-volume, repetitive
transactions and (2) distributed, client-server processing and
desktop PC interfaces to support port office analysis and decision
support needs.  The transaction-intensive applications are to run on
IBM mainframes at Customs' national data center and are to be written
in COBOL and C++.  The distributed processing is to occur on UNIX
servers and the applications are to be written in C++ and Java.  To
support this processing environment, Customs plans to design and
implement a single, integrated agencywide database.  It also plans to
connect port offices to the national data center through existing
communications networks comprising the Treasury Communications
System, and to connect the trade community to the data center through
the Internet and a combination of dial-up lines and dedicated lines. 

According to Customs, ACE will facilitate increased compliance of
individuals, businesses, and governments with the trade laws and
regulations of the United States and increased communication with the
trade community.  It will also provide an integrated, account-based,
automated information system for collecting, disseminating, and
analyzing trade-related data and ensuring the proper collection and
allocation of revenues. 

Customs began developing ACE in 1994 and plans to develop and deploy
ACE in 21 increments from 1998 through 2005.  Each increment consists
of the software and hardware necessary to perform a discrete portion
of the total ACE functionality.  The first four increments, known
collectively as the NCAP prototype, will process pre-identified
merchandise shipped into the U.S.  via truck by three selected
importers.\3 The first two increments, NCAP 0.1 and NCAP 0.2, were
deployed for operational evaluation at three pilot border port
locations (Detroit, Michigan; Laredo, Texas; and Port Huron,
Michigan) during May 1998 and October 1998, respectively. 

The succeeding 17 increments are intended to build upon the
foundational capabilities contained within the four NCAP prototype
increments, providing additional functionality including the
capability to (1) process merchandise imported via rail, air, sea,
and couriers, (2) process collections and refunds associated with
imported cargo, (3) process warehouse entry cargo, and (4) process
cargo subject to drawbacks.  Until all of these deployments are
completed, Customs intends to operate and maintain ACS. 

Customs' estimate of ACE costs has escalated since August 1997, when
it initially estimated that ACE's 10-year life-cycle cost would be
$895 million for the period 1998 through 2007.  In November 1997,
Customs revised this 10-year cost estimate to $987.6 million and also
estimated that, over 15 years, from 1994 through 2008, it would spend
$1.05 billion on ACE.  It then used the $1.05 billion estimate along
with estimated expected savings of $1.9 billion to justify ACE. 
Customs is currently reevaluating this estimate. 

Figure 1.1 compares planned and actual deployments of the NCAP
increments.  As the figure illustrates, Customs is well over 2 years
behind its original schedule for NCAP. 

   Figure 1.1:  Planned Versus
   Actual NCAP Deployment
   Schedules

   (See figure in printed
   edition.)

Note:  NCAP 0.3 and NCAP 0.4 current dates are unknown pending budget
approval. 


--------------------
\3 The importers participating in the NCAP prototype are General
Motors Corporation, Ford Motor Company, and Chrysler Corporation. 


   OBJECTIVES, SCOPE, AND
   METHODOLOGY
---------------------------------------------------------- Chapter 1:4

The Chairman, Subcommittee on Treasury, Postal Service, and General
Government, House Committee on Appropriations, and the Chairman,
Subcommittee on Treasury, General Government, and Civil Service,
Senate Appropriations Committee, requested that we review ACE.  Our
objective was to determine whether Customs is effectively managing
ACE, including whether Customs has adequately justified ACE
cost-effectiveness.  In making these determinations, we relied
primarily on the following criteria:  (1) the Clinger-Cohen Act of
1996\4 and other legislative reforms that require federal agencies to
develop and maintain integrated systems architectures and improve
their information technology investment management processes, (2)
Office of Management and Budget guidance related to the acquisition
and management of information resources, (3) GAO and Treasury systems
architecture guidance, and (4) related Software Engineering Institute
system engineering standards concerning cost estimating and software
development and acquisition maturity. 

To address our objective, we reviewed pertinent documentation (e.g.,
ACE project plan, system/software development process
practices/standards, ACE technical architecture descriptions, and ACE
project status reports) and interviewed responsible Treasury
officials and Customs ACE project officials in order to identify (1)
Customs' management structure for developing and deploying ACE, (2)
the ACE system/software development methodology, (3) the planned ACE
hardware/software and communications architecture and configuration,
and (4) the current ACE system development status. 

Additionally, we met with Customs ACE officials to determine the
status of the agency's implementation of recommendations we made in
May 1998 to complete its systems architecture.  These recommendations
included (1) ensuring that the architecture fully described Customs'
business functions, (2) defining the information needs and flows
among these functions, (3) establishing the technical standards,
products, and services that will be used to build systems that
support these business functions and needs, and (4) enforcing
compliance with the architecture. 

Further, we obtained and reviewed supporting documentation and
interviewed Treasury officials, Customs ACE officials, and Customs
Investment Review Board (IRB) officials to (1) identify the current
ACE project cost and life-cycle cost estimate baselines and determine
the current actual expenditures to date, (2) identify the current ACE
project schedule baseline and determine the actual progress to date
against scheduled milestones, (3) define what ACE is intended to do
and how it is expected to benefit Customs' mission, (4) determine the
extent to which ACE mission-related goals/benefits have been achieved
to date and how Customs is measuring the accrued benefits, and (5)
determine Customs' strategic approach to managing the
development/acquisition, integration, and deployment of ACE.  The
documentation we analyzed included the ACE project plan and strategic
plan, functional/performance requirements, and NCAP technical
assessment documents, budget/financial data, cost-benefit policy and
guidance, cost-benefit analyses, various life-cycle and project cost
estimates, project status reports, testing problem reports, and
configuration management documentation. 

Also, we reviewed project documentation to determine how the
life-cycle cost baseline was estimated and how this estimating
approach compared to criteria established by SEI in A Manager's
Checklist for Validating Software Cost and Schedule Estimates.\5 The
SEI criteria defines seven primary questions, each supported by more
detailed secondary questions, that can be used to determine whether
defined and disciplined processes were used to derive a given cost
estimate.  We also interviewed ACE project officials. 

In addition, we assessed whether Customs thoroughly analyzed
technical alternatives to ACE including the possibility of (1)
enhancing and continuing to use the legacy trade system, ACS, (2)
using different architectural designs for ACE, (3) following
different development/acquisition strategies, and/or (4) using the
Department of the Treasury's planned multiagency International Trade
Data System (ITDS), instead of ACE, for some functions, such as
collecting, editing, and validating trade data and collecting and
accounting for trade revenue. 

Last, we used the Software Engineering Institute's (SEI's) Software
Development Capability Maturity Model\SM (SW-CMM),\6

Software Acquisition Capability Maturity Model\SM (SA-CMM),
and its Software Capability Evaluation Method, to evaluate Customs'
ability to manage its NCAP 0.1 software development project and NCAP
0.2 software acquisition effort, respectively.  These models and
methods provide a logical and widely accepted framework for
baselining an organization's current process capabilities (i.e.,
strengths and weaknesses) and assessing whether an organization has
the necessary process discipline in place to repeat earlier successes
on similar projects.  Organizations that do not satisfy the
requirements for the "repeatable" level are by default judged to be
at the "initial" level of maturity, meaning that their processes are
ad hoc, sometimes even chaotic, with few of the processes defined and
success dependent mainly on the heroic efforts of individuals. 

In following the SEI method, GAO staff trained at SEI evaluated
Customs' ACE project software development/maintenance maturity in
five of the six key process areas (KPA) that are necessary to attain
a "repeatable" level of process maturity.\7 GAO did not evaluate
Customs in the sixth repeatable level KPA, software subcontract
management, because Customs did not use subcontractors on the ACE
project.  Additionally, GAO staff trained at SEI evaluated Customs'
ACE project software acquisition maturity in the seven KPAs that are
necessary to attain a repeatable level of process capability, and one
KPA associated with the "defined" level of process
maturity--acquisition risk management.\8 The purpose of acquisition
risk management is to formally identify risks as early as possible
and adjust the acquisition to mitigate those risks.  Many software
experts consider acquisition risk management to be an integral part
of the solicitation, project performance management, and contract
performance management processes. 

Customs provided written comments on a draft of this report.  These
comments are presented in chapter 5, and are reprinted in appendix I. 
We performed our work at Customs and Department of the Treasury
headquarters offices in Washington, D.C., and at the Customs Data
Center facility in Newington, Virginia between February 1998 and
November 1998, in accordance with generally accepted government
auditing standards. 


--------------------
\4 Although the Clinger-Cohen Act (Public Law 104-106) was passed
after Customs began developing ACE, its principles are based on
practices that are widely considered to be integral to successful IT
investments.  For an analysis of the management practices of several
leading private and public sector organizations on which the
Clinger-Cohen Act is based see Executive Guide:  Improving Mission
Performance Through Strategic Information Management and Technology,
(GAO/AIMD-94-115, May 1994).  For an overview of the IT management
process envisioned by Clinger-Cohen see Assessing Risk and Returns: 
A Guide for Evaluating Federal Agencies' IT Investment
Decision-making (GAO/AIMD-10.1.13, February 1997). 

\5 CMU/SEI-95-SR-004, January 1995. 

\6 Capability Maturity Model\SM is a service mark of Carnegie Mellon
University, and CMM is registered in the U.S.  Patent and
Trademark Office. 

\7 The five KPAs are requirements management, software project
planning, software project tracking and oversight, software quality
assurance, and software configuration management.  According to the
SW-CMM, (1) requirements management is the process of establishing a
common understanding between the customer and the software developer
of the customer's requirements, (2) software project planning is the
process of establishing reasonable plans for engineering the software
and managing the software project, (3) software project tracking and
oversight is the process of providing adequate visibility into the
software project's progress to permit effective action when
deviations from plans occur, (4) software quality assurance is the
process of verifying for management that software process and product
procedures and standards are being followed, and (5) software
configuration management is the process of establishing and
maintaining the integrity of the software products throughout their
life-cycle. 

\8 The seven KPA's relating to the repeatable level are software
acquisition planning, solicitation, requirements development and
management, project office management, contract tracking and
oversight, evaluation, and transition and support.  The KPA relating
to the defined level is acquisition risk management.  According to
the SA-CMM, (1) software acquisition planning is the process of
ensuring that reasonable planning for all elements of the software
acquisition occur, (2) solicitation is the process of ensuring that
award is made to the contractor most capable of satisfying the
specified requirements, (3) requirements development and management
is the process of establishing an unambiguous and agreed upon set of
software requirements, (4) project office management is the process
of effective and efficient management of project office activities,
(5) contract tracking and oversight is the process of ensuring that
contractor activities, products, and services satisfy contract
requirements, (6) evaluation is the process of determining that
acquired software products and services satisfy contract requirements
prior to acceptance, (7) transition and support is the process of
transferring acquired software products to the eventual support
organization, and (8) acquisition risk management is the process of
identifying software risks early and adjusting the acquisition
strategy to mitigate those risks. 


ACE IS BEING DEVELOPED WITHOUT A
COMPLETE ENTERPRISE SYSTEMS
ARCHITECTURE
============================================================ Chapter 2

Architectures are critical for designing and developing large and
complex information systems.  These comprehensive "construction
plans" systematically and completely describe an organization's
target business environment, both in logical (e.g., missions,
business functions, information flows) terms and technical (e.g.,
software, hardware, communications) terms.  Without a target
architecture to guide and constrain IT investment, there is no
systematic way to preclude either inconsistent system design and
development decisions or the resulting suboptimal performance and
added cost associated with incompatible systems. 

The Clinger-Cohen Act of 1996 requires agency CIOs to develop and
maintain an integrated system architecture.  In addition, OMB issued
guidance in 1996 that, among other things, requires agency IT
investments to be consistent with federal, agency, and bureau
architectures.\1

Customs does not currently have a complete target architecture but
has recently established a process for enforcing an architecture once
one is completed.  Customs has plans for completing its target
architecture by May 1999 and ensuring its enforcement.  Thus far,
however, it has only defined its current architectural environment. 


--------------------
\1 OMB Memorandum M-97-02, Funding Information Systems Investments,
October 25, 1996. 


   A FRAMEWORK FOR EFFECTIVE
   SYSTEMS ARCHITECTURES
---------------------------------------------------------- Chapter 2:1

Reflecting the general consensus in the industry that large, complex
systems development and acquisition efforts should be guided by
explicit architectures, we issued a report in 1992 defining a
comprehensive framework for designing and developing systems
architectures.\2 This framework, which is consistent with guidance
that Treasury has provided to its bureaus,\3 divides systems
architectures into (1) a logical or business component, which is
developed first, and (2) a technical or systems component, which is
based on the first component. 

The logical component ensures that the systems meet the business
needs of the organization.  It provides a high-level description of
the organization's mission and target concept of operations; the
business functions being performed and the relationships among
functions; the information needed to perform the functions; the users
and locations of the functions and information; and the information
systems needed to support the agency's business needs. 

The technical component ensures that systems are interoperable,
function together efficiently, and are cost-effective over their
life-cycles (including maintenance costs).  The technical component
details specific standards and approaches that will be used to build
systems, including hardware, software, communications, data
management, security, and performance characteristics. 


--------------------
\2 Strategic Information Planning:  Framework for Designing and
Developing System Architectures (GAO/IMTEC-92-51, June 1992). 

\3 This guidance--Treasury Information Systems Architecture Framework
(TISAF) version 1.0, January 3, 1997--is also included in OMB's
guidance on developing system architecture.  See, OMB Memorandum
M-97-16, Information Technology Architectures, June 18, 1997. 


   CUSTOMS IS DEVELOPING, BUT HAS
   NOT YET COMPLETED, ITS TARGET
   SYSTEMS ARCHITECTURE
---------------------------------------------------------- Chapter 2:2

Customs does not currently have a complete enterprise systems
architecture.  In May 1998, we reported that for five of its six
business areas (outbound, passenger, finance, human resources, and
investigations), Customs' architecture does not (1) describe all of
the agency's business functions, (2) completely identify the users
and locations of the functions, or (3) define the information needed
to perform the functions.\4 Further, while the architecture and
related documentation describe business functions and users and work
locations for the sixth business area (trade compliance), they do not
identify all of the information needs and flows for all of the trade
functions.  Nonetheless, Customs had defined many characteristics of
its systems' hardware, software, communications, data management, and
security components.  Because these characteristics were not based on
a complete understanding of its enterprisewide functional and
information needs, as specified in both best practice and Treasury
guidance, we concluded that Customs did not have adequate assurance
that its information systems will optimally support its needs across
all business areas. 

We recommended that the Commissioner of Customs direct the Customs
CIO, in consultation with the Treasury CIO, to complete the
architecture.  Specifically, we recommended that, at a minimum, the
architecture should (1) describe Customs' target business operations,
(2) fully define Customs' interrelated business functions to support
these target operations, (3) clearly describe information needs and
flows among these functions, (4) identify the systems that will
provide these functions and support these information needs and
flows, and (5) use this information to specify the technical
standards and related characteristics that these systems should
possess to ensure that they interoperate, function together
efficiently, and are cost-effective to maintain. 

Customs and Treasury officials have acknowledged the limitations in
Customs' architecture, and agreed to define the agency's logical
architecture (e.g., functions, information needs and flows and users)
across its six business areas and, in light of this definition,
reevaluate the technical characteristics it has specified for its
technical architecture (i.e., systems environment).  Thus far,
Customs has defined its current (i.e., its "as-is" or "in-state")
architectural environment, including its current business operations,
its current supporting business functions and their information needs
and flows, the systems currently supporting these functions, and the
technical characteristics (e.g., hardware, software, and
communications) of these supporting systems.  However, Customs has
not defined an architecture for its target (i.e., future) business
and systems environment.  Customs originally planned to have
completed its target architecture by September 1998, but that date is
now May 1999. 


--------------------
\4 GAO/AIMD-98-70, May 5, 1998. 


   CUSTOMS HAS RECENTLY ANNOUNCED
   PROCESS FOR ENSURING
   ARCHITECTURAL COMPLIANCE
---------------------------------------------------------- Chapter 2:3

In May 1998, we reported that Customs' investment management
process\5

did not ensure that all new systems would conform to the
architecture.  In particular, we reported that Customs' investment
review board (IRB) used four criteria in scoring competing investment
options and allocating funding among them.  The four criteria were

(1) risk (e.g., technical, schedule, and cost);

(2) strategic alignment (e.g., cross-functional benefits, linkage to
Customs' business plan, and compliance with legislative mandates);

(3) mission effectiveness (e.g., contributions to service delivery);
and

(4) cost-benefit ratio (e.g., tangible and intangible benefits, and
costs). 

Because compliance with the architecture was considered under the
risk criterion but was not required, the process did not preclude
funding projects that were inconsistent with the enterprise
architecture.  Moreover, the process did not require that such
deviations from the architecture be rigorously justified. 

To ensure that Customs effectively enforced its information systems
architecture, we recommended that Customs require that all new
projects comply with the architecture unless an exception could be
justified by careful, thorough, and documented analysis.\6 Customs
agreed and, in January 1999, changed its investment management
process to explicitly require that proposed IT investments comply
with its architecture, unless an exception is justified and a waiver
is granted by the technical review committee. 


--------------------
\5 IT Investment Management Process, U.S.  Customs Service, August
28, 1997. 

\6 GAO/AIMD-98-70, May 5, 1998. 


   CONCLUSIONS
---------------------------------------------------------- Chapter 2:4

Until Customs completes and enforces its enterprise systems
architecture, it will not have adequate assurance that ACE and other
systems it plans to build and operationally deploy (1) will
effectively meet the agency's business needs and (2) are compatible,
efficient, and cost-effective to develop, integrate, and maintain. 


CUSTOMS' ACE INVESTMENT MANAGEMENT
PRACTICES ARE INEFFECTIVE
============================================================ Chapter 3

The Clinger-Cohen Act and OMB guidance provide an effective framework
for IT investment management.  Together, they set requirements for
(1) identifying all potential alternative system solutions, (2)
developing reliable estimates of project life-cycle costs and
benefits, and investing in the most cost-beneficial alternative, and
(3) to the maximum extent practical, structuring major projects into
a series of increments to ensure that each increment constitutes a
wise investment. 

Customs has not effectively implemented any of these investment
management practices on ACE.  Specifically, (1) Customs' investment
analysis did not address alternatives to its chosen ACE system
solution, (2) Customs' did not use rigorous cost estimating
techniques in preparing its ACE cost estimate and its cost-benefit
analysis for ACE omitted substantial costs and inflated benefits, and
(3) Customs is not justifying and validating the costs and benefits
for each ACE increment.  As a result, Customs lacks a sound basis for
making ACE investment decisions. 


   ACE ALTERNATIVES WERE NOT
   CONSIDERED
---------------------------------------------------------- Chapter 3:1

Before embarking on a major, costly systems initiative such as ACE,
agencies should thoroughly assess the full range of technical
options.  In the case of ACE, Customs had several alternatives to
satisfying its mission needs as specified in the Mod Act.  For
example, it could enhance ACS, use different architectural designs
for ACE, follow different development/ acquisition strategies for
ACE, and/or use Treasury's planned governmentwide trade system,
International Trade Data System (ITDS), to supplement some ACE
functions, such as collecting and disseminating trade data.  ITDS was
initiated in 1995 as a project to implement the National Performance
Review recommendation to develop a coordinated, governmentwide system
for the collection, use, and dissemination of trade data.  At that
time, a multiagency board of directors was established, headed by the
Department of the Treasury with Customs and other major trade-related
agencies represented.  The Department of the Treasury is responsible
for designing and developing ITDS, which is expected to reduce the
burden that federal agencies place on the international trading
organizations by requiring that they respond to duplicative data
requests.  Treasury intends for the system to serve as the single
point for collecting, editing, and validating trade data as well as
collecting and accounting for trade revenue.\1

By thoroughly considering these and other choices, Customs would have
ensured that the most cost-effective and beneficial alternative was
chosen before deciding to invest $1.05 billion in ACE.  In fact, OMB
requires that agencies consider alternative system solutions to meet
mission needs, including different system architectures, upgrading
existing systems, or contracting for development and integration of
major systems. 

Customs did not identify and evaluate a full range of alternatives to
ACE.  In fact, Customs considered only (1) enhancing and operating
ACS to provide the same functionality of ACE, (2) operating ACS
without any enhancement, (3) operating ACS with limited enhancements,
and (4) developing and deploying part of ACE's planned functionality. 
Customs discarded the last three because none provided for meeting
all of the Mod Act requirements.  With respect to the first, Customs
compared ACS to ACE and decided that ACE was the more cost-effective
alternative. 

Customs did not evaluate other alternatives to its ACE solution,
including (1) adopting a different ACE architectural design (e.g.,
using a combination mainframe and client-server configuration instead
of a mainframe only system), (2) contracting out for ACE development
rather than developing ACE in-house, or (3) using ITDS to satisfy
part of its functional needs.  As a result, Customs committed to and
began investing in ACE development without knowing whether it had
chosen the most cost-effective alternative. 

Since making its decision to develop ACE, Customs has changed ACE's
architectural design, and it has decided to acquire the second ACE
increment rather than develop it in-house.  However, these decisions
are not supported by any verifiable alternatives analysis, meaning
that Customs still lacks adequate assurance that it is following the
most cost-effective approach to satisfying its mission needs. 


--------------------
\1 Treasury has developed an ITDS project plan, system specification,
cost-benefit analysis, and related documentation.  It plans to begin
developing ITDS in 1999 and to fully implement it by 2003.  It
estimates that ITDS will cost $256 million to develop, deploy, and
maintain through 2005. 


   CUSTOMS' $1.05 BILLION ACE COST
   ESTIMATE IS NOT RELIABLE
---------------------------------------------------------- Chapter 3:2

Reliable estimates of IT projects' expected costs are essential to
determining a project's cost-effectiveness.  Without such
information, the likelihood of poor investment decisions is
increased, not only when a project is initiated but also throughout
its life-cycle. 

To assist management in assessing the credibility of a given
project's cost estimate, SEI developed the following seven questions
for decisionmakers to use to assess the reliability of a project's
cost estimate and detailed criteria to assist in evaluating how well
a project satisfies each question.\2

(1) Are the objectives of the estimate clear and correct? 

(2) Has the task been appropriately sized? 

(3) Is the estimated cost consistent with demonstrated
accomplishments on other projects? 

(4) Have the factors that affect the estimate been identified and
explained? 

(5) Have steps been taken to ensure the integrity of the estimating
process? 

(6) Is the organization's historical evidence capable of supporting a
reliable estimate? 

(7) Has the situation remained unchanged since the estimate was
prepared? 

We analyzed the approach that Customs followed in deriving its $1.05
billion ACE life-cycle cost estimate using SEI's criteria.  Among
these criteria were several very significant and closely intertwined
requirements that are at the core of effective cost estimating. 
Specifically, embedded in several of the aforementioned seven
questions are requirements for using (1) formal cost models, (2)
structured and documented processes for determining the software size
and reuse inputs to the models, and (3) relevant, measured, and
normalized historical cost data (estimated and actual) to calibrate
the models.\3

We found that Customs did not satisfy any of these requirements.  In
particular, Customs did not use a cost model.  Instead, it used an
unsophisticated spreadsheet to extrapolate the cost of each ACE
increment.  Further, Customs' approach to determining software size
and reuse was not documented and was not well supported or
convincing.  Customs estimated the size of each ACE software
increment (most of which were still undefined) by extrapolating from
the estimated size of the first increment based on individuals'
undocumented best judgments about increment functionality and
complexity.  Last, Customs did not have any historical project cost
data at the time it developed the $1.05 billion estimate, and it did
not account for relevant, measured, and normalized differences among
the increments.  For example, it did not account for (1) the change
in ACE's architecture from a mainframe system written in COBOL and
C++ to a combined mainframe and Internet-based system written in C++
and Java and (2) the change in ACE from an in-house software
development project (NCAP 0.1) to a software acquisition (NCAP 0.2). 
Clearly, such fundamental changes can have a dramatic effect on
system costs, and should have been addressed explicitly in Customs'
cost estimates. 

Table 3.1 summarizes the complete results of our assessment of
Customs' cost estimating process.  For each of SEI's questions and
supporting criteria, Customs was rated as demonstrating a strength,
i.e., effectively implementing the criterion; a weakness, i.e.,
ineffectively implementing the criterion; or, where evidence was
inconclusive and could not support characterization as either a
strength or a weakness, an observation was noted.  (See appendix II
for further detail on SEI's criteria and our findings). 



                               Table 3.1
                
                 Summary of ACE Project Cost Estimate's
                Satisfaction of SEI's Checklist Criteria

                                                  Number of  Number of
                                       Number of  weaknesse  observati
SEI checklist questions                strengths          s        ons
-------------------------------------  ---------  ---------  ---------
Are the objectives of the estimate             4          0          0
 clear and correct?
Has the task been appropriately                0          7          1
 sized?
Is the estimated cost consistent with          1         10          0
 demonstrated accomplishments on
 other projects?
Have the factors that affect the               3          2          0
 estimate been identified and
 explained?
Have steps been taken to ensure the            3          1          3
 integrity of the estimating process?
Is the organization's historical               2          8          0
 evidence capable of supporting a
 reliable estimate?
Has the situation remained unchanged           0          3          0
 since the estimate was prepared?
======================================================================
Totals                                        13         31          4
----------------------------------------------------------------------

--------------------
\2 A Manager's Checklist for Validating Software Cost and Schedule
Estimates (CMU/SEI-95-SR-004, January 1995). 

\3 Examples of such data are the productivity value of a
"staff-month," that is, the time and cost to produce a specified unit
of code, such as a line of code. 


      CUSTOMS' COST ESTIMATE
      IMPLIES AN UNJUSTIFIED LEVEL
      OF PRECISION
-------------------------------------------------------- Chapter 3:2.1

Software and systems development experts agree that early project
estimates are by definition imprecise, and that this inherent
imprecision decreases during the project's life-cycle as more
information becomes known about the system.  These experts emphasize
that, to be useful, each cost estimate should include an indication
of its degree of uncertainty, possibly as an estimated range or
qualified by some factor of confidence.  For example, a cost estimate
of $1 million could be presented as a range from $750,000 to $1.25
million or as $1 million with a confidence level of 90 percent,
indicating that there is a 10 percent chance that costs will exceed
this estimate. 

Customs did not reveal the degree of uncertainty of its cost estimate
for ACE to managers involved in investment decisions.  For example,
Customs did not disclose that it made the estimate before fully
defining ACE functionality.  Instead, Customs presented its $1.05
billion ACE life-cycle cost estimate as an unqualified point
estimate.  This suggests an element of precision that cannot exist
for such an undefined system and obscures the investment risk
remaining in this project. 


   CUSTOMS' ANALYSIS OF ACE COSTS
   AND BENEFITS IS ALSO UNRELIABLE
---------------------------------------------------------- Chapter 3:3

ACE cost estimates are understated, benefit estimates are overstated,
and both are unreliable.  Customs' August 1997 cost-benefit analysis
estimated that ACE would produce cumulative savings of $1.9 billion
over a 10-year life-cycle beginning in fiscal year 1998 and ending in
fiscal year 2007.  However, this analysis was unreliable because
certain benefits and costs were unsupported, other benefits were
overstated, and relevant costs were omitted.  For example: 

  -- The analysis identified $650 million (35 percent of the total
     ACE estimated savings) in savings to the trade community from
     more accurate and timely data, improved customer compliance,
     reduced support staff, reduced transportation costs, elimination
     of duplicate recordkeeping systems, and lower processing costs
     associated with periodic payments.  However, Customs could not
     produce any verifiable data or analysis to support this claim. 
     According to Customs officials, this benefit amount was
     projected on the basis of undocumented information supplied by
     one company.  The officials stated that other companies
     considered such data confidential and would not provide them to
     Customs. 

  -- The analysis identified an additional $644 million (33 percent
     of the total savings) resulting from increased productivity. 
     Because this estimate is driven by Customs' assumption that
     every minute "saved" by processing transactions or analyzing
     data faster using ACE, rather than ACS, would be productively
     used by all workers, it can be viewed as a "best case," upper
     limit on estimated productivity improvements.  Given the
     magnitude of the potential savings, even a small change in the
     assumption translates into a large reduction in benefits.  For
     example, conservatively assuming that three-fourths of each
     minute saved would be productively utilized by three-fourths of
     all workers, the expected benefits would be reduced by about
     $282 million. 

  -- The analysis excluded costs for hardware and system software
     upgrades (e.g., desktop workstations and operating systems,
     application and data servers, data base management systems) at
     each port office.  Using Customs' estimate for acquiring the
     initial suite of port office hardware and system software, and
     assuming a technology refreshment cycle of every 3 to 5 years,
     we estimated this cost to be between $72.9 million and $171.8
     million. 

  -- The analysis excluded $52.8 million of costs needed to support
     the data center and maintain ACS through fiscal year 2002. 
     Since Customs intends to operate and maintain ACS in parallel
     with ACE through 2002, these costs should be included. 

  -- The analysis excluded costs to conduct security analysis,
     project planning and management, and independent verification
     and validation.  Customs estimates that these costs collectively
     total $23 million. 

  -- The analysis excluded other relevant cost items, such as the
     cost of defining ACE requirements, integrating ACE components,
     and conducting ACE regression testing and training.  Customs did
     not have an estimated value for these costs. 

  -- The analysis included annual telecommunications costs of $60.3
     million once ACE is deployed to all sites.  However, Customs
     could not provide us with any supporting data or analysis for
     this estimate. 


   CUSTOMS IS NOT INVESTING
   INCREMENTALLY IN ACE
---------------------------------------------------------- Chapter 3:4

Incremental project management involves three fundamental components: 
(1) developing/acquiring a large system in a series of smaller
projects or system increments, (2) individually justifying investment
in each separate increment on the basis of return-on-investment, and
(3) monitoring actual benefits achieved and costs incurred on
completed increments and modifying subsequent increments/investments
to reflect lessons learned.  By doing so, agencies avoid discovering
too late that their system is not cost-beneficial, and they can
reduce the enormous risks associated with large, expensive projects. 
The Clinger-Cohen Act requires that agencies acquire information
technology incrementally, to the maximum extent practicable, and have
milestones for senior managers to obtain information on the cost,
timeliness, quality, and capability of information system investments
to meet requirements.  Additionally, OMB policy requires that
investments in major information systems be implemented in increments
with each increment delivering measureable benefits.\4

Customs is developing and acquiring ACE in a series of 21 increments. 
It has, to date, defined the functionality of only the first two
increments and will define later increments in the future. 
Nonetheless Customs has estimated costs and benefits for, and has
committed to, investing in the entire system (i.e., all 21
increments).  It has not estimated the costs and benefits for each
increment and does not know whether each increment will produce a
reasonable return on investment.  Furthermore, once it has deployed
an increment at a pilot site for evaluation, Customs is not
validating that estimated benefits were actually achieved.  As a
result, Customs will not even know whether the first ACE increment,
now being piloted at three sites, is producing expected benefits or
is cost-effective.  Instead, Customs will only determine whether the
first increment is performing at a level "equal to or better than"
ACS. 

Customs "grand design" approach to ACE does not constitute wise
investment management.  Customs will not know whether earlier
increments were cost beneficial before it invests in later
increments; it does not have reliable cost or benefit data upon which
to base investment decisions; and it is committing substantial
resources with no assurance that it will achieve a reasonable
return-on-investment. 


--------------------
\4 Memorandum For Heads Of Executive Department And Agencies, October
25, 1996 (M-97-02). 


   CONCLUSIONS
---------------------------------------------------------- Chapter 3:5

Because Customs does not have reliable information on ACE costs and
benefits and has not analyzed other viable alternatives to the system
solution it selected, it does not have adequate assurance that ACE is
the optimal approach.  In fact, it has no assurance at all that ACE
is cost effective.  Furthermore, because it is not justifying the
return on its investment in each ACE increment, Customs will not be
able to demonstrate whether ACE is cost-effective until it has
already spent hundreds of millions of dollars to develop/acquire the
entire system. 


ACE SOFTWARE DEVELOPMENT AND
ACQUISITION PROCESSES ARE NOT
EFFECTIVE
============================================================ Chapter 4

The Clinger-Cohen Act requires agency CIOs to establish effective IT
management processes, such as processes for developing or acquiring
software.  Customs has developed the first ACE software increment
(NCAP 0.1) in-house and is acquiring the second increment (NCAP 0.2)
from a contractor.  Customs has not yet decided whether it will
develop or acquire future ACE software increments.  If it is to do
either effectively, Customs must use mature software processes. 

SEI, recognized for its expertise in software processes, has
developed models and methods that define and determine, respectively,
software development and software acquisition process maturity.  We
evaluated both ACE software development and ACE software acquisition
processes using SEI's software development capability maturity model
(SW-CMM) and software acquisition capability maturity model (SA-CMM),
respectively, and SEI's software capability evaluation (SCE) method. 
Our evaluations focused on the key process areas (KPA) necessary to
obtain a "repeatable" level of maturity, the second level of SEI's
five-level process maturity models.  Organizations that do not
satisfy these second-level KPA requirements are by default at the
"initial" or first level, meaning that their processes are ad hoc,
sometimes even chaotic. 

Our evaluations found that Customs lacks the capability to either
develop or acquire ACE software effectively, and that it lacks a
software process improvement program.\1 Because of the number and
severity of the KPA weaknesses found, Customs did not achieve the
"repeatable" level of process maturity as either a software developer
or acquirer, meaning that any attempts to do so are likely to produce
software that does not perform as intended, costs more than expected,
and is delivered late.  Further, without a software process
improvement program, it is unlikely that Customs can effectively
address its software process weaknesses. 


--------------------
\1 The results of our evaluation of ACE software development maturity
and Customs' software process improvement efforts were first
published in our report on Customs-wide software development
capability (Customs Service Modernization:  Immature Software
Development Processes Increase Customs Systems Development Risks
(GAO/AIMD-99-35, February 11, 1999)). 


   SEI'S SOFTWARE CAPABILITY
   MODELS AND METHOD:  A BRIEF
   DESCRIPTION
---------------------------------------------------------- Chapter 4:1

The SW-CMM and the SA-CMM rank organizational maturity according to
five levels.  Maturity levels 2 through 5 require verifiable
existence and use of certain key process areas, known as KPAs.  The
SW-CMM includes six level 2 KPAs.  We evaluated Customs' against five
of the six.  The sixth level 2 KPA, software subcontract management,
was not evaluated because Customs did not use subcontractors on NCAP
0.1 (see table 4.1 for a description of each KPA). 



                               Table 4.1
                
                         Description of SW-CMM
                              Level 2 KPAs

SW-CMM Level 2
KPAs                Description
------------------  --------------------------------------------------
Requirements        Defining, validating, and prioritizing
management          requirements, such as functions, performance, and
                    delivery dates.

Software project    Developing estimates for the work to be performed,
planning            establishing the necessary commitments, and
                    defining the plan to perform the work.

Software project    Tracking and reviewing software accomplishments
tracking and        and results against documented estimates,
oversight           commitments, and plans and adjusting these based
                    on the actual accomplishments and results.

Software            Selecting qualified contractors and managing them
subcontract         effectively.
management

Software quality    Reviewing and auditing the software products and
assurance           activities to ensure that they comply with the
                    applicable processes, standards, and procedures
                    and providing the staff and managers with the
                    results of these reviews and audits.

Software            Selecting project baseline items, such as
configuration       specifications, systematically controlling these
management          items and changes to them, and recording and
                    reporting status and change activity for these
                    items.
----------------------------------------------------------------------
The SA-CMM includes seven level 2 KPAs.  We evaluated Customs against
all seven level 2 KPAs.  We also evaluated Customs against one level
3 KPA--acquisition risk management--because it is considered by
software experts to be an integral part of the solicitation, project
performance management, and contract performance management processes
(see table 4.2 for a description of each KPA). 



                               Table 4.2
                
                Description of SA-CMM Level 2 KPA's and
                    the Risk Management Level 3 KPA

SA-CMM Level 2
KPAs                Description
------------------  --------------------------------------------------
Software            Ensuring that reasonable planning for the software
acquisition         acquisition is conducted and that all elements of
planning            the project are included.

Solicitation        Ensuring that award is made to the contractor most
                    capable of satisfying the specified requirements.

Requirements        Establishing a common and unambiguous definition
development and     of software acquisition requirements understood by
management          the acquisition team, system user, and the
                    contractor.

Project office      Managing the activities of the project office and
management          supporting contractor(s) to ensure a timely,
                    efficient, and effective software acquisition.

Contract tracking   Ensuring that the software activities under
and oversight       contract are being performed in accordance with
                    contract requirements, and that products and
                    services will satisfy contract requirements.

Evaluation          Determining that the acquired software products
                    and services satisfy contract requirements prior
                    to acceptance and transition to support.

Transition and      Providing for the transition of the software
support             products being acquired to their eventual support
                    organization.

SA-CMM Level 3 KPA  Description

Acquisition risk    Identifying risks as early as possible, adjusting
management          acquisition strategy to mitigate those risks, and
                    developing and implementing a risk management
                    process as an integral part of the acquisition
                    process.
----------------------------------------------------------------------
For both models, each KPA contains five common attributes that
indicate whether the implementation and institutionalization of a KPA
can be effective, repeatable, and lasting.  The five common features
are: 

  -- Commitment to perform:  The actions that the organization must
     take to establish the process and ensure that it can endure. 
     Commitment to perform typically involves establishing
     organizational policies and sponsorship. 

  -- Ability to perform:  The preconditions that must exist in the
     project or organization to implement the process competently. 
     Ability to perform typically involves allocating resources,
     establishing effective organizational structures, and ensuring
     that personnel have the needed training. 

  -- Activities performed:  The roles and procedures necessary to
     implement the process.  Activities performed typically involve
     establishing plans and procedures, performing the work, tracking
     it, and taking appropriate management actions. 

  -- Measurement and analysis:  The activities performed to measure
     the process and analyze the measurements.  Measurement and
     analysis typically include defining the measurements to be taken
     and the analyses to be conducted to determine the status and
     effectiveness of the activities performed. 

  -- Verifying implementation:  The steps to ensure that the
     activities are performed in compliance with the process that has
     been established.  Verification typically includes reviews by
     management. 

In accordance with SEI's SCE method, for each KPA in level 2 and the
one KPA in level 3 (risk management), we evaluated institutional
policies and practices and compared project-specific guidance and
practices against the five common attributes.  This project-specific
comparison can result in one of four possible outcomes:  (1) project
strength--effective implementation of the key practice, (2) project
weakness--ineffective implementation of a key practice or failure to
implement a key practice, (3) project observation--key practice
evaluated but evidence is inconclusive and cannot support
characterization as either a strength or a weakness, and (4) not
rated--key practice not currently relevant to project, therefore not
evaluated. 


   CUSTOMS LACKS THE CAPABILITY TO
   DEVELOP ACE SOFTWARE
   EFFECTIVELY
---------------------------------------------------------- Chapter 4:2

In February 1999, we reported that NCAP 0.1 did not fully satisfy any
of the KPAs that SEI's SW-CMM requires to be CMM level 2 or
repeatable.\2 While NCAP 0.1 exhibited many strengths in three
KPA's--requirements management, software project planning, and
software project tracking and oversight--it had numerous and
significant weaknesses in the remaining two KPA's--software quality
assurance and software configuration management.  As a result,
Customs' ACE software development capability is immature. 

In our February 1999 report, we also stated that Customs lacked a
software process improvement program.  Without one, it is unlikely
that Customs can effectively address its software process weaknesses. 
Accordingly, we recommended that, after ensuring that its
mission-critical systems are Year 2000 compliant, but before
investing in major software development efforts like ACE, Customs (1)
assign responsibility and authority for software process improvement,
(2) develop and implement a formal plan for software process
improvement that, among other things, was based on our software
development process maturity findings, (3) ensure that every new
software development effort in Customs adopts processes that satisfy
at least SW-CMM level 2 requirements, and (4) ensure that process
improvement activities be initiated for all ongoing essential
software maintenance projects.  Customs agreed with our findings and
stated its commitment to software process improvement and maturity,
including stating its plans for establishing a software process
improvement program and addressing the weaknesses that we identified. 

Each of the five SW-CMM KPAs, along with examples of how the ACE
software development organization compares to each KPA practices, is
summarized below.  (See appendix III for more detailed information on
the KPAs and our findings.)


--------------------
\2 GAO/AIMD-99-35, February 11, 1999. 


      REQUIREMENTS MANAGEMENT
-------------------------------------------------------- Chapter 4:2.1

The purpose of requirements management is to establish a common
understanding of the requirements between the customer and the
software developers.  According to the SW-CMM, a repeatable
requirements management process, among other things, includes (1)
documenting the system requirements allocated to software, (2)
providing adequate resources and funding for managing the allocated
requirements, (3) training members of the software engineering group
to perform their requirements management activities, (4) using the
allocated requirements as the basis for software plans, work
products, and activities, (5) following a written organizational
policy for requirements management, and (6) having a quality
assurance group that reviews the activities and work products for
managing allocated requirements and reports the results. 

The NCAP 0.1 project exhibited strengths in almost all key practices
within the requirements management KPA.  For example, (1) the system
requirements allocated to software were documented, (2) adequate
resources and funding were provided for managing the allocated
requirements, (3) members of the software engineering group were
trained to perform their requirements management activities, and (4)
the software developers used the allocated requirements as the basis
for software plans, work products, and activities. 

Nevertheless, two very important requirements management key
practices were not being performed on the NCAP 0.1 project. 
Specifically, there was no written organizational policy for
requirements management and there was no quality assurance group to
review the activities.  In the absence of these practices, management
is missing important means for ensuring that software requirements
are managed in a prescribed manner. 


      SOFTWARE PROJECT PLANNING
-------------------------------------------------------- Chapter 4:2.2

The purpose of software project planning is to establish reasonable
plans for performing software engineering and for managing the
software project.  According to the SW-CMM, a repeatable software
project planning process, among other things, includes (1)
documenting the software project plan, and preparing plans for
software engineering facilities and support tools, (2) identifying
the work products needed to establish and maintain control of the
software project, (3) making and using measurements to determine the
status of planning activities, (4) following a written organizational
policy for planning a software project, (5) training personnel in
software project planning and estimating, (6) estimating the software
project's efforts, costs, critical computer resources, and schedule
according to documented procedures, and (7) having a quality
assurance group that reviews the activities and work products for
software project planning and reports the results. 

The NCAP 0.1 project exhibited strengths in many of the key practices
within the software project planning KPA.  For example, (1) the
software project plan and plans for the software engineering
facilities and support tools were documented, (2) software work
products that are needed to establish and maintain control of the
project were identified in the project plan, and (3) measurements
were made and used to determine the status of software planning
activities. 

However, we also found several significant key practice weaknesses. 
Specifically, (1) NCAP 0.1 did not follow a written policy for
planning a software project, (2) project personnel were not trained
in project planning and estimating procedures, (3) estimates for the
software project's effort and costs, critical computer resources, and
project schedule were not derived according to documented procedures,
and (4) there was no software quality assurance group to review or
audit the activities and work products for software project planning. 
Such project planning weaknesses mean that management has no
assurance that it will get the consistent, complete, and reliable
information about the project's expected costs and schedules needed
to make expeditious and informed investment decisions. 


      SOFTWARE PROJECT TRACKING
      AND OVERSIGHT
-------------------------------------------------------- Chapter 4:2.3

The purpose of software project tracking and oversight is to provide
adequate visibility into the progress of the software development so
that management can take effective actions when the software
project's performance deviates significantly from the software plans. 
According to the SW-CMM, effective software project tracking and
oversight, among other things, includes (1) designating a project
software manager to be responsible for the project's software
activities and results, (2) having a documented software development
plan for tracking software activities and communicating status, (3)
conducting periodic internal reviews to track technical progress,
plans, performance, and issues against the software development plan,
(4) periodically reviewing the activities for software project
tracking and oversight with senior management, (5) following a
written organizational policy for managing the project, (6) training
software project managers in the technical and personnel aspects of
the software project, (7) reviewing software project commitments and
changes to commitments with senior management, and (8) independently
reviewing or auditing the activities and work products for software
project tracking and oversight. 

NCAP 0.1 exhibited strengths in most of the key practices within the
software project tracking and oversight KPA.  For example, (1) a
project manager had been designated to be responsible for the
project's software activities and results, (2) the project had a
documented software development plan, (3) the software engineering
group conducted periodic internal reviews to track technical
progress, plans, performance, and issues against the software
development plan (e.g., the sizes of software work products and the
risks associated with software costs, resources, and schedule), and
(4) software project tracking and oversight activities were reviewed
with senior management on a weekly basis. 

However, significant weaknesses also existed.  For example, (1) there
was no written organizational policy for managing the software
project, (2) the software managers were not trained in managing the
technical and personnel aspects of the software project, (3) software
project commitments and changes to commitments were not reviewed with
senior management, and (4) because no software quality assurance
group exists, there were no independent reviews or audits of the
activities and work products for software project tracking and
oversight. 


      SOFTWARE QUALITY ASSURANCE
-------------------------------------------------------- Chapter 4:2.4

The purpose of software quality assurance is to independently review
and audit software products and activities to verify that they comply
with applicable procedures and standards and to provide the software
project and higher level managers with the results of these
independent reviews and audits.  According to the SW-CMM, a
repeatable software quality assurance process, among other things,
includes (1) preparing a software quality assurance plan according to
a documented procedure, (2) having a written organizational policy
for implementing software quality assurance, (3) conducting audits of
designated work processes and products to verify compliance, (4)
documenting deviations identified in the software activities and
software work products and handling them according to a documented
procedure, and (5) having experts independent of the software quality
assurance group periodically review the activities and work products
of the project's software quality assurance group. 

NCAP 0.1 did not satisfy any of the key practices within the software
quality assurance KPA.  Because its software quality assurance is
ineffective, ACE is at risk that (1) process and product standards
will not be met and (2) software will not perform as intended, and
will cost more and take longer to develop than expected. 


      SOFTWARE CONFIGURATION
      MANAGEMENT
-------------------------------------------------------- Chapter 4:2.5

The purpose of software configuration management is to establish and
maintain the integrity of the products of the software project
throughout the project's software life- cycle.  According to the
SW-CMM, a repeatable software configuration management process, among
other things, includes (1) preparing a software configuration
management plan according to a documented procedure, (2) establishing
a configuration management library system as a repository for the
software baselines, (3) identifying software work products to be
placed under configuration management, (4) controlling the release of
products from the software baseline library according to a documented
procedure, (5) following a written organizational policy for
implementing software configuration management, (6) recording the
status of configuration items/units according to a documented
procedure, (7) making and using measurements to determine the status
of software configuration management activities, and (8) periodically
reviewing software configuration management activities with senior
management. 

NCAP 0.1 had strengths in several of the key practices within the
software configuration management KPA.  For example, (1) a software
configuration management plan was developed according to a documented
procedure, (2) a configuration management library system was
established as a repository for the software baselines, (3) software
work products to be placed under configuration management were
identified, and (4) the release of products from the software
baseline library was controlled according to a documented procedure. 

However, NCAP 0.1 exhibited weaknesses in most of the software
configuration management key practices, which collectively jeopardize
Customs' ability to maintain the integrity of the project's software
products.  Specifically, (1) there was no organizational policy for
implementing software configuration management, (2) the status of
configuration management items was not recorded according to a
documented procedure, (3) no measurements were taken to determine the
status of software configuration management activities, and (4)
software configuration management activities were not periodically
reviewed with senior management. 


   CUSTOMS LACKS THE CAPABILITY TO
   ACQUIRE ACE SOFTWARE
   EFFECTIVELY
---------------------------------------------------------- Chapter 4:3

NCAP 0.2 did not fully satisfy any of the KPA's that SEI's SA-CMM
requires to be a CMM level 2 or repeatable.  We found extensive
weaknesses in all KPA's except one--requirements development and
management.  Each of the eight KPA's, along with examples of how
Customs' software acquisition processes compare to the KPA goals, is
summarized below.  (See appendix IV for more detailed information on
KPAs and our findings.)


      SOFTWARE ACQUISITION
      PLANNING
-------------------------------------------------------- Chapter 4:3.1

The purpose of software acquisition planning is to ensure that
reasonable planning for the software acquisition is conducted and
that all aspects of the total software acquisition effort are
included in these plans.  According to the SA-CMM, a repeatable
software acquisition planning process, among other things, includes
(1) having a written software acquisition policy, (2) developing and
documenting the software acquisition strategy and plan, (3) having
management review software acquisition planning activities, and (4)
making and using measurements to determine the status of software
acquisition planning activities. 

NCAP 0.2 had no strengths in this KPA.  For example, (1) the project
followed no written policy for planning the software acquisition, (2)
there was no documented software acquisition strategy or plan, (3)
software acquisition planning activities were not being reviewed by
management, and (4) software acquisition planning activities were not
being measured.  As a result, management lacks an effective means to
identify problems in project performance and take corrective action
expeditiously. 


      SOLICITATION
-------------------------------------------------------- Chapter 4:3.2

The purpose of solicitation is to prepare a request for proposal that
delineates a project's software-related requirements, and, consistent
with relevant solicitation laws and regulations, select a contractor
that can most cost-effectively satisfy these requirements.  According
to the SA-CMM, specific requirements for a repeatable solicitation
process include, among other things, (1) designating responsibility
for the software portion of the solicitation, (2) designating a
selection official to be responsible for the selection process and
decision, (3) preparing cost and schedule estimates for the software
products and services being acquired, (4) having a written policy for
the conduct of the software portion of the solicitation, (5) having
and following a solicitation plan, (6) independently reviewing cost
and schedule estimates for the software products and services being
acquired, and (7) making and using measurements to determine the
status of the solicitation activities and resultant products. 

NCAP 0.2 had some strengths in this area, including (1) designating
responsibility for the software solicitation, (2) designating a
selection official for the selection process and decision, and (3)
preparing cost and schedule estimates for the software products and
services being acquired. 

However, we found many weaknesses, including (1) no written policy
for the conduct of the software solicitation and (2) no solicitation
plans to guide the performance of solicitation activities.  These
weaknesses increase the risk of Customs not adequately or uniformly
evaluating the offerors' proposals, and making a suboptimal
selection.  Other weaknesses included (1) no independent review of
software cost and schedule estimates and (2) no measurements to
determine the status of solicitation activities and resultant
products.  As a result, management cannot identify solicitation
problems early and resolve them expeditiously. 


      REQUIREMENTS DEVELOPMENT AND
      MANAGEMENT
-------------------------------------------------------- Chapter 4:3.3

The purpose of requirements development and management is to
establish and maintain a common and unambiguous definition of
software requirements among the acquisition team, the system users,
and the software development contractor.  This KPA involves two
subprocesses:  (1) developing a baseline set of software-related
contractual requirements and (2) managing these requirements and
changes to these requirements for the duration of the acquisition. 

SA-CMM requirements development and management practices necessary to
achieve a repeatable maturity include (1) having a group that is
responsible for performing requirements development and management
activities, (2) ensuring that individuals performing requirements
development and management activities have experience or receive
training, (3) measuring and reporting on the status of requirements
development and management activities and resultant products to
management, (4) periodically reviewing requirements development and
management activities by acquisition organization management, (5)
having a written organizational policy for establishing and managing
the software-related contractual requirements, (6) performing
requirements development and management activities in accordance with
documented plans, (7) appraising the impact of system requirements
change requests on the software being acquired, and (8) maintaining
traceability between the software-related contractual requirements
and the contractor's software work products. 

NCAP 0.2 had many strengths in requirements development and
management.  For example, (1) there was a group that is responsible
for performing requirements development and management, (2) group
members had experience or have received training in the activities,
(3) measurements of requirements development and management
activities were made, and (4) management reviewed the activities
periodically. 

However, we found weaknesses in important key practices that
jeopardize effective control of the requirements baseline and can
result in software products that do not meet cost, schedule, or
performance objectives.  Specifically, (1) there was no written
policy for establishing and managing the software-related contractual
requirements, (2) there was no documented requirements development
and management plan, (3) system requirements change requests were not
appraised for their impact on the software being acquired, and (4)
traceability between the software-related contractual requirements
and the contractor's software work products was not maintained. 


      PROJECT OFFICE MANAGEMENT
-------------------------------------------------------- Chapter 4:3.4

The purpose of project office management is to manage the activities
of the project office and supporting contractors to ensure a timely,
efficient, and effective software acquisition.  According to the
SA-CMM, effective project office management requires, among other
things, (1) designating responsibility for project management, (2)
providing project teams with authority to alter either the project's
performance, cost, or schedule baseline, (3) having a written policy
for the management of the software project, (4) using a software
acquisition management plan, and (5) using a corrective action system
for identifying, recording, tracking, and correcting problems. 

NCAP 0.2 had several practice strengths in the KPA including (1)
designating a project manager as responsible for the project and (2)
providing project teams with authority to change either the
performance, cost, or schedule software acquisition baseline when
necessary.  However, we found numerous weaknesses, including (1) no
written policy for management of the software project, (2) no
software acquisition management plan, and (3) no corrective action
system.  These weaknesses jeopardize the project's ability to ensure
that important project office and contractor activities are defined,
understood, and completed. 


      CONTRACT TRACKING AND
      OVERSIGHT
-------------------------------------------------------- Chapter 4:3.5

The purpose of contract tracking and oversight is to ensure that the
software development contractor performs according to the terms of
the contract; needed contract changes are identified, negotiated, and
incorporated into the contract; and contractor performance issues are
identified early, when they are easier to address.  According to the
SA-CMM, a repeatable contract tracking and oversight process, among
other things, includes (1) designating responsibility for contract
tracking and oversight, (2) providing support for the project team by
contracting specialists, (3) ensuring that individuals performing
contract tracking and oversight are suitably experienced or trained,
(4) having a written organizational policy for contract tracking and
oversight, (5) having a documented plan for contract tracking and
oversight, (6) reviewing required contractor software planning
documents to oversee the contractor's software engineering effort,
and (7) tracking problems or issues found by the project team during
contract tracking and oversight in a corrective action system. 

NCAP 0.2 had several strengths in this KPA, including (1) designating
responsibility for contract tracking and oversight activities, (2)
providing contracting specialist support to the project team, and (3)
ensuring that personnel responsible for contract tracking and
oversight are suitably experienced or trained.  However, we found
significant weaknesses, including (1) no written organizational
policy for contract tracking and oversight, (2) no documented
contract tracking and oversight plan, (3) no required contractor
software planning documents that could be used to oversee the
contractor's software engineering effort, and (4) no corrective
action tracking system for recording problems or issues found by the
project team.  Because of these weaknesses, Customs' contractor
tracking and oversight activities are undisciplined and unstructured,
thereby increasing the chances of its software acquisitions being
late, costing more than expected, and not performing as intended. 


      EVALUATION
-------------------------------------------------------- Chapter 4:3.6

The purpose of evaluation is to determine that the acquired software
products and services satisfy contract requirements prior to
acceptance.  According to the SA-CMM, a repeatable evaluation process
includes (1) designating responsibility for planning, managing, and
performing evaluation activities, (2) ensuring that individuals
performing evaluation activities have experience or receive training,
(3) managing the evaluation of the acquired software according to a
written policy, (4) documenting evaluation plans and conducting
evaluation activities in accordance with the plan, (5) developing and
managing evaluation requirements in conjunction with developing
software technical requirements, (6) tracking contractor performance
of evaluation activities for compliance with the contract, and (7)
measuring and reporting on the status of evaluation activities to
management. 

NCAP 0.2 had some evaluation strengths, including (1) designating
responsibility for planning, managing, and performing evaluation
activities and (2) ensuring that individuals performing evaluation
activities have experience or receive training.  However, we found
many significant weaknesses, including, (1) there was no written
policy for managing the evaluation of the acquired software, (2)
there was no documented evaluation plan to use as the basis for
conducting evaluation activities, (3) evaluation requirements were
not developed in conjunction with development of the system or
software technical requirements, (4) contractor evaluation activity
performance was not tracked for compliance with the contract, and (5)
no measurements were made to determine the status of evaluation
activities.  Because of these pervasive evaluation weaknesses,
Customs has no assurance that contractor-developed ACE software will
satisfy defined requirements. 


      TRANSITION AND SUPPORT
-------------------------------------------------------- Chapter 4:3.7

The purpose of transition and support is to provide for the effective
and efficient "hand-off" of the acquired software products to the
support organization responsible for software maintenance.  According
to the SA-CMM, repeatable transition and support processes, among
other things, include (1) having a written policy for transitioning
software products to the support organization, (2) designating a
group that is responsible for coordinating transition and support
activities, (3) performing project activities in accordance with a
documented transition and support plan, and (4) measuring the status
of the transition and support activities. 

NCAP 0.2 had no practice strengths in the transition and support KPA. 
In particular, (1) there was no written policy for the transitioning
of software products to the software support organization, (2) there
was no designated group responsible for coordinating transition and
support activities, (3) there was no transition and support plan, and
(4) the status of transition and support activities were not
measured.  As a result, Customs is not effectively positioned to
assume support responsibility for the ACE software that it acquires. 


      ACQUISITION RISK MANAGEMENT
-------------------------------------------------------- Chapter 4:3.8

The purpose of acquisition risk management is to formally identify
risks as early as possible and adjust the acquisition to mitigate
those risks.  According to the SA-CMM, an effective risk management
process, among other things, includes (1) having a written policy on
acquisition risk management, (2) developing a software acquisition
risk management plan, (3) conducting software risk management
activities in accordance with the plan (e.g., identifying risks,
taking mitigation actions, and tracking risk mitigation actions to
completion), and (4) measuring and reporting on the status of
acquisition risk management activities to management. 

NCAP 0.2 was weak in all key practices of acquisition risk
management.  In particular, (1) there was no written policy on
acquisition risk management, (2) there was no software acquisition
risk management plan, (3) no risk management activities were being
performed, and (4) no measurements were taken to determine the status
of risk management activities.  Because of these weaknesses, Customs
has no assurance that it will identify risks early and effectively
mitigate them. 


   CONCLUSIONS
---------------------------------------------------------- Chapter 4:4

Customs' software acquisition and development processes are
insufficiently mature to support the effective development or
acquisition of ACE.  Until these processes are strengthened, there is
no basis for confidence that ACE will be delivered on time, within
budget, or perform as specified. 


OVERALL CONCLUSIONS,
RECOMMENDATIONS, AND AGENCY
COMMENTS
============================================================ Chapter 5

Successful systems modernization is critical to Customs' ability to
implement the provisions of its legislative mandates and to achieve
its goals of increased compliance with trade laws and speedier
processing of imports.  However, Customs currently lacks the full
management and technical capability needed to successfully modernize
its systems.  As a result, satisfaction of its legislative mandate
and realization of the attendant benefits by both Customs and the
trade community are in jeopardy. 

With ACE, Customs has not adequately demonstrated that it is "doing
the right thing," i.e., that the system that it is building is the
most cost effective system for meeting the needs of Customs and the
trade community.  Specifically, by not having a complete information
systems architecture to guide the development and evolution of its
systems, including ACE, Customs runs the risk that its efforts will
not provide optimum performance and will be incompatible and
expensive to maintain.  Also, by using unreliable and incomplete ACE
cost and benefit data, Customs does not know if ACE is a cost
effective system solution to its needs, or if its investments in ACE
are wise. 

Customs also has not clearly demonstrated that it is developing ACE
"the right way." In particular, without incrementally managing such a
mammoth system as ACE, Customs will not know whether its cost and
benefit expectations are being met until it has already spent
hundreds of millions of dollars.  Such a "grand design" approach has
proven repeatedly to be ineffective and has been abandoned by
successful IT organizations.  Additionally, because its ability to
develop or acquire software is immature, Customs can do neither
effectively.  Therefore, its software acquisition and development
processes must be strengthened before it can hope to acquire or
develop ACE software on time and within budget, as well as to achieve
expected benefits. 


   RECOMMENDATIONS
---------------------------------------------------------- Chapter 5:1

In addition to previous recommendations to improve Customs'
management of information technology,\1 we recommend that Customs
correct the management and technical weaknesses discussed in this
report before building ACE.  To accomplish this, GAO recommends that
the Commissioner of Customs, with the support of Customs' CIO, ensure
that Customs

(1) rigorously analyze alternative approaches to building ACE,
including ITDS as an alternative to developing ACE entirely within
Customs and;

(2) make investment decisions incrementally, i.e., for each
increment: 

  -- use disciplined processes to prepare a rigorous life-cycle cost
     estimate, including an explicit discussion of its inherent
     uncertainty;

  -- prepare realistic and supportable benefit expectations;

  -- require a favorable return-on-investment and compliance with
     Customs' architecture before making any investment; and

  -- validate actual costs and benefits once an increment is piloted,
     compare these with estimates, use the results in making further
     decisions on subsequent increments, and report the results to
     Customs' House and Senate appropriations and authorizing
     committees; and

(3) strengthen ACE software acquisition management by: 

  -- establishing an effective process improvement program and
     correcting the weaknesses in ACE software acquisition processes
     identified in this report, thereby bringing ACE processes to at
     least SEI level 2 and

  -- requiring at least SEI level 2 processes of all ACE software
     contractors. 


--------------------
\1 GAO/AIMD-99-35, February 11, 1999 and GAO/AIMD-98-70, May 5, 1998. 


   AGENCY COMMENTS
---------------------------------------------------------- Chapter 5:2

In its written comments on a draft of this report, Customs agreed
with our conclusions and recommendations and stated that it is
committed to addressing the problems discussed in the report.  To
this end, Customs cited a number of actions that it has underway and
planned to improve management of information technology in general,
and ACE specifically.  For example, Customs stated that it has
recruited a CIO with extensive experience in enterprise architecture
and major systems acquisition, reorganized its Office of Information
and Technology, and revised its System Development Life Cycle
process.  Also, Customs stated that it has engaged a contractor to
update and improve the ACE cost-benefit analysis and plans for
another contractor to independently review the new ACE cost-benefit
analysis.  Additionally, Customs reported that it revised its
investment management procedures to provide for effective systems
architecture enforcement, as we recommended in our May 1998 report.\2
The report has been modified to reflect this information.  To fully
correct the management and technical weaknesses discussed in this
report, Customs must follow through and effectively implement its
plans to address all of our recommendations.  Appendix I provides the
full text of Customs' comments. 



(See figure in printed edition.)Appendix I

--------------------
\2 GAO/AIMD-98-70, May 5, 1998. 


COMMENTS FROM THE U.S.  CUSTOMS
SERVICE
============================================================ Chapter 5



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)


DETAILED COMPARISON OF SEI'S
CHECKLIST FOR RELIABLE COST
ESTIMATES AND CUSTOMS' PRACTICES
FOR ESTIMATING ACE COSTS
========================================================== Appendix II

The following seven tables rate Customs' practices for estimating
life-cycle costs of $1.05 billion against SEI's criteria for reliable
cost estimating.  A strength indicates that Customs satisfied the SEI
criterion; a weakness indicates that it did not.  Where evidence was
inconclusive and did not clearly support a determination of either
strength or weakness, a rating of "observation" was given. 



                                    Table II.1
                     
                     Are the Objectives of the Estimate Clear
                                   and Correct?

SEI checklist item                Finding                           Rating
--------------------------------  --------------------------------  ------------
The objectives of the estimate    The objectives of the estimates   Strength
are stated in writing.            are stated in writing.

The life-cycle to which the       The 10-year life-cycle is         Strength
estimate applies is clearly       clearly defined for the ACE cost
defined.                          estimate, and consists of 6
                                  years of development and
                                  deployment and 4 years of
                                  operations and maintenance.

The tasks and activities          The tasks and activities that     Strength
included in (and excluded from)   Customs included in (and
the estimate are clearly          excluded from) the estimate are
identified.                       clearly identified.

The tasks and activities          The tasks and activities          Strength
included in the estimate are      included in the estimate are
consistent with the objectives    consistent with the stated
of the estimate.                  objectives of the estimate.
--------------------------------------------------------------------------------


                                    Table II.2
                     
                      Has the Task Been Appropriately Sized?

SEI checklist item                 Finding                            Rating
---------------------------------  ---------------------------------  ----------
A structured process has been      A structured process has not been  Weakness
used to estimate and describe the  used to estimate and describe the
size of the software project.      size of the entire software
                                   project. Customs followed a
                                   structured process for estimating
                                   the size of NCAP 0.1, the first
                                   ACE software release; however,
                                   the methodology used to estimate
                                   the size of NCAP 0.2 and all
                                   subsequent ACE software releases
                                   was not structured and lacked
                                   critical elements, such as
                                   defined functional requirements
                                   for NCAP 0.2 and all subsequent
                                   releases.

A structured process has been      A structured process has not been  Weakness
used to estimate and describe the  used to estimate and describe the
extent of software reuse.          extent of software reuse.
                                   According to Customs officials,
                                   software reuse was discussed
                                   during an ACE planning
                                   conference; however, the results
                                   were not documented.

The processes for estimating size  The processes for estimating size  Weakness
and reuse are documented.          and reuse are not documented. The
                                   documentation available on the
                                   estimate, including the ACE/NCAP
                                   workload estimate spreadsheets,
                                   does not document the size and
                                   reuse estimating processes
                                   followed by Customs.

The descriptions of size and       The description of size            Weakness
reuse identify what is included    identifies what is included in
in (and excluded from) the size    the size measures used (e.g.,
and reuse measures used.           screens, business objects, and
                                   operations); however, no
                                   description of reuse measures
                                   exists.

The measures of reuse distinguish  The measures of reuse are not      Observatio
between code that will be          documented; however, the ACE       n
modified and code that will be     workload estimate spreadsheets do
integrated as is into the system.  include an estimate of existing
                                   code that will be modified for
                                   use in subsequent NCAP software
                                   releases.

The definitions, measures, and     Customs did not use a model to     Weakness
rules used to describe size and    estimate the cost of ACE.
reuse are consistent with the      Furthermore, the reuse
requirements (and calibrations)    definitions, measures, and rules
of the models used to estimate     are not documented.
cost.

The size estimate was checked by   The size estimate for the entire   Weakness
relating it to measured sizes of   ACE project was not checked by
other software products or         relating it to measured sizes of
components.                        other software products or
                                   components. Customs did not have
                                   a comparable internal software
                                   product to benchmark against at
                                   the time of the estimate.
                                   According to Customs officials,
                                   the NCAP 0.1 size estimate was
                                   compared (via informal and
                                   undocumented telephone inquiries)
                                   to similar products and
                                   components previously developed
                                   outside of Customs. The size
                                   estimate for NCAP 0.2 and
                                   subsequent releases was based
                                   upon the NCAP 0.1 size estimate.

The size estimating process was    The size estimating process was    Weakness
checked by testing its predictive  not checked by testing its
capabilities against measured      predictive capabilities against
sizes of completed products.       measured sizes of completed
                                   projects.
--------------------------------------------------------------------------------


                                    Table II.3
                     
                      Is the Estimated Cost Consistent With
                      Demonstrated Accomplishments on Other
                                    Projects?

SEI checklist item                 Finding                            Rating
---------------------------------  ---------------------------------  ----------
The organization has a structured  Customs does not have a            Weakness
process for relating estimates to  structured, documented process
actual costs of completed work     for relating estimates to actual
--the process is documented        costs of completed work.
--the process was followed.

The cost models that were used     Customs did not use a cost model   Weakness
have been calibrated to relevant   and did not have any relevant
historical data. (Models of some   historical data to calibrate.
sort are needed to provide
consistent rules for
extrapolating from previous
experience.)

The cost models quantify           Customs did not use a cost model.  Weakness
demonstrated organizational        Further, Customs changed its
performance in ways that           software development approach
normalize for differences among    from an in-house development
software products and projects.    (NCAP 0.1) to an acquisition
(So that a simple, unnormalized,   (NCAP 0.2). Nonetheless, Customs
lines-of-code per staff-month      used the NCAP 0.1 estimate as the
extrapolation is NOT the basis     basis for extrapolating the
for the estimate.)                 estimate for the remaining ACE
                                   releases without normalizing the
                                   NCAP 0.1 data to account for the
                                   differences in the approaches.

The consistency achieved when      Customs did not use a cost model.  Weakness
fitting the cost models to         At the time that the ACE cost
historical data has been measured  estimate was developed, Customs
and reported.                      had no historical data with which
                                   to evaluate consistency. Cost
                                   data are currently being
                                   collected, but the collected data
                                   has not been used to measure and
                                   report the consistency achieved
                                   when using the cost models.

The values used for cost models'   Customs did not use a cost model;  Weakness
parameters appear valid when       therefore, comparison of cost
compared to values that fit the    models' parameter values to those
models well to past projects.      used on past projects was not
                                   done.

The calibration of cost models     Customs did not use cost models;   Weakness
was done with the same versions    therefore, calibration was not
of the models that were used to    done.
prepare the estimate.

The methods used to account for    The methods used to account for    Strength
reuse recognize that reuse is not  reuse recognize that reuse is not
free. (The estimate accounts for   free. The Customs estimating
activities such as interface       spreadsheets account for
design, modification,              activities associated with NCAP
integration, testing, and          software reuse, such as
documentation that are associated  modification of existing code,
with effective reuse.)             integration, and testing.

Extrapolations from past projects  Extrapolations from NCAP 0.1 to    Weakness
account for differences in         NCAP 0.2 and subsequent releases
application technology. (For       did not account for differences
example, data from projects that   in application technology (e.g.,
implemented traditional mainframe  the change in architecture from
applications require adjustments   the legacy mainframe system to a
if used as a basis for estimating  combined mainframe and Internet-
client-server implementation.      based client-server system).
Some cost models provide
capabilities for this, others do
not.)

Extrapolations from past projects  Extrapolations from NCAP 0.1 to    Weakness
account for observed, long-term    NCAP 0.2 and subsequent releases
trends in software technology      did not account for differences
improvement. (Although some cost   in tools, languages, and
models attempt this internally,    personnel due to the change in
the best methods are usually       architecture from the legacy
based on extrapolating measured    mainframe system written in COBOL
trends in calibrated               and C++ to a combined mainframe
organizational performance.)       and Internet-based system written
                                   in C++ and Java.

Extrapolations from past projects  Extrapolations from NCAP 0.1 to    Weakness
account for the effects            NCAP 0.2 and subsequent releases
of introducing new software        did not account for the change in
technology or processes.           process from an in-house software
(Introducing a new technology or   development project (NCAP 0.1) to
process can initially reduce an    a software acquisition (NCAP
organization's productivity.)      0.2).

Work-flow schematics have been     Work-flow schematics have not      Weakness
used to evaluate how this project  been used to evaluate how this
is similar to (and how it differs  project is similar to (and how it
from) projects used to             differs from) projects used to
characterize the organization's    characterize the organization's
past performance.                  past performance.
--------------------------------------------------------------------------------


                                    Table II.4
                     
                         Have the Factors That Affect the
                     Estimate Been Identified and Explained?

SEI checklist item                 Finding                            Rating
---------------------------------  ---------------------------------  ----------
A written summary of parameter     A written summary of the           Weakness
values and their                   parameters affecting the cost
rationales accompanies the         estimate exists; however, a
estimate.                          written summary of the values
                                   used and their rationales does
                                   not exist.

Assumptions have been identified   The assumptions have been          Strength
and explained.                     identified and explained.

A structured process such as a     Customs has used spreadsheets      Strength
template or format has been used   designed specifically for the ACE
to ensure that key factors have    NCAP project in an effort to
not been overlooked.               ensure that key factors have not
                                   been overlooked.

Uncertainties in parameter values  No uncertainties in parameter      Weakness
have been identified and           values were identified or
quantified.                        quantified.

A risk analysis has been           A risk analysis has been           Strength
performed, and risks that affect   performed, and risks that affect
cost have been identified and      cost have been identified and
documented. (Elements addressed    documented (e.g., lack of
include issues such as             requirement definition for later
probability of occurrence,         releases, technical
effects on parameter values, cost  complications, stability of
impacts, schedule impacts, and     development and configuration
interactions with other            management environments and
organizations.)                    process).
--------------------------------------------------------------------------------


                                    Table II.5
                     
                       Have Steps Been Taken to Ensure the
                       Integrity of the Estimating Process?

SEI checklist item                 Finding                            Rating
---------------------------------  ---------------------------------  ----------
Management reviewed and agreed to  According to Customs project       Observatio
the values for all descriptive     officials, management reviewed     n
parameters before costs were       and agreed to the values for all
estimated.                         descriptive parameters before
                                   costs were estimated; however,
                                   management approval has not been
                                   documented.

Adjustments to parameter values    Not applicable. According to       N/A
to meet a desired cost             Customs project officials, no
or schedule have been documented.  adjustments to parameter values
                                   to meet a desired cost or
                                   schedule were made.

If a dictated schedule has been    Not applicable. According to       N/A
imposed, the estimate is           Customs project officials, no
accompanied by an estimate of (1)  dictated schedule has been
the normal schedule and (2) the    imposed.
additional expenditures required
to meet the dictated schedule.

Adjustments to parameter values    Not applicable. No adjustments to  N/A
to meet a desired cost             parameter values were made, and
or schedule are accompanied by     thus no accompanying management
management action that makes the   action occurred.
values realistic.

More than one cost model or        Two estimating approaches          Strength
estimating approach has been       (spreadsheets) were used for
used, and the differences in       NCAP, and the resulting
results have been analyzed and     differences in the estimates
explained.                         (within 15 percent) have been
                                   analyzed and explained.

People from related but different  People from the requirements,      Strength
projects or disciplines were       design, systems engineering,
involved in preparing the          integration, testing, and project
estimate.                          management teams were involved in
                                   preparing the estimate.

At least one member of the         Two members of the estimating      Observatio
estimating team is an experienced  team have practical experience,    n
estimator, trained in the cost     but none have been trained in
models that were used.             cost estimating and cost models.

Estimators independent of the      According to Customs project       Observatio
performing organization concur     officials, estimators independent  n
with the reasonableness of the     of the performing organization
parameter values and estimating    concurred with the reasonableness
methodology.                       of the parameter values and
                                   estimating methodology, but their
                                   analysis is not documented.

The groups that will be doing the  Customs (NCAP 0.1 developer) and   Strength
work accept the estimate as an     the contractor (NCAP 0.2
achievable target.                 developer) accepted the estimates
                                   as achievable targets.

Memorandums of agreement have      No memorandums of agreement have   Weakness
been completed and signed with     been completed and signed with
the other organizations whose      the requirements, design, systems
contributions affect cost.         engineering, integration,
                                   testing, project management team,
                                   and other organizations whose
                                   contributions affect cost.
--------------------------------------------------------------------------------


                                    Table II.6
                     
                         Is the Organization's Historical
                         Evidence Capable of Supporting a
                                Reliable Estimate?

SEI checklist item                 Finding                            Rating
---------------------------------  ---------------------------------  ----------
The estimating organization has a  Customs did not have a method for  Weakness
method for organizing and          organizing and retaining
retaining information on           information on completed projects
completed projects (a historical   (a historical database) at the
database).                         time that this estimate was
                                   developed. (Note: Customs has
                                   since developed a historical
                                   database which contains planned
                                   versus actual results for
                                   schedule, effort, and cost and
                                   plans to use this in future
                                   estimates.)

The database contains a useful     Customs did not have a historical  Weakness
set of completed projects.         database at the time this
                                   estimate was developed. (Note:
                                   The database developed later
                                   contains NCAP 0.1 and 0.2 data
                                   only.
                                   No data from other completed
                                   projects is included.)

Elements included in (and          Customs did not have a historical  Weakness
excluded from) the effort, cost,   database at the time this
schedule, size, and reuse          estimate was developed. Elements
measures in the database are       included in the effort, cost, and
clearly identified. (See, for      schedule measures in the database
example, the SEI checklists for    Customs has since developed are
defining effort, schedule, and     clearly identified. However, size
size measures.)                    and reuse measures are not
                                   identified in the database.

Schedule milestones (start and     The schedule milestones (start     Strength
finish dates) are described in     and finish dates) are accompanied
terms of criteria for initiation   by task and subtask descriptions.
or completion, so that work
accomplished between milestones
is clearly bounded.

Records for completed projects     Not applicable. According to       N/A
indicate whether or not unpaid     Customs project officials, unpaid
overtime was used.                 overtime was insignificant and
                                   did not warrant recording for
                                   NCAP 0.1.

Unpaid overtime, if used, has      Not applicable. According to       N/A
been quantified, so that recorded  Customs project officials, unpaid
data provide a valid basis for     overtime was insignificant and
estimating future effort.          did not warrant recording for
                                   NCAP 0.1.

Cost models that were used for     Customs did not use any cost       Weakness
estimating have been used also to  models for estimating.
provide consistent frameworks for  The spreadsheets that were used
recording historical data. (This   for cost and workload estimating
helps ensure that comparable       do not provide a consistent
terms and parameters are used      framework for recording
across all projects, and that      historical data for NCAP releases
recorded data are suitable for     0.1 and 0.2.
use in the estimating models.)

The data in the historical         At the time of the estimate,       Weakness
database have been examined to     there was no historical database.
identify inconsistencies and       (Note: The data in the historical
anomalies have been corrected or   database prepared later have not
explained. (This is best done      yet been examined to identify
with the same cost models used     inconsistencies.)
for estimating.)

The organization has a structured  Customs has a structured process   Strength
process for capturing effort and   for capturing effort and cost
cost data from ongoing projects.   data from the ACE project on a
                                   weekly basis.

The producing organization holds   According to project officials,    Weakness
postmortems at the completion of   Customs plans to hold postmortems
its projects to (1) ensure that    at the completion of each ACE/
recorded data are valid, and (2)   NCAP software release. NCAP 0.1
ensure that events that affected   was completed but a postmortem
costs get recorded and described   has not yet been performed. No
while they are still fresh in      documentation exists which
people's minds.                    describes the postmortem process
                                   or structure.

Information on completed projects  According to Customs project       Weakness
includes                           officials, all this information
--the life-cycle model used,       has not yet been collected, but
together with the portion covered  they plan to do so.
by the recorded cost and
schedule;
--actual (measured) size, cost,
and schedule;
--the actual staffing profile;
--an estimate at completion,
together with the values for cost
model parameters that map the
estimate to the actual cost and
schedule;
--a work breakdown structure or
alternative description of the
tasks included in the recorded
cost;
--a work-flow schematic that
illustrates the software process
used;
--nonlabor costs;
--management costs;
--a summary or list of
significant deliverables
(software and documentation)
produced by the project; and
--a summary of any unusual issues
that affected cost.

Evolution in the organization's    Customs' historical database on    Weakness
work-flow schematics shows steady  completed software projects
improvement in the understanding   represents an effort to
and measurement of its software    understand and measure its
processes.                         software processes. Customs
                                   currently has no other ongoing
                                   software process improvement
                                   efforts.
--------------------------------------------------------------------------------


                                    Table II.7
                     
                       Has the Situation Remained Unchanged
                         Since the Estimate Was Prepared?

SEI checklist item                 Finding                            Rating
---------------------------------  ---------------------------------  ----------
The estimate has not been          The estimate has been invalidated  Weakness
invalidated by recent events,      by the following recent actions
changing requirements, or          and events: (1) The approved
management action (or inaction).   fiscal year 1999 ACE appropriated
                                   funding totals $6 million, which
                                   is significantly less than the
                                   $75.1 million fiscal year 1999
                                   funding requirement stated in the
                                   ACE cost estimate;
                                   (2) NCAP 0.1 was developed in-
                                   house and the cost estimate for
                                   subsequent releases was based on
                                   the assumption that they would be
                                   developed in-house as well.
                                   Customs has since decided to
                                   acquire NCAP 0.2 and perhaps
                                   subsequent releases from a
                                   contractor; and (3) NCAP 0.2 and
                                   subsequent releases will utilize
                                   a combined mainframe and
                                   internet-based client-server
                                   architecture instead of the NCAP
                                   0.1 mainframe architecture upon
                                   which the estimate was based.

The estimate is being used as the  The estimate is not being used as  Weakness
basis for assigning resources,     the basis for
deploying schedules, and making    assigning resources, deploying
commitments.                       schedules, and making
                                   commitments. Instead, Customs is
                                   using a revised estimate that
                                   reflects actual fiscal year 1998
                                   and 1999 ACE appropriated
                                   funding, which is significantly
                                   less than the funding
                                   requirements identified in the
                                   ACE cost estimate for fiscal
                                   years 1998 and 1999.

The estimate is the current        The estimate is not the current    Weakness
baseline for project tracking and  baseline for project tracking and
oversight.                         oversight. Instead, Customs is
                                   using a revised estimate that
                                   reflects actual fiscal year 1998
                                   and 1999 ACE appropriated
                                   funding, which is significantly
                                   less than the funding
                                   requirements identified in the
                                   ACE cost estimate for fiscal
                                   years 1998 and 1999. Customs is
                                   currently tracking actual project
                                   costs against these appropriated
                                   funds.
--------------------------------------------------------------------------------

RESULTS OF SOFTWARE DEVELOPMENT
CAPABILITY MATURITY MODEL (SW-CMM)
EVALUATION FOR NCAP 0.1
========================================================= Appendix III



                                   Table III.1
                     
                          Key Process Area: Requirements
                                    Management

              Key practice               Finding                    Rating
------------  -------------------------  -------------------------  ------------
Commitment 1  The project follows a      There is no written        Weakness
              written organizational     organizational policy for
              policy for managing the    managing the system
              system requirements        requirements allocated to
              allocated to software.     software.

Ability 1     For each project,          The Process Analysis and   Strength
              responsibility is          Requirements Team is
              established for analyzing  responsible for analyzing
              the system requirements    system requirements and
              and allocating them to     allocating them to
              hardware, software, and    hardware, software, and
              other system components.   other system components.

Ability 2     The allocated              Allocated requirements     Strength
              requirements are           are documented.
              documented.

Ability 3     Adequate resources and     Adequate resources and     Strength
              funding are provided for   funding are provided for
              managing the allocated     managing the allocated
              requirements.              requirements.

Ability 4     Members of the software    Members of the software    Strength
              engineering group and      engineering group and
              other software-related     other software-related
              groups are trained to      groups are trained to
              perform their              perform their
              requirements management    requirements management
              activities.                activities.

Activity 1    The software engineering   The software engineering   Strength
              group reviews the          group reviews the
              allocated requirements     allocated requirements
              before they are            before they are
              incorporated into the      incorporated into the
              software project.          software project.

Activity 2    The software engineering   The software engineering   Strength
              group uses the allocated   group uses the allocated
              requirements as the basis  requirements as the basis
              for software plans, work   for software plans, work
              products, and activities.  products, and activities.

Activity 3    Changes to the allocated   Changes to the allocated   Strength
              requirements are reviewed  requirements are reviewed
              and incorporated into the  and incorporated into the
              software project.          software project.

Measurement   Measurements are made and  Measurements are made and  Strength
1             used to determine the      used to determine the
              status of the activities   status of the activities
              for managing the           for managing the
              allocated requirements.    allocated requirements.

Verification  The activities for         Periodic meetings with     Strength
1             managing the allocated     senior management include
              requirements are reviewed  reviews of allocated
              with senior management on  requirements.
              a periodic basis.

Verification  The activities for         The activities for         Strength
2             managing the allocated     managing the allocated
              requirements are reviewed  requirements are reviewed
              with the project manager   with the project manager
              on both a periodic and     on both a periodic and
              event-driven basis.        event-driven basis.

Verification  The software quality       There is no software       Weakness
3             assurance group reviews    quality assurance group;
              and/or audits the          therefore, no reviews
              activities and work        and/or audits are done.
              products for managing the
              allocated requirements
              and reports the results.
--------------------------------------------------------------------------------


                                   Table III.2
                     
                        Key Process Area: Software Project
                                     Planning

              Key practice               Finding                    Rating
------------  -------------------------  -------------------------  ------------
Commitment 1  A project software         The NCAP project has a     Strength
              manager is designated to   software manager
              be responsible for         designated to be
              negotiating commitments    responsible for
              and developing the         negotiating commitments
              project's software         and developing the
              development plan.          project's software
                                         development plan.

Commitment 2  The project follows a      The project does not       Weakness
              written organizational     follow a written
              policy for planning a      organizational policy for
              software project.          planning a software
                                         project.

Ability 1     A documented and approved  The approved project plan  Strength
              statement of work exists   meets the requirements
              for the software project.  for a statement of work
                                         for the project.

Ability 2     Responsibilities for       The project manager has    Strength
              developing the software    been assigned
              development plan are       responsibility for
              assigned.                  developing the software
                                         development plan.

Ability 3     Adequate resources and     Adequate resources and     Strength
              funding are provided for   funding have been
              planning the software      provided for planning the
              project.                   software project.

Ability 4     The software managers,     Project personnel are not  Weakness
              software engineers, and    trained in software
              other individuals          project planning and
              involved in the software   estimating procedures.
              project planning are
              trained in the software
              estimating and planning
              procedures applicable to
              their areas of
              responsibility.

Activity 1    The software engineering   The software engineering   Strength
              group participates on the  group participates on the
              project proposal team.     project proposal team.

Activity 2    Software project planning  Software project planning  Strength
              is initiated in the early  is initiated in the early
              stages of, and in          stages of, and in
              parallel with, the         parallel with, the
              overall project planning.  overall project planning.

Activity 3    The software engineering   The software engineering   Strength
              group participates with    group participates with
              other affected groups in   other affected groups in
              the overall project        the overall project
              planning throughout the    planning throughout the
              project's life.            project's life.

Activity 4    Software project           Software project           Weakness
              commitments made to        commitments made to
              individuals and groups     individuals and groups
              external to the            external to the
              organization are reviewed  organization are not
              with senior management     reviewed with senior
              according to a documented  management and there is
              procedure.                 no documented procedure
                                         for such reviews.

Activity 5    A software life-cycle      There is no documented     Weakness
              with predefined stages of  evidence that a software
              manageable size is         life-cycle was selected
              identified or defined.     for the project.

Activity 6    The project's software     The project has a          Strength
              development plan is        software development plan
              developed according to a   that is developed
              documented procedure.      according to a documented
                                         procedure.

Activity 7    The plan for the software  The plan for the software  Strength
              project is documented.     project is documented in
                                         the project plan.

Activity 8    Software work products     Software work products     Strength
              that are needed to         that are needed to
              establish and maintain     establish and maintain
              control of the software    control of the software
              project are identified.    project are identified in
                                         the project plan.

Activity 9    Estimates for the size of  Estimates for the size of  Strength
              the software work          the software work
              products (or changes to    products (or changes to
              the size of software work  the size of software work
              products) are derived      products) are derived
              according to a documented  according to a documented
              procedure.                 procedure.

Activity 10   Estimates for the          Estimates for the          Weakness
              software project's effort  software project's effort
              and costs are derived      and costs are not derived
              according to a documented  according to a documented
              procedure.                 procedure.

Activity 11   Estimates for the          Estimates for the          Weakness
              project's critical         project's critical
              computer resources are     computer resources are
              derived according to a     not derived according to
              documented procedure.      a documented procedure.

Activity 12   The project's software     There is no documented     Weakness
              schedule is derived        procedure for deriving
              according to a documented  the software schedule.
              procedure.

Activity 13   The software risks         The software risks         Strength
              associated with the cost,  associated with the cost,
              resource, schedule, and    resource, schedule, and
              technical aspects of the   technical aspects
              project are identified,    of the project are
              assessed, and documented.  identified, assessed, and
                                         documented.

Activity 14   Plans for the project's    Plans for the project's    Strength
              software engineering       software engineering
              facilities and support     facilities and support
              tools are prepared.        tools are prepared.

Activity 15   Software planning data     Software planning data     Strength
              are recorded.              are recorded.

Measurement   Measurements are made and  Measurements are made and  Strength
1             used to determine the      used to determine the
              status of the software     status of the software
              planning activities.       planning activities
                                         (status and MS Project
                                         reports).

Verification  The activities for         The activities for         Strength
1             software project planning  software project planning
              are reviewed with senior   are periodically reviewed
              management on a periodic   with senior management,
              basis.                     including reports to
                                         Treasury and Customs
                                         investment review board
                                         (IRB).

Verification  The activities for         The activities for         Strength
2             software project planning  software project planning
              are reviewed with the      are reviewed with the
              project manager on both a  project manager on both a
              periodic and event-        periodic and event-
              driven basis.              driven basis through
                                         weekly status reports and
                                         meetings.

Verification  The software quality       There is no SQA group.     Weakness
3             assurance group reviews
              and/or audits the
              activities and work
              products for software
              project planning and
              reports the results.
--------------------------------------------------------------------------------


                                   Table III.3
                     
                        Key Process Area: Software Project
                              Tracking and Oversight

              Key practice               Finding                    Rating
------------  -------------------------  -------------------------  ------------
Commitment 1  A project software         The project software       Strength
              manager is designated to   manager is designated to
              be responsible for the     be responsible for the
              project's software         project's software
              activities and results.    activities and results.

Commitment 2  The project follows a      There is no written        Weakness
              written organizational     organizational policy for
              policy for managing the    managing the software
              software project.          project.

Ability 1     A software development     An approved and            Strength
              plan for the software      documented software
              project is documented and  development plan is
              approved.                  contained in the project
                                         plan.

Ability 2     The project software       The project software       Strength
              manager explicitly         manager explicitly
              assigns responsibility     assigns responsibility
              for software work          for software work
              products and activities.   products and activities.

Ability 3     Adequate resources and     Adequate resources and     Strength
              funding are provided for   funding are provided for
              tracking the software      tracking the software
              project.                   project.

Ability 4     The software managers are  The software managers are  Weakness
              trained in managing the    not trained in managing
              technical and personnel    the technical and
              aspects of the software    personnel aspects of the
              project.                   software project.

Ability 5     First-line software        First-line software        Strength
              managers receive           managers receive
              orientation in the         orientation in the
              technical aspects of the   technical aspects of the
              software project.          software project.

Activity 1    A documented software      A documented software      Strength
              development plan is used   development plan
              for tracking the software  is used for tracking the
              activities and             software activities and
              communicating status.      communicating status.

Activity 2    The project's software     No documented procedure    Weakness
              development plan is        exists for
              revised according to a     revising the software
              documented procedure.      development plan.

Activity 3    Software project           Software project           Weakness
              commitments and changes    commitments and changes
              to commitments made to     to commitments made to
              individuals and groups     individuals and groups
              external to the            external to the
              organization are reviewed  organization are not
              with senior management     reviewed with senior
              according to a documented  management. Also, there
              procedure.                 is no documented
                                         procedure for such
                                         reviews.

Activity 4    Approved changes to        Approved changes to        Strength
              commitments that affect    commitments that affect
              the software project are   the software project are
              communicated to the        communicated to the
              members of the software    members of the software
              engineering group and      engineering group and
              other software-related     other software-related
              groups.                    groups through weekly
                                         staff meetings.

Activity 5    The sizes of the software  The size of the software   Strength
              work products (or sizes    work products (or size of
              of the changes to the      the changes to the
              software work products)    software work products)
              are tracked, and           are tracked; however, at
              corrective actions are     the time of our
              taken as necessary.        evaluation, no corrective
                                         actions were needed.

Activity 6    The project's software     The project's software     Strength
              effort and costs are       effort and costs are
              tracked and corrective     tracked; however, at the
              actions are taken as       time of our evaluation,
              necessary.                 no corrective actions
                                         were needed.

Activity 7    The project's critical     The project's critical     Strength
              computer resources are     computer resources are
              tracked, and corrective    tracked; however, at the
              actions are taken as       time of our evaluation,
              necessary.                 no corrective actions
                                         were needed.

Activity 8    The project's software     The project's software     Strength
              schedule is tracked, and   schedule is tracked;
              corrective actions are     however, at the time of
              taken as necessary.        our evaluation, no
                                         corrective actions were
                                         needed.

Activity 9    Software engineering       Software engineering       Strength
              technical activities are   technical activities are
              tracked, and corrective    tracked, and corrective
              actions are taken as       actions are taken as
              necessary.                 necessary.

Activity 10   The software risks         The software risks         Strength
              associated with cost,      associated with cost,
              resource, schedule, and    resource, schedule, and
              technical aspects          technical aspects
              of the project are         of the project are
              tracked.                   tracked.

Activity 11   Actual measurement data    Actual measurement data    Strength
              and replanning data for    and replanning data for
              the software project are   the software project are
              recorded.                  recorded.

Activity 12   The software engineering   The software engineering   Strength
              group conducts periodic    group conducts periodic
              internal reviews to track  internal reviews to track
              technical progress,        technical progress,
              plans, performance, and    plans, performance, and
              issues against the         issues against the
              software development       software development
              plan.                      plan.

Activity 13   Formal reviews to address  Formal reviews to address  Strength
              the accomplishments and    the accomplishments and
              results of the software    results of the software
              project are conducted at   project are conducted at
              selected project           selected project
              milestones according to a  milestones according to a
              documented procedure.      documented procedure.

Measurement   Measurements are made and  Measurements are made and  Strength
1             used to determine the      used to determine the
              status of the software     status of the software
              tracking and oversight     tracking and oversight
              activities.                activities.

Verification  The activities for         The activities for         Strength
1             software project tracking  software project tracking
              and oversight are          and oversight are
              reviewed with senior       reviewed with senior
              management periodically.   management on a weekly
                                         basis.

Verification  The activities for         The activities for         Strength
2             software project tracking  software project tracking
              and oversight are          and oversight are
              reviewed with the project  reviewed with the project
              manager on both a          manager on both a
              periodic and event-        periodic and event-
              driven basis.              driven basis.

Verification  The software quality       No software quality        Weakness
3             assurance group reviews    assurance group exists;
              and/or audits the          therefore, there are no
              activities and work        reviews and/or audits of
              products for software      the activities and work
              project tracking and       products for software
              oversight and reports the  project tracking and
              results.                   oversight.
--------------------------------------------------------------------------------


                                   Table III.4
                     
                        Key Process Area: Software Quality
                                    Assurance

              Key practice               Finding                    Rating
------------  -------------------------  -------------------------  ------------
Commitment 1  The project follows a      There is no written        Weakness
              written organizational     organizational policy for
              policy for implementing    implementing SQA.
              software quality
              assurance (SQA).

Ability 1     A group that is            Although there is no       Observation
              responsible for            group responsible for
              coordinating and           coordinating and
              implementing SQA for the   implementing SQA for the
              project (the SQA group)    project, there are plans
              exists.                    to establish one and
                                         assign responsibility.

Ability 2     Adequate resources and     There is no SQA group,     Weakness
              funding are provided for   and no resources and
              performing the SQA         funding are provided for
              activities.                performing SQA
                                         activities.

Ability 3     Members of the SQA group   There is no SQA group or   Weakness
              are trained to perform     plan to provide SQA
              their SQA activities.      training.

Ability 4     The members of the         Project staff do not       Weakness
              software project receive   receive orientation on
              orientation on the role,   the role,
              responsibilities,          responsibilities,
              authority, and value of    authority, and value of
              the SQA group.             the SQA group.

Activity 1    The SQA plan is prepared   There is no SQA plan or    Weakness
              for the software project   documented procedure for
              according to a documented  preparing one.
              procedure.

Activity 2    The SQA group's            There is no SQA group.     Weakness
              activities are performed
              in accordance with the
              SQA plan.

Activity 3    The SQA group              There is no SQA group.     Weakness
              participates in the
              preparation and review of
              the project's software
              development plan,
              standards, and
              procedures.

Activity 4    The SQA group reviews the  There is no SQA group.     Weakness
              software engineering
              activities to verify
              compliance.

Activity 5    The SQA group audits       There is no SQA group.     Weakness
              designated software work
              products to verify
              compliance.

Activity 6    The SQA group              There is no SQA group.     Weakness
              periodically reports the
              results of its activities
              to the software
              engineering group.

Activity 7    Deviations identified in   Procedures for handling    Weakness
              the software activities    deviations in
              and software work          testing activities are
              products are documented    documented. However,
              and handled according to   procedures for handling
              a documented procedure.    deviations in other
                                         software development
                                         activities (such as
                                         compliance with
                                         organizational policy and
                                         standards and adherence
                                         to software development
                                         plan) are not documented.

Activity 8    The SQA group conducts     There is no SQA group.     Weakness
              periodic reviews of its
              activities and findings
              with the customer's SQA
              personnel, as
              appropriate.

Measurement   Measurements are made and  There is no SQA group.     Weakness
1             used to determine the
              cost and schedule status
              of the SQA activities.

Verification  The SQA activities are     There is no SQA group.     Weakness
1             reviewed with senior
              management on a periodic
              basis.

Verification  The SQA activities are     There is no SQA group.     Weakness
2             reviewed with the project
              manager on both a
              periodic and event-
              driven basis.

Verification  Experts independent of     There is no SQA group.     Weakness
3             the SQA group
              periodically review the
              activities and
              software work products of
              the project's SQA group.
--------------------------------------------------------------------------------


                                   Table III.5
                     
                     Key Process Area: Software Configuration
                                    Management

              Key practice               Finding                    Rating
------------  -------------------------  -------------------------  ------------
Commitment 1  The project follows a      The project has no         Weakness
              written organizational     organizational policy for
              policy for implementing    implementing SCM.
              software configuration
              management (SCM).

Ability 1     A board having the         The project does not have  Weakness
              authority for managing     a SCM board with
              the project's software     authority for managing
              baselines (i.e., a         software baselines.
              software configuration
              control board) exists or
              is established.

Ability 2     A group that is            The project does not have  Weakness
              responsible for            a group that is
              coordinating and           responsible for
              implementing SCM for the   coordinating and
              project (i.e., the SCM     implementing SCM
              group) exists.             functions.

Ability 3     Adequate resources and     Adequate resources and     Weakness
              funding are provided for   funding are not provided
              performing the SCM         for performing the SCM
              activities.                activities.

Ability 4     Members of the SCM group   There is no SCM group.     Weakness
              are trained in the
              objectives, procedures,
              and methods for
              performing their SCM
              activities.

Ability 5     Members of the software    Members of the software    Weakness
              engineering group and      engineering group and
              other software-related     other software-related
              groups are trained to      groups are not trained to
              perform their SCM          perform their SCM
              activities.                activities.

Activity 1    A SCM plan is prepared     A SCM plan was prepared    Strength
              for each software project  according to procedures
              according to a documented  documented in the October
              procedure.                 1996 SDLC.

Activity 2    A documented and approved  The documented and         Weakness
              SCM plan is used as the    approved SCM plan is used
              basis for performing the   as the basis for
              SCM activities.            performing code control;
                                         however, the plan is not
                                         used as a basis for doing
                                         SCM on software
                                         documentation and other
                                         software engineering
                                         products such as cost
                                         estimates and schedules.

Activity 3    A configuration            A configuration            Strength
              management library system  management library system
              is established as a        is established as a
              repository for the         repository for the
              software baselines.        software baselines.

Activity 4    The software work          The software work          Strength
              products to be placed      products to be placed
              under configuration        under configuration
              management are             management are
              identified.                identified.

Activity 5    Change requests and        Change requests and        Strength
              problem reports for all    problem items/units are
              configuration items/       initiated, recorded,
              risks are initiated,       reviewed, approved, and
              recorded, reviewed,        tracked according to a
              approved, and tracked      procedure documented in
              according to a documented  the SCM plan.
              procedure.

Activity 6    Changes to baselines are   Changes to baselines are   Strength
              controlled according to a  controlled according to a
              documented procedure.      documented procedure in
                                         the SCM plan.

Activity 7    Products from the          Products from the          Strength
              software baseline library  software baseline library
              are created, and their     are created, and their
              release is controlled      release is controlled
              according to a documented  according to the SCM
              procedure.                 plan.

Activity 8    The status of software     The status of SCM items/   Weakness
              configuration items/       units are not recorded
              units is recorded          according to a documented
              according to a documented  procedure.
              procedure.

Activity 9    Standard reports           Standard reports           Weakness
              documenting SCM            documenting SCM
              activities and the         activities and contents
              contents of the software   of the software baseline
              baseline are developed     are not developed.
              and made available to
              affected groups and
              individuals.

Activity 10   Software baseline audits   Software baseline audits   Weakness
              are conducted according    are not conducted.
              to a documented
              procedure.

Measurement   Measurements are made and  No measurements are taken  Weakness
1             used to determine the      to determine the status
              status of SCM activities.  of SCM activities.

Verification  SCM activities are         SCM activities are not     Weakness
1             reviewed with senior       reviewed with senior
              management periodically.   management periodically.

Verification  SCM activities are         SCM activities are not     Weakness
2             reviewed with the project  reviewed with the project
              manager on both a          manager on both a
              periodic and               periodic and event-
              event-driven basis.        driven basis.

Verification  SCM group periodically     No SCM audits are done to  Weakness
3             audits software baselines  verify that software
              to verify that they        baselines conform to the
              conform to the             documentation that
              documentation that         defines them.
              defines them.

Verification  The software quality       There is no quality        Weakness
4             assurance group reviews    assurance group;
              and/or audits the          therefore, no one reviews
              activities and work        and/or audits the
              products for SCM and       activities and work
              reports the results.       products for SCM
                                         activities performed.
--------------------------------------------------------------------------------

RESULTS OF SOFTWARE ACQUISITION
CAPABILITY MATURITY MODEL (SA-CMM)
EVALUATION FOR NCAP 0.2
========================================================== Appendix IV



                                    Table IV.1
                     
                      Key Process Area: Software Acquisition
                                     Planning

            Key practice                 Finding                      Rating
----------  ---------------------------  ---------------------------  ----------
Commitment  The acquisition              There is no written policy   Weakness
1           organization has a written   for planning the software
            policy for planning the      acquisition.
            software acquisition.

Commitment  Responsibility for software  Responsibility for software  Weakness
2           acquisition planning         acquisition planning is not
            activities is designated.    designated.

Ability 1   The acquisition              The acquisition              Weakness
            organization has             organization does not have
            experienced software         experienced software
            acquisition management       acquisition management
            personnel.                   personnel.

Ability 2   Adequate resources are       Resources for software       Weakness
            provided for software        acquisition planning are
            acquisition planning         not adequate.
            activities.

Activity 1  Software acquisition         Software acquisition         Weakness
            planning personnel are       planning personnel are not
            involved in system           involved in system
            acquisition planning.        acquisition planning.

Activity 2  The software acquisition     A software acquisition       Weakness
            strategy for the             strategy for NCAP 0.2 was
            project is developed and     not developed and
            documented.                  documented.

Activity 3  The project's software       Software acquisition         Weakness
            acquisition planning is      planning for NCAP 0.2 has
            documented, and the          not been documented.
            planning documentation
            is maintained over the life
            of the project.

Activity 4  Life-cycle support of the    Life-cycle support of the    Weakness
            software is included in      software has not been
            software acquisition         included in any document.
            planning documentation.

Activity 5  Life-cycle cost and          Life-cycle cost and          Observatio
            schedule estimates for the   schedule estimates were      n
            software products and        prepared, but there is no
            services being acquired are  evidence that they were
            prepared and independently   independently reviewed.
            reviewed.

Measuremen  Measurements are made and    Software acquisition         Weakness
t 1         used to determine the        planning activities are not
            status of the software       measured.
            acquisition planning
            activities and resultant
            products.

Verificati  Software acquisition         Software acquisition         Weakness
on 1        planning activities are      planning activities are not
            reviewed by acquisition      reviewed by management.
            organization management
            periodically.

Verificati  Software acquisition         There is no documented       Weakness
on 2        planning activities are      evidence that the project
            reviewed by the project      manager reviewed software
            manager on both a periodic   acquisition planning
            and event-driven basis.      activities.
--------------------------------------------------------------------------------


                                    Table IV.2
                     
                          Key Process Area: Solicitation

           Key practice                   Finding                        Rating
---------  -----------------------------  -----------------------------  -------
Commitmen  The acquisition organization   There is no written policy     Weaknes
t 1        has a written policy for the   for the conduct of the         s
           conduct of the software        software portion of the
           portion of the solicitation.   solicitation.

Commitmen  Responsibility for the         Responsibility for the         Strengt
t 2        software portion of the        software portion of the        h
           solicitation is designated.    solicitation was designated
                                          to the project manager/
                                          contract officer's technical
                                          representative (COTR).

Commitmen  A selection official has been  A selection official has been  Strengt
t 3        designated to be responsible   designated.                    h
           for the selection process and
           the decision.

Ability 1  A group that is responsible    There is no documented         Weaknes
           for coordinating and           evidence that a group          s
           conducting solicitation        responsible for coordinating
           activities exists.             and conducting solicitation
                                          activities exists.

Ability 2  Adequate resources are         Resources for solicitation     Weaknes
           provided for solicitation      activities are not adequate.   s
           activities.

Ability 3  Individuals performing         There is only one individual   Weaknes
           solicitation activities have   performing solicitation        s
           experience or receive          activities. Although he was
           training.                      trained as a COTR, he has no
                                          experience or training in
                                          acquisition management or
                                          costing methodologies and
                                          tools.

Ability 4  The groups supporting the      No orientation is provided to  Weaknes
           solicitation (e.g.,            any other groups. End users,   s
           end user, systems              system engineers, and
           engineering, software support  software support
           organization, and application  organizations did not
           domain experts) receive        participate in the
           orientation on the             solicitation.
           solicitation's objectives and
           procedures.

Activity   The project team performs its  There are no documented        Weaknes
1          activities in accordance with  solicitation plans to guide    s
           its documented solicitation    the performance of
           plans.                         solicitation activities.

Activity   The project team performs its  There are no documented        Weaknes
2          activities in accordance with  proposal evaluation plans.     s
           its documented proposal
           evaluation plans.

Activity   Cost and schedule estimates    Cost and schedule estimates    Strengt
3          for the software products and  for the software products and  h
           services being sought are      services being sought are
           prepared.                      prepared.

Activity   Software cost and schedule     Software cost and schedule     Weaknes
4          estimates are independently    estimates are not              s
           reviewed for                   independently reviewed.
           comprehensiveness and
           realism.

Activity   The project team takes action  The project team takes action  Strengt
5          to ensure the mutual           to ensure the mutual           h
           understanding of software      understanding of software
           requirements and plans prior   requirements and plans prior
           to contract award.             to contract award.

Measureme  Measurements are made and      No measurements were made to   Weaknes
nt 1       used to                        determine the status of        s
           determine the status of the    solicitation activities and
           solicitation activities and    resultant products.
           resultant products.

Verificat  Solicitation activities are    Solicitation activities were   Weaknes
ion 1      reviewed periodically by the   not reviewed by the            s
           designated selection official  designated selection official
           or acquisition organization    or acquisition organization
           management.                    management on a periodic
                                          basis.

Verificat  Solicitation activities are    Solicitation activities are    Weaknes
ion 2      reviewed by the project        not reviewed by the project    s
           manager on both a periodic     manager on both a periodic
           and event-driven basis.        and event-driven basis.
--------------------------------------------------------------------------------


                                    Table IV.3
                     
                          Key Process Area: Requirements
                            Development and Management

            Key practice                 Finding                      Rating
----------  ---------------------------  ---------------------------  ----------
Commitment  The acquisition              There is no written policy   Weakness
1           organization has a written   for establishing and
            policy for establishing and  managing the software-
            managing the software-       related contractual
            related contractual          requirements.
            requirements.

Commitment  Responsibility for           Responsibility for           Strength
2           requirements development     requirements development
            and management is            and management is
            designated.                  designated to the Process
                                         Analysis and Requirements
                                         Team (PART).

Ability 1   A group that is responsible  The PART team is             Strength
            for performing requirements  responsible for performing
            development and management   requirements development
            activities exists.           and management.

Ability 2   Adequate resources are       The PART team has adequate   Strength
            provided for requirements    resources for requirements
            development and management   development and management
            activities.                  activities.

Ability 3   Individuals performing       Individuals performing       Strength
            requirements development     requirements development
            and management activities    and management activities
            have experience or receive   have experience or receive
            training.                    training.

Activity 1  The project team performs    There are no documented      Weakness
            its activities in            requirements development
            accordance with its          and management plans.
            documented requirements
            development and management
            plans.

Activity 2  The project team develops    The project team develops    Strength
            and baselines the software-  and baselines the software-
            related contractual          related contractual
            requirements and places      requirements and places
            them under change control    them under change control
            early in the project, but    early in the project, but
            not later than release of    not later than release of
            the solicitation package.    the solicitation package.

Activity 3  The project team appraises   To date there have been no   Observatio
            system requirements change   system requirements change   n
            requests for their impact    requests.
            on the software being
            acquired.

Activity 4  The project team appraises   To date there have been no   Observatio
            all changes to the           changes to the software-     n
            software-related             related contractual
            contractual requirements     requirements.
            for their impact on
            performance, architecture,
            supportability, system
            resource utilization, and
            contract schedule and cost.

Activity 5  Bidirectional traceability   Bidirectional traceability   Weakness
            between the software-        between the software-
            related contractual          related contractual
            requirements and the         requirements and the
            contractor's software work   contractor's software work
            products and services is     products and services is
            maintained throughout the    not maintained throughout
            effort.                      the effort.

Measuremen  Measurements are made and    Measurements are made and    Strength
t 1         used to determine the        used to determine the
            status of the requirements   status of the requirements
            development and management   development and management
            activities and resultant     activities and resultant
            products.                    products.

Verificati  Requirements development     Requirements development     Strength
on 1        and management activities    and management activities
            are reviewed periodically    are reviewed periodically
            by acquisition organization  by acquisition management
            management (and the          (and the contractor).
            contractor).

Verificati  Requirements development     Requirements development     Weakness
on 2        and management activities    and management activities
            are reviewed by the project  are not reviewed by the
            manager on both a periodic   project manager.
            and event-driven basis.
--------------------------------------------------------------------------------


                                    Table IV.4
                     
                       Key Process Area: Project Management

            Key practice                 Finding                      Rating
----------  ---------------------------  ---------------------------  ----------
Commitment  The acquisition              There is no written policy   Weakness
1           organization has a written   for execution of the
            policy for execution of the  software project.
            software project.

Commitment  Responsibility for project   Responsibility for project   Strength
2           management is designated.    management has been
                                         designated.

Ability 1   A team that is responsible   A team that is responsible   Strength
            for performing the           for performing the
            project's software           project's software
            acquisition management       acquisition management
            activities exists.           activities exists.

Ability 2   Adequate resources for the   Adequate resources for the   Weakness
            project team and matrix      project team and matrix
            support persons are          support persons are not
            provided for the duration    provided for the duration
            of the software acquisition  of the software acquisition
            project.                     project.

Ability 3   When project trade-offs are  The project manager, who is  Strength
            necessary, the project       also the contracting
            manager is permitted to      officer's technical
            alter either the             representative, is
            performance, cost, or        permitted to alter either
            schedule software            the performance, cost, or
            acquisition baseline.        schedule software
                                         acquisition baseline.

Ability 4   The project team and matrix  The project team members do  Weakness
            support individual(s) have   not have experience and
            experience or receive        have not received training
            training in project          in project software
            software acquisition         acquisition management
            management activities.       activities.

Activity 1  The project team performs    There is no software         Weakness
            its activities in            acquisition management
            accordance with its          plan; therefore, the
            documented software          project team does not
            acquisition management       perform its activities in
            plans.                       accordance with a plan.

Activity 2  The organization of the      The organization of the      Weakness.
            project provides for the     project does not provide
            management of all project    for the management of all
            functions.                   the project's functions.
                                         For example, acquisition
                                         planning and risk
                                         management are not
                                         provided.

Activity 3  The software acquisition     The software acquisition     Weakness
            management activities of     management activities of
            the project team are         the project team are not
            directed to accomplish the   directed to accomplish the
            project's objectives.        project's objectives.

Activity 4  The software acquisition     The activities of the        Weakness.
            management activities of     project team are not
            the project team are         controlled.
            controlled.

Activity 5  The project team implements  There is no corrective       Weakness
            a corrective action system   action system for the
            for the identification,      identification, recording,
            recording, tracking, and     tracking, and correction of
            correction of problems       problems discovered during
            discovered during the        the software acquisition.
            software acquisition.

Activity 6  The project team tracks      The project team tracks      Observatio
            project status, execution,   project status, execution,   n
            funding, and expenditures    funding, and expenditures.
            and takes action.            At the time of our
                                         evaluation, however, no
                                         actions
                                         were needed.

Measuremen  Measurements are made and    Measurements are taken and   Strength
t 1         used to                      used to determine the
            determine the status of the  status of project
            project management           management activities and
            activities and resultant     resultant products.
            products.

Verificati  Project management           Project management           Weakness
on 1        activities are reviewed by   activities are not reviewed
            acquisition organization     by acquisition organization
            management periodically.     management periodically.

Verificati  Project management           There was no evidence        Weakness
on 2        activities are reviewed by   provided to show that the
            the project manager on both  project management
            a periodic and event-        activities are reviewed by
            driven basis.                the project manager.
--------------------------------------------------------------------------------


                                    Table IV.5
                     
                     Key Process Area: Contract Tracking and
                                Oversight Findings

            Key practice                 Finding                      Rating
----------  ---------------------------  ---------------------------  ----------
Commitment  The acquisition              There is no organizational   Weakness
1           organization has a written   policy for the
            policy for the contract      contract tracking and
            tracking and oversight of    oversight of the contracted
            the contracted software      software.
            effort.

Commitment  Responsibility for contract  Responsibility for contract  Strength
2           tracking and oversight       tracking and oversight has
            activities is designated.    been designated to the
                                         project manager/COTR.

Commitment  The project team is          The contracting officer      Strength
3           supported by contracting     supports the project team
            specialists in the           in the execution of the
            execution of the contract.   contract.


Ability 1   A group that is responsible  The project team is          Strength
            for managing contract        responsible for contract
            tracking and oversight       tracking and oversight
            activities exists.           activities.

Ability 2   Adequate resources are       Adequate resources are not   Weakness
            provided for contract        provided for contract
            tracking and oversight       tracking and oversight
            activities.                  activities.

Ability 3   Individuals performing       The project manager has      Strength
            contract tracking and        received training
            oversight activities have    conducting his duties as a
            experience or receive        COTR.
            training.

Activity 1  The project team performs    There is no contract         Weakness
            its activities in            tracking and oversight
            accordance with its          plan.
            documented contract
            tracking and oversight
            plans.

Activity 2  The project team reviews     The contractor was not       Weakness
            required contractor          required to submit any
            software planning documents  software planning documents
            which, when satisfactory,    that could be used to
            are used to oversee the      oversee the contractor's
            contractor's software        software engineering
            engineering effort.          effort.

Activity 3  The project team conducts    The project team conducts    Strength
            periodic reviews             periodic reviews and
            and interchanges with the    interchanges with the
            contractor.                  contractor.

Activity 4  The project team reviews     Customs provided the         Not rated
            and tracks the development   software engineering
            of the software engineering  environment to the
            environment required to      contractor; therefore,
            provide life-cycle support   there was no need to track
            for the acquired software.   the development of the
                                         software engineering
                                         environment required to
                                         provide life-cycle support
                                         for the acquired software.

Activity 5  Any problems or issues       Problems or issues found by  Weakness
            found by the project team    the project team during
            during contract tracking     contract tracking and
            and oversight are recorded   oversight are not tracked.
            in the appropriate
            corrective action system
            and tracked to closure.

Activity 6  The project team maintains   The project team maintains   Strength
            the integrity of the         the integrity of the
            contract throughout the      contract throughout the
            contract performance         contract performance
            period.                      period.

Measuremen  Measurements are made and    Measurement are not made to  Weakness
t 1         used to determine the        determine the status of
            status of the contract       contract tracking and
            tracking and oversight       oversight activities and
            activities and resultant     resultant products.
            products.

Verificati  Contract tracking and        Contract tracking and        Weakness
on 1        oversight activities are     oversight activities are
            reviewed by acquisition      not reviewed by the
            organization management      acquisition organization
            periodically.                management.

Verificati  Contract tracking and        Contract tracking and        Weakness
on 2        oversight activities are     oversight activities are
            reviewed by the project      not reviewed by the project
            manager on both a periodic   manager.
            and event-driven basis.
--------------------------------------------------------------------------------


                                    Table IV.6
                     
                           Key Process Area: Evaluation

            Key practice                 Finding                      Rating
----------  ---------------------------  ---------------------------  ----------
Commitment  The acquisition              There is no written policy   Weakness
1           organization has a written   for managing the evaluation
            policy for managing the      of the acquired software
            evaluation of the acquired   products and services.
            software products and
            services.

Commitment  Responsibility for           Responsibility for           Strength
2           evaluation activities is     evaluation activities is
            clearly designated.          clearly designated.

Ability 1   A group that is responsible  A group that is responsible  Strength
            for planning, managing, and  for planning, managing, and
            performing evaluation        performing evaluation
            activities for the project   activities for the project
            exists.                      exists.

Ability 2   Adequate resources are       We found no evidence that a  Observatio
            provided for evaluation      lack of resources precluded  n
            activities.                  the performance of
                                         evaluation activities.

Ability 3   Individuals performing       Individuals performing       Strength
            evaluation activities have   evaluation activities have
            experience or receive        experience or receive
            training.                    training.

Ability 4   Members of the project team  No orientation is provided   Weakness
            and groups supporting the    to members of the project
            software acquisition         team and groups supporting
            receive orientation on the   the software acquisition.
            objectives of the
            evaluation approach.

Activity 1  The project team performs    There are no documented      Weakness
            its activities in            evaluation plans.
            accordance with its
            documented evaluation
            plans.

Activity 2  The project's evaluation     The project's evaluation     Weakness
            requirements are developed   requirements are not
            in conjunction with the      developed in conjunction
            development of the system    with the development of the
            or software technical        system or software
            requirements.                technical requirements.

Activity 3  The evaluation requirements  Some evaluation              Weakness
            are incorporated into the    requirements have been
            solicitation package and     incorporated into the
            resulting contract.          solicitation package and
                                         resulting contract. Others,
                                         such as a mechanism that
                                         provides the project team
                                         visibility into the
                                         contractor's evaluation
                                         program and requirements
                                         for the contractor to
                                         establish a corrective
                                         action system, have not.

Activity 4  The project team assesses    The project team does not    Weakness
            contractor's performance     assess the contractor's
            for compliance with          performance for compliance
            evaluation requirements.     with evaluation
                                         requirements.

Activity 5  Planned evaluations are      No products or services      Observatio
            performed on the acquired    have yet been accepted for   n
            software products and        operational use. The
            services prior to            project team plans to
            acceptance for operational   evaluate acquired software
            use.                         products and services prior
                                         to acceptance for
                                         operational use.

Activity 6  Results of the evaluations   No products or services      Observatio
            are analyzed and compared    have yet been delivered by   n
            to the contract's            the contractor. The project
            requirements to establish    team plans to analyze the
            an objective basis to        results of the evaluations
            support the decision to      and compare the results to
            accept the products and      contract requirements as an
            services                     objective basis to support
            or to take further action.   the decision to accept the
                                         products and services or to
                                         take further action.

Measuremen  Measurements are made and    No measurements are made to  Weakness
t 1         used to determine the        determine the status of
            status of the evaluation     evaluation activities and
            activities and resultant     resultant products.
            products.

Verificati  Evaluation activities are    Evaluation activities are    Weakness
on 1        reviewed by acquisition      not reviewed by acquisition
            organization management      organization management.
            periodically.

Verificati  Evaluation activities are    Evaluation activities are    Weakness
on 2        reviewed by the project      not reviewed by the project
            manager on both a periodic   manager.
            and event-driven basis.
--------------------------------------------------------------------------------


                                    Table IV.7
                     
                     Key Process Area: Transition to Support

            Key practice                 Finding                      Rating
----------  ---------------------------  ---------------------------  ----------
Commitment  The acquisition              There is no policy for the   Weakness
1           organization has a written   transitioning of software
            policy for transitioning     products to the software
            software products to the     support organization.
            software support
            organization.

Commitment  The acquisition              A software support           Weakness
2           organization ensures that    organization has not been
            the software support         established.
            organization is involved in
            planning for transition to
            support.

Commitment  Responsibility for           Responsibility for           Weakness
3           transition to support        transition to support
            activities is designated.    activities has not been
                                         designated.

Ability 1   A group that is responsible  No group is responsible for  Weakness
            for coordinating the         coordinating transition to
            transition to support        support activities.
            activities exists.

Ability 2   Adequate resources are       Adequate resources have not  Weakness
            provided for transition to   been provided
            support activities.          for transition to support
                                         activities.

Ability 3   The organization             The organization             Weakness
            responsible for providing    responsible for providing
            support of the software      support of the software
            products is identified no    products was not identified
            later than initiation of     before initiation of the
            the solicitation package's   solicitation package's
            development.                 development.

Ability 4   The software support         At the time of the audit,    Not rated
            organization, prior to       the system was still in
            transition, has a complete   development and had not
            inventory of all software    reached the
            and related items that are   transition stage.
            to be transitioned.

Ability 5   Individuals performing       Individuals responsible for  Weakness
            transition to support        transition to support
            activities have experience   activities have not been
            or receive training.         selected.

Ability 6   The members of               The members of               Weakness
            organizations interfacing    organizations interfacing
            with the transition to       with the transition to
            support activities receive   support activities have not
            orientation on the salient   received orientation on the
            aspects of transition to     salient aspects of
            support activities.          transition to support
                                         activities.

Activity 1  The project team performs    There are no transition to   Weakness
            its activities in            support plans.
            accordance with its
            documented transition to
            support plans.

Activity 2  Responsibility for the       No products have been        Not rated
            software products is         transferred yet.
            transferred only after the
            software support
            organization demonstrates
            its capability to modify
            and support the software
            products.

Activity 3  The project team oversees    Transition has not yet       Not rated
            the configuration control    occurred.
            of the software products
            throughout the transition.

Measuremen  Measurements are made and    There are no plans to take   Weakness
t 1         used to                      measurements to determine
            determine the status of the  the status of the
            transition to support        transition to support
            activities and resultant     activities and resultant
            products.                    products.

Verificati  Transition to support        There are no plans for       Weakness
on 1        activities are reviewed by   acquisition and software
            acquisition and software     support organizations'
            support organizations'       managements to review
            managements periodically.    transition to support
                                         activities periodically.

Verificati  Transition to support        No evidence was provided     Weakness
on 2        activities are reviewed by   that transition to support
            the project manager on both  activities are reviewed by
            a periodic and event-        the project manager.
            driven basis.
--------------------------------------------------------------------------------


                                    Table IV.8
                     
                        Key Process Area: Acquisition Risk
                                    Management

            Key practice                 Finding                      Rating
----------  ---------------------------  ---------------------------  ----------
Commitment  The acquisition              There is no policy for the   Weakness
1           organization has a written   management of software
            policy for the management    acquisition risk.
            of software acquisition
            risk.

Commitment  Responsibility for software  Responsibility for software  Weakness
2           acquisition risk management  acquisition risk management
            activities is designated.    activities is not
                                         designated.

Ability 1   A group that is responsible  No group responsible for     Weakness
            for coordinating software    coordinating software
            acquisition risk management  acquisition risk management
            activities exists.           activities exists.

Ability 2   Adequate resources are       Adequate resources are not   Weakness
            provided for software        provided for software
            acquisition risk management  acquisition risk management
            activities.                  activities.

Ability 3   Individuals performing       No individual or group is    Weakness
            software acquisition risk    performing software
            management activities have   acquisition risk
            experience or receive        management.
            required training.

Activity 1  Software acquisition risk    Software acquisition risk    Weakness
            management activities are    management activities are
            integrated into software     not integrated into
            acquisition planning.        software acquisition
                                         planning.

Activity 2  The software acquisition     There is no software         Weakness
            risk management plan is      acquisition risk management
            developed in accordance      plan.
            with the project's defined
            software acquisition
            process.

Activity 3  The project team performs    No documented software       Weakness
            its software acquisition     acquisition risk management
            risk management activities   plans exists. No risk
            in accordance with its       management activities are
            documented plans.            being performed.

Activity 4  Risk management is           Risk management is not       Weakness
            conducted as an integral     conducted as an integral
            part of the solicitation,    part of the solicitation,
            project performance          project performance
            management, and contract     management, and contract
            performance management       performance management
            processes.                   processes.

Activity 5  Software acquisition risk    There is no record of        Weakness
            handling actions are         tracking and controlling
            tracked and controlled       software acquisition risk
            until the risks are          handling actions.
            mitigated.

Measuremen  Measurements are made and    Measurements are not made    Weakness
t 1         used to                      and used to determine the
            determine the status of the  status of the acquisition
            acquisition risk management  risk management activities
            activities and resultant     and resultant products.
            products.

Verificati  Acquisition risk management  There are no acquisition     Weakness
on 1        activities are reviewed by   risk management activities
            acquisition organization     for management to review.
            management periodically.

Verificati  Acquisition risk management  There are no acquisition     Weakness
on 2        activities are reviewed by   risk management activities
            the project manager on both  for the project manager to
            a periodic and event-        review.
            driven basis.
--------------------------------------------------------------------------------

MAJOR CONTRIBUTORS TO THIS REPORT
=========================================================== Appendix V

ACCOUNTING AND INFORMATION
MANAGEMENT DIVISION, WASHINGTON,
D.C. 

Dr.  Rona Stillman, Chief Scientist for Computers and
 Telecommunications
Jack L.  Brock, Jr., Director, Governmentwide and Defense Information
 Systems
Mark T.  Bird, Assistant Director
Madhav S.  Panwar, Technical Assistant Director
Dr.  Nabajyoti Barkakati, Technical Assistant Director
Prithviraj Mukerji, Assistant Director
Carl M.  Urie, Assistant Director
Bernard R.  Anderson, Senior Information Systems Analyst
Suzanne M.  Burns, Senior Information Systems Analyst
Timothy D.  Hopkins, Senior Information Systems Analyst
Ona M.  Noble, Senior Information Systems Analyst
Karen A.  Richey, Senior Information Systems Analyst
Cristina T.  Chaplain, Communications Analyst

OFFICE OF THE CHIEF ECONOMIST,
WASHINGTON, D.C. 

Harold J.  Brumm, Economist

ATLANTA FIELD OFFICE

Teresa F.  Tucker, Senior Information Systems Analyst


*** End of document. ***