Customs Service Modernization: Serious Management and Technical
Weaknesses Must Be Corrected (Chapter Report, 02/26/99, GAO/AIMD-99-41).
Pursuant to a congressional request, GAO reviewed the Customs Service's
management of the Automated Commercial Environment (ACE), focusing on
whether Customs has adequately justified ACE cost-effectiveness.
GAO noted that: (1) Customs is not managing ACE effectively, and it does
not have a firm basis for concluding that ACE is a cost-effective
solution to modernizing its commercial environment; (2) GAO found
serious weaknesses relating to architectural definition, investment
management, and software development and acquisition that must be
corrected before further investment in ACE is justified; (3) Customs is
not building ACE within the context of a complete and enforced systems
architecture; (4) in May 1998, GAO reported that Customs' architecture
was incomplete because it was not based on a complete understanding of
its enterprisewide functional and information needs; (5) GAO also
reported that Customs had not yet instituted effective procedures for
ensuring compliance with the architecture once it is completed; (6)
until its architecture is completed and effectively enforced, Customs
will not have adequate assurance that information systems like ACE will
optimally support its needs across all business areas; (7) further,
Customs lacks a reliable estimate of what ACE will cost to build,
deploy, and maintain, and Customs has neither adequately justified, nor
is it effectively monitoring, ACE's cost-effectiveness; (8)
specifically, Customs did not use rigorous cost estimating techniques in
preparing its cost estimate, and did not disclose the inherent
imprecision of the estimate; (9) additionally, Customs omitted costs and
inflated benefits in preparing its cost-benefit analysis; (10) moreover,
Customs is not using effective incremental investment management
practices; (11) while Customs plans to develop/acquire ACE in 21
increments, these increments are not individually cost-benefit
justified, and Customs is not determining what benefits each increment,
once operational, actually provides; (12) as a result, Customs will not
know if ACE's expected return-on-investment is actually being realized
until it has already spent hundreds of millions of dollars
developing/acquiring the entire system; and (13) GAO found that Customs
has neither the capability to effectively develop nor acquire ACE and
that its processes for doing both, according to widely accepted and
proven software capability maturity models, are ad hoc, immature, and
ineffective.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: AIMD-99-41
TITLE: Customs Service Modernization: Serious Management and
Technical Weaknesses Must Be Corrected
DATE: 02/26/99
SUBJECT: ADP procurement
Customs administration
Information resources management
Strategic information systems planning
Cost effectiveness analysis
Systems conversions
Systems design
Life cycle costs
Cost analysis
Computer software verification and validation
IDENTIFIER: Customs Service Automated Commercial Environment System
Customs Service National Customs Automation Program
Software Capability Maturity Model
Treasury International Trade Data System
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to Congressional Requesters
February 1999
CUSTOMS SERVICE MODERNIZATION -
SERIOUS MANAGEMENT AND TECHNICAL
WEAKNESSES MUST BE CORRECTED
GAO/AIMD-99-41
Customs Service Modernization
(511114)
Abbreviations
=============================================================== ABBREV
ACE - Automated Commercial Environment
ACS - Automated Commercial System
CIO - chief information officer
CMM - Capability Maturity Model\SM
IRB - Investment Review Board
IT - information technology
ITDS - International Trade Data System
KPA - key process area
NAFTA - North American Free Trade Agreement
NCAP - National Customs Automation Program
OMB - Office of Management and Budget
SA-CMM - Software Acquisition Capability Maturity Model\SM
SCE - software capability evaluation
SEI - Software Engineering Institute
SW-CMM - Software Development Capability Maturity Model\SM
SW-CMM -
SA-CMM -
Letter
=============================================================== LETTER
B-280416
February 26, 1999
The Honorable Ben Nighthorse Campbell
Chairman, Subcommittee on Treasury,
General Government, and Civil Service
Committee on Appropriations
United States Senate
The Honorable Jim Kolbe
Chairman, Subcommittee on Treasury,
Postal Service, and General Government
Committee on Appropriations
House of Representatives
This report responds to your requests that we review the U. S.
Customs Service's management of the Automated Commercial Environment
(ACE), including whether Customs has adequately justified ACE
cost-effectiveness. Customs plans to spend over $1 billion on ACE,
which is planned to support modernized import processing. We found
that Customs' is not managing ACE effectively and it does not have a
firm basis for concluding that ACE is cost-effective. Accordingly,
we are making recommendations to the Commissioner of Customs for
strengthening the management and technical weaknesses we identified.
We are sending copies of this report to the Secretary of the
Treasury; Commissioner of Customs; Director of the Office of
Management and Budget; and Ranking Minority Members of the
Subcommittee on Treasury, General Government, and Civil Service of
the Senate Committee on Appropriations and the Subcommittee on
Treasury, Postal Service, and General Government of the House
Committee on Appropriations. We are also sending copies of this
report to the Chairmen and Ranking Minority Members of the Senate
Committee on Finance and the House Committee on Ways and Means and to
other congressional committees. We will also make copies available
to other interested parties upon request. If you have questions or
wish to discuss the issues in this report, please contact me at (202)
512-6240. Major contributors to this report are listed in appendix
V.
Randolph C. Hite
Associate Director, Governmentwide
and Defense Information Systems
EXECUTIVE SUMMARY
============================================================ Chapter 0
PURPOSE
---------------------------------------------------------- Chapter 0:1
The U.S. Customs Service plans to spend well over $1 billion to
modernize its systems environment for certain core missions:
facilitating international trade, enforcing laws governing the flow
of goods across U.S. borders, and assessing and collecting about $22
billion annually on imported merchandise. The Clinger-Cohen Act of
1996, and related system and software engineering best practices,
provide federal agencies with a framework for effectively managing
such modernization efforts.
The Customs modernization effort, known as the Automated Commercial
Environment or ACE, is of longstanding concern to the House
Appropriations Committee, Subcommittee on Treasury, Postal Service,
and General Government, and the Senate Appropriations Committee,
Subcommittee on Treasury, General Government, and Civil Service.
Accordingly, the Subcommittee Chairmen asked GAO to determine whether
Customs is effectively managing ACE, including whether Customs has
adequately justified ACE cost-effectiveness.
BACKGROUND
---------------------------------------------------------- Chapter 0:2
Title VI of the North American Free Trade Agreement Implementation
Act, Public Law 103-182, enabled Customs to speed the processing of
imports and improve compliance with trade laws. Customs refers to
this legislation as the "Customs Modernization and Informed
Compliance Act" or "Mod" Act. The primary purpose of the act is to
streamline and automate Customs' commercial operations. According to
Customs, modernized commercial operations will permit it to more
efficiently handle its burgeoning import workloads and expedite the
movement of merchandise at more than 300 ports of entry. For 1995
through 2001, Customs estimates that the annual dollar volume of
import trade will increase from
$761 billion to $1.1 trillion, with the number of commercial entries
processed annually increasing from 13.1 million to 20.6 million.
ACE is Customs' system solution to a modernized commercial
environment. In November 1997, Customs estimated that it would cost
$1.05 billion to develop, operate, and maintain ACE over the 15 year
period between fiscal year 1994 and 2008. Customs plans to develop
and deploy ACE in multiple increments. The first four increments are
known collectively as the National Customs Automation Program (NCAP).
The first increment, NCAP 0.1, was deployed for field operation and
evaluation in May 1998. As of the end of fiscal year 1998, Customs
reported that it had spent $62.1 million on ACE.
RESULTS IN BRIEF
---------------------------------------------------------- Chapter 0:3
Customs is not managing ACE effectively, and it does not have a firm
basis for concluding that ACE is a cost-effective solution to
modernizing its commercial environment. GAO found serious weaknesses
relating to architectural definition, investment management, and
software development and acquisition that must be corrected before
further investment in ACE is justified.
First, Customs is not building ACE within the context of a complete
and enforced systems architecture or "construction plan" that
precludes inconsistent system design and development decisions. In
May 1998, GAO reported that Customs' architecture was incomplete
because it was not based on a complete understanding of its
enterprisewide functional and information needs. GAO also reported
that Customs had not yet instituted effective procedures for ensuring
compliance with the architecture once it is completed. Customs is
attempting to complete its architecture, but it has not yet done so.
Until its architecture is completed and effectively enforced, Customs
will not have adequate assurance that information systems like ACE
will optimally support its needs across all business areas.
Further, Customs lacks a reliable estimate of what ACE will cost to
build, deploy, and maintain; and Customs has neither adequately
justified, nor is it effectively monitoring, ACE's
cost-effectiveness. Specifically, Customs did not use rigorous cost
estimating techniques in preparing its cost estimate, and did not
disclose the inherent imprecision of the estimate. Additionally,
Customs omitted costs and inflated benefits in preparing its
cost-benefit analysis. Moreover, Customs is not using effective
incremental investment management practices. While Customs plans to
develop/acquire ACE in 21 increments, these increments are not
individually cost-benefit justified, and Customs is not determining
what benefits each increment, once operational, actually provides.
As a result, Customs will not know if ACE's expected
return-on-investment is actually being realized until it has already
spent hundreds of millions of dollars developing/acquiring the entire
system. This combination of unreliable investment information and
analysis, and a "grand design" approach to justifying and managing
system investments,\1
has failed consistently on other agency modernization efforts over
the past two decades, has been abandoned by successful organizations,
and was a major reason for the information technology investment
management reforms in the Clinger-Cohen Act of 1996.
These investment risks and uncertainties are compounded by the fact
that Customs is not employing effective software engineering
practices on ACE. Specifically, Customs developed the first ACE
increment in-house using its own software developers, but because of
cost and schedule delays, it decided to acquire the second ACE
increment from a software development contractor. GAO found that
Customs has neither the capability to effectively develop nor acquire
ACE and that its processes for doing both, according to widely
accepted and proven software capability maturity models, are ad hoc,
immature, and ineffective.
--------------------
\1 The "grand design" approach involves investing in a large,
long-term, expensive project based on cost and benefit estimates
prepared at the outset and attempting to deliver the entire project
years later as a single increment.
PRINCIPAL FINDINGS
---------------------------------------------------------- Chapter 0:4
CUSTOMS IS DEVELOPING ACE
WITHOUT A COMPLETE
ENTERPRISE SYSTEMS
ARCHITECTURE
-------------------------------------------------------- Chapter 0:4.1
The Clinger-Cohen Act recognizes the importance of architectures by
requiring agency Chief Information Officers (CIO) to develop,
maintain, and facilitate an integrated systems architecture. Customs
does not have a complete target systems architecture. In May 1998,
GAO reported that Customs' target systems architecture was not
effective because it was neither complete nor enforced, and GAO made
several recommendations for needed improvements.\2 For example, GAO
reported that the architecture did not (1) fully describe Customs'
business functions, (2) define the information needs and flows among
these functions, and (3) establish the technical standards, products,
and services that will be used to build systems that support these
defined business functions and needs. GAO also reported that Customs
did not require that its systems be architecturally compliant or that
architectural deviations be justified and documented.
Customs officials acknowledged the limitations in the agency's
architecture and its enforcement. They agreed to define functions,
information needs, and flows across Customs' six business areas and,
in light of this definition, reevaluate the technical characteristics
the architecture specified for its system environment. Customs
originally planned to have completed its target architecture by
September 1998, but that date has slipped to May 1999. Since
receiving a draft of this report, Customs has changed its investment
management procedures with the intent of ensuring that its
architecture, once completed, can be enforced effectively. Until its
architecture is complete, however, Customs risks spending millions of
dollars to develop, acquire, and maintain information systems,
including ACE, that do not effectively and efficiently support the
agency's mission needs.
--------------------
\2 Customs Service Modernization: Architecture Must Be Complete and
Enforced to Effectively Build and Maintain Systems (GAO/AIMD-98-70,
May 1998).
ACE INVESTMENT MANAGEMENT IS
NOT EFFECTIVE
-------------------------------------------------------- Chapter 0:4.2
The Clinger-Cohen Act and Office of Management and Budget (OMB)
guidance provide an effective framework for information technology
(IT) investment management. Together, they establish requirements
for (1) identifying all promising alternative system solutions, (2)
developing reliable estimates of project life-cycle costs and
benefits, and investing in the most cost-beneficial alternative, and
(3) to the maximum extent practical, structuring major projects into
a series of increments to ensure that each increment constitutes a
wise investment.
Customs did not consider potential alternatives, such as using the
International Trade Data System (ITDS), to perform certain critical
functions, before deciding to invest in ACE.\3 ITDS was initiated in
1995 as a project to implement the National Performance Review
recommendation to develop a coordinated, governmentwide system for
the collection, use, and dissemination of trade data. At that time,
a multiagency board of directors was established, headed by the
Department of the Treasury with Customs and other major trade-related
agencies represented. The Department of the Treasury is responsible
for designing and developing ITDS, which is expected to reduce the
burden that federal agencies place on international organizations by
requiring that they respond to duplicative data requests. Treasury
intends for the system to serve as the single point for collecting,
editing, and validating trade data as well as collecting and
accounting for trade revenue--functions that are also planned for
ACE.
Further, for the alternative it selected, i.e., ACE, Customs did not
develop a reliable life-cycle cost estimate. Carnegie Mellon
University's Software Engineering Institute (SEI) has developed
criteria by which the reliability of project cost estimates can be
assessed. Using SEI's criteria, GAO found that the processes used by
Customs to develop its $1.05 billion ACE life-cycle cost estimate
were neither thorough nor disciplined, and as a result, Customs' ACE
cost estimate is not reliable and does not provide a sound basis for
investment decision-making. For example, Customs did not use a cost
model, did not account for changes in its approach to building
different ACE increments, did not account for changes to ACE software
and hardware architecture, and did not have the requisite historical
project cost data upon which to compare its ACE estimate.
Additionally, the $1.05 billion cost estimate omits various relevant
cost elements, such as requirements definition, data warehouse
development, system documentation development, system integration,
training, and hardware/software technology refreshment. Customs then
exacerbated these problems by representing its ACE cost estimate as a
precise, point estimate, rather than explicitly describing the
estimate's inherent uncertainty to ensure that it would be used
appropriately.
Moreover, Customs' projections of ACE benefits are overstated by at
least $52.8 million. The analysis includes $203.5 million in savings
attributable to 10 years of avoided maintenance and support costs on
the Automated Commercial System (ACS)--the system that ACE is to
replace. However, Customs will not avoid maintenance and support
costs for 10 years. Because Customs plans to run both systems in
parallel for 4 years, it will expend $52.8 million on continued
maintenance and support of ACS during this period.
Lastly, although Customs has decided to implement ACE as a series of
21 increments, it is not making its investment decisions
incrementally as required by the Clinger-Cohen Act and OMB.
Specifically, Customs is not justifying investing in each increment
on the basis of measurable benefits, and once it has deployed an
increment at a pilot site for evaluation, it is not validating the
benefits that the increment actually provides. Instead, Customs has
estimated costs and benefits for the entire system (i.e., all 21
increments). Such estimates of many system increments to be
delivered over many years are impossible to make accurately because
later increments are not well understood or defined, and are subject
to change in light of experiences on nearer term increments and
changing business needs. By using an inaccurate, aggregated estimate
that is not refined as increments are developed, Customs is
committing enormous resources with no assurance that it will achieve
a reasonable return on its investment. This "grand design" or "big
bang" approach to managing large system modernization projects has
repeatedly proven to be ineffective, resulting in huge sums invested
in systems that do not provide expected benefits.
--------------------
\3 The U.S. Department of the Treasury has developed an ITDS project
plan, system specification, cost-benefit analysis, and related
documentation. It plans to begin developing ITDS in 1999 and to
fully implement it by 2003. It estimates that ITDS will cost $256
million to develop, deploy, and maintain through 2005.
CUSTOMS LACKS THE CAPABILITY
TO DEVELOP OR ACQUIRE ACE
SOFTWARE EFFECTIVELY
-------------------------------------------------------- Chapter 0:4.3
The Clinger-Cohen Act requires the establishment of effective IT
management processes, including processes for managing software
development and acquisition. SEI has developed criteria for
determining organizations' software development and acquisition
effectiveness or maturity.\4
Using SEI criteria for process maturity at the "repeatable" level,
which is the second level on SEI's five-level scale and means that an
organization has the software development/acquisition rigor and
discipline to repeat project successes, GAO evaluated ACE software
processes. In February 1999, GAO reported that Customs lacked the
capability to develop software effectively on several projects,
including ACE.\5 For example, GAO reported that NCAP 0.1 lacked an
effective software configuration management process, which is
important for establishing and maintaining the integrity of the
software products during development, and NCAP 0.1 did not have any
type of software quality assurance program, which greatly increases
the risk of ACE software not meeting process and product standards.
GAO also reported that Customs lacked a software process improvement
program to effectively address these and other software process
weaknesses. Accordingly, GAO made several recommendations for needed
improvements. Customs agreed with GAO's findings and stated that it
initiated steps to implement GAO's recommendations, including
assigning responsibility for software process improvement.
Additionally, GAO found that Customs lacks the capability to acquire
ACE software effectively. For example, Customs did not have an
effective software acquisition planning process, and therefore could
not effectively establish reasonable plans for performing software
engineering and for managing the software project. Also, Customs did
not have an effective evaluation process, which means that it lacked
the means for assuring that contractor-developed software satisfied
defined requirements.
Because of these and other serious software process weaknesses,
Customs' ability to either develop or acquire ACE software is
immature, and therefore project success is unlikely.
--------------------
\4 Software Development Capability Maturity Model\SM (SW-CMM
) and Software Acquisition Capability Maturity Model\SM
(SA-CMM ). Capability Maturity Model\SM is a service mark
of Carnegie Mellon University, and CMM is registered in
the U.S. Patent and Trademark Office.
\5 Customs Service Modernization: Immature Software Development
Processes Increase Customs System Development Risks (GAO/AIMD-99-35,
February 11, 1999).
RECOMMENDATIONS
---------------------------------------------------------- Chapter 0:5
In addition to previous recommendations to improve Customs'
management of information technology,L\6 GAO recommends that Customs
correct the management and technical weaknesses discussed in this
report before building ACE. To accomplish this, GAO recommends that
the Commissioner of Customs, with the support of Customs' CIO, ensure
that Customs
(1) rigorously analyze alternative approaches to building ACE,
including ITDS as an alternative to developing ACE entirely within
Customs;
(2) make investment decisions incrementally, i.e., for each
increment:
use disciplined processes to prepare a rigorous life-cycle cost
estimate, including an explicit discussion of its inherent
uncertainty;
prepare realistic and supportable benefit expectations;
require a favorable return-on-investment and compliance with Customs'
architecture before making any investment; and
validate actual costs and benefits once an increment is piloted,
compare these with estimates, use the results in making further
decisions on subsequent increments, and report the results to
Customs' House and Senate appropriations and authorizing committees;
and
(3) strengthen ACE software acquisition management by:
establishing an effective process improvement program and correcting
the weaknesses in ACE software acquisition processes identified in
this report, thereby bringing ACE processes to at least SEI level 2
and
requiring at least SEI level 2 processes of all ACE software
contractors.
--------------------
\6 GAO/AIMD-99-35, February 11, 1999 and GAO/AIMD-98-70, May 5, 1998.
AGENCY COMMENTS
---------------------------------------------------------- Chapter 0:6
In its written comments on a draft of this report, Customs agreed
with GAO's conclusions and recommendations and stated that it is
committed to addressing the problems discussed in the report. To
this end, Customs cited a number of actions that it has underway and
planned over the next few years to improve management of information
technology in general, and ACE in specific. To fully correct the
management and technical weaknesses discussed in this report, Customs
must follow through and effectively implement actions to address all
of GAO's recommendations. Customs' comments are discussed in greater
detail in chapter 5 and the full text of its comments are reproduced
in appendix I of this report.
INTRODUCTION
============================================================ Chapter 1
The mission of the Customs Service is to ensure that all goods and
persons entering and exiting the United States do so in compliance
with all U.S. laws and regulations. It does this by (1) enforcing
the laws governing the flow of goods and persons across U.S. borders
and (2) assessing and collecting duties, taxes, and fees on imported
merchandise. During fiscal year 1997, Customs collected $22.1
billion in revenue\1 at more than 300 ports of entry, and it
processed nearly 450 million passengers entering the United States.
--------------------
\1 Includes tariff duty, user fees, Internal Revenue Service excise
taxes, and other assessments.
CUSTOMS' CORE BUSINESS
OPERATIONS: A BRIEF
DESCRIPTION
---------------------------------------------------------- Chapter 1:1
To accomplish its mission, Customs is organized into the following
six business areas: trade compliance, outbound, passenger, finance,
human resources, and investigations. Each business area is described
below.
-- Trade compliance includes enforcement of laws and regulations
associated with the importation of goods into the United States.
To do so, Customs (1) works with the trade community to promote
understanding of applicable laws and regulations, (2)
selectively examines cargo to ensure that only eligible goods
enter the country, (3) reviews documentation associated with
cargo entries to ensure that they are properly valued and
classified, (4) collects billions of dollars annually in duties,
taxes, and fees associated with imported cargo, (5) assesses
fines and penalties for noncompliance with trade laws and
regulation, (6) seizes and accounts for illegal cargo, and (7)
manages the collection of these moneys to ensure that all
trade-related debts due to Customs are paid and properly
accounted for.
-- Outbound includes Customs enforcement of laws and regulations
associated with the movement of merchandise and conveyances from
the United States. To do so, Customs (1) selectively inspects
cargo at U.S. ports to guard against the exportation of illegal
goods, such as protected technologies, stolen vehicles, and
illegal currency, (2) collects, disseminates, and uses
intelligence to identify high-risk cargo and passengers, (3)
assesses and collects fines and penalties associated with the
exportation of illegal cargo, and (4) physically examines
baggage and cargo at airport facilities for explosive and
nuclear materials. In addition, the outbound business includes
collecting and disseminating trade data within the federal
government. Accurate trade data are crucial to establishing
accurate trade statistics on which to base trade policy
decisions and negotiate trade agreements with other countries.
By the year 2000, Customs estimates that exports will be valued
at $1.2 trillion, as compared to a reported $696 billion in
1994.
-- Passenger includes processing all passengers and crew of
arriving and departing air, sea, and land conveyances and
pedestrians. In fiscal year 1997, Customs reported that it
processed nearly 450 million travelers, and by the year 2000,
expects almost 500 million passengers to arrive in the United
States annually. Many of Customs' passenger activities are
coordinated with other federal agencies, such as the Immigration
and Naturalization Service and the Department of Agriculture's
Animal and Plant Health Inspection Service. Activities include
targeting high-risk passengers, which requires timely and
accurate information, and physically inspecting selected
passengers, baggage, and vehicles to determine compliance with
laws and regulations.
-- Finance includes asset and revenue management activities. Asset
management consists of activities to (1) formulate Customs'
budget, (2) properly allocate and distribute funds, and (3)
acquire, manage, and account for personnel, goods, and services.
Revenue management encompasses all Customs activities to
identify and establish amounts owed Customs, collect these
amounts, and accurately report the status of revenue from all
sources. Sources of revenue include duties, fees, taxes, other
user fees, and forfeited currency and property. The revenue
management activities interrelate closely with the revenue
collection activities in the trade compliance, outbound, and
passenger business areas.
-- Human resources is responsible for filling positions, providing
employee benefits and services, training employees, facilitating
workforce effectiveness, and processing personnel actions for
Customs' 18,000 employees and managers.
-- Investigations includes activities to detect and eliminate
narcotics and money laundering operations. Customs works with
other agencies and foreign governments to reduce drug-related
activity by interdicting (seizing and destroying) narcotics,
investigating organizations involved in drug smuggling, and
deterring smuggling efforts through various other methods.
Customs also develops and provides information to the trade and
carrier communities to assist them in their efforts to prevent
smuggling organizations from using cargo containers and
commercial conveyances to introduce narcotics into the United
States.
To carry out its responsibilities, Customs relies on information
systems and processes to assist its staff in (1) documenting,
inspecting, and accounting for the movement and disposition of
imported goods and (2) collecting and accounting for the related
revenues. Customs expects its reliance on information systems to
increase as a result of its burgeoning workload. For 1995 through
2001, Customs estimates that the annual volume of import trade
between the United States and other countries will increase from $761
billion to $1.1 trillion. This will result in Customs processing an
estimated increase of 7.5 million import entries--from 13.1 million
to 20.6 million annually--during the same period. Recent trade
agreements, such as the North American Free Trade Agreement (NAFTA),
have also increased the number and complexity of trade provisions
that Customs must enforce.
CUSTOMS IS LEGISLATIVELY
MANDATED TO MODERNIZE
OPERATIONS AND SYSTEMS
---------------------------------------------------------- Chapter 1:2
Both Customs and the Congress recognize that the ability to process
the growing volume of imports while improving compliance with trade
laws depends heavily on successfully modernizing Customs' trade
compliance process and its supporting automated systems. To speed
the processing of imports and improve compliance with trade laws, the
Congress enacted Title VI of the North American Free Trade Agreement
Implementation Act in December 1993.\2 Customs refers to this
legislation as the "Customs Modernization and Informed Compliance
Act" or "Mod" Act.
The primary purpose of the act is to streamline and automate Customs'
commercial operations by eliminating certain legislatively mandated
paper requirements and requiring Customs to establish the National
Customs Automation Program (NCAP). The legislation also specified
certain functions that NCAP must provide, including giving members of
the trade community the capability to electronically file import
entries at remote locations as well as enabling Customs to
electronically process "drawback" claims, which are refunds of duties
and taxes paid on imported goods that are subsequently exported or
destroyed. According to Customs, the act provides the legal
framework to automate commercial operations and thereby streamline
the processing, and expedite the movement of merchandise at more than
300 ports of entry.
--------------------
\2 Public Law 103-182, 19 U.S.C. 1411 et seq.
ACE PURPOSE, PLANS, AND STATUS
---------------------------------------------------------- Chapter 1:3
ACE is intended to support the trade compliance business process,
implement Mod Act requirements, and replace the existing import
system, the Automated Commercial System (ACS). ACS is nearly 15
years old and Customs reports that it is becoming increasingly
difficult and expensive to operate, maintain, and enhance due to the
system's antiquated hardware and software and limited processing
capacity.
Currently, ACE's architecture is to include both (1) mainframe-based,
centralized processing to support high-volume, repetitive
transactions and (2) distributed, client-server processing and
desktop PC interfaces to support port office analysis and decision
support needs. The transaction-intensive applications are to run on
IBM mainframes at Customs' national data center and are to be written
in COBOL and C++. The distributed processing is to occur on UNIX
servers and the applications are to be written in C++ and Java. To
support this processing environment, Customs plans to design and
implement a single, integrated agencywide database. It also plans to
connect port offices to the national data center through existing
communications networks comprising the Treasury Communications
System, and to connect the trade community to the data center through
the Internet and a combination of dial-up lines and dedicated lines.
According to Customs, ACE will facilitate increased compliance of
individuals, businesses, and governments with the trade laws and
regulations of the United States and increased communication with the
trade community. It will also provide an integrated, account-based,
automated information system for collecting, disseminating, and
analyzing trade-related data and ensuring the proper collection and
allocation of revenues.
Customs began developing ACE in 1994 and plans to develop and deploy
ACE in 21 increments from 1998 through 2005. Each increment consists
of the software and hardware necessary to perform a discrete portion
of the total ACE functionality. The first four increments, known
collectively as the NCAP prototype, will process pre-identified
merchandise shipped into the U.S. via truck by three selected
importers.\3 The first two increments, NCAP 0.1 and NCAP 0.2, were
deployed for operational evaluation at three pilot border port
locations (Detroit, Michigan; Laredo, Texas; and Port Huron,
Michigan) during May 1998 and October 1998, respectively.
The succeeding 17 increments are intended to build upon the
foundational capabilities contained within the four NCAP prototype
increments, providing additional functionality including the
capability to (1) process merchandise imported via rail, air, sea,
and couriers, (2) process collections and refunds associated with
imported cargo, (3) process warehouse entry cargo, and (4) process
cargo subject to drawbacks. Until all of these deployments are
completed, Customs intends to operate and maintain ACS.
Customs' estimate of ACE costs has escalated since August 1997, when
it initially estimated that ACE's 10-year life-cycle cost would be
$895 million for the period 1998 through 2007. In November 1997,
Customs revised this 10-year cost estimate to $987.6 million and also
estimated that, over 15 years, from 1994 through 2008, it would spend
$1.05 billion on ACE. It then used the $1.05 billion estimate along
with estimated expected savings of $1.9 billion to justify ACE.
Customs is currently reevaluating this estimate.
Figure 1.1 compares planned and actual deployments of the NCAP
increments. As the figure illustrates, Customs is well over 2 years
behind its original schedule for NCAP.
Figure 1.1: Planned Versus
Actual NCAP Deployment
Schedules
(See figure in printed
edition.)
Note: NCAP 0.3 and NCAP 0.4 current dates are unknown pending budget
approval.
--------------------
\3 The importers participating in the NCAP prototype are General
Motors Corporation, Ford Motor Company, and Chrysler Corporation.
OBJECTIVES, SCOPE, AND
METHODOLOGY
---------------------------------------------------------- Chapter 1:4
The Chairman, Subcommittee on Treasury, Postal Service, and General
Government, House Committee on Appropriations, and the Chairman,
Subcommittee on Treasury, General Government, and Civil Service,
Senate Appropriations Committee, requested that we review ACE. Our
objective was to determine whether Customs is effectively managing
ACE, including whether Customs has adequately justified ACE
cost-effectiveness. In making these determinations, we relied
primarily on the following criteria: (1) the Clinger-Cohen Act of
1996\4 and other legislative reforms that require federal agencies to
develop and maintain integrated systems architectures and improve
their information technology investment management processes, (2)
Office of Management and Budget guidance related to the acquisition
and management of information resources, (3) GAO and Treasury systems
architecture guidance, and (4) related Software Engineering Institute
system engineering standards concerning cost estimating and software
development and acquisition maturity.
To address our objective, we reviewed pertinent documentation (e.g.,
ACE project plan, system/software development process
practices/standards, ACE technical architecture descriptions, and ACE
project status reports) and interviewed responsible Treasury
officials and Customs ACE project officials in order to identify (1)
Customs' management structure for developing and deploying ACE, (2)
the ACE system/software development methodology, (3) the planned ACE
hardware/software and communications architecture and configuration,
and (4) the current ACE system development status.
Additionally, we met with Customs ACE officials to determine the
status of the agency's implementation of recommendations we made in
May 1998 to complete its systems architecture. These recommendations
included (1) ensuring that the architecture fully described Customs'
business functions, (2) defining the information needs and flows
among these functions, (3) establishing the technical standards,
products, and services that will be used to build systems that
support these business functions and needs, and (4) enforcing
compliance with the architecture.
Further, we obtained and reviewed supporting documentation and
interviewed Treasury officials, Customs ACE officials, and Customs
Investment Review Board (IRB) officials to (1) identify the current
ACE project cost and life-cycle cost estimate baselines and determine
the current actual expenditures to date, (2) identify the current ACE
project schedule baseline and determine the actual progress to date
against scheduled milestones, (3) define what ACE is intended to do
and how it is expected to benefit Customs' mission, (4) determine the
extent to which ACE mission-related goals/benefits have been achieved
to date and how Customs is measuring the accrued benefits, and (5)
determine Customs' strategic approach to managing the
development/acquisition, integration, and deployment of ACE. The
documentation we analyzed included the ACE project plan and strategic
plan, functional/performance requirements, and NCAP technical
assessment documents, budget/financial data, cost-benefit policy and
guidance, cost-benefit analyses, various life-cycle and project cost
estimates, project status reports, testing problem reports, and
configuration management documentation.
Also, we reviewed project documentation to determine how the
life-cycle cost baseline was estimated and how this estimating
approach compared to criteria established by SEI in A Manager's
Checklist for Validating Software Cost and Schedule Estimates.\5 The
SEI criteria defines seven primary questions, each supported by more
detailed secondary questions, that can be used to determine whether
defined and disciplined processes were used to derive a given cost
estimate. We also interviewed ACE project officials.
In addition, we assessed whether Customs thoroughly analyzed
technical alternatives to ACE including the possibility of (1)
enhancing and continuing to use the legacy trade system, ACS, (2)
using different architectural designs for ACE, (3) following
different development/acquisition strategies, and/or (4) using the
Department of the Treasury's planned multiagency International Trade
Data System (ITDS), instead of ACE, for some functions, such as
collecting, editing, and validating trade data and collecting and
accounting for trade revenue.
Last, we used the Software Engineering Institute's (SEI's) Software
Development Capability Maturity Model\SM (SW-CMM),\6
Software Acquisition Capability Maturity Model\SM (SA-CMM),
and its Software Capability Evaluation Method, to evaluate Customs'
ability to manage its NCAP 0.1 software development project and NCAP
0.2 software acquisition effort, respectively. These models and
methods provide a logical and widely accepted framework for
baselining an organization's current process capabilities (i.e.,
strengths and weaknesses) and assessing whether an organization has
the necessary process discipline in place to repeat earlier successes
on similar projects. Organizations that do not satisfy the
requirements for the "repeatable" level are by default judged to be
at the "initial" level of maturity, meaning that their processes are
ad hoc, sometimes even chaotic, with few of the processes defined and
success dependent mainly on the heroic efforts of individuals.
In following the SEI method, GAO staff trained at SEI evaluated
Customs' ACE project software development/maintenance maturity in
five of the six key process areas (KPA) that are necessary to attain
a "repeatable" level of process maturity.\7 GAO did not evaluate
Customs in the sixth repeatable level KPA, software subcontract
management, because Customs did not use subcontractors on the ACE
project. Additionally, GAO staff trained at SEI evaluated Customs'
ACE project software acquisition maturity in the seven KPAs that are
necessary to attain a repeatable level of process capability, and one
KPA associated with the "defined" level of process
maturity--acquisition risk management.\8 The purpose of acquisition
risk management is to formally identify risks as early as possible
and adjust the acquisition to mitigate those risks. Many software
experts consider acquisition risk management to be an integral part
of the solicitation, project performance management, and contract
performance management processes.
Customs provided written comments on a draft of this report. These
comments are presented in chapter 5, and are reprinted in appendix I.
We performed our work at Customs and Department of the Treasury
headquarters offices in Washington, D.C., and at the Customs Data
Center facility in Newington, Virginia between February 1998 and
November 1998, in accordance with generally accepted government
auditing standards.
--------------------
\4 Although the Clinger-Cohen Act (Public Law 104-106) was passed
after Customs began developing ACE, its principles are based on
practices that are widely considered to be integral to successful IT
investments. For an analysis of the management practices of several
leading private and public sector organizations on which the
Clinger-Cohen Act is based see Executive Guide: Improving Mission
Performance Through Strategic Information Management and Technology,
(GAO/AIMD-94-115, May 1994). For an overview of the IT management
process envisioned by Clinger-Cohen see Assessing Risk and Returns:
A Guide for Evaluating Federal Agencies' IT Investment
Decision-making (GAO/AIMD-10.1.13, February 1997).
\5 CMU/SEI-95-SR-004, January 1995.
\6 Capability Maturity Model\SM is a service mark of Carnegie Mellon
University, and CMM is registered in the U.S. Patent and
Trademark Office.
\7 The five KPAs are requirements management, software project
planning, software project tracking and oversight, software quality
assurance, and software configuration management. According to the
SW-CMM, (1) requirements management is the process of establishing a
common understanding between the customer and the software developer
of the customer's requirements, (2) software project planning is the
process of establishing reasonable plans for engineering the software
and managing the software project, (3) software project tracking and
oversight is the process of providing adequate visibility into the
software project's progress to permit effective action when
deviations from plans occur, (4) software quality assurance is the
process of verifying for management that software process and product
procedures and standards are being followed, and (5) software
configuration management is the process of establishing and
maintaining the integrity of the software products throughout their
life-cycle.
\8 The seven KPA's relating to the repeatable level are software
acquisition planning, solicitation, requirements development and
management, project office management, contract tracking and
oversight, evaluation, and transition and support. The KPA relating
to the defined level is acquisition risk management. According to
the SA-CMM, (1) software acquisition planning is the process of
ensuring that reasonable planning for all elements of the software
acquisition occur, (2) solicitation is the process of ensuring that
award is made to the contractor most capable of satisfying the
specified requirements, (3) requirements development and management
is the process of establishing an unambiguous and agreed upon set of
software requirements, (4) project office management is the process
of effective and efficient management of project office activities,
(5) contract tracking and oversight is the process of ensuring that
contractor activities, products, and services satisfy contract
requirements, (6) evaluation is the process of determining that
acquired software products and services satisfy contract requirements
prior to acceptance, (7) transition and support is the process of
transferring acquired software products to the eventual support
organization, and (8) acquisition risk management is the process of
identifying software risks early and adjusting the acquisition
strategy to mitigate those risks.
ACE IS BEING DEVELOPED WITHOUT A
COMPLETE ENTERPRISE SYSTEMS
ARCHITECTURE
============================================================ Chapter 2
Architectures are critical for designing and developing large and
complex information systems. These comprehensive "construction
plans" systematically and completely describe an organization's
target business environment, both in logical (e.g., missions,
business functions, information flows) terms and technical (e.g.,
software, hardware, communications) terms. Without a target
architecture to guide and constrain IT investment, there is no
systematic way to preclude either inconsistent system design and
development decisions or the resulting suboptimal performance and
added cost associated with incompatible systems.
The Clinger-Cohen Act of 1996 requires agency CIOs to develop and
maintain an integrated system architecture. In addition, OMB issued
guidance in 1996 that, among other things, requires agency IT
investments to be consistent with federal, agency, and bureau
architectures.\1
Customs does not currently have a complete target architecture but
has recently established a process for enforcing an architecture once
one is completed. Customs has plans for completing its target
architecture by May 1999 and ensuring its enforcement. Thus far,
however, it has only defined its current architectural environment.
--------------------
\1 OMB Memorandum M-97-02, Funding Information Systems Investments,
October 25, 1996.
A FRAMEWORK FOR EFFECTIVE
SYSTEMS ARCHITECTURES
---------------------------------------------------------- Chapter 2:1
Reflecting the general consensus in the industry that large, complex
systems development and acquisition efforts should be guided by
explicit architectures, we issued a report in 1992 defining a
comprehensive framework for designing and developing systems
architectures.\2 This framework, which is consistent with guidance
that Treasury has provided to its bureaus,\3 divides systems
architectures into (1) a logical or business component, which is
developed first, and (2) a technical or systems component, which is
based on the first component.
The logical component ensures that the systems meet the business
needs of the organization. It provides a high-level description of
the organization's mission and target concept of operations; the
business functions being performed and the relationships among
functions; the information needed to perform the functions; the users
and locations of the functions and information; and the information
systems needed to support the agency's business needs.
The technical component ensures that systems are interoperable,
function together efficiently, and are cost-effective over their
life-cycles (including maintenance costs). The technical component
details specific standards and approaches that will be used to build
systems, including hardware, software, communications, data
management, security, and performance characteristics.
--------------------
\2 Strategic Information Planning: Framework for Designing and
Developing System Architectures (GAO/IMTEC-92-51, June 1992).
\3 This guidance--Treasury Information Systems Architecture Framework
(TISAF) version 1.0, January 3, 1997--is also included in OMB's
guidance on developing system architecture. See, OMB Memorandum
M-97-16, Information Technology Architectures, June 18, 1997.
CUSTOMS IS DEVELOPING, BUT HAS
NOT YET COMPLETED, ITS TARGET
SYSTEMS ARCHITECTURE
---------------------------------------------------------- Chapter 2:2
Customs does not currently have a complete enterprise systems
architecture. In May 1998, we reported that for five of its six
business areas (outbound, passenger, finance, human resources, and
investigations), Customs' architecture does not (1) describe all of
the agency's business functions, (2) completely identify the users
and locations of the functions, or (3) define the information needed
to perform the functions.\4 Further, while the architecture and
related documentation describe business functions and users and work
locations for the sixth business area (trade compliance), they do not
identify all of the information needs and flows for all of the trade
functions. Nonetheless, Customs had defined many characteristics of
its systems' hardware, software, communications, data management, and
security components. Because these characteristics were not based on
a complete understanding of its enterprisewide functional and
information needs, as specified in both best practice and Treasury
guidance, we concluded that Customs did not have adequate assurance
that its information systems will optimally support its needs across
all business areas.
We recommended that the Commissioner of Customs direct the Customs
CIO, in consultation with the Treasury CIO, to complete the
architecture. Specifically, we recommended that, at a minimum, the
architecture should (1) describe Customs' target business operations,
(2) fully define Customs' interrelated business functions to support
these target operations, (3) clearly describe information needs and
flows among these functions, (4) identify the systems that will
provide these functions and support these information needs and
flows, and (5) use this information to specify the technical
standards and related characteristics that these systems should
possess to ensure that they interoperate, function together
efficiently, and are cost-effective to maintain.
Customs and Treasury officials have acknowledged the limitations in
Customs' architecture, and agreed to define the agency's logical
architecture (e.g., functions, information needs and flows and users)
across its six business areas and, in light of this definition,
reevaluate the technical characteristics it has specified for its
technical architecture (i.e., systems environment). Thus far,
Customs has defined its current (i.e., its "as-is" or "in-state")
architectural environment, including its current business operations,
its current supporting business functions and their information needs
and flows, the systems currently supporting these functions, and the
technical characteristics (e.g., hardware, software, and
communications) of these supporting systems. However, Customs has
not defined an architecture for its target (i.e., future) business
and systems environment. Customs originally planned to have
completed its target architecture by September 1998, but that date is
now May 1999.
--------------------
\4 GAO/AIMD-98-70, May 5, 1998.
CUSTOMS HAS RECENTLY ANNOUNCED
PROCESS FOR ENSURING
ARCHITECTURAL COMPLIANCE
---------------------------------------------------------- Chapter 2:3
In May 1998, we reported that Customs' investment management
process\5
did not ensure that all new systems would conform to the
architecture. In particular, we reported that Customs' investment
review board (IRB) used four criteria in scoring competing investment
options and allocating funding among them. The four criteria were
(1) risk (e.g., technical, schedule, and cost);
(2) strategic alignment (e.g., cross-functional benefits, linkage to
Customs' business plan, and compliance with legislative mandates);
(3) mission effectiveness (e.g., contributions to service delivery);
and
(4) cost-benefit ratio (e.g., tangible and intangible benefits, and
costs).
Because compliance with the architecture was considered under the
risk criterion but was not required, the process did not preclude
funding projects that were inconsistent with the enterprise
architecture. Moreover, the process did not require that such
deviations from the architecture be rigorously justified.
To ensure that Customs effectively enforced its information systems
architecture, we recommended that Customs require that all new
projects comply with the architecture unless an exception could be
justified by careful, thorough, and documented analysis.\6 Customs
agreed and, in January 1999, changed its investment management
process to explicitly require that proposed IT investments comply
with its architecture, unless an exception is justified and a waiver
is granted by the technical review committee.
--------------------
\5 IT Investment Management Process, U.S. Customs Service, August
28, 1997.
\6 GAO/AIMD-98-70, May 5, 1998.
CONCLUSIONS
---------------------------------------------------------- Chapter 2:4
Until Customs completes and enforces its enterprise systems
architecture, it will not have adequate assurance that ACE and other
systems it plans to build and operationally deploy (1) will
effectively meet the agency's business needs and (2) are compatible,
efficient, and cost-effective to develop, integrate, and maintain.
CUSTOMS' ACE INVESTMENT MANAGEMENT
PRACTICES ARE INEFFECTIVE
============================================================ Chapter 3
The Clinger-Cohen Act and OMB guidance provide an effective framework
for IT investment management. Together, they set requirements for
(1) identifying all potential alternative system solutions, (2)
developing reliable estimates of project life-cycle costs and
benefits, and investing in the most cost-beneficial alternative, and
(3) to the maximum extent practical, structuring major projects into
a series of increments to ensure that each increment constitutes a
wise investment.
Customs has not effectively implemented any of these investment
management practices on ACE. Specifically, (1) Customs' investment
analysis did not address alternatives to its chosen ACE system
solution, (2) Customs' did not use rigorous cost estimating
techniques in preparing its ACE cost estimate and its cost-benefit
analysis for ACE omitted substantial costs and inflated benefits, and
(3) Customs is not justifying and validating the costs and benefits
for each ACE increment. As a result, Customs lacks a sound basis for
making ACE investment decisions.
ACE ALTERNATIVES WERE NOT
CONSIDERED
---------------------------------------------------------- Chapter 3:1
Before embarking on a major, costly systems initiative such as ACE,
agencies should thoroughly assess the full range of technical
options. In the case of ACE, Customs had several alternatives to
satisfying its mission needs as specified in the Mod Act. For
example, it could enhance ACS, use different architectural designs
for ACE, follow different development/ acquisition strategies for
ACE, and/or use Treasury's planned governmentwide trade system,
International Trade Data System (ITDS), to supplement some ACE
functions, such as collecting and disseminating trade data. ITDS was
initiated in 1995 as a project to implement the National Performance
Review recommendation to develop a coordinated, governmentwide system
for the collection, use, and dissemination of trade data. At that
time, a multiagency board of directors was established, headed by the
Department of the Treasury with Customs and other major trade-related
agencies represented. The Department of the Treasury is responsible
for designing and developing ITDS, which is expected to reduce the
burden that federal agencies place on the international trading
organizations by requiring that they respond to duplicative data
requests. Treasury intends for the system to serve as the single
point for collecting, editing, and validating trade data as well as
collecting and accounting for trade revenue.\1
By thoroughly considering these and other choices, Customs would have
ensured that the most cost-effective and beneficial alternative was
chosen before deciding to invest $1.05 billion in ACE. In fact, OMB
requires that agencies consider alternative system solutions to meet
mission needs, including different system architectures, upgrading
existing systems, or contracting for development and integration of
major systems.
Customs did not identify and evaluate a full range of alternatives to
ACE. In fact, Customs considered only (1) enhancing and operating
ACS to provide the same functionality of ACE, (2) operating ACS
without any enhancement, (3) operating ACS with limited enhancements,
and (4) developing and deploying part of ACE's planned functionality.
Customs discarded the last three because none provided for meeting
all of the Mod Act requirements. With respect to the first, Customs
compared ACS to ACE and decided that ACE was the more cost-effective
alternative.
Customs did not evaluate other alternatives to its ACE solution,
including (1) adopting a different ACE architectural design (e.g.,
using a combination mainframe and client-server configuration instead
of a mainframe only system), (2) contracting out for ACE development
rather than developing ACE in-house, or (3) using ITDS to satisfy
part of its functional needs. As a result, Customs committed to and
began investing in ACE development without knowing whether it had
chosen the most cost-effective alternative.
Since making its decision to develop ACE, Customs has changed ACE's
architectural design, and it has decided to acquire the second ACE
increment rather than develop it in-house. However, these decisions
are not supported by any verifiable alternatives analysis, meaning
that Customs still lacks adequate assurance that it is following the
most cost-effective approach to satisfying its mission needs.
--------------------
\1 Treasury has developed an ITDS project plan, system specification,
cost-benefit analysis, and related documentation. It plans to begin
developing ITDS in 1999 and to fully implement it by 2003. It
estimates that ITDS will cost $256 million to develop, deploy, and
maintain through 2005.
CUSTOMS' $1.05 BILLION ACE COST
ESTIMATE IS NOT RELIABLE
---------------------------------------------------------- Chapter 3:2
Reliable estimates of IT projects' expected costs are essential to
determining a project's cost-effectiveness. Without such
information, the likelihood of poor investment decisions is
increased, not only when a project is initiated but also throughout
its life-cycle.
To assist management in assessing the credibility of a given
project's cost estimate, SEI developed the following seven questions
for decisionmakers to use to assess the reliability of a project's
cost estimate and detailed criteria to assist in evaluating how well
a project satisfies each question.\2
(1) Are the objectives of the estimate clear and correct?
(2) Has the task been appropriately sized?
(3) Is the estimated cost consistent with demonstrated
accomplishments on other projects?
(4) Have the factors that affect the estimate been identified and
explained?
(5) Have steps been taken to ensure the integrity of the estimating
process?
(6) Is the organization's historical evidence capable of supporting a
reliable estimate?
(7) Has the situation remained unchanged since the estimate was
prepared?
We analyzed the approach that Customs followed in deriving its $1.05
billion ACE life-cycle cost estimate using SEI's criteria. Among
these criteria were several very significant and closely intertwined
requirements that are at the core of effective cost estimating.
Specifically, embedded in several of the aforementioned seven
questions are requirements for using (1) formal cost models, (2)
structured and documented processes for determining the software size
and reuse inputs to the models, and (3) relevant, measured, and
normalized historical cost data (estimated and actual) to calibrate
the models.\3
We found that Customs did not satisfy any of these requirements. In
particular, Customs did not use a cost model. Instead, it used an
unsophisticated spreadsheet to extrapolate the cost of each ACE
increment. Further, Customs' approach to determining software size
and reuse was not documented and was not well supported or
convincing. Customs estimated the size of each ACE software
increment (most of which were still undefined) by extrapolating from
the estimated size of the first increment based on individuals'
undocumented best judgments about increment functionality and
complexity. Last, Customs did not have any historical project cost
data at the time it developed the $1.05 billion estimate, and it did
not account for relevant, measured, and normalized differences among
the increments. For example, it did not account for (1) the change
in ACE's architecture from a mainframe system written in COBOL and
C++ to a combined mainframe and Internet-based system written in C++
and Java and (2) the change in ACE from an in-house software
development project (NCAP 0.1) to a software acquisition (NCAP 0.2).
Clearly, such fundamental changes can have a dramatic effect on
system costs, and should have been addressed explicitly in Customs'
cost estimates.
Table 3.1 summarizes the complete results of our assessment of
Customs' cost estimating process. For each of SEI's questions and
supporting criteria, Customs was rated as demonstrating a strength,
i.e., effectively implementing the criterion; a weakness, i.e.,
ineffectively implementing the criterion; or, where evidence was
inconclusive and could not support characterization as either a
strength or a weakness, an observation was noted. (See appendix II
for further detail on SEI's criteria and our findings).
Table 3.1
Summary of ACE Project Cost Estimate's
Satisfaction of SEI's Checklist Criteria
Number of Number of
Number of weaknesse observati
SEI checklist questions strengths s ons
------------------------------------- --------- --------- ---------
Are the objectives of the estimate 4 0 0
clear and correct?
Has the task been appropriately 0 7 1
sized?
Is the estimated cost consistent with 1 10 0
demonstrated accomplishments on
other projects?
Have the factors that affect the 3 2 0
estimate been identified and
explained?
Have steps been taken to ensure the 3 1 3
integrity of the estimating process?
Is the organization's historical 2 8 0
evidence capable of supporting a
reliable estimate?
Has the situation remained unchanged 0 3 0
since the estimate was prepared?
======================================================================
Totals 13 31 4
----------------------------------------------------------------------
--------------------
\2 A Manager's Checklist for Validating Software Cost and Schedule
Estimates (CMU/SEI-95-SR-004, January 1995).
\3 Examples of such data are the productivity value of a
"staff-month," that is, the time and cost to produce a specified unit
of code, such as a line of code.
CUSTOMS' COST ESTIMATE
IMPLIES AN UNJUSTIFIED LEVEL
OF PRECISION
-------------------------------------------------------- Chapter 3:2.1
Software and systems development experts agree that early project
estimates are by definition imprecise, and that this inherent
imprecision decreases during the project's life-cycle as more
information becomes known about the system. These experts emphasize
that, to be useful, each cost estimate should include an indication
of its degree of uncertainty, possibly as an estimated range or
qualified by some factor of confidence. For example, a cost estimate
of $1 million could be presented as a range from $750,000 to $1.25
million or as $1 million with a confidence level of 90 percent,
indicating that there is a 10 percent chance that costs will exceed
this estimate.
Customs did not reveal the degree of uncertainty of its cost estimate
for ACE to managers involved in investment decisions. For example,
Customs did not disclose that it made the estimate before fully
defining ACE functionality. Instead, Customs presented its $1.05
billion ACE life-cycle cost estimate as an unqualified point
estimate. This suggests an element of precision that cannot exist
for such an undefined system and obscures the investment risk
remaining in this project.
CUSTOMS' ANALYSIS OF ACE COSTS
AND BENEFITS IS ALSO UNRELIABLE
---------------------------------------------------------- Chapter 3:3
ACE cost estimates are understated, benefit estimates are overstated,
and both are unreliable. Customs' August 1997 cost-benefit analysis
estimated that ACE would produce cumulative savings of $1.9 billion
over a 10-year life-cycle beginning in fiscal year 1998 and ending in
fiscal year 2007. However, this analysis was unreliable because
certain benefits and costs were unsupported, other benefits were
overstated, and relevant costs were omitted. For example:
-- The analysis identified $650 million (35 percent of the total
ACE estimated savings) in savings to the trade community from
more accurate and timely data, improved customer compliance,
reduced support staff, reduced transportation costs, elimination
of duplicate recordkeeping systems, and lower processing costs
associated with periodic payments. However, Customs could not
produce any verifiable data or analysis to support this claim.
According to Customs officials, this benefit amount was
projected on the basis of undocumented information supplied by
one company. The officials stated that other companies
considered such data confidential and would not provide them to
Customs.
-- The analysis identified an additional $644 million (33 percent
of the total savings) resulting from increased productivity.
Because this estimate is driven by Customs' assumption that
every minute "saved" by processing transactions or analyzing
data faster using ACE, rather than ACS, would be productively
used by all workers, it can be viewed as a "best case," upper
limit on estimated productivity improvements. Given the
magnitude of the potential savings, even a small change in the
assumption translates into a large reduction in benefits. For
example, conservatively assuming that three-fourths of each
minute saved would be productively utilized by three-fourths of
all workers, the expected benefits would be reduced by about
$282 million.
-- The analysis excluded costs for hardware and system software
upgrades (e.g., desktop workstations and operating systems,
application and data servers, data base management systems) at
each port office. Using Customs' estimate for acquiring the
initial suite of port office hardware and system software, and
assuming a technology refreshment cycle of every 3 to 5 years,
we estimated this cost to be between $72.9 million and $171.8
million.
-- The analysis excluded $52.8 million of costs needed to support
the data center and maintain ACS through fiscal year 2002.
Since Customs intends to operate and maintain ACS in parallel
with ACE through 2002, these costs should be included.
-- The analysis excluded costs to conduct security analysis,
project planning and management, and independent verification
and validation. Customs estimates that these costs collectively
total $23 million.
-- The analysis excluded other relevant cost items, such as the
cost of defining ACE requirements, integrating ACE components,
and conducting ACE regression testing and training. Customs did
not have an estimated value for these costs.
-- The analysis included annual telecommunications costs of $60.3
million once ACE is deployed to all sites. However, Customs
could not provide us with any supporting data or analysis for
this estimate.
CUSTOMS IS NOT INVESTING
INCREMENTALLY IN ACE
---------------------------------------------------------- Chapter 3:4
Incremental project management involves three fundamental components:
(1) developing/acquiring a large system in a series of smaller
projects or system increments, (2) individually justifying investment
in each separate increment on the basis of return-on-investment, and
(3) monitoring actual benefits achieved and costs incurred on
completed increments and modifying subsequent increments/investments
to reflect lessons learned. By doing so, agencies avoid discovering
too late that their system is not cost-beneficial, and they can
reduce the enormous risks associated with large, expensive projects.
The Clinger-Cohen Act requires that agencies acquire information
technology incrementally, to the maximum extent practicable, and have
milestones for senior managers to obtain information on the cost,
timeliness, quality, and capability of information system investments
to meet requirements. Additionally, OMB policy requires that
investments in major information systems be implemented in increments
with each increment delivering measureable benefits.\4
Customs is developing and acquiring ACE in a series of 21 increments.
It has, to date, defined the functionality of only the first two
increments and will define later increments in the future.
Nonetheless Customs has estimated costs and benefits for, and has
committed to, investing in the entire system (i.e., all 21
increments). It has not estimated the costs and benefits for each
increment and does not know whether each increment will produce a
reasonable return on investment. Furthermore, once it has deployed
an increment at a pilot site for evaluation, Customs is not
validating that estimated benefits were actually achieved. As a
result, Customs will not even know whether the first ACE increment,
now being piloted at three sites, is producing expected benefits or
is cost-effective. Instead, Customs will only determine whether the
first increment is performing at a level "equal to or better than"
ACS.
Customs "grand design" approach to ACE does not constitute wise
investment management. Customs will not know whether earlier
increments were cost beneficial before it invests in later
increments; it does not have reliable cost or benefit data upon which
to base investment decisions; and it is committing substantial
resources with no assurance that it will achieve a reasonable
return-on-investment.
--------------------
\4 Memorandum For Heads Of Executive Department And Agencies, October
25, 1996 (M-97-02).
CONCLUSIONS
---------------------------------------------------------- Chapter 3:5
Because Customs does not have reliable information on ACE costs and
benefits and has not analyzed other viable alternatives to the system
solution it selected, it does not have adequate assurance that ACE is
the optimal approach. In fact, it has no assurance at all that ACE
is cost effective. Furthermore, because it is not justifying the
return on its investment in each ACE increment, Customs will not be
able to demonstrate whether ACE is cost-effective until it has
already spent hundreds of millions of dollars to develop/acquire the
entire system.
ACE SOFTWARE DEVELOPMENT AND
ACQUISITION PROCESSES ARE NOT
EFFECTIVE
============================================================ Chapter 4
The Clinger-Cohen Act requires agency CIOs to establish effective IT
management processes, such as processes for developing or acquiring
software. Customs has developed the first ACE software increment
(NCAP 0.1) in-house and is acquiring the second increment (NCAP 0.2)
from a contractor. Customs has not yet decided whether it will
develop or acquire future ACE software increments. If it is to do
either effectively, Customs must use mature software processes.
SEI, recognized for its expertise in software processes, has
developed models and methods that define and determine, respectively,
software development and software acquisition process maturity. We
evaluated both ACE software development and ACE software acquisition
processes using SEI's software development capability maturity model
(SW-CMM) and software acquisition capability maturity model (SA-CMM),
respectively, and SEI's software capability evaluation (SCE) method.
Our evaluations focused on the key process areas (KPA) necessary to
obtain a "repeatable" level of maturity, the second level of SEI's
five-level process maturity models. Organizations that do not
satisfy these second-level KPA requirements are by default at the
"initial" or first level, meaning that their processes are ad hoc,
sometimes even chaotic.
Our evaluations found that Customs lacks the capability to either
develop or acquire ACE software effectively, and that it lacks a
software process improvement program.\1 Because of the number and
severity of the KPA weaknesses found, Customs did not achieve the
"repeatable" level of process maturity as either a software developer
or acquirer, meaning that any attempts to do so are likely to produce
software that does not perform as intended, costs more than expected,
and is delivered late. Further, without a software process
improvement program, it is unlikely that Customs can effectively
address its software process weaknesses.
--------------------
\1 The results of our evaluation of ACE software development maturity
and Customs' software process improvement efforts were first
published in our report on Customs-wide software development
capability (Customs Service Modernization: Immature Software
Development Processes Increase Customs Systems Development Risks
(GAO/AIMD-99-35, February 11, 1999)).
SEI'S SOFTWARE CAPABILITY
MODELS AND METHOD: A BRIEF
DESCRIPTION
---------------------------------------------------------- Chapter 4:1
The SW-CMM and the SA-CMM rank organizational maturity according to
five levels. Maturity levels 2 through 5 require verifiable
existence and use of certain key process areas, known as KPAs. The
SW-CMM includes six level 2 KPAs. We evaluated Customs' against five
of the six. The sixth level 2 KPA, software subcontract management,
was not evaluated because Customs did not use subcontractors on NCAP
0.1 (see table 4.1 for a description of each KPA).
Table 4.1
Description of SW-CMM
Level 2 KPAs
SW-CMM Level 2
KPAs Description
------------------ --------------------------------------------------
Requirements Defining, validating, and prioritizing
management requirements, such as functions, performance, and
delivery dates.
Software project Developing estimates for the work to be performed,
planning establishing the necessary commitments, and
defining the plan to perform the work.
Software project Tracking and reviewing software accomplishments
tracking and and results against documented estimates,
oversight commitments, and plans and adjusting these based
on the actual accomplishments and results.
Software Selecting qualified contractors and managing them
subcontract effectively.
management
Software quality Reviewing and auditing the software products and
assurance activities to ensure that they comply with the
applicable processes, standards, and procedures
and providing the staff and managers with the
results of these reviews and audits.
Software Selecting project baseline items, such as
configuration specifications, systematically controlling these
management items and changes to them, and recording and
reporting status and change activity for these
items.
----------------------------------------------------------------------
The SA-CMM includes seven level 2 KPAs. We evaluated Customs against
all seven level 2 KPAs. We also evaluated Customs against one level
3 KPA--acquisition risk management--because it is considered by
software experts to be an integral part of the solicitation, project
performance management, and contract performance management processes
(see table 4.2 for a description of each KPA).
Table 4.2
Description of SA-CMM Level 2 KPA's and
the Risk Management Level 3 KPA
SA-CMM Level 2
KPAs Description
------------------ --------------------------------------------------
Software Ensuring that reasonable planning for the software
acquisition acquisition is conducted and that all elements of
planning the project are included.
Solicitation Ensuring that award is made to the contractor most
capable of satisfying the specified requirements.
Requirements Establishing a common and unambiguous definition
development and of software acquisition requirements understood by
management the acquisition team, system user, and the
contractor.
Project office Managing the activities of the project office and
management supporting contractor(s) to ensure a timely,
efficient, and effective software acquisition.
Contract tracking Ensuring that the software activities under
and oversight contract are being performed in accordance with
contract requirements, and that products and
services will satisfy contract requirements.
Evaluation Determining that the acquired software products
and services satisfy contract requirements prior
to acceptance and transition to support.
Transition and Providing for the transition of the software
support products being acquired to their eventual support
organization.
SA-CMM Level 3 KPA Description
Acquisition risk Identifying risks as early as possible, adjusting
management acquisition strategy to mitigate those risks, and
developing and implementing a risk management
process as an integral part of the acquisition
process.
----------------------------------------------------------------------
For both models, each KPA contains five common attributes that
indicate whether the implementation and institutionalization of a KPA
can be effective, repeatable, and lasting. The five common features
are:
-- Commitment to perform: The actions that the organization must
take to establish the process and ensure that it can endure.
Commitment to perform typically involves establishing
organizational policies and sponsorship.
-- Ability to perform: The preconditions that must exist in the
project or organization to implement the process competently.
Ability to perform typically involves allocating resources,
establishing effective organizational structures, and ensuring
that personnel have the needed training.
-- Activities performed: The roles and procedures necessary to
implement the process. Activities performed typically involve
establishing plans and procedures, performing the work, tracking
it, and taking appropriate management actions.
-- Measurement and analysis: The activities performed to measure
the process and analyze the measurements. Measurement and
analysis typically include defining the measurements to be taken
and the analyses to be conducted to determine the status and
effectiveness of the activities performed.
-- Verifying implementation: The steps to ensure that the
activities are performed in compliance with the process that has
been established. Verification typically includes reviews by
management.
In accordance with SEI's SCE method, for each KPA in level 2 and the
one KPA in level 3 (risk management), we evaluated institutional
policies and practices and compared project-specific guidance and
practices against the five common attributes. This project-specific
comparison can result in one of four possible outcomes: (1) project
strength--effective implementation of the key practice, (2) project
weakness--ineffective implementation of a key practice or failure to
implement a key practice, (3) project observation--key practice
evaluated but evidence is inconclusive and cannot support
characterization as either a strength or a weakness, and (4) not
rated--key practice not currently relevant to project, therefore not
evaluated.
CUSTOMS LACKS THE CAPABILITY TO
DEVELOP ACE SOFTWARE
EFFECTIVELY
---------------------------------------------------------- Chapter 4:2
In February 1999, we reported that NCAP 0.1 did not fully satisfy any
of the KPAs that SEI's SW-CMM requires to be CMM level 2 or
repeatable.\2 While NCAP 0.1 exhibited many strengths in three
KPA's--requirements management, software project planning, and
software project tracking and oversight--it had numerous and
significant weaknesses in the remaining two KPA's--software quality
assurance and software configuration management. As a result,
Customs' ACE software development capability is immature.
In our February 1999 report, we also stated that Customs lacked a
software process improvement program. Without one, it is unlikely
that Customs can effectively address its software process weaknesses.
Accordingly, we recommended that, after ensuring that its
mission-critical systems are Year 2000 compliant, but before
investing in major software development efforts like ACE, Customs (1)
assign responsibility and authority for software process improvement,
(2) develop and implement a formal plan for software process
improvement that, among other things, was based on our software
development process maturity findings, (3) ensure that every new
software development effort in Customs adopts processes that satisfy
at least SW-CMM level 2 requirements, and (4) ensure that process
improvement activities be initiated for all ongoing essential
software maintenance projects. Customs agreed with our findings and
stated its commitment to software process improvement and maturity,
including stating its plans for establishing a software process
improvement program and addressing the weaknesses that we identified.
Each of the five SW-CMM KPAs, along with examples of how the ACE
software development organization compares to each KPA practices, is
summarized below. (See appendix III for more detailed information on
the KPAs and our findings.)
--------------------
\2 GAO/AIMD-99-35, February 11, 1999.
REQUIREMENTS MANAGEMENT
-------------------------------------------------------- Chapter 4:2.1
The purpose of requirements management is to establish a common
understanding of the requirements between the customer and the
software developers. According to the SW-CMM, a repeatable
requirements management process, among other things, includes (1)
documenting the system requirements allocated to software, (2)
providing adequate resources and funding for managing the allocated
requirements, (3) training members of the software engineering group
to perform their requirements management activities, (4) using the
allocated requirements as the basis for software plans, work
products, and activities, (5) following a written organizational
policy for requirements management, and (6) having a quality
assurance group that reviews the activities and work products for
managing allocated requirements and reports the results.
The NCAP 0.1 project exhibited strengths in almost all key practices
within the requirements management KPA. For example, (1) the system
requirements allocated to software were documented, (2) adequate
resources and funding were provided for managing the allocated
requirements, (3) members of the software engineering group were
trained to perform their requirements management activities, and (4)
the software developers used the allocated requirements as the basis
for software plans, work products, and activities.
Nevertheless, two very important requirements management key
practices were not being performed on the NCAP 0.1 project.
Specifically, there was no written organizational policy for
requirements management and there was no quality assurance group to
review the activities. In the absence of these practices, management
is missing important means for ensuring that software requirements
are managed in a prescribed manner.
SOFTWARE PROJECT PLANNING
-------------------------------------------------------- Chapter 4:2.2
The purpose of software project planning is to establish reasonable
plans for performing software engineering and for managing the
software project. According to the SW-CMM, a repeatable software
project planning process, among other things, includes (1)
documenting the software project plan, and preparing plans for
software engineering facilities and support tools, (2) identifying
the work products needed to establish and maintain control of the
software project, (3) making and using measurements to determine the
status of planning activities, (4) following a written organizational
policy for planning a software project, (5) training personnel in
software project planning and estimating, (6) estimating the software
project's efforts, costs, critical computer resources, and schedule
according to documented procedures, and (7) having a quality
assurance group that reviews the activities and work products for
software project planning and reports the results.
The NCAP 0.1 project exhibited strengths in many of the key practices
within the software project planning KPA. For example, (1) the
software project plan and plans for the software engineering
facilities and support tools were documented, (2) software work
products that are needed to establish and maintain control of the
project were identified in the project plan, and (3) measurements
were made and used to determine the status of software planning
activities.
However, we also found several significant key practice weaknesses.
Specifically, (1) NCAP 0.1 did not follow a written policy for
planning a software project, (2) project personnel were not trained
in project planning and estimating procedures, (3) estimates for the
software project's effort and costs, critical computer resources, and
project schedule were not derived according to documented procedures,
and (4) there was no software quality assurance group to review or
audit the activities and work products for software project planning.
Such project planning weaknesses mean that management has no
assurance that it will get the consistent, complete, and reliable
information about the project's expected costs and schedules needed
to make expeditious and informed investment decisions.
SOFTWARE PROJECT TRACKING
AND OVERSIGHT
-------------------------------------------------------- Chapter 4:2.3
The purpose of software project tracking and oversight is to provide
adequate visibility into the progress of the software development so
that management can take effective actions when the software
project's performance deviates significantly from the software plans.
According to the SW-CMM, effective software project tracking and
oversight, among other things, includes (1) designating a project
software manager to be responsible for the project's software
activities and results, (2) having a documented software development
plan for tracking software activities and communicating status, (3)
conducting periodic internal reviews to track technical progress,
plans, performance, and issues against the software development plan,
(4) periodically reviewing the activities for software project
tracking and oversight with senior management, (5) following a
written organizational policy for managing the project, (6) training
software project managers in the technical and personnel aspects of
the software project, (7) reviewing software project commitments and
changes to commitments with senior management, and (8) independently
reviewing or auditing the activities and work products for software
project tracking and oversight.
NCAP 0.1 exhibited strengths in most of the key practices within the
software project tracking and oversight KPA. For example, (1) a
project manager had been designated to be responsible for the
project's software activities and results, (2) the project had a
documented software development plan, (3) the software engineering
group conducted periodic internal reviews to track technical
progress, plans, performance, and issues against the software
development plan (e.g., the sizes of software work products and the
risks associated with software costs, resources, and schedule), and
(4) software project tracking and oversight activities were reviewed
with senior management on a weekly basis.
However, significant weaknesses also existed. For example, (1) there
was no written organizational policy for managing the software
project, (2) the software managers were not trained in managing the
technical and personnel aspects of the software project, (3) software
project commitments and changes to commitments were not reviewed with
senior management, and (4) because no software quality assurance
group exists, there were no independent reviews or audits of the
activities and work products for software project tracking and
oversight.
SOFTWARE QUALITY ASSURANCE
-------------------------------------------------------- Chapter 4:2.4
The purpose of software quality assurance is to independently review
and audit software products and activities to verify that they comply
with applicable procedures and standards and to provide the software
project and higher level managers with the results of these
independent reviews and audits. According to the SW-CMM, a
repeatable software quality assurance process, among other things,
includes (1) preparing a software quality assurance plan according to
a documented procedure, (2) having a written organizational policy
for implementing software quality assurance, (3) conducting audits of
designated work processes and products to verify compliance, (4)
documenting deviations identified in the software activities and
software work products and handling them according to a documented
procedure, and (5) having experts independent of the software quality
assurance group periodically review the activities and work products
of the project's software quality assurance group.
NCAP 0.1 did not satisfy any of the key practices within the software
quality assurance KPA. Because its software quality assurance is
ineffective, ACE is at risk that (1) process and product standards
will not be met and (2) software will not perform as intended, and
will cost more and take longer to develop than expected.
SOFTWARE CONFIGURATION
MANAGEMENT
-------------------------------------------------------- Chapter 4:2.5
The purpose of software configuration management is to establish and
maintain the integrity of the products of the software project
throughout the project's software life- cycle. According to the
SW-CMM, a repeatable software configuration management process, among
other things, includes (1) preparing a software configuration
management plan according to a documented procedure, (2) establishing
a configuration management library system as a repository for the
software baselines, (3) identifying software work products to be
placed under configuration management, (4) controlling the release of
products from the software baseline library according to a documented
procedure, (5) following a written organizational policy for
implementing software configuration management, (6) recording the
status of configuration items/units according to a documented
procedure, (7) making and using measurements to determine the status
of software configuration management activities, and (8) periodically
reviewing software configuration management activities with senior
management.
NCAP 0.1 had strengths in several of the key practices within the
software configuration management KPA. For example, (1) a software
configuration management plan was developed according to a documented
procedure, (2) a configuration management library system was
established as a repository for the software baselines, (3) software
work products to be placed under configuration management were
identified, and (4) the release of products from the software
baseline library was controlled according to a documented procedure.
However, NCAP 0.1 exhibited weaknesses in most of the software
configuration management key practices, which collectively jeopardize
Customs' ability to maintain the integrity of the project's software
products. Specifically, (1) there was no organizational policy for
implementing software configuration management, (2) the status of
configuration management items was not recorded according to a
documented procedure, (3) no measurements were taken to determine the
status of software configuration management activities, and (4)
software configuration management activities were not periodically
reviewed with senior management.
CUSTOMS LACKS THE CAPABILITY TO
ACQUIRE ACE SOFTWARE
EFFECTIVELY
---------------------------------------------------------- Chapter 4:3
NCAP 0.2 did not fully satisfy any of the KPA's that SEI's SA-CMM
requires to be a CMM level 2 or repeatable. We found extensive
weaknesses in all KPA's except one--requirements development and
management. Each of the eight KPA's, along with examples of how
Customs' software acquisition processes compare to the KPA goals, is
summarized below. (See appendix IV for more detailed information on
KPAs and our findings.)
SOFTWARE ACQUISITION
PLANNING
-------------------------------------------------------- Chapter 4:3.1
The purpose of software acquisition planning is to ensure that
reasonable planning for the software acquisition is conducted and
that all aspects of the total software acquisition effort are
included in these plans. According to the SA-CMM, a repeatable
software acquisition planning process, among other things, includes
(1) having a written software acquisition policy, (2) developing and
documenting the software acquisition strategy and plan, (3) having
management review software acquisition planning activities, and (4)
making and using measurements to determine the status of software
acquisition planning activities.
NCAP 0.2 had no strengths in this KPA. For example, (1) the project
followed no written policy for planning the software acquisition, (2)
there was no documented software acquisition strategy or plan, (3)
software acquisition planning activities were not being reviewed by
management, and (4) software acquisition planning activities were not
being measured. As a result, management lacks an effective means to
identify problems in project performance and take corrective action
expeditiously.
SOLICITATION
-------------------------------------------------------- Chapter 4:3.2
The purpose of solicitation is to prepare a request for proposal that
delineates a project's software-related requirements, and, consistent
with relevant solicitation laws and regulations, select a contractor
that can most cost-effectively satisfy these requirements. According
to the SA-CMM, specific requirements for a repeatable solicitation
process include, among other things, (1) designating responsibility
for the software portion of the solicitation, (2) designating a
selection official to be responsible for the selection process and
decision, (3) preparing cost and schedule estimates for the software
products and services being acquired, (4) having a written policy for
the conduct of the software portion of the solicitation, (5) having
and following a solicitation plan, (6) independently reviewing cost
and schedule estimates for the software products and services being
acquired, and (7) making and using measurements to determine the
status of the solicitation activities and resultant products.
NCAP 0.2 had some strengths in this area, including (1) designating
responsibility for the software solicitation, (2) designating a
selection official for the selection process and decision, and (3)
preparing cost and schedule estimates for the software products and
services being acquired.
However, we found many weaknesses, including (1) no written policy
for the conduct of the software solicitation and (2) no solicitation
plans to guide the performance of solicitation activities. These
weaknesses increase the risk of Customs not adequately or uniformly
evaluating the offerors' proposals, and making a suboptimal
selection. Other weaknesses included (1) no independent review of
software cost and schedule estimates and (2) no measurements to
determine the status of solicitation activities and resultant
products. As a result, management cannot identify solicitation
problems early and resolve them expeditiously.
REQUIREMENTS DEVELOPMENT AND
MANAGEMENT
-------------------------------------------------------- Chapter 4:3.3
The purpose of requirements development and management is to
establish and maintain a common and unambiguous definition of
software requirements among the acquisition team, the system users,
and the software development contractor. This KPA involves two
subprocesses: (1) developing a baseline set of software-related
contractual requirements and (2) managing these requirements and
changes to these requirements for the duration of the acquisition.
SA-CMM requirements development and management practices necessary to
achieve a repeatable maturity include (1) having a group that is
responsible for performing requirements development and management
activities, (2) ensuring that individuals performing requirements
development and management activities have experience or receive
training, (3) measuring and reporting on the status of requirements
development and management activities and resultant products to
management, (4) periodically reviewing requirements development and
management activities by acquisition organization management, (5)
having a written organizational policy for establishing and managing
the software-related contractual requirements, (6) performing
requirements development and management activities in accordance with
documented plans, (7) appraising the impact of system requirements
change requests on the software being acquired, and (8) maintaining
traceability between the software-related contractual requirements
and the contractor's software work products.
NCAP 0.2 had many strengths in requirements development and
management. For example, (1) there was a group that is responsible
for performing requirements development and management, (2) group
members had experience or have received training in the activities,
(3) measurements of requirements development and management
activities were made, and (4) management reviewed the activities
periodically.
However, we found weaknesses in important key practices that
jeopardize effective control of the requirements baseline and can
result in software products that do not meet cost, schedule, or
performance objectives. Specifically, (1) there was no written
policy for establishing and managing the software-related contractual
requirements, (2) there was no documented requirements development
and management plan, (3) system requirements change requests were not
appraised for their impact on the software being acquired, and (4)
traceability between the software-related contractual requirements
and the contractor's software work products was not maintained.
PROJECT OFFICE MANAGEMENT
-------------------------------------------------------- Chapter 4:3.4
The purpose of project office management is to manage the activities
of the project office and supporting contractors to ensure a timely,
efficient, and effective software acquisition. According to the
SA-CMM, effective project office management requires, among other
things, (1) designating responsibility for project management, (2)
providing project teams with authority to alter either the project's
performance, cost, or schedule baseline, (3) having a written policy
for the management of the software project, (4) using a software
acquisition management plan, and (5) using a corrective action system
for identifying, recording, tracking, and correcting problems.
NCAP 0.2 had several practice strengths in the KPA including (1)
designating a project manager as responsible for the project and (2)
providing project teams with authority to change either the
performance, cost, or schedule software acquisition baseline when
necessary. However, we found numerous weaknesses, including (1) no
written policy for management of the software project, (2) no
software acquisition management plan, and (3) no corrective action
system. These weaknesses jeopardize the project's ability to ensure
that important project office and contractor activities are defined,
understood, and completed.
CONTRACT TRACKING AND
OVERSIGHT
-------------------------------------------------------- Chapter 4:3.5
The purpose of contract tracking and oversight is to ensure that the
software development contractor performs according to the terms of
the contract; needed contract changes are identified, negotiated, and
incorporated into the contract; and contractor performance issues are
identified early, when they are easier to address. According to the
SA-CMM, a repeatable contract tracking and oversight process, among
other things, includes (1) designating responsibility for contract
tracking and oversight, (2) providing support for the project team by
contracting specialists, (3) ensuring that individuals performing
contract tracking and oversight are suitably experienced or trained,
(4) having a written organizational policy for contract tracking and
oversight, (5) having a documented plan for contract tracking and
oversight, (6) reviewing required contractor software planning
documents to oversee the contractor's software engineering effort,
and (7) tracking problems or issues found by the project team during
contract tracking and oversight in a corrective action system.
NCAP 0.2 had several strengths in this KPA, including (1) designating
responsibility for contract tracking and oversight activities, (2)
providing contracting specialist support to the project team, and (3)
ensuring that personnel responsible for contract tracking and
oversight are suitably experienced or trained. However, we found
significant weaknesses, including (1) no written organizational
policy for contract tracking and oversight, (2) no documented
contract tracking and oversight plan, (3) no required contractor
software planning documents that could be used to oversee the
contractor's software engineering effort, and (4) no corrective
action tracking system for recording problems or issues found by the
project team. Because of these weaknesses, Customs' contractor
tracking and oversight activities are undisciplined and unstructured,
thereby increasing the chances of its software acquisitions being
late, costing more than expected, and not performing as intended.
EVALUATION
-------------------------------------------------------- Chapter 4:3.6
The purpose of evaluation is to determine that the acquired software
products and services satisfy contract requirements prior to
acceptance. According to the SA-CMM, a repeatable evaluation process
includes (1) designating responsibility for planning, managing, and
performing evaluation activities, (2) ensuring that individuals
performing evaluation activities have experience or receive training,
(3) managing the evaluation of the acquired software according to a
written policy, (4) documenting evaluation plans and conducting
evaluation activities in accordance with the plan, (5) developing and
managing evaluation requirements in conjunction with developing
software technical requirements, (6) tracking contractor performance
of evaluation activities for compliance with the contract, and (7)
measuring and reporting on the status of evaluation activities to
management.
NCAP 0.2 had some evaluation strengths, including (1) designating
responsibility for planning, managing, and performing evaluation
activities and (2) ensuring that individuals performing evaluation
activities have experience or receive training. However, we found
many significant weaknesses, including, (1) there was no written
policy for managing the evaluation of the acquired software, (2)
there was no documented evaluation plan to use as the basis for
conducting evaluation activities, (3) evaluation requirements were
not developed in conjunction with development of the system or
software technical requirements, (4) contractor evaluation activity
performance was not tracked for compliance with the contract, and (5)
no measurements were made to determine the status of evaluation
activities. Because of these pervasive evaluation weaknesses,
Customs has no assurance that contractor-developed ACE software will
satisfy defined requirements.
TRANSITION AND SUPPORT
-------------------------------------------------------- Chapter 4:3.7
The purpose of transition and support is to provide for the effective
and efficient "hand-off" of the acquired software products to the
support organization responsible for software maintenance. According
to the SA-CMM, repeatable transition and support processes, among
other things, include (1) having a written policy for transitioning
software products to the support organization, (2) designating a
group that is responsible for coordinating transition and support
activities, (3) performing project activities in accordance with a
documented transition and support plan, and (4) measuring the status
of the transition and support activities.
NCAP 0.2 had no practice strengths in the transition and support KPA.
In particular, (1) there was no written policy for the transitioning
of software products to the software support organization, (2) there
was no designated group responsible for coordinating transition and
support activities, (3) there was no transition and support plan, and
(4) the status of transition and support activities were not
measured. As a result, Customs is not effectively positioned to
assume support responsibility for the ACE software that it acquires.
ACQUISITION RISK MANAGEMENT
-------------------------------------------------------- Chapter 4:3.8
The purpose of acquisition risk management is to formally identify
risks as early as possible and adjust the acquisition to mitigate
those risks. According to the SA-CMM, an effective risk management
process, among other things, includes (1) having a written policy on
acquisition risk management, (2) developing a software acquisition
risk management plan, (3) conducting software risk management
activities in accordance with the plan (e.g., identifying risks,
taking mitigation actions, and tracking risk mitigation actions to
completion), and (4) measuring and reporting on the status of
acquisition risk management activities to management.
NCAP 0.2 was weak in all key practices of acquisition risk
management. In particular, (1) there was no written policy on
acquisition risk management, (2) there was no software acquisition
risk management plan, (3) no risk management activities were being
performed, and (4) no measurements were taken to determine the status
of risk management activities. Because of these weaknesses, Customs
has no assurance that it will identify risks early and effectively
mitigate them.
CONCLUSIONS
---------------------------------------------------------- Chapter 4:4
Customs' software acquisition and development processes are
insufficiently mature to support the effective development or
acquisition of ACE. Until these processes are strengthened, there is
no basis for confidence that ACE will be delivered on time, within
budget, or perform as specified.
OVERALL CONCLUSIONS,
RECOMMENDATIONS, AND AGENCY
COMMENTS
============================================================ Chapter 5
Successful systems modernization is critical to Customs' ability to
implement the provisions of its legislative mandates and to achieve
its goals of increased compliance with trade laws and speedier
processing of imports. However, Customs currently lacks the full
management and technical capability needed to successfully modernize
its systems. As a result, satisfaction of its legislative mandate
and realization of the attendant benefits by both Customs and the
trade community are in jeopardy.
With ACE, Customs has not adequately demonstrated that it is "doing
the right thing," i.e., that the system that it is building is the
most cost effective system for meeting the needs of Customs and the
trade community. Specifically, by not having a complete information
systems architecture to guide the development and evolution of its
systems, including ACE, Customs runs the risk that its efforts will
not provide optimum performance and will be incompatible and
expensive to maintain. Also, by using unreliable and incomplete ACE
cost and benefit data, Customs does not know if ACE is a cost
effective system solution to its needs, or if its investments in ACE
are wise.
Customs also has not clearly demonstrated that it is developing ACE
"the right way." In particular, without incrementally managing such a
mammoth system as ACE, Customs will not know whether its cost and
benefit expectations are being met until it has already spent
hundreds of millions of dollars. Such a "grand design" approach has
proven repeatedly to be ineffective and has been abandoned by
successful IT organizations. Additionally, because its ability to
develop or acquire software is immature, Customs can do neither
effectively. Therefore, its software acquisition and development
processes must be strengthened before it can hope to acquire or
develop ACE software on time and within budget, as well as to achieve
expected benefits.
RECOMMENDATIONS
---------------------------------------------------------- Chapter 5:1
In addition to previous recommendations to improve Customs'
management of information technology,\1 we recommend that Customs
correct the management and technical weaknesses discussed in this
report before building ACE. To accomplish this, GAO recommends that
the Commissioner of Customs, with the support of Customs' CIO, ensure
that Customs
(1) rigorously analyze alternative approaches to building ACE,
including ITDS as an alternative to developing ACE entirely within
Customs and;
(2) make investment decisions incrementally, i.e., for each
increment:
-- use disciplined processes to prepare a rigorous life-cycle cost
estimate, including an explicit discussion of its inherent
uncertainty;
-- prepare realistic and supportable benefit expectations;
-- require a favorable return-on-investment and compliance with
Customs' architecture before making any investment; and
-- validate actual costs and benefits once an increment is piloted,
compare these with estimates, use the results in making further
decisions on subsequent increments, and report the results to
Customs' House and Senate appropriations and authorizing
committees; and
(3) strengthen ACE software acquisition management by:
-- establishing an effective process improvement program and
correcting the weaknesses in ACE software acquisition processes
identified in this report, thereby bringing ACE processes to at
least SEI level 2 and
-- requiring at least SEI level 2 processes of all ACE software
contractors.
--------------------
\1 GAO/AIMD-99-35, February 11, 1999 and GAO/AIMD-98-70, May 5, 1998.
AGENCY COMMENTS
---------------------------------------------------------- Chapter 5:2
In its written comments on a draft of this report, Customs agreed
with our conclusions and recommendations and stated that it is
committed to addressing the problems discussed in the report. To
this end, Customs cited a number of actions that it has underway and
planned to improve management of information technology in general,
and ACE specifically. For example, Customs stated that it has
recruited a CIO with extensive experience in enterprise architecture
and major systems acquisition, reorganized its Office of Information
and Technology, and revised its System Development Life Cycle
process. Also, Customs stated that it has engaged a contractor to
update and improve the ACE cost-benefit analysis and plans for
another contractor to independently review the new ACE cost-benefit
analysis. Additionally, Customs reported that it revised its
investment management procedures to provide for effective systems
architecture enforcement, as we recommended in our May 1998 report.\2
The report has been modified to reflect this information. To fully
correct the management and technical weaknesses discussed in this
report, Customs must follow through and effectively implement its
plans to address all of our recommendations. Appendix I provides the
full text of Customs' comments.
(See figure in printed edition.)Appendix I
--------------------
\2 GAO/AIMD-98-70, May 5, 1998.
COMMENTS FROM THE U.S. CUSTOMS
SERVICE
============================================================ Chapter 5
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
DETAILED COMPARISON OF SEI'S
CHECKLIST FOR RELIABLE COST
ESTIMATES AND CUSTOMS' PRACTICES
FOR ESTIMATING ACE COSTS
========================================================== Appendix II
The following seven tables rate Customs' practices for estimating
life-cycle costs of $1.05 billion against SEI's criteria for reliable
cost estimating. A strength indicates that Customs satisfied the SEI
criterion; a weakness indicates that it did not. Where evidence was
inconclusive and did not clearly support a determination of either
strength or weakness, a rating of "observation" was given.
Table II.1
Are the Objectives of the Estimate Clear
and Correct?
SEI checklist item Finding Rating
-------------------------------- -------------------------------- ------------
The objectives of the estimate The objectives of the estimates Strength
are stated in writing. are stated in writing.
The life-cycle to which the The 10-year life-cycle is Strength
estimate applies is clearly clearly defined for the ACE cost
defined. estimate, and consists of 6
years of development and
deployment and 4 years of
operations and maintenance.
The tasks and activities The tasks and activities that Strength
included in (and excluded from) Customs included in (and
the estimate are clearly excluded from) the estimate are
identified. clearly identified.
The tasks and activities The tasks and activities Strength
included in the estimate are included in the estimate are
consistent with the objectives consistent with the stated
of the estimate. objectives of the estimate.
--------------------------------------------------------------------------------
Table II.2
Has the Task Been Appropriately Sized?
SEI checklist item Finding Rating
--------------------------------- --------------------------------- ----------
A structured process has been A structured process has not been Weakness
used to estimate and describe the used to estimate and describe the
size of the software project. size of the entire software
project. Customs followed a
structured process for estimating
the size of NCAP 0.1, the first
ACE software release; however,
the methodology used to estimate
the size of NCAP 0.2 and all
subsequent ACE software releases
was not structured and lacked
critical elements, such as
defined functional requirements
for NCAP 0.2 and all subsequent
releases.
A structured process has been A structured process has not been Weakness
used to estimate and describe the used to estimate and describe the
extent of software reuse. extent of software reuse.
According to Customs officials,
software reuse was discussed
during an ACE planning
conference; however, the results
were not documented.
The processes for estimating size The processes for estimating size Weakness
and reuse are documented. and reuse are not documented. The
documentation available on the
estimate, including the ACE/NCAP
workload estimate spreadsheets,
does not document the size and
reuse estimating processes
followed by Customs.
The descriptions of size and The description of size Weakness
reuse identify what is included identifies what is included in
in (and excluded from) the size the size measures used (e.g.,
and reuse measures used. screens, business objects, and
operations); however, no
description of reuse measures
exists.
The measures of reuse distinguish The measures of reuse are not Observatio
between code that will be documented; however, the ACE n
modified and code that will be workload estimate spreadsheets do
integrated as is into the system. include an estimate of existing
code that will be modified for
use in subsequent NCAP software
releases.
The definitions, measures, and Customs did not use a model to Weakness
rules used to describe size and estimate the cost of ACE.
reuse are consistent with the Furthermore, the reuse
requirements (and calibrations) definitions, measures, and rules
of the models used to estimate are not documented.
cost.
The size estimate was checked by The size estimate for the entire Weakness
relating it to measured sizes of ACE project was not checked by
other software products or relating it to measured sizes of
components. other software products or
components. Customs did not have
a comparable internal software
product to benchmark against at
the time of the estimate.
According to Customs officials,
the NCAP 0.1 size estimate was
compared (via informal and
undocumented telephone inquiries)
to similar products and
components previously developed
outside of Customs. The size
estimate for NCAP 0.2 and
subsequent releases was based
upon the NCAP 0.1 size estimate.
The size estimating process was The size estimating process was Weakness
checked by testing its predictive not checked by testing its
capabilities against measured predictive capabilities against
sizes of completed products. measured sizes of completed
projects.
--------------------------------------------------------------------------------
Table II.3
Is the Estimated Cost Consistent With
Demonstrated Accomplishments on Other
Projects?
SEI checklist item Finding Rating
--------------------------------- --------------------------------- ----------
The organization has a structured Customs does not have a Weakness
process for relating estimates to structured, documented process
actual costs of completed work for relating estimates to actual
--the process is documented costs of completed work.
--the process was followed.
The cost models that were used Customs did not use a cost model Weakness
have been calibrated to relevant and did not have any relevant
historical data. (Models of some historical data to calibrate.
sort are needed to provide
consistent rules for
extrapolating from previous
experience.)
The cost models quantify Customs did not use a cost model. Weakness
demonstrated organizational Further, Customs changed its
performance in ways that software development approach
normalize for differences among from an in-house development
software products and projects. (NCAP 0.1) to an acquisition
(So that a simple, unnormalized, (NCAP 0.2). Nonetheless, Customs
lines-of-code per staff-month used the NCAP 0.1 estimate as the
extrapolation is NOT the basis basis for extrapolating the
for the estimate.) estimate for the remaining ACE
releases without normalizing the
NCAP 0.1 data to account for the
differences in the approaches.
The consistency achieved when Customs did not use a cost model. Weakness
fitting the cost models to At the time that the ACE cost
historical data has been measured estimate was developed, Customs
and reported. had no historical data with which
to evaluate consistency. Cost
data are currently being
collected, but the collected data
has not been used to measure and
report the consistency achieved
when using the cost models.
The values used for cost models' Customs did not use a cost model; Weakness
parameters appear valid when therefore, comparison of cost
compared to values that fit the models' parameter values to those
models well to past projects. used on past projects was not
done.
The calibration of cost models Customs did not use cost models; Weakness
was done with the same versions therefore, calibration was not
of the models that were used to done.
prepare the estimate.
The methods used to account for The methods used to account for Strength
reuse recognize that reuse is not reuse recognize that reuse is not
free. (The estimate accounts for free. The Customs estimating
activities such as interface spreadsheets account for
design, modification, activities associated with NCAP
integration, testing, and software reuse, such as
documentation that are associated modification of existing code,
with effective reuse.) integration, and testing.
Extrapolations from past projects Extrapolations from NCAP 0.1 to Weakness
account for differences in NCAP 0.2 and subsequent releases
application technology. (For did not account for differences
example, data from projects that in application technology (e.g.,
implemented traditional mainframe the change in architecture from
applications require adjustments the legacy mainframe system to a
if used as a basis for estimating combined mainframe and Internet-
client-server implementation. based client-server system).
Some cost models provide
capabilities for this, others do
not.)
Extrapolations from past projects Extrapolations from NCAP 0.1 to Weakness
account for observed, long-term NCAP 0.2 and subsequent releases
trends in software technology did not account for differences
improvement. (Although some cost in tools, languages, and
models attempt this internally, personnel due to the change in
the best methods are usually architecture from the legacy
based on extrapolating measured mainframe system written in COBOL
trends in calibrated and C++ to a combined mainframe
organizational performance.) and Internet-based system written
in C++ and Java.
Extrapolations from past projects Extrapolations from NCAP 0.1 to Weakness
account for the effects NCAP 0.2 and subsequent releases
of introducing new software did not account for the change in
technology or processes. process from an in-house software
(Introducing a new technology or development project (NCAP 0.1) to
process can initially reduce an a software acquisition (NCAP
organization's productivity.) 0.2).
Work-flow schematics have been Work-flow schematics have not Weakness
used to evaluate how this project been used to evaluate how this
is similar to (and how it differs project is similar to (and how it
from) projects used to differs from) projects used to
characterize the organization's characterize the organization's
past performance. past performance.
--------------------------------------------------------------------------------
Table II.4
Have the Factors That Affect the
Estimate Been Identified and Explained?
SEI checklist item Finding Rating
--------------------------------- --------------------------------- ----------
A written summary of parameter A written summary of the Weakness
values and their parameters affecting the cost
rationales accompanies the estimate exists; however, a
estimate. written summary of the values
used and their rationales does
not exist.
Assumptions have been identified The assumptions have been Strength
and explained. identified and explained.
A structured process such as a Customs has used spreadsheets Strength
template or format has been used designed specifically for the ACE
to ensure that key factors have NCAP project in an effort to
not been overlooked. ensure that key factors have not
been overlooked.
Uncertainties in parameter values No uncertainties in parameter Weakness
have been identified and values were identified or
quantified. quantified.
A risk analysis has been A risk analysis has been Strength
performed, and risks that affect performed, and risks that affect
cost have been identified and cost have been identified and
documented. (Elements addressed documented (e.g., lack of
include issues such as requirement definition for later
probability of occurrence, releases, technical
effects on parameter values, cost complications, stability of
impacts, schedule impacts, and development and configuration
interactions with other management environments and
organizations.) process).
--------------------------------------------------------------------------------
Table II.5
Have Steps Been Taken to Ensure the
Integrity of the Estimating Process?
SEI checklist item Finding Rating
--------------------------------- --------------------------------- ----------
Management reviewed and agreed to According to Customs project Observatio
the values for all descriptive officials, management reviewed n
parameters before costs were and agreed to the values for all
estimated. descriptive parameters before
costs were estimated; however,
management approval has not been
documented.
Adjustments to parameter values Not applicable. According to N/A
to meet a desired cost Customs project officials, no
or schedule have been documented. adjustments to parameter values
to meet a desired cost or
schedule were made.
If a dictated schedule has been Not applicable. According to N/A
imposed, the estimate is Customs project officials, no
accompanied by an estimate of (1) dictated schedule has been
the normal schedule and (2) the imposed.
additional expenditures required
to meet the dictated schedule.
Adjustments to parameter values Not applicable. No adjustments to N/A
to meet a desired cost parameter values were made, and
or schedule are accompanied by thus no accompanying management
management action that makes the action occurred.
values realistic.
More than one cost model or Two estimating approaches Strength
estimating approach has been (spreadsheets) were used for
used, and the differences in NCAP, and the resulting
results have been analyzed and differences in the estimates
explained. (within 15 percent) have been
analyzed and explained.
People from related but different People from the requirements, Strength
projects or disciplines were design, systems engineering,
involved in preparing the integration, testing, and project
estimate. management teams were involved in
preparing the estimate.
At least one member of the Two members of the estimating Observatio
estimating team is an experienced team have practical experience, n
estimator, trained in the cost but none have been trained in
models that were used. cost estimating and cost models.
Estimators independent of the According to Customs project Observatio
performing organization concur officials, estimators independent n
with the reasonableness of the of the performing organization
parameter values and estimating concurred with the reasonableness
methodology. of the parameter values and
estimating methodology, but their
analysis is not documented.
The groups that will be doing the Customs (NCAP 0.1 developer) and Strength
work accept the estimate as an the contractor (NCAP 0.2
achievable target. developer) accepted the estimates
as achievable targets.
Memorandums of agreement have No memorandums of agreement have Weakness
been completed and signed with been completed and signed with
the other organizations whose the requirements, design, systems
contributions affect cost. engineering, integration,
testing, project management team,
and other organizations whose
contributions affect cost.
--------------------------------------------------------------------------------
Table II.6
Is the Organization's Historical
Evidence Capable of Supporting a
Reliable Estimate?
SEI checklist item Finding Rating
--------------------------------- --------------------------------- ----------
The estimating organization has a Customs did not have a method for Weakness
method for organizing and organizing and retaining
retaining information on information on completed projects
completed projects (a historical (a historical database) at the
database). time that this estimate was
developed. (Note: Customs has
since developed a historical
database which contains planned
versus actual results for
schedule, effort, and cost and
plans to use this in future
estimates.)
The database contains a useful Customs did not have a historical Weakness
set of completed projects. database at the time this
estimate was developed. (Note:
The database developed later
contains NCAP 0.1 and 0.2 data
only.
No data from other completed
projects is included.)
Elements included in (and Customs did not have a historical Weakness
excluded from) the effort, cost, database at the time this
schedule, size, and reuse estimate was developed. Elements
measures in the database are included in the effort, cost, and
clearly identified. (See, for schedule measures in the database
example, the SEI checklists for Customs has since developed are
defining effort, schedule, and clearly identified. However, size
size measures.) and reuse measures are not
identified in the database.
Schedule milestones (start and The schedule milestones (start Strength
finish dates) are described in and finish dates) are accompanied
terms of criteria for initiation by task and subtask descriptions.
or completion, so that work
accomplished between milestones
is clearly bounded.
Records for completed projects Not applicable. According to N/A
indicate whether or not unpaid Customs project officials, unpaid
overtime was used. overtime was insignificant and
did not warrant recording for
NCAP 0.1.
Unpaid overtime, if used, has Not applicable. According to N/A
been quantified, so that recorded Customs project officials, unpaid
data provide a valid basis for overtime was insignificant and
estimating future effort. did not warrant recording for
NCAP 0.1.
Cost models that were used for Customs did not use any cost Weakness
estimating have been used also to models for estimating.
provide consistent frameworks for The spreadsheets that were used
recording historical data. (This for cost and workload estimating
helps ensure that comparable do not provide a consistent
terms and parameters are used framework for recording
across all projects, and that historical data for NCAP releases
recorded data are suitable for 0.1 and 0.2.
use in the estimating models.)
The data in the historical At the time of the estimate, Weakness
database have been examined to there was no historical database.
identify inconsistencies and (Note: The data in the historical
anomalies have been corrected or database prepared later have not
explained. (This is best done yet been examined to identify
with the same cost models used inconsistencies.)
for estimating.)
The organization has a structured Customs has a structured process Strength
process for capturing effort and for capturing effort and cost
cost data from ongoing projects. data from the ACE project on a
weekly basis.
The producing organization holds According to project officials, Weakness
postmortems at the completion of Customs plans to hold postmortems
its projects to (1) ensure that at the completion of each ACE/
recorded data are valid, and (2) NCAP software release. NCAP 0.1
ensure that events that affected was completed but a postmortem
costs get recorded and described has not yet been performed. No
while they are still fresh in documentation exists which
people's minds. describes the postmortem process
or structure.
Information on completed projects According to Customs project Weakness
includes officials, all this information
--the life-cycle model used, has not yet been collected, but
together with the portion covered they plan to do so.
by the recorded cost and
schedule;
--actual (measured) size, cost,
and schedule;
--the actual staffing profile;
--an estimate at completion,
together with the values for cost
model parameters that map the
estimate to the actual cost and
schedule;
--a work breakdown structure or
alternative description of the
tasks included in the recorded
cost;
--a work-flow schematic that
illustrates the software process
used;
--nonlabor costs;
--management costs;
--a summary or list of
significant deliverables
(software and documentation)
produced by the project; and
--a summary of any unusual issues
that affected cost.
Evolution in the organization's Customs' historical database on Weakness
work-flow schematics shows steady completed software projects
improvement in the understanding represents an effort to
and measurement of its software understand and measure its
processes. software processes. Customs
currently has no other ongoing
software process improvement
efforts.
--------------------------------------------------------------------------------
Table II.7
Has the Situation Remained Unchanged
Since the Estimate Was Prepared?
SEI checklist item Finding Rating
--------------------------------- --------------------------------- ----------
The estimate has not been The estimate has been invalidated Weakness
invalidated by recent events, by the following recent actions
changing requirements, or and events: (1) The approved
management action (or inaction). fiscal year 1999 ACE appropriated
funding totals $6 million, which
is significantly less than the
$75.1 million fiscal year 1999
funding requirement stated in the
ACE cost estimate;
(2) NCAP 0.1 was developed in-
house and the cost estimate for
subsequent releases was based on
the assumption that they would be
developed in-house as well.
Customs has since decided to
acquire NCAP 0.2 and perhaps
subsequent releases from a
contractor; and (3) NCAP 0.2 and
subsequent releases will utilize
a combined mainframe and
internet-based client-server
architecture instead of the NCAP
0.1 mainframe architecture upon
which the estimate was based.
The estimate is being used as the The estimate is not being used as Weakness
basis for assigning resources, the basis for
deploying schedules, and making assigning resources, deploying
commitments. schedules, and making
commitments. Instead, Customs is
using a revised estimate that
reflects actual fiscal year 1998
and 1999 ACE appropriated
funding, which is significantly
less than the funding
requirements identified in the
ACE cost estimate for fiscal
years 1998 and 1999.
The estimate is the current The estimate is not the current Weakness
baseline for project tracking and baseline for project tracking and
oversight. oversight. Instead, Customs is
using a revised estimate that
reflects actual fiscal year 1998
and 1999 ACE appropriated
funding, which is significantly
less than the funding
requirements identified in the
ACE cost estimate for fiscal
years 1998 and 1999. Customs is
currently tracking actual project
costs against these appropriated
funds.
--------------------------------------------------------------------------------
RESULTS OF SOFTWARE DEVELOPMENT
CAPABILITY MATURITY MODEL (SW-CMM)
EVALUATION FOR NCAP 0.1
========================================================= Appendix III
Table III.1
Key Process Area: Requirements
Management
Key practice Finding Rating
------------ ------------------------- ------------------------- ------------
Commitment 1 The project follows a There is no written Weakness
written organizational organizational policy for
policy for managing the managing the system
system requirements requirements allocated to
allocated to software. software.
Ability 1 For each project, The Process Analysis and Strength
responsibility is Requirements Team is
established for analyzing responsible for analyzing
the system requirements system requirements and
and allocating them to allocating them to
hardware, software, and hardware, software, and
other system components. other system components.
Ability 2 The allocated Allocated requirements Strength
requirements are are documented.
documented.
Ability 3 Adequate resources and Adequate resources and Strength
funding are provided for funding are provided for
managing the allocated managing the allocated
requirements. requirements.
Ability 4 Members of the software Members of the software Strength
engineering group and engineering group and
other software-related other software-related
groups are trained to groups are trained to
perform their perform their
requirements management requirements management
activities. activities.
Activity 1 The software engineering The software engineering Strength
group reviews the group reviews the
allocated requirements allocated requirements
before they are before they are
incorporated into the incorporated into the
software project. software project.
Activity 2 The software engineering The software engineering Strength
group uses the allocated group uses the allocated
requirements as the basis requirements as the basis
for software plans, work for software plans, work
products, and activities. products, and activities.
Activity 3 Changes to the allocated Changes to the allocated Strength
requirements are reviewed requirements are reviewed
and incorporated into the and incorporated into the
software project. software project.
Measurement Measurements are made and Measurements are made and Strength
1 used to determine the used to determine the
status of the activities status of the activities
for managing the for managing the
allocated requirements. allocated requirements.
Verification The activities for Periodic meetings with Strength
1 managing the allocated senior management include
requirements are reviewed reviews of allocated
with senior management on requirements.
a periodic basis.
Verification The activities for The activities for Strength
2 managing the allocated managing the allocated
requirements are reviewed requirements are reviewed
with the project manager with the project manager
on both a periodic and on both a periodic and
event-driven basis. event-driven basis.
Verification The software quality There is no software Weakness
3 assurance group reviews quality assurance group;
and/or audits the therefore, no reviews
activities and work and/or audits are done.
products for managing the
allocated requirements
and reports the results.
--------------------------------------------------------------------------------
Table III.2
Key Process Area: Software Project
Planning
Key practice Finding Rating
------------ ------------------------- ------------------------- ------------
Commitment 1 A project software The NCAP project has a Strength
manager is designated to software manager
be responsible for designated to be
negotiating commitments responsible for
and developing the negotiating commitments
project's software and developing the
development plan. project's software
development plan.
Commitment 2 The project follows a The project does not Weakness
written organizational follow a written
policy for planning a organizational policy for
software project. planning a software
project.
Ability 1 A documented and approved The approved project plan Strength
statement of work exists meets the requirements
for the software project. for a statement of work
for the project.
Ability 2 Responsibilities for The project manager has Strength
developing the software been assigned
development plan are responsibility for
assigned. developing the software
development plan.
Ability 3 Adequate resources and Adequate resources and Strength
funding are provided for funding have been
planning the software provided for planning the
project. software project.
Ability 4 The software managers, Project personnel are not Weakness
software engineers, and trained in software
other individuals project planning and
involved in the software estimating procedures.
project planning are
trained in the software
estimating and planning
procedures applicable to
their areas of
responsibility.
Activity 1 The software engineering The software engineering Strength
group participates on the group participates on the
project proposal team. project proposal team.
Activity 2 Software project planning Software project planning Strength
is initiated in the early is initiated in the early
stages of, and in stages of, and in
parallel with, the parallel with, the
overall project planning. overall project planning.
Activity 3 The software engineering The software engineering Strength
group participates with group participates with
other affected groups in other affected groups in
the overall project the overall project
planning throughout the planning throughout the
project's life. project's life.
Activity 4 Software project Software project Weakness
commitments made to commitments made to
individuals and groups individuals and groups
external to the external to the
organization are reviewed organization are not
with senior management reviewed with senior
according to a documented management and there is
procedure. no documented procedure
for such reviews.
Activity 5 A software life-cycle There is no documented Weakness
with predefined stages of evidence that a software
manageable size is life-cycle was selected
identified or defined. for the project.
Activity 6 The project's software The project has a Strength
development plan is software development plan
developed according to a that is developed
documented procedure. according to a documented
procedure.
Activity 7 The plan for the software The plan for the software Strength
project is documented. project is documented in
the project plan.
Activity 8 Software work products Software work products Strength
that are needed to that are needed to
establish and maintain establish and maintain
control of the software control of the software
project are identified. project are identified in
the project plan.
Activity 9 Estimates for the size of Estimates for the size of Strength
the software work the software work
products (or changes to products (or changes to
the size of software work the size of software work
products) are derived products) are derived
according to a documented according to a documented
procedure. procedure.
Activity 10 Estimates for the Estimates for the Weakness
software project's effort software project's effort
and costs are derived and costs are not derived
according to a documented according to a documented
procedure. procedure.
Activity 11 Estimates for the Estimates for the Weakness
project's critical project's critical
computer resources are computer resources are
derived according to a not derived according to
documented procedure. a documented procedure.
Activity 12 The project's software There is no documented Weakness
schedule is derived procedure for deriving
according to a documented the software schedule.
procedure.
Activity 13 The software risks The software risks Strength
associated with the cost, associated with the cost,
resource, schedule, and resource, schedule, and
technical aspects of the technical aspects
project are identified, of the project are
assessed, and documented. identified, assessed, and
documented.
Activity 14 Plans for the project's Plans for the project's Strength
software engineering software engineering
facilities and support facilities and support
tools are prepared. tools are prepared.
Activity 15 Software planning data Software planning data Strength
are recorded. are recorded.
Measurement Measurements are made and Measurements are made and Strength
1 used to determine the used to determine the
status of the software status of the software
planning activities. planning activities
(status and MS Project
reports).
Verification The activities for The activities for Strength
1 software project planning software project planning
are reviewed with senior are periodically reviewed
management on a periodic with senior management,
basis. including reports to
Treasury and Customs
investment review board
(IRB).
Verification The activities for The activities for Strength
2 software project planning software project planning
are reviewed with the are reviewed with the
project manager on both a project manager on both a
periodic and event- periodic and event-
driven basis. driven basis through
weekly status reports and
meetings.
Verification The software quality There is no SQA group. Weakness
3 assurance group reviews
and/or audits the
activities and work
products for software
project planning and
reports the results.
--------------------------------------------------------------------------------
Table III.3
Key Process Area: Software Project
Tracking and Oversight
Key practice Finding Rating
------------ ------------------------- ------------------------- ------------
Commitment 1 A project software The project software Strength
manager is designated to manager is designated to
be responsible for the be responsible for the
project's software project's software
activities and results. activities and results.
Commitment 2 The project follows a There is no written Weakness
written organizational organizational policy for
policy for managing the managing the software
software project. project.
Ability 1 A software development An approved and Strength
plan for the software documented software
project is documented and development plan is
approved. contained in the project
plan.
Ability 2 The project software The project software Strength
manager explicitly manager explicitly
assigns responsibility assigns responsibility
for software work for software work
products and activities. products and activities.
Ability 3 Adequate resources and Adequate resources and Strength
funding are provided for funding are provided for
tracking the software tracking the software
project. project.
Ability 4 The software managers are The software managers are Weakness
trained in managing the not trained in managing
technical and personnel the technical and
aspects of the software personnel aspects of the
project. software project.
Ability 5 First-line software First-line software Strength
managers receive managers receive
orientation in the orientation in the
technical aspects of the technical aspects of the
software project. software project.
Activity 1 A documented software A documented software Strength
development plan is used development plan
for tracking the software is used for tracking the
activities and software activities and
communicating status. communicating status.
Activity 2 The project's software No documented procedure Weakness
development plan is exists for
revised according to a revising the software
documented procedure. development plan.
Activity 3 Software project Software project Weakness
commitments and changes commitments and changes
to commitments made to to commitments made to
individuals and groups individuals and groups
external to the external to the
organization are reviewed organization are not
with senior management reviewed with senior
according to a documented management. Also, there
procedure. is no documented
procedure for such
reviews.
Activity 4 Approved changes to Approved changes to Strength
commitments that affect commitments that affect
the software project are the software project are
communicated to the communicated to the
members of the software members of the software
engineering group and engineering group and
other software-related other software-related
groups. groups through weekly
staff meetings.
Activity 5 The sizes of the software The size of the software Strength
work products (or sizes work products (or size of
of the changes to the the changes to the
software work products) software work products)
are tracked, and are tracked; however, at
corrective actions are the time of our
taken as necessary. evaluation, no corrective
actions were needed.
Activity 6 The project's software The project's software Strength
effort and costs are effort and costs are
tracked and corrective tracked; however, at the
actions are taken as time of our evaluation,
necessary. no corrective actions
were needed.
Activity 7 The project's critical The project's critical Strength
computer resources are computer resources are
tracked, and corrective tracked; however, at the
actions are taken as time of our evaluation,
necessary. no corrective actions
were needed.
Activity 8 The project's software The project's software Strength
schedule is tracked, and schedule is tracked;
corrective actions are however, at the time of
taken as necessary. our evaluation, no
corrective actions were
needed.
Activity 9 Software engineering Software engineering Strength
technical activities are technical activities are
tracked, and corrective tracked, and corrective
actions are taken as actions are taken as
necessary. necessary.
Activity 10 The software risks The software risks Strength
associated with cost, associated with cost,
resource, schedule, and resource, schedule, and
technical aspects technical aspects
of the project are of the project are
tracked. tracked.
Activity 11 Actual measurement data Actual measurement data Strength
and replanning data for and replanning data for
the software project are the software project are
recorded. recorded.
Activity 12 The software engineering The software engineering Strength
group conducts periodic group conducts periodic
internal reviews to track internal reviews to track
technical progress, technical progress,
plans, performance, and plans, performance, and
issues against the issues against the
software development software development
plan. plan.
Activity 13 Formal reviews to address Formal reviews to address Strength
the accomplishments and the accomplishments and
results of the software results of the software
project are conducted at project are conducted at
selected project selected project
milestones according to a milestones according to a
documented procedure. documented procedure.
Measurement Measurements are made and Measurements are made and Strength
1 used to determine the used to determine the
status of the software status of the software
tracking and oversight tracking and oversight
activities. activities.
Verification The activities for The activities for Strength
1 software project tracking software project tracking
and oversight are and oversight are
reviewed with senior reviewed with senior
management periodically. management on a weekly
basis.
Verification The activities for The activities for Strength
2 software project tracking software project tracking
and oversight are and oversight are
reviewed with the project reviewed with the project
manager on both a manager on both a
periodic and event- periodic and event-
driven basis. driven basis.
Verification The software quality No software quality Weakness
3 assurance group reviews assurance group exists;
and/or audits the therefore, there are no
activities and work reviews and/or audits of
products for software the activities and work
project tracking and products for software
oversight and reports the project tracking and
results. oversight.
--------------------------------------------------------------------------------
Table III.4
Key Process Area: Software Quality
Assurance
Key practice Finding Rating
------------ ------------------------- ------------------------- ------------
Commitment 1 The project follows a There is no written Weakness
written organizational organizational policy for
policy for implementing implementing SQA.
software quality
assurance (SQA).
Ability 1 A group that is Although there is no Observation
responsible for group responsible for
coordinating and coordinating and
implementing SQA for the implementing SQA for the
project (the SQA group) project, there are plans
exists. to establish one and
assign responsibility.
Ability 2 Adequate resources and There is no SQA group, Weakness
funding are provided for and no resources and
performing the SQA funding are provided for
activities. performing SQA
activities.
Ability 3 Members of the SQA group There is no SQA group or Weakness
are trained to perform plan to provide SQA
their SQA activities. training.
Ability 4 The members of the Project staff do not Weakness
software project receive receive orientation on
orientation on the role, the role,
responsibilities, responsibilities,
authority, and value of authority, and value of
the SQA group. the SQA group.
Activity 1 The SQA plan is prepared There is no SQA plan or Weakness
for the software project documented procedure for
according to a documented preparing one.
procedure.
Activity 2 The SQA group's There is no SQA group. Weakness
activities are performed
in accordance with the
SQA plan.
Activity 3 The SQA group There is no SQA group. Weakness
participates in the
preparation and review of
the project's software
development plan,
standards, and
procedures.
Activity 4 The SQA group reviews the There is no SQA group. Weakness
software engineering
activities to verify
compliance.
Activity 5 The SQA group audits There is no SQA group. Weakness
designated software work
products to verify
compliance.
Activity 6 The SQA group There is no SQA group. Weakness
periodically reports the
results of its activities
to the software
engineering group.
Activity 7 Deviations identified in Procedures for handling Weakness
the software activities deviations in
and software work testing activities are
products are documented documented. However,
and handled according to procedures for handling
a documented procedure. deviations in other
software development
activities (such as
compliance with
organizational policy and
standards and adherence
to software development
plan) are not documented.
Activity 8 The SQA group conducts There is no SQA group. Weakness
periodic reviews of its
activities and findings
with the customer's SQA
personnel, as
appropriate.
Measurement Measurements are made and There is no SQA group. Weakness
1 used to determine the
cost and schedule status
of the SQA activities.
Verification The SQA activities are There is no SQA group. Weakness
1 reviewed with senior
management on a periodic
basis.
Verification The SQA activities are There is no SQA group. Weakness
2 reviewed with the project
manager on both a
periodic and event-
driven basis.
Verification Experts independent of There is no SQA group. Weakness
3 the SQA group
periodically review the
activities and
software work products of
the project's SQA group.
--------------------------------------------------------------------------------
Table III.5
Key Process Area: Software Configuration
Management
Key practice Finding Rating
------------ ------------------------- ------------------------- ------------
Commitment 1 The project follows a The project has no Weakness
written organizational organizational policy for
policy for implementing implementing SCM.
software configuration
management (SCM).
Ability 1 A board having the The project does not have Weakness
authority for managing a SCM board with
the project's software authority for managing
baselines (i.e., a software baselines.
software configuration
control board) exists or
is established.
Ability 2 A group that is The project does not have Weakness
responsible for a group that is
coordinating and responsible for
implementing SCM for the coordinating and
project (i.e., the SCM implementing SCM
group) exists. functions.
Ability 3 Adequate resources and Adequate resources and Weakness
funding are provided for funding are not provided
performing the SCM for performing the SCM
activities. activities.
Ability 4 Members of the SCM group There is no SCM group. Weakness
are trained in the
objectives, procedures,
and methods for
performing their SCM
activities.
Ability 5 Members of the software Members of the software Weakness
engineering group and engineering group and
other software-related other software-related
groups are trained to groups are not trained to
perform their SCM perform their SCM
activities. activities.
Activity 1 A SCM plan is prepared A SCM plan was prepared Strength
for each software project according to procedures
according to a documented documented in the October
procedure. 1996 SDLC.
Activity 2 A documented and approved The documented and Weakness
SCM plan is used as the approved SCM plan is used
basis for performing the as the basis for
SCM activities. performing code control;
however, the plan is not
used as a basis for doing
SCM on software
documentation and other
software engineering
products such as cost
estimates and schedules.
Activity 3 A configuration A configuration Strength
management library system management library system
is established as a is established as a
repository for the repository for the
software baselines. software baselines.
Activity 4 The software work The software work Strength
products to be placed products to be placed
under configuration under configuration
management are management are
identified. identified.
Activity 5 Change requests and Change requests and Strength
problem reports for all problem items/units are
configuration items/ initiated, recorded,
risks are initiated, reviewed, approved, and
recorded, reviewed, tracked according to a
approved, and tracked procedure documented in
according to a documented the SCM plan.
procedure.
Activity 6 Changes to baselines are Changes to baselines are Strength
controlled according to a controlled according to a
documented procedure. documented procedure in
the SCM plan.
Activity 7 Products from the Products from the Strength
software baseline library software baseline library
are created, and their are created, and their
release is controlled release is controlled
according to a documented according to the SCM
procedure. plan.
Activity 8 The status of software The status of SCM items/ Weakness
configuration items/ units are not recorded
units is recorded according to a documented
according to a documented procedure.
procedure.
Activity 9 Standard reports Standard reports Weakness
documenting SCM documenting SCM
activities and the activities and contents
contents of the software of the software baseline
baseline are developed are not developed.
and made available to
affected groups and
individuals.
Activity 10 Software baseline audits Software baseline audits Weakness
are conducted according are not conducted.
to a documented
procedure.
Measurement Measurements are made and No measurements are taken Weakness
1 used to determine the to determine the status
status of SCM activities. of SCM activities.
Verification SCM activities are SCM activities are not Weakness
1 reviewed with senior reviewed with senior
management periodically. management periodically.
Verification SCM activities are SCM activities are not Weakness
2 reviewed with the project reviewed with the project
manager on both a manager on both a
periodic and periodic and event-
event-driven basis. driven basis.
Verification SCM group periodically No SCM audits are done to Weakness
3 audits software baselines verify that software
to verify that they baselines conform to the
conform to the documentation that
documentation that defines them.
defines them.
Verification The software quality There is no quality Weakness
4 assurance group reviews assurance group;
and/or audits the therefore, no one reviews
activities and work and/or audits the
products for SCM and activities and work
reports the results. products for SCM
activities performed.
--------------------------------------------------------------------------------
RESULTS OF SOFTWARE ACQUISITION
CAPABILITY MATURITY MODEL (SA-CMM)
EVALUATION FOR NCAP 0.2
========================================================== Appendix IV
Table IV.1
Key Process Area: Software Acquisition
Planning
Key practice Finding Rating
---------- --------------------------- --------------------------- ----------
Commitment The acquisition There is no written policy Weakness
1 organization has a written for planning the software
policy for planning the acquisition.
software acquisition.
Commitment Responsibility for software Responsibility for software Weakness
2 acquisition planning acquisition planning is not
activities is designated. designated.
Ability 1 The acquisition The acquisition Weakness
organization has organization does not have
experienced software experienced software
acquisition management acquisition management
personnel. personnel.
Ability 2 Adequate resources are Resources for software Weakness
provided for software acquisition planning are
acquisition planning not adequate.
activities.
Activity 1 Software acquisition Software acquisition Weakness
planning personnel are planning personnel are not
involved in system involved in system
acquisition planning. acquisition planning.
Activity 2 The software acquisition A software acquisition Weakness
strategy for the strategy for NCAP 0.2 was
project is developed and not developed and
documented. documented.
Activity 3 The project's software Software acquisition Weakness
acquisition planning is planning for NCAP 0.2 has
documented, and the not been documented.
planning documentation
is maintained over the life
of the project.
Activity 4 Life-cycle support of the Life-cycle support of the Weakness
software is included in software has not been
software acquisition included in any document.
planning documentation.
Activity 5 Life-cycle cost and Life-cycle cost and Observatio
schedule estimates for the schedule estimates were n
software products and prepared, but there is no
services being acquired are evidence that they were
prepared and independently independently reviewed.
reviewed.
Measuremen Measurements are made and Software acquisition Weakness
t 1 used to determine the planning activities are not
status of the software measured.
acquisition planning
activities and resultant
products.
Verificati Software acquisition Software acquisition Weakness
on 1 planning activities are planning activities are not
reviewed by acquisition reviewed by management.
organization management
periodically.
Verificati Software acquisition There is no documented Weakness
on 2 planning activities are evidence that the project
reviewed by the project manager reviewed software
manager on both a periodic acquisition planning
and event-driven basis. activities.
--------------------------------------------------------------------------------
Table IV.2
Key Process Area: Solicitation
Key practice Finding Rating
--------- ----------------------------- ----------------------------- -------
Commitmen The acquisition organization There is no written policy Weaknes
t 1 has a written policy for the for the conduct of the s
conduct of the software software portion of the
portion of the solicitation. solicitation.
Commitmen Responsibility for the Responsibility for the Strengt
t 2 software portion of the software portion of the h
solicitation is designated. solicitation was designated
to the project manager/
contract officer's technical
representative (COTR).
Commitmen A selection official has been A selection official has been Strengt
t 3 designated to be responsible designated. h
for the selection process and
the decision.
Ability 1 A group that is responsible There is no documented Weaknes
for coordinating and evidence that a group s
conducting solicitation responsible for coordinating
activities exists. and conducting solicitation
activities exists.
Ability 2 Adequate resources are Resources for solicitation Weaknes
provided for solicitation activities are not adequate. s
activities.
Ability 3 Individuals performing There is only one individual Weaknes
solicitation activities have performing solicitation s
experience or receive activities. Although he was
training. trained as a COTR, he has no
experience or training in
acquisition management or
costing methodologies and
tools.
Ability 4 The groups supporting the No orientation is provided to Weaknes
solicitation (e.g., any other groups. End users, s
end user, systems system engineers, and
engineering, software support software support
organization, and application organizations did not
domain experts) receive participate in the
orientation on the solicitation.
solicitation's objectives and
procedures.
Activity The project team performs its There are no documented Weaknes
1 activities in accordance with solicitation plans to guide s
its documented solicitation the performance of
plans. solicitation activities.
Activity The project team performs its There are no documented Weaknes
2 activities in accordance with proposal evaluation plans. s
its documented proposal
evaluation plans.
Activity Cost and schedule estimates Cost and schedule estimates Strengt
3 for the software products and for the software products and h
services being sought are services being sought are
prepared. prepared.
Activity Software cost and schedule Software cost and schedule Weaknes
4 estimates are independently estimates are not s
reviewed for independently reviewed.
comprehensiveness and
realism.
Activity The project team takes action The project team takes action Strengt
5 to ensure the mutual to ensure the mutual h
understanding of software understanding of software
requirements and plans prior requirements and plans prior
to contract award. to contract award.
Measureme Measurements are made and No measurements were made to Weaknes
nt 1 used to determine the status of s
determine the status of the solicitation activities and
solicitation activities and resultant products.
resultant products.
Verificat Solicitation activities are Solicitation activities were Weaknes
ion 1 reviewed periodically by the not reviewed by the s
designated selection official designated selection official
or acquisition organization or acquisition organization
management. management on a periodic
basis.
Verificat Solicitation activities are Solicitation activities are Weaknes
ion 2 reviewed by the project not reviewed by the project s
manager on both a periodic manager on both a periodic
and event-driven basis. and event-driven basis.
--------------------------------------------------------------------------------
Table IV.3
Key Process Area: Requirements
Development and Management
Key practice Finding Rating
---------- --------------------------- --------------------------- ----------
Commitment The acquisition There is no written policy Weakness
1 organization has a written for establishing and
policy for establishing and managing the software-
managing the software- related contractual
related contractual requirements.
requirements.
Commitment Responsibility for Responsibility for Strength
2 requirements development requirements development
and management is and management is
designated. designated to the Process
Analysis and Requirements
Team (PART).
Ability 1 A group that is responsible The PART team is Strength
for performing requirements responsible for performing
development and management requirements development
activities exists. and management.
Ability 2 Adequate resources are The PART team has adequate Strength
provided for requirements resources for requirements
development and management development and management
activities. activities.
Ability 3 Individuals performing Individuals performing Strength
requirements development requirements development
and management activities and management activities
have experience or receive have experience or receive
training. training.
Activity 1 The project team performs There are no documented Weakness
its activities in requirements development
accordance with its and management plans.
documented requirements
development and management
plans.
Activity 2 The project team develops The project team develops Strength
and baselines the software- and baselines the software-
related contractual related contractual
requirements and places requirements and places
them under change control them under change control
early in the project, but early in the project, but
not later than release of not later than release of
the solicitation package. the solicitation package.
Activity 3 The project team appraises To date there have been no Observatio
system requirements change system requirements change n
requests for their impact requests.
on the software being
acquired.
Activity 4 The project team appraises To date there have been no Observatio
all changes to the changes to the software- n
software-related related contractual
contractual requirements requirements.
for their impact on
performance, architecture,
supportability, system
resource utilization, and
contract schedule and cost.
Activity 5 Bidirectional traceability Bidirectional traceability Weakness
between the software- between the software-
related contractual related contractual
requirements and the requirements and the
contractor's software work contractor's software work
products and services is products and services is
maintained throughout the not maintained throughout
effort. the effort.
Measuremen Measurements are made and Measurements are made and Strength
t 1 used to determine the used to determine the
status of the requirements status of the requirements
development and management development and management
activities and resultant activities and resultant
products. products.
Verificati Requirements development Requirements development Strength
on 1 and management activities and management activities
are reviewed periodically are reviewed periodically
by acquisition organization by acquisition management
management (and the (and the contractor).
contractor).
Verificati Requirements development Requirements development Weakness
on 2 and management activities and management activities
are reviewed by the project are not reviewed by the
manager on both a periodic project manager.
and event-driven basis.
--------------------------------------------------------------------------------
Table IV.4
Key Process Area: Project Management
Key practice Finding Rating
---------- --------------------------- --------------------------- ----------
Commitment The acquisition There is no written policy Weakness
1 organization has a written for execution of the
policy for execution of the software project.
software project.
Commitment Responsibility for project Responsibility for project Strength
2 management is designated. management has been
designated.
Ability 1 A team that is responsible A team that is responsible Strength
for performing the for performing the
project's software project's software
acquisition management acquisition management
activities exists. activities exists.
Ability 2 Adequate resources for the Adequate resources for the Weakness
project team and matrix project team and matrix
support persons are support persons are not
provided for the duration provided for the duration
of the software acquisition of the software acquisition
project. project.
Ability 3 When project trade-offs are The project manager, who is Strength
necessary, the project also the contracting
manager is permitted to officer's technical
alter either the representative, is
performance, cost, or permitted to alter either
schedule software the performance, cost, or
acquisition baseline. schedule software
acquisition baseline.
Ability 4 The project team and matrix The project team members do Weakness
support individual(s) have not have experience and
experience or receive have not received training
training in project in project software
software acquisition acquisition management
management activities. activities.
Activity 1 The project team performs There is no software Weakness
its activities in acquisition management
accordance with its plan; therefore, the
documented software project team does not
acquisition management perform its activities in
plans. accordance with a plan.
Activity 2 The organization of the The organization of the Weakness.
project provides for the project does not provide
management of all project for the management of all
functions. the project's functions.
For example, acquisition
planning and risk
management are not
provided.
Activity 3 The software acquisition The software acquisition Weakness
management activities of management activities of
the project team are the project team are not
directed to accomplish the directed to accomplish the
project's objectives. project's objectives.
Activity 4 The software acquisition The activities of the Weakness.
management activities of project team are not
the project team are controlled.
controlled.
Activity 5 The project team implements There is no corrective Weakness
a corrective action system action system for the
for the identification, identification, recording,
recording, tracking, and tracking, and correction of
correction of problems problems discovered during
discovered during the the software acquisition.
software acquisition.
Activity 6 The project team tracks The project team tracks Observatio
project status, execution, project status, execution, n
funding, and expenditures funding, and expenditures.
and takes action. At the time of our
evaluation, however, no
actions
were needed.
Measuremen Measurements are made and Measurements are taken and Strength
t 1 used to used to determine the
determine the status of the status of project
project management management activities and
activities and resultant resultant products.
products.
Verificati Project management Project management Weakness
on 1 activities are reviewed by activities are not reviewed
acquisition organization by acquisition organization
management periodically. management periodically.
Verificati Project management There was no evidence Weakness
on 2 activities are reviewed by provided to show that the
the project manager on both project management
a periodic and event- activities are reviewed by
driven basis. the project manager.
--------------------------------------------------------------------------------
Table IV.5
Key Process Area: Contract Tracking and
Oversight Findings
Key practice Finding Rating
---------- --------------------------- --------------------------- ----------
Commitment The acquisition There is no organizational Weakness
1 organization has a written policy for the
policy for the contract contract tracking and
tracking and oversight of oversight of the contracted
the contracted software software.
effort.
Commitment Responsibility for contract Responsibility for contract Strength
2 tracking and oversight tracking and oversight has
activities is designated. been designated to the
project manager/COTR.
Commitment The project team is The contracting officer Strength
3 supported by contracting supports the project team
specialists in the in the execution of the
execution of the contract. contract.
Ability 1 A group that is responsible The project team is Strength
for managing contract responsible for contract
tracking and oversight tracking and oversight
activities exists. activities.
Ability 2 Adequate resources are Adequate resources are not Weakness
provided for contract provided for contract
tracking and oversight tracking and oversight
activities. activities.
Ability 3 Individuals performing The project manager has Strength
contract tracking and received training
oversight activities have conducting his duties as a
experience or receive COTR.
training.
Activity 1 The project team performs There is no contract Weakness
its activities in tracking and oversight
accordance with its plan.
documented contract
tracking and oversight
plans.
Activity 2 The project team reviews The contractor was not Weakness
required contractor required to submit any
software planning documents software planning documents
which, when satisfactory, that could be used to
are used to oversee the oversee the contractor's
contractor's software software engineering
engineering effort. effort.
Activity 3 The project team conducts The project team conducts Strength
periodic reviews periodic reviews and
and interchanges with the interchanges with the
contractor. contractor.
Activity 4 The project team reviews Customs provided the Not rated
and tracks the development software engineering
of the software engineering environment to the
environment required to contractor; therefore,
provide life-cycle support there was no need to track
for the acquired software. the development of the
software engineering
environment required to
provide life-cycle support
for the acquired software.
Activity 5 Any problems or issues Problems or issues found by Weakness
found by the project team the project team during
during contract tracking contract tracking and
and oversight are recorded oversight are not tracked.
in the appropriate
corrective action system
and tracked to closure.
Activity 6 The project team maintains The project team maintains Strength
the integrity of the the integrity of the
contract throughout the contract throughout the
contract performance contract performance
period. period.
Measuremen Measurements are made and Measurement are not made to Weakness
t 1 used to determine the determine the status of
status of the contract contract tracking and
tracking and oversight oversight activities and
activities and resultant resultant products.
products.
Verificati Contract tracking and Contract tracking and Weakness
on 1 oversight activities are oversight activities are
reviewed by acquisition not reviewed by the
organization management acquisition organization
periodically. management.
Verificati Contract tracking and Contract tracking and Weakness
on 2 oversight activities are oversight activities are
reviewed by the project not reviewed by the project
manager on both a periodic manager.
and event-driven basis.
--------------------------------------------------------------------------------
Table IV.6
Key Process Area: Evaluation
Key practice Finding Rating
---------- --------------------------- --------------------------- ----------
Commitment The acquisition There is no written policy Weakness
1 organization has a written for managing the evaluation
policy for managing the of the acquired software
evaluation of the acquired products and services.
software products and
services.
Commitment Responsibility for Responsibility for Strength
2 evaluation activities is evaluation activities is
clearly designated. clearly designated.
Ability 1 A group that is responsible A group that is responsible Strength
for planning, managing, and for planning, managing, and
performing evaluation performing evaluation
activities for the project activities for the project
exists. exists.
Ability 2 Adequate resources are We found no evidence that a Observatio
provided for evaluation lack of resources precluded n
activities. the performance of
evaluation activities.
Ability 3 Individuals performing Individuals performing Strength
evaluation activities have evaluation activities have
experience or receive experience or receive
training. training.
Ability 4 Members of the project team No orientation is provided Weakness
and groups supporting the to members of the project
software acquisition team and groups supporting
receive orientation on the the software acquisition.
objectives of the
evaluation approach.
Activity 1 The project team performs There are no documented Weakness
its activities in evaluation plans.
accordance with its
documented evaluation
plans.
Activity 2 The project's evaluation The project's evaluation Weakness
requirements are developed requirements are not
in conjunction with the developed in conjunction
development of the system with the development of the
or software technical system or software
requirements. technical requirements.
Activity 3 The evaluation requirements Some evaluation Weakness
are incorporated into the requirements have been
solicitation package and incorporated into the
resulting contract. solicitation package and
resulting contract. Others,
such as a mechanism that
provides the project team
visibility into the
contractor's evaluation
program and requirements
for the contractor to
establish a corrective
action system, have not.
Activity 4 The project team assesses The project team does not Weakness
contractor's performance assess the contractor's
for compliance with performance for compliance
evaluation requirements. with evaluation
requirements.
Activity 5 Planned evaluations are No products or services Observatio
performed on the acquired have yet been accepted for n
software products and operational use. The
services prior to project team plans to
acceptance for operational evaluate acquired software
use. products and services prior
to acceptance for
operational use.
Activity 6 Results of the evaluations No products or services Observatio
are analyzed and compared have yet been delivered by n
to the contract's the contractor. The project
requirements to establish team plans to analyze the
an objective basis to results of the evaluations
support the decision to and compare the results to
accept the products and contract requirements as an
services objective basis to support
or to take further action. the decision to accept the
products and services or to
take further action.
Measuremen Measurements are made and No measurements are made to Weakness
t 1 used to determine the determine the status of
status of the evaluation evaluation activities and
activities and resultant resultant products.
products.
Verificati Evaluation activities are Evaluation activities are Weakness
on 1 reviewed by acquisition not reviewed by acquisition
organization management organization management.
periodically.
Verificati Evaluation activities are Evaluation activities are Weakness
on 2 reviewed by the project not reviewed by the project
manager on both a periodic manager.
and event-driven basis.
--------------------------------------------------------------------------------
Table IV.7
Key Process Area: Transition to Support
Key practice Finding Rating
---------- --------------------------- --------------------------- ----------
Commitment The acquisition There is no policy for the Weakness
1 organization has a written transitioning of software
policy for transitioning products to the software
software products to the support organization.
software support
organization.
Commitment The acquisition A software support Weakness
2 organization ensures that organization has not been
the software support established.
organization is involved in
planning for transition to
support.
Commitment Responsibility for Responsibility for Weakness
3 transition to support transition to support
activities is designated. activities has not been
designated.
Ability 1 A group that is responsible No group is responsible for Weakness
for coordinating the coordinating transition to
transition to support support activities.
activities exists.
Ability 2 Adequate resources are Adequate resources have not Weakness
provided for transition to been provided
support activities. for transition to support
activities.
Ability 3 The organization The organization Weakness
responsible for providing responsible for providing
support of the software support of the software
products is identified no products was not identified
later than initiation of before initiation of the
the solicitation package's solicitation package's
development. development.
Ability 4 The software support At the time of the audit, Not rated
organization, prior to the system was still in
transition, has a complete development and had not
inventory of all software reached the
and related items that are transition stage.
to be transitioned.
Ability 5 Individuals performing Individuals responsible for Weakness
transition to support transition to support
activities have experience activities have not been
or receive training. selected.
Ability 6 The members of The members of Weakness
organizations interfacing organizations interfacing
with the transition to with the transition to
support activities receive support activities have not
orientation on the salient received orientation on the
aspects of transition to salient aspects of
support activities. transition to support
activities.
Activity 1 The project team performs There are no transition to Weakness
its activities in support plans.
accordance with its
documented transition to
support plans.
Activity 2 Responsibility for the No products have been Not rated
software products is transferred yet.
transferred only after the
software support
organization demonstrates
its capability to modify
and support the software
products.
Activity 3 The project team oversees Transition has not yet Not rated
the configuration control occurred.
of the software products
throughout the transition.
Measuremen Measurements are made and There are no plans to take Weakness
t 1 used to measurements to determine
determine the status of the the status of the
transition to support transition to support
activities and resultant activities and resultant
products. products.
Verificati Transition to support There are no plans for Weakness
on 1 activities are reviewed by acquisition and software
acquisition and software support organizations'
support organizations' managements to review
managements periodically. transition to support
activities periodically.
Verificati Transition to support No evidence was provided Weakness
on 2 activities are reviewed by that transition to support
the project manager on both activities are reviewed by
a periodic and event- the project manager.
driven basis.
--------------------------------------------------------------------------------
Table IV.8
Key Process Area: Acquisition Risk
Management
Key practice Finding Rating
---------- --------------------------- --------------------------- ----------
Commitment The acquisition There is no policy for the Weakness
1 organization has a written management of software
policy for the management acquisition risk.
of software acquisition
risk.
Commitment Responsibility for software Responsibility for software Weakness
2 acquisition risk management acquisition risk management
activities is designated. activities is not
designated.
Ability 1 A group that is responsible No group responsible for Weakness
for coordinating software coordinating software
acquisition risk management acquisition risk management
activities exists. activities exists.
Ability 2 Adequate resources are Adequate resources are not Weakness
provided for software provided for software
acquisition risk management acquisition risk management
activities. activities.
Ability 3 Individuals performing No individual or group is Weakness
software acquisition risk performing software
management activities have acquisition risk
experience or receive management.
required training.
Activity 1 Software acquisition risk Software acquisition risk Weakness
management activities are management activities are
integrated into software not integrated into
acquisition planning. software acquisition
planning.
Activity 2 The software acquisition There is no software Weakness
risk management plan is acquisition risk management
developed in accordance plan.
with the project's defined
software acquisition
process.
Activity 3 The project team performs No documented software Weakness
its software acquisition acquisition risk management
risk management activities plans exists. No risk
in accordance with its management activities are
documented plans. being performed.
Activity 4 Risk management is Risk management is not Weakness
conducted as an integral conducted as an integral
part of the solicitation, part of the solicitation,
project performance project performance
management, and contract management, and contract
performance management performance management
processes. processes.
Activity 5 Software acquisition risk There is no record of Weakness
handling actions are tracking and controlling
tracked and controlled software acquisition risk
until the risks are handling actions.
mitigated.
Measuremen Measurements are made and Measurements are not made Weakness
t 1 used to and used to determine the
determine the status of the status of the acquisition
acquisition risk management risk management activities
activities and resultant and resultant products.
products.
Verificati Acquisition risk management There are no acquisition Weakness
on 1 activities are reviewed by risk management activities
acquisition organization for management to review.
management periodically.
Verificati Acquisition risk management There are no acquisition Weakness
on 2 activities are reviewed by risk management activities
the project manager on both for the project manager to
a periodic and event- review.
driven basis.
--------------------------------------------------------------------------------
MAJOR CONTRIBUTORS TO THIS REPORT
=========================================================== Appendix V
ACCOUNTING AND INFORMATION
MANAGEMENT DIVISION, WASHINGTON,
D.C.
Dr. Rona Stillman, Chief Scientist for Computers and
Telecommunications
Jack L. Brock, Jr., Director, Governmentwide and Defense Information
Systems
Mark T. Bird, Assistant Director
Madhav S. Panwar, Technical Assistant Director
Dr. Nabajyoti Barkakati, Technical Assistant Director
Prithviraj Mukerji, Assistant Director
Carl M. Urie, Assistant Director
Bernard R. Anderson, Senior Information Systems Analyst
Suzanne M. Burns, Senior Information Systems Analyst
Timothy D. Hopkins, Senior Information Systems Analyst
Ona M. Noble, Senior Information Systems Analyst
Karen A. Richey, Senior Information Systems Analyst
Cristina T. Chaplain, Communications Analyst
OFFICE OF THE CHIEF ECONOMIST,
WASHINGTON, D.C.
Harold J. Brumm, Economist
ATLANTA FIELD OFFICE
Teresa F. Tucker, Senior Information Systems Analyst
*** End of document. ***