Information Technology: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule Problems Needs to Be
Addressed (14-MAY-04, GAO-04-719).
The Department of Homeland Security's (DHS) Bureau of Customs and
Border Protection (CBP) is conducting a multiyear,
multibillion-dollar acquisition of a new trade processing system
planned to support the movement of legitimate imports and exports
and strengthen border security. By congressional mandate,
expenditure plans for this system, called the Automated
Commercial Environment (ACE), must meet certain conditions,
including GAO review. This study addresses the extent to which
the latest plan, for fiscal year 2004, satisfies these
conditions, provides information about DHS's efforts to implement
GAO's recommendations for improving ACE management, and makes
observations about ACE.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-04-719
ACCNO: A10067
TITLE: Information Technology: Early Releases of Customs Trade
System Operating, but Pattern of Cost and Schedule Problems Needs
to Be Addressed
DATE: 05/14/2004
SUBJECT: Budget activities
Trade policies
Transportation safety
Export regulation
Exporting
Import regulation
Import restriction
Importing
DHS Visitor and Immigrant Status
Indicator Technology Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-04-719
United States General Accounting Office
GAO Report to Congressional Committees
May 2004
INFORMATION
TECHNOLOGY
Early Releases of Customs Trade System Operating, but Pattern of Cost and
Schedule Problems Needs to Be Addressed
a
GAO-04-719
Highlights of GAO-04-719, a report to the Subcommittees on Homeland
Security, Senate and House Committees on Appropriations
The Department of Homeland Security's (DHS) Bureau of Customs and Border
Protection (CBP) is conducting a multiyear, multibillion-dollar
acquisition of a new trade processing system planned to support the
movement of legitimate imports and exports and strengthen border security.
By congressional mandate, expenditure plans for this system, called the
Automated Commercial Environment (ACE), must meet certain conditions,
including GAO review. This study addresses the extent to which the latest
plan, for fiscal year 2004, satisfies these conditions, provides
information about DHS's efforts to implement GAO's recommendations for
improving ACE management, and makes observations about ACE.
To assist DHS in managing ACE and increasing the chances that future
releases will deliver promised capabilities on time and within budget, GAO
is making recommendations to the Secretary of Homeland Security aimed at
addressing recurring cost and schedule problems. DHS concurred with GAO's
recommendations and described specific actions that it is taking to
respond to each.
www.gao.gov/cgi-bin/getrpt?GAO-04-719
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Randolph C. Hite at (202)
512-3439 or [email protected].
May 2004
INFORMATION TECHNOLOGY
Early Releases of Customs Trade System Operating, but Pattern of Cost and
Schedule Problems Needs to Be Addressed
The DHS fiscal year 2004 ACE expenditure plan provides for certain
activities, including system implementation infrastructure and support,
operations and maintenance, and the definition and design of two future
releases. This plan, including related program documentation and program
officials' statements, largely satisfies the legislative conditions
imposed by the Congress. Some of the recommendations that GAO has
previously made to strengthen ACE management have been addressed, and DHS
has committed to addressing those that remain. However, much remains to be
done before these recommendations are fully implemented. Specifically,
progress on overcoming human capital challenges has been slow.
GAO made several observations about ACE. Specifically, the first two ACE
releases, which have been deployed, are operating largely as intended.
However, achieving this initial operating capability has been difficult.
DHS has established a pattern of borrowing resources from future releases
to address problems of near-term releases, which has adversely affected
the cost, schedule, and capability commitments of ongoing releases. In
particular, the delay in completing the second ACE release has introduced
a pattern of increased reliance on concurrent activities to meet the
dictated schedule, continued release schedule delays, and cost overruns
for the ongoing releases, as the figure below indicates. This domino
effect is continuing into Release 3 and beyond. This pattern is not likely
to change unless the degree of concurrence among activities within and
between releases is better controlled, which will require that the reasons
for release quality problems that led to the concurrent activity be
addressed. Until the reasons for this pattern are pinpointed and
corrected, in part through implementation of GAO's unaddressed
recommendations, DHS will not be positioned to deliver promised
capabilities on time and within budget, or produce mission value
commensurate with investment cost.
Cost Overruns and Schedule Delays, Releases 1-4
Contents
Letter 1
Compliance with Legislative Conditions 2
Status of Open Recommendations 2
Observations on Management of ACE 5
Conclusions 7
Recommendations for Executive Action 7
Agency Comments and Our Evaluation 8
Appendixes
Appendix I:
Appendix II: Appendix III:
IT: Early Releases of Customs Trade System Operating, but
Pattern of Cost and Schedule Problems Needs to be
Addressed 10
Comments from the U.S. Department of Homeland
Security 103
GAO Contact and Staff Acknowledgments 107
GAO Contact 107
Staff Acknowledgments 107
Contents
Abbreviations
ACE Automated Commercial Environment
CBP Customs and Border Protection
CBPMO Customs and Border Protection Modernization Office
DHS Department of Homeland Security
IDIQ indefinite delivery/indefinite quantity
ITDS International Trade Data System
IV&V independent verification and validation
I2V2 integrated independent verification and validation
OIG Office of Inspector General
OMB Office of Management and Budget
PTR program trouble report
SA-CMM Software Acquisition Capability Maturity Model
SAT system acceptance test
SEI Software Engineering Institute
SIT system integration test
SWIT software integration test
UAT user acceptance test
US-VISIT United States Visitor and Immigrant Status Indicator
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
A
United States General Accounting Office Washington, D.C. 20548
May 14, 2004
The Honorable Thad Cochran
Chairman
The Honorable Robert C. Byrd
Ranking Minority Member
Subcommittee on Homeland Security
Committee on Appropriations
United States Senate
The Honorable Harold Rogers
Chairman
The Honorable Martin Olav Sabo
Ranking Minority Member
Subcommittee on Homeland Security
Committee on Appropriations
House of Representatives
In January 2004, U.S. Customs and Border Protection (CBP) within the
Department of Homeland Security (DHS) submitted to the Congress its
fiscal year 2004 expenditure plan, seeking release of $318.7 million for
its
Automated Commercial Environment (ACE) program. ACE is to be CBP's
new trade system. The program's goals include facilitating the movement of
legitimate trade through more effective trade account management and
strengthening border security by identifying import and export
transactions that have an elevated risk of posing a threat to the United
States. As required by CBP's fiscal year 2004 appropriations,1 we reviewed
the expenditure plan. Our objectives were to (1) determine whether the
ACE fiscal year 2004 expenditure plan satisfies certain legislative
conditions, (2) determine the status of our open ACE recommendations,
and (3) provide any other observations about the expenditure plan and
DHS's management of the ACE program.
On March 8, 2004, we briefed your offices on the results of this review.
This
report transmits the results of our work. The full briefing, including our
scope and methodology, is reprinted as appendix I.
1P.L. 108-90 (October 1, 2003).
Compliance with Legislative Conditions
The expenditure plan satisfied or partially satisfied the conditions
specified in DHS's appropriations act. Specifically, the plan, including
related program documentation and program officials' statements, satisfied
or provided for satisfying all key aspects of (1) meeting the capital
planning and investment control review requirements of the Office of
Management and Budget (OMB), (2) complying with the DHS enterprise
architecture,2 and (3) review and approval by DHS and OMB. The plan
partially satisfied the condition that specifies compliance with the
acquisition rules, requirements, guidelines, and systems acquisition
management practices of the federal government.3
Status of Open Recommendations
CBP is making progress in addressing our open recommendations. Each
recommendation, along with the status of CBP's actions to address it, is
summarized below.
o Before building each ACE release (i.e., beginning detailed design and
development), certify to Customs' House and Senate appropriations
subcommittees that the enterprise architecture has been sufficiently
extended to provide the requisite enterprise design content, and has been
updated to ensure consistency and integration across business areas.
With respect to the fiscal year 2004 expenditure plan, CBP has completed
action to implement this recommendation by using a process developed by
the former Customs Service to certify ACE releases against the former
Customs enterprise architecture. Further, DHS is developing a process to
ensure alignment of ACE with the departmentwide enterprise architecture it
is developing to replace DHS component agency and bureau architectures.
2An enterprise architecture is an institutional blueprint for guiding and
constraining investments in business process change and systems.
3In June 2003, the Department of the Treasury's Office of Inspector
General (OIG) issued a report on the ACE program's contract, concluding
that the former Customs Service, now CBP, did not fully comply with
Federal Acquisition Regulation requirements in the solicitation and award
of the contract. CBP disagrees with the Treasury OIG conclusion. To
resolve the disagreement, DHS asked us to render a formal decision. We are
currently reviewing the matter.
o Develop and implement a rigorous and analytically verifiable
costestimating program that embodied the tenets of effective estimating as
defined in the Software Engineering Institute's (SEI) institutional and
project-specific estimating guidance. 4
CBP is in the process of implementing this recommendation. It has
developed and begun to implement a cost-estimating program that embodies
SEI's models. For example, CBP hired a contractor to define and implement
a cost-estimating program and develop independent cost estimates.
Additionally, it has defined and documented processes for estimating
expenditure plan costs, and tasked contractors with ensuring that
expenditure plan estimates are evaluated against SEI criteria for
validating software cost and schedule estimates.
o Immediately develop and implement a human capital management strategy
that provides both near-and long-term solutions to program office human
capital capacity limitations, and report quarterly to the appropriations
committees on the progress of efforts to do so. These efforts should
include defining the office's skill and capability needs in terms that
will allow the program office to attract qualified individuals and that
will provide sufficient rewards and training, linked to performance, to
promote their retention.
CBP is in the process of implementing this recommendation, and it has
reported on these actions to the Congress. In particular, the program
office developed and began implementing a human capital management plan
that called for addressing several areas, including filling vacant
positions. However, the program office has continued to experience
difficulty in filling key positions and has begun implementing a new
staffing plan intended to address DHS's concern that the program office
has insufficient government program management staff.
4SEI's institutional and project-specific estimating guidelines are
defined in Robert E. Park, Checklists and Criteria for Evaluating the Cost
and Schedule Estimating Capabilities of Software Organizations,
CMU/SEI-95-SR-005 (Pittsburgh, Pa.: Carnegie Mellon University Software
Engineering Institute, 1995) and A Manager's Checklist for Validating
Software Cost and Schedule Estimates, CMU/SEI-95-SR-004 (Pittsburg, Pa:
1995), respectively.
o Develop and implement process controls for SEI's Software Acquisition
Capability Maturity Model (SA-CMM5) level 2 key process areas and for the
level 3 acquisition risk management key process area; develop and
implement each of the missing SA-CMM key practices for these key process
areas, and until this is accomplished, report to the appropriations
committees quarterly on the progress of efforts to do so.
CBP has implemented this recommendation. In November 2003, SEI assigned
the program a level 2 rating, meaning that the program had established
basic acquisition management processes. Further, CBP had addressed the two
weaknesses SEI identified in the level 3 acquisition risk management area,
and has reported to the Congress on the status of its acquisition
management improvement activities.
o Establish an independent verification and validation (IV&V) function to
assist CBP in overseeing contractor efforts, such as testing.6
CBP is in the process of implementing this recommendation. The program
office has established an IV&V function and has designated its IV&V
contractor. However, program documentation describes roles for the IV&V
contractor that are so integral to ACE development, testing, and
deployment, as well as to ACE program management, that they raise
questions about how the contractor could also perform independent
ACErelated assessments. CBP officials stated that they plan to address
this situation.
o Have future ACE expenditure plans specifically address any proposals or
plans, whether tentative or approved, for extending and using ACE
infrastructure to support other homeland security applications, including
any impact on ACE of such proposals and plans.
5Capability Maturity Model (CMM) is a service mark of Carnegie Mellon
University, and CMM is registered in the U.S. Patent and Trademark Office.
The SA-CMM identifies key process areas that are necessary to effectively
manage software-intensive system acquisitions. Level 2 is the second level
of the SA-CMM's five-level scale; achieving this level means that an
organization has the software acquisition rigor and discipline to repeat
project successes.
6IV&V involves having an independent organization conduct unbiased reviews
of management processes, products, and results with the goal of verifying
and validating that these meet stated requirements and standards.
CBP plans to implement this recommendation. Program officials acknowledged
the potential for ACE infrastructure to support other DHS system
applications and the potential for integrating ACE data and applications
with those systems. According to program officials, to begin preparing for
this potential, they have focused initially on discussing collaboration
opportunities with DHS's United States Visitor and Immigrant Status
Indicator Technology (US-VISIT) program.7
Observations on Management of ACE
We recognize accomplishments to date, including the deployment of the
first 2 of 10 planned ACE releases, and address the need for rigorous and
disciplined program management practices relating to managing ACE's costs,
schedule, and requirements, as well as the degree of concurrent system
development activities. An overview of the observations follows:
o Release 1 and 2 testing revealed a sufficient volume and significance
of system defects to affect schedule commitments. Release 1 testing
revealed system defects, which were corrected about 3 months after the
testing was concluded. Release 2 testing revealed more system defects than
expected; addressing these defects required more time than originally
planned for Release 2 testing.
o Requirements planned for Releases 1 and 2 have been deferred to later
releases. Releases 1 and 2 were originally planned as a single release
that would satisfy 465 requirements. Subsequently, 103 of these
requirements, which included capabilities related to document management,
data recovery, and security, were deferred to Releases 3 and 4, and 14
were deferred to later releases, for a total of 117 deferred requirements.
o Release 2 delays have set in motion a pattern of increased reliance on
concurrent activities, continued release delays, and cost overruns. The
time required to resolve defects resulted in overlap of Release 2 test
phases. Delays in completing Release 2 increased its overlap with Release
3 and have contributed to Release 3 delays. Likewise, Release 3 delays
will increase overlap with Release 4 and could cause Release 4 delays.
7US-VISIT is a governmentwide program to collect, maintain, and share
information on foreign nationals in order to enhance national security and
facilitate legitimate trade and travel while adhering to U.S. privacy laws
and policies.
Finally, the increased concurrency and schedule delays resulted in
Releases 1 through 4 costing more than planned.8
o Releases 1 and 2 are largely operating according to adjusted
requirements, defect density has stabilized, and the system is mostly up
and running when needed. Defects prevalent in Releases 1 and 2 have
stabilized, but nontrivial defects remain. Releases 1 and 2, however, are
mostly available when needed.
o Following problems with Release 1, steps were taken to avoid future
problems, but the success of these actions is unclear. As a result of
Release 1 cost overruns and schedule delays, CBP directed its contractor
to develop and implement a plan to improve program performance. The
objective of this plan was to demonstrate serious, tangible, and
measurable actions to resolve program issues. However, program officials
could not point to metrics that demonstrated improved performance, and the
cost to develop and implement the corrective action plan is unknown.
o Initial releases represent a small fraction of the total ACE software.
Releases 1, 2, 3, and 4 account for only about 14 percent of ACE software
and are to be designed, developed, and deployed within about 3 years. In
contrast, the remaining six ACE releases constitute 86 percent of ACE
software but are to be designed, developed, and deployed within about
3-1/2 years. CBP officials stated that they are currently studying the
extent to which off-the-shelf software can be employed in the remaining
six releases.
o The reliability of key cost estimates in the expenditure plan is
unclear. The cost estimates in the expenditure plan varied from
independent cost estimates, and the derivation of the expenditure plan
estimates only partially satisfied SEI guidance for assessing the
effectiveness of processes used to derive cost estimates.
8Our March 8, 2004, briefing to the staffs of the Subcommittees on
Homeland Security, Senate and House Committees on Appropriations, reported
a $23.3 million overrun for Releases 1 and 2 in addition to a $36.2
million overrun for Releases 3 and 4. Subsequently, CBP's ACE development
contractor revised the estimated overrun for Releases 3 and 4 to $46
million.
Conclusions The fiscal year 2004 ACE expenditure plan, along with program
documentation and officials' commitments, largely satisfies the
legislative conditions imposed by the Congress. Further, many of the
recommendations that we have made to strengthen ACE management have been
addressed, and CBP leadership has committed to addressing those that
remain. However, much must still be done before these recommendations are
fully implemented. Particularly, progress has been slow on overcoming
human capital challenges.
To CBP's credit, the first two ACE releases are operating largely as
intended. Achieving this initial operating capability, however, has not
occurred without difficulty, such as borrowing resources from future
releases to overcome problems on near-term releases; similar difficulties
that could affect cost, schedule, and capability commitments are being
experienced on ongoing releases. This pattern is not likely to change
unless the degree of concurrency among activities within and between
releases is better controlled, and the underlying reasons for introducing
this concurrency are addressed. While time and resources were invested in
management improvements over a year ago to address root causes, the
absence of meaningful measures for determining whether these investments
will be successful, coupled with limited progress on some of our previous
recommendations, makes CBP's chances of delivering future ACE release
capabilities on time and within budget uncertain at best.
Recommendations for Executive Action
To assist CBP in managing ACE and increasing the chances that future
releases will deliver promised capabilities on time and within budget, we
recommend that the Secretary of Homeland Security, through the Under
Secretary for Border and Transportation Security and the CBP Commissioner,
direct the CBP Chief Information Officer to take the following actions:
o Ensure the independence of its IV&V agent.
o Ensure that future expenditure plans are based on cost estimates that
are reconciled with independent cost estimates.
o Reconsider the ACE acquisition schedule and cost estimates in light of
early release problems, including these early releases' cascading effects
on future releases and their relatively small size compared with later
releases and in light of the need to avoid the past levels of concurrency
among activities within and between releases.
o Define measures, and collect and use associated metrics, for
determining whether prior and future program management improvements are
successful.
o Report quarterly to the House and Senate Committees on Appropriations
on efforts to address the above, as well as on our previous but
unaddressed recommendations.
Agency Comments and In written comments on a draft of this report signed
by the director, DHS Bankcard Programs and GAO/OIG Liaison, DHS concurred
with our
Our Evaluation recommendations and stated actions that it is taking or
plans to take to implement each. DHS also provided recommendations for
disclosing certain sensitive information included in our draft report. We
modified this report in accordance with DHS's recommendations. DHS's
comments are reprinted in appendix II.
We are sending copies of this report to the Chairmen and Ranking Minority
Members of other Senate and House committees and subcommittees that have
authorization and oversight responsibilities for homeland security. We are
also sending copies to the Secretary of Homeland Security, the Under
Secretary for Border and Transportation Security, the CBP Commissioner,
and the Director of OMB. In addition, the report will be available at no
charge on the GAO Web site at http://www.gao.gov.
Should you or your offices have any questions on matters discussed in this
report, please contact me at (202) 512-3439 or by e-mail at [email protected].
Other contacts and key contributors to this report are listed in appendix
III.
Randolph C. Hite Director, Information Technology Architecture and Systems
Issues
Appendix I
IT: Early Releases of Customs Trade System Operating, but Pattern of Cost
and Schedule Problems Needs to Be Addressed
Information Technology: Early Releases of Customs Trade System Operating,
but Pattern of Cost and Schedule Problems Needs to Be Addressed
Briefing to the Staffs of the
Subcommittees on Homeland Security,
Senate and House Committees on Appropriations
March 8, 2004
This briefing has been modified to exclude sensitive material that the
Bureau of Customs and Border Protection deemed to be
acquisition-sensitive.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Introduction Objectives Results in Brief Background Results
o Legislative Conditions
o Status of Recommendations
o Observations Conclusions Recommendations Agency Comments Attachment 1:
Scope and Methodology
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
The Department of Homeland Security (DHS) Bureau of Customs and Border
Protection (CBP), formed from the former U.S. Customs Service and other
entities with border protection responsibility, is acquiring a new trade
processing system, known as the Automated Commercial Environment (ACE).
The goals of ACE are to
o facilitate the movement of legitimate trade through more effective
trade account management;
o strengthen border security by identifying import/export transactions
that have an elevated risk of posing a threat to the United States or of
violating a trade law or regulation; and
o provide a single system interface between the trade community1 and the
federal government,2 known as the International Trade Data System (ITDS),
and thereby reduce the data reporting burden placed on the trade community
while also providing federal agencies with the data and various
capabilities to support their respective international trade and
transportation missions.
1 Members of the trade community include importers and exporters, brokers
and trade advisors, and carriers.
2 Federal agencies include CBP and other federal agencies responsible for
managing international trade and transportation processes.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
The Department of Homeland Security Appropriations Act, 2004,1 limits
obligating any funds for ACE until DHS submits for approval to the House
and Senate Committees on Appropriations a plan for expenditure that
1. meets the capital planning and investment control review requirements
established by the Office of Management and Budget (OMB), including
Circular A-11, part 3;2
2. complies with CBP's enterprise architecture;
3. complies with the acquisition rules, requirements, guidelines, and
systems acquisition management practices of the federal government;
4. is reviewed and approved by the CBP Investment Review Board,3 DHS, and
OMB; and
5. is reviewed by GAO. 1 Pub. L. 108-90 (Oct. 1, 2003).
2 OMB Circular A-11 establishes policy for planning, budgeting,
acquisition, and management of federal capital assets.
3 The purpose of the IRB is to integrate capital planning and investment
control, budgeting, acquisition, and management of investments. It is also
to ensure that spending on investments directly supports and furthers the
mission and that this spending provides optimal benefits and capabilities
to stakeholders and customers.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
In the Department of Homeland Security Appropriations Act, the Congress
appropriated approximately $318.7 million for the ACE program.1
DHS submitted its fiscal year 2004 expenditure plan for $318.7 million on
January 21, 2004, to its House and Senate Appropriations Subcommittees on
Homeland Security.
1 Pub. L. 108-90 (Oct. 1, 2003).
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Objectives
As agreed, our objectives were to
o determine whether the ACE fiscal year 2004 expenditure plan satisfies
the legislative conditions,
o determine the status of our open recommendations made on ACE, and
o provide any other observations about the expenditure plan and DHS's
management of the ACE program.
We conducted our work at CBP headquarters and contractor facilities in the
Washington, D.C., metropolitan area from January through March 2004 in
accordance with generally accepted government auditing standards. Details
of our scope and methodology are given in attachment 1.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Observations
Observations:
o Release 1 and 2 testing revealed a sufficient volume and significance
of system defects to impact schedule commitments.
o Requirements planned for Releases 1 and 2 have been deferred.
o Release 2 delays have set in motion a pattern of increased reliance on
concurrent activities, continued release delays, and cost overruns.
o Releases 1 and 2 are largely operating according to adjusted
requirements; defect density has stabilized and system is mostly up and
running when needed.
o Following Release 1 problems, steps were taken to avoid future
problems, but success of actions is unclear.
o Initial releases represent a small fraction of the total ACE software.
o Reliability of key cost estimates in expenditure plan is unclear.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
To better ensure that future ACE releases deliver promised capability
within budget and on time, we are making recommendations to the DHS
Secretary to address a recurring pattern of cost and schedule problems.
In their comments on a draft of this briefing, DHS, CBP, and ACE
officials, including the DHS Chief Information Officer and the CBP Acting
Chief Information Officer, generally agreed with the briefing and stated
that it was fair and balanced.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
ACE is to support CBP administration, enforcement, and trade compliance
business operations.
o Supported administration business operations include (1) processing and
recording of revenue generated from trade compliance, and (2) maintaining
the general ledger and accounting.
o Supported enforcement business operations include identifying import
and export transactions (i.e., cargo, conveyances, crew, and passengers)
that have an elevated risk of threatening the United States or violating a
trade law or regulation.
o Supported trade compliance business operations include (1) establishing
and maintaining trade accounts; (2) processing and releasing cargo for
import or export; tracking and monitoring of conveyances, cargo, and
individuals involved in importing and exporting; (3) liquidating import
entries including the processing of protests and decisions and the
potential authorization of refunds; and (4) ruling on import and export
legal, regulatory, and policy issues.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
The ACE technical architecture is to consist of layers or tiers of
computer technology:
o The Client Tier includes user workstations and external system
interfaces.
o The Web & Interface Tier provides the mechanisms for the user
workstations and external systems to access ACE.
o The Enterprise Application Integration Tier provides the middleware for
integrating and routing information between ACE software applications and
legacy systems.
o The Applications Tier includes software applications comprising
commercial products (e.g., SAP1), custom-developed software, and existing
legacy systems that provide the applications supporting business
processes.
o The Data Tier includes the business data, storage resources, and access
services used by the ACE software applications and the legacy systems.
Security and data privacy are embedded in all five layers.
1 SAP is a commercial enterprise resource planning software product that
has multiple modules, each performing separate but integrated business
functions. CBP is using SAP as part of a joint project between its Office
of Finance to support financial and property management and the CBP
Modernization Office (CBPMO) to support such ACE functions as customer
relationship management.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Simplified view of ACE technical architecture
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
CBP's initial strategy provided for the acquisition of ACE in four
increments over 4 years. In September 2002, CBP modified this strategy to
acquire and deploy the first two increments in two releases, with all
increments and releases acquired over 4 years. In October 2003, CBP
changed plans to acquire ACE through the acquisition and deployment of 10
releases over 6 years.
The following figure maps the initial four increments to the 10 releases
now planned, 7 of which are called "releases" (e.g., Release 1, Release 2,
... Release
7) and 3 of which are called "selectivity" (e.g., Selectivity 1,
Selectivity 2, and Selectivity 3).
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Evolution of ACE from 4 increments to 10 releases
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
The functionality associated with, status of, and plans for the 10 ACE
releases are as follows.
Release 1: Provide IT infrastructure-computer hardware and system
software-to simultaneously support operating the subsequent system
releases. This release was deployed in October 2003 and is operating.
Release 2: Provide initial group of CBP national account managers1 and 41
importers access to account information, such as trade activity. This
release was deployed in October 2003 and is operating.
Release 3: Provide additional account managers and importers, as well as
brokers and carriers,2 access to account information; provide initial
financial transaction processing and CBP revenue collection capability,
allowing importers and their brokers to make monthly payments of duties
and fees. This release is under development and scheduled for deployment
in August 2004.
1 CBP national account managers work with the largest importers to ensure
their compliance with trade laws.
2 Brokers obtain licenses from CBP to conduct business on behalf of the
importers by filling out paperwork and obtaining a bond; carriers are
individuals or organizations engaged in transporting goods for hire.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Release 4: Provide truck manifest1 processing and interfacing to legacy
enforcement systems and transaction history data. This release is under
development and scheduled for deployment in February 2005.
Release 5: Provide legacy systems interface for airport, seaport, and rail
crossing inspectors for accessing supply chain, enforcement, and
compliance data, as well as additional manifest functionality for selected
federal government agencies (including the Food and Drug Administration)
and the trade community. This release is scheduled for deployment in
August 2005.
Selectivity 1: Identify shipments that pose a security risk using new and
different combinations of data than current legacy systems permit, as well
as end-user defined threat identification criteria. This release is
scheduled for deployment in September 2005.
1 Manifests are lists of passengers or invoices of cargo for a vehicle,
such as a truck, ship, or plane.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Release 6: Provide additional account management functionality for daily
statement creation, refund identification and generation, and expanded
payment acceptance; license, permit, and certificate granting, tracking,
and revoking; and cargo, conveyance, and equipment tracking. This release
is scheduled for deployment in December 2006.
Selectivity 2: Identify shipments that pose a security risk during
authorized movements. This release is scheduled for deployment in May
2006.
Release 7: Provide manifest processing for cargo moving between modes of
transportation; additional cargo and conveyance tracking and initial
export processing functions; importer activity summary; and regulatory
audit, protest, and drawback1 processing. This release is scheduled for
deployment in June 2007.
Selectivity 3: Identify imports and exports (e.g., mail, courier,
hand-carried items, and shipments transitioning between modes of
transportation) that pose a security risk. This release is scheduled for
deployment in November 2006.
1 A refund of duties on an imported product subsequently exported or used
to produce a product for export.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
CBP is acquiring and implementing ACE through an indefinite-delivery,
indefinitequantity contract1 awarded on April 27, 2001, to IBM Global
Services. IBM and its subcontractors are collectively called the e-Customs
Partnership (eCP). Through a series of contract task orders, CBP plans to
acquire and implement ACE.
The following table describes and provides the status of the executed eCP
task orders.
1 An indefinite-delivery, indefinite-quantity contract provides for an
indefinite quantity, within stated limits, of supplies or services during
a fixed period of time. The government schedules deliveries or performance
by placing orders with the contractor.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
ACE Schedule
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Since March 2001, five ACE expenditure plans have been submitted.1
o On March 26, 2001, CBP submitted to its appropriations committees the
first expenditure plan seeking $45 million for the modernization contract
to sustain CBPMO operations, including contractor support. The
appropriations committees subsequently approved the use of $45 million,
bringing the total ACE funding to $50 million.
o On February 1, 2002, the second expenditure plan sought $206.9 million
to sustain CBPMO operations; define, design, develop, and deploy Increment
1, Release 1 (now Releases 1 and 2); and identify requirements for
Increment 2 (now Release 5 and Selectivity 1 in addition to part of
Release 6 and Selectivity 2). The appropriations committees subsequently
approved the use of $188.6 million, bringing total ACE funding to $238.6
million.
1 In March 2001, appropriations committees approved the use of $5 million
in stopgap funding to fund program management office operations.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Chronology of Five ACE Expenditure Plans
o On May 24, 2002, the third expenditure plan sought $190.2 million to
define, design, develop, and implement Increment 1, Release 2 (now
Releases 3 and 4). The appropriations committees subsequently approved the
use of $190.2 million, bringing the total ACE funding to $428.8 million.
o On November 22, 2002, the fourth expenditure plan sought $314 million
to operate and maintain Increment 1 (now Releases 1, 2, 3, and 4), design
and develop Increment 2, Release 1 (now Release 5 and Selectivity 1), and
to define requirements and plan Increment 3 (now part of Releases 6 and 7
and Selectivity 2 and 3). The appropriations committees subsequently
approved the use of $306.4 million, bringing total ACE funding to $735.2
million.
o On January 21, 2004, CBP submitted its fifth expenditure plan, seeking
$318.7 million for ACE implementation infrastructure and support, ACE
operations and maintenance, and definition and design of Release 6 and
Selectivity 2.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
The fiscal year 2004 ACE expenditure plan satisfied or partially satisfied
each of the legislative conditions.
Condition 1. The plan, in conjunction with related program documentation
and program officials' statements, satisfied the capital planning and
investment control review requirements established by OMB, including
Circular A-11, part 7, which establishes policy for planning, budgeting,
acquisition, and management of federal capital assets.
Details of our analysis are shown on the table that follows.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Condition 2. The plan and related program officials' statements provide
for complying with the DHS enterprise architecture.
DHS released version 1 of the architecture in October 2003,1 and plans to
issue version 2 in September 2004.
According to the DHS Chief Information Officer (CIO), DHS is developing a
process to align its systems modernization efforts, such as ACE, to its
enterprise architecture. Alignment of ACE to the enterprise architecture
has yet to be determined, but DHS CIO and CBP officials stated that they
plan to address this issue. Moreover, the fiscal year 2004 expenditure
plan states that Release 6 and Selectivity 2 will be aligned with the DHS
enterprise architecture.
1 Department of Homeland Security Enterprise Architecture Compendium
Version 1.0 and Transitional Strategy.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Condition 3. The plan, in conjunction with related program documentation,
partially satisfies the condition of compliance with the acquisition
rules, requirements, guidelines, and systems acquisition management
practices of the federal government.
The Software Acquisition Capability Maturity Model (SA-CMM), developed by
Carnegie Mellon University's Software Engineering Institute (SEI), is
consistent with the acquisition guidelines and systems acquisition
management practices of the federal government, and it provides a
management framework that defines processes for acquisition planning,
solicitation, requirements development and management, project management,
contract tracking and oversight, and evaluation.
On November 5, 2003, SEI assessed ACE acquisition management against the
SA-CMM and assigned a level 2 rating, indicating that CBPMO has instituted
basic acquisition management processes and controls in the following
areas: acquisition planning, solicitation, requirements development and
management, project management, contract tracking and oversight, and
evaluation.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
In June 2003, the Department of the Treasury's Office of Inspector General
(OIG) issued a report on the ACE program's contract, concluding that the
former Customs Service, now CBP, did not fully comply with Federal
Acquisition Regulation requirements in the solicitation and award of its
contract because the ACE contract is a multiyear contract and not an
indefinite delivery /indefinite quantity (IDIQ) contract. Further, the
Treasury OIG found that the ACE contract type, which it determined to be a
multiyear contract, is not compatible with the program's stated needs for
a contract that can be extended to a total of 15 years because multiyear
contracts are limited to 5 years. Additionally, the Treasury OIG found
that Customs combined multiyear contracting with IDIQ contracting
practices. For example, it plans to use contract options to extend the
initial 5-year performance period.
CBP disagrees with the Treasury OIG conclusion.
To resolve the disagreement, DHS asked GAO to render a formal decision. We
are currently reviewing the matter.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Satisfies condition 4. The plan satisfies the condition that it be
reviewed and approved by the CBP IRB, DHS, and OMB.
On July 15, 2003, the CBP IRB approved the plan. DHS subsequently approved
the plan on November 20, 2003, and OMB approved the plan on January 20,
2004.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Satisfies condition 5. The plan satisfies the condition that it be
reviewed by GAO. Our review was completed on March 4, 2004.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Open recommendation 1: Before building each ACE release (i.e., beginning
detailed design and development), certify to Customs' House and Senate
appropriations subcommittees that the enterprise architecture has been
sufficiently extended to provide the requisite enterprise design content
and has been updated to ensure consistency and integration across business
areas.
Status: Complete.
In response to our recommendation made in March 2002, the U.S. Customs
Service established a process for certifying to its appropriations
committees that the Customs enterprise architecture had been sufficiently
extended and updated before detailed design and development of each ACE
release was begun.
In March 2003, the U.S. Customs Service business operations supported by
ACE became part of the DHS CBP.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
In May 2003, CBP used the process developed by the former Customs Service
to certify Releases 3 and 4 against the former Customs enterprise
architecture. At this time, DHS also committed to developing a single,
departmentwide enterprise architecture to replace DHS component
agency/bureau architectures. DHS released version 1 of its enterprise
architecture in October 2003,1 and plans to issue version 2 of its
architecture in September 2004.
According to the DHS CIO, DHS is developing a process to ensure alignment
of its systems modernization efforts, including ACE, to its enterprise
architecture. While this alignment step for ACE has not occurred yet, the
fiscal year 2004 expenditure plan states that Release 6 and Selectivity 2
will be aligned with the DHS architecture.
1 Department of Homeland Security Enterprise Architecture Compendium
Version 1.0 and Transitional Strategy.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Open recommendation 2: Develop and implement a rigorous and analytically
verifiable cost estimating program that embodies the tenets of effective
estimating
as defined in SEI's institutional and project-specific estimating models.1
Status: In progress.
The program office has developed and begun to implement a cost estimating
program that embodies SEI's models. For example, CBP hired a contractor to
define and implement a cost estimating program. The contractor has
responsibility
for development of ACE life cycle cost estimates and independent cost
estimates to
support budget planning, budget execution, and contract task order
evaluation.
Additionally, the program office has defined and documented processes for
estimating expenditure plan costs and management reserve in addition to
preparing independent cost estimates. Further, CBP tasked its support and
modernization contractors with ensuring that the expenditure plan
estimates are
evaluated against SEI criteria for validating software cost and schedule
estimates.
1SEI's institutional and project-specific estimating guidelines are
defined in Checklists and Criteria for Evaluating the Cost and Schedule
Estimating Capabilities of Software Organizations and A Manager's
Checklist for Validating Software Cost and Schedule Estimates,
respectively.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Open recommendation 3: Immediately develop and implement a human capital
management strategy that provides both near-and long-term solutions to
program office human capital capacity limitations, and report quarterly to
the appropriations committees on the progress of efforts to do so. These
efforts should include defining the office's skill and capability needs in
terms that will allow the program office to attract qualified individuals
and that will provide sufficient rewards and training, linked to
performance, to promote their retention.
Status: In progress
The program office developed and began implementing a human capital
management plan. This plan called for addressing several areas, including
identification of required core competencies (i.e., defined knowledge and
skill requirements), assessing existing capabilities, developing a
training program, and filling vacant positions.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
In its June and October 2003 reports to its appropriations committees on
progress in addressing these areas, the program office indicated that it
had defined core competencies, was assessing capabilities and developing
an annual training plan, and had filled three vacancies and developed a
plan for filling its remaining vacancies. However, the program office has
continued to experience difficulty in filling key positions, including
three responsible for organizational change associated with adopting the
business process changes embedded in ACE functionality.
Currently, program officials told us that the capability assessment and
the annual training plan were completed and both are awaiting approval.
They also told us that they have suspended development of the original
plan for filling vacancies, and in January 2004 began implementing a new
staffing plan that was developed in response to a DHS IRB statement that
the program office has insufficient government program management staff.
We made this same observation, along with recommendations to address it, 2
years ago.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Open recommendation 4: Develop and implement process controls for SEI's
SA-CMM level 2 key process areas and for the level 3 acquisition risk
management key process area; develop and implement each of the missing
SA-CMM key practices for these key process areas, and until this is
accomplished, report to the appropriations committees quarterly on the
progress of efforts to do so.
Status: Complete
CBP has taken steps to develop and implement each of the key process areas
for the SA-CMM level 2 key process areas and the level 3 acquisition risk
management area. SEI assessed the acquisition management of Releases 1
through 4 against the SA-CMM level 2 criteria. On November 5, 2003, SEI
assigned the program a level 2 rating, meaning that the program has
established basic acquisition management processes.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
SEI also conducted an unrated assessment of the program's acquisition risk
management capability in May 2003. The assessment identified two
weaknesses: (1) risk management was not integrated into acquisition
planning and (2) a defined organizational process was not used to produce
the program's acquisition risk management plan. CBP has addressed these
weaknesses by updating its acquisition planning process to include risk
management activities and by documenting an acquisition risk management
process.
CBP's October 2003 report to its appropriations committees reported on the
status of these acquisition management improvement efforts.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Open recommendation 5: Establish an independent verification and
validation (IV&V) function to assist CBP in overseeing contractor efforts,
such as testing.
Status: In progress
The program office has established an integrated independent verification
and validation (I2V2) function, and it has designated its I2V2 contractor.
According to CBP's I2V2 strategy, the contractor is to review key eCP
deliverables and products and provide assessments and reports at various
life-cycle stages and milestones. Further, the strategy recognizes the
importance of maintaining independence in the conduct of I2V2 and outlines
(1) a reporting structure that allows the contractor to report its
findings to CBP executives and (2) an approach to assigning staff to I2V2
tasks that is to ensure contractor independence from program management
pressures.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
However, other program documentation, such as the fiscal year 2004
expenditure plan, the contractor's statement of objectives, and the
modernization program work breakdown structure, describes roles for the
I2V2 contractor that are so integral to ACE development, testing, and
deployment, as well as ACE program management, that they raise questions
about how the contractor could also perform independent ACE-related
assessments. For example, they assign the I2V2 contractor responsibility
for
o supporting development of long-range and short-term strategic plans for
the modernization program;
o developing the task orders that contractually specify eCP's work; and
o developing incentives for improving eCP's performance.
CBP officials stated that they recognize the importance of ensuring the
independence of their verification and validation function and plan to
address this situation.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Open recommendation 6: Have future ACE expenditure plans specifically
address any proposals or plans, whether tentative or approved, for
extending and using ACE infrastructure to support other homeland security
applications, including any impact on ACE of such proposals and plans.
Status: Planned
Program officials acknowledge the potential for ACE infrastructure to
support other DHS system applications and the potential for integrating
ACE data and applications with those systems. To begin preparing for this
potential, program officials told us they have focused initially on
discussing collaboration opportunities with DHS's United States Visitor
and Immigrant Status Indicator Technology (US-VISIT) system.1 To date,
those discussions have included
o sharing lessons learned on leveraging contractors and
o coordinating enterprise architecture activities.
1US-VISIT is a governmentwide program to collect, maintain, and share
information on foreign nationals for enhancing national security and
facilitating legitimate trade and travel, while adhering to U.S. privacy
laws and polices.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Release 1 user acceptance testing revealed problems.
According to CBP's test results report, user acceptance testing (UAT) for
Release 1 was intended to ensure that the design of the users' workstation
screens, the capabilities for navigating among screens, and the
functionality associated with these screens satisfied user needs.1
Additionally, CBP's stated goal of Release 1 UAT was to collect feedback
from users and identify defects and improvement opportunities.
UAT began on January 8, 2003, and was completed on January 24, 2003. It
consisted of users (1) completing 28 test cases and (2) responding to 10
questions designed to solicit their satisfaction (on a scale of 1 to 5,
with 1 being very poor and 5 being excellent) about different aspects of
Release 1, such as screen design, quality, reliability, availability,
response time, and training.
1 In GAO-03-406, we reported that the first ACE release had completed SAT.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Release 2 integration, acceptance, and user acceptance testing revealed
problems that required additional time to address.
System integration testing (SIT). As previously stated, SIT is intended to
verify that related system, subsystem, or module components are capable of
integrating and interfacing with each other. Successfully completing SIT
requires that all test cases achieve a "test passed" status, meaning that
the test procedures were successfully executed and completed, and the test
objectives were met.
Release 2 SIT, which began on December 31, 2002, and was to end on April
4, 2003, consisted of 46 test cases. Of these 46, 33 were reported to be
successfully completed. According to the test results, of the remaining 13
test cases, either they could not be executed for various reasons (such as
a capability not being available in the current release) or a test case
was changed during attempts to execute it. The 46 test cases, according to
eCP's projection, were expected to disclose 389 defects; however, they
resulted in 745.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
When SIT concluded in June 2003 (about 2.5 months later than planned), 49
PTRs remained open (6 severe, 36 moderate, and 7 minor). However, because
none of the tested items failed the SIT and the defects identified were
either resolved or deferred to SAT or a later release, the SIT test
procedures were judged to be successfully executed.
As of January 2004, the program office reports that all the critical and
severe PTRs have been resolved, and that 16 PTRs are still open, of which
11 are moderate and 5 are minor.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
System acceptance testing (SAT). According to CBP's test results report,
SAT is intended to show that a system performs according to its specified
functional and performance requirements. To pass SAT, all test cases must
pass without identification of critical or severe defects.
Release 2 SAT was to begin April 7, 2003, and conclude on April 28, 2003,
but because of the large number of unexpected defects, SAT began on May 1,
2003, and ended on June 5, 2003, which is about a 1 month delay. Of the
SAT test cases executed, all but 19 reportedly passed. Of these 19, one
resulted in a critical PTR, which was closed within 2 days, and 18
produced moderate or minor PTRs. The program office reports that those 18
were either corrected or deferred to future releases based on
implementation of acceptable procedural workarounds.
As of January 2004, 11 of the deferred PTRs reportedly remain open. An
example of a deferred PTR is Release 2's inability to identify duplicate
accounts. According to a system integration manager, this defect was not
categorized as critical or severe because an acceptable workaround was
available (i.e., establishing a manual process to identify possible
duplicate accounts until Release 3 is ready).
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
User acceptance testing (UAT). According to CBP's test results report, UAT
is intended to verify that the system meets the needs of the users.
Release 2 UAT began on July 10, 2003, and was to end on August 15, 2003.
It consisted of CBP and the trade community users executing 19 and 23 test
cases, respectively, and being asked to rate the release in several areas
using a 1 to 5 scale (with 1 indicating very dissatisfied and 5 indicating
very satisfied). The test areas addressed the account management
functionality included in Release 2, such as creating a new account,
managing account details, and creating reports. The average user
satisfaction rating was slightly more than 4 out of 5, which indicated
user satisfaction, according to the test results report.
UAT nevertheless resulted in the identification of 17 defects. As of
January 2004, 15 of the associated PTRs are reported to have been
resolved, and the remaining 2 are categorized as moderate.
The shaded portion of the graph on the following page shows the number of
PTR's open (i.e., unresolved) each week during Release 2 testing.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
SIT SAT UAT and Pilot Period
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Observation 2: Requirements planned for Releases 1 and 2 have been
deferred.
As previously described, each ACE release is expected to contain specific
functionality and performance capabilities. These capabilities are defined
in a system requirements document, which lists the overall requirements
for the entire ACE system. The requirements specification set breaks these
into more detailed requirements and assigns them to releases.
Releases 1 and 2 were originally planned as a single release that was to
satisfy 465 requirements. Subsequently, 103 of these requirements were
deferred to Release 3 and 4, and 14 requirements were deferred to later
releases, for a total of 117 deferred requirements. Also, 25 of 465
requirements were deleted from the baseline. In total, 142 requirements
were either deferred or deleted from the original baseline, representing
about 31 percent of the original baseline of 465 requirements.
At the same time, 36 new requirements were added to the baseline for
Release 1 and 2, meaning that Release 1 and 2 satisfied 359 total
requirements. The next slide summarizes the requirements changes in
Release 1 and 2.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
According to the requirements specification set for Release 3 and 4, the
following categories of capabilities and functionality were deferred from
Releases 1 and 2 to Releases 3 and 4:
o document management, document retention, workflow, and data recovery;
o network response time; and
o security (remote access authentication, data security during network
transfer, audit trails).
According to program officials, the deferred security requirements must be
implemented by Release 4.
Program change requests cite the reasons for the respective requirements
deferral. Reasons include the following:
o Document management and retention requirements were deferred because
neither Release 1 nor 2 needed to be able to accommodate either paper or
electronic documents.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Observation 2
o Workflow requirements were deferred because eCP and CBP needed
additional time to evaluate and select among commercial product options.
o Data recovery functionality was deferred because CBP chose to delay
implementation of disaster recovery for ACE until Releases 3 and 4.
o Broker and carrier account functionality was deferred because full
support for the functionality was not needed until Releases 3 and 4.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Observation 3: Release 2 delays have set in motion a pattern of increased
reliance on concurrent activities, continued release delays, and cost
overruns.
As we previously reported,1 the importance of ACE to our country's
security and trade facilitation prompted CBP to take steps aimed at
expediting ACE acquisition and deployment, including introducing greater
reliance on performing release design, development, and testing activities
concurrently. While such concurrency can permit faster acquisition and
deployment, it also introduces risks, which the ACE contractor has said
include limited understanding of requirements before design and
development activities begin, uncertainty regarding the timely
availability of commercial hardware and software products, and increased
nearterm funding requirements. Other risks include contention for limited
resources (such as key personnel, as well as development and testing
equipment and facilities) and dependencies among releases not being met.
1U.S. General Accounting Office, Customs Service Modernization: Management
Improvements Needed on High-Risk Automated Commercial Environment Project,
GAO-02-545 (Washington, D.C.: May 13, 2002).
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Time required to resolve defects resulted in overlap of Release 2
integration and acceptance testing.
Resolving Release 2 defects also resulted in more overlap between testing
activities. The original Release 2 testing schedule projected an overlap
of about 1.5 months between SWIT and SIT, and no overlap between SIT and
SAT. However, due to the extension of testing periods caused by the high
number of defects, the overlap between SWIT and SIT was about 4.5 months,
and the SAT period was completely concurrent with SIT. The next slide
shows the overlap of Release 2 testing activities.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Delays in completing Release 2 increased its overlap with Release 3 and
have contributed to Release 3 delays.
According to program officials and variance reports, some ongoing Release
3 work had to be curtailed when key resources slated for Release 3 (e.g.,
test and development environment, equipment, and key personnel) had to be
diverted to address Release 2 PTRs and complete Release 2 test activities.
Although some resource diversion continued through June 2003, as Release 3
resources were used to complete Release 2 SIT (including regression
testing of software changes made to address PTRs), one report states that
after June 2003, about 50 percent of the planned Release 3 SIT resources
were redirected to support Release 2 deployment activities.
According to program officials, Release 1 delays, coupled with Release 2
dependencies on Release 1 and subsequent contention for key resources
between Releases 2 and 3, have had a domino effect, causing Release 3
resource shortfalls and consequent delays. For example, as of January
2004, a delay of 5 to 6 months in the March 2004 milestone for determining
Release 3 operational readiness was reported.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Release 3 delays will increase overlap with Release 4 and could cause
Release 4 delays.
According to program officials and documentation, Release 3 and 4
development began in August 2001, concurrent with ongoing Release 1 and 2
testing, but before they were aware of the large number of Release 1 and 2
defects. According to these officials, work was undertaken concurrently to
minimize the time between releases and thereby maximize the efficiency in
deploying releases.
While these officials agreed that the cascading Release 1 and 2 problems
will continue to impact Release 3, they stated that Release 4 will not be
affected. One reason that they cited is that Release 4 will be tested in a
recently acquired separate laboratory environment from Release 3, thus
eliminating the contention for these key resources.
Despite these statements, program variance reports show that Release 4 is
being impacted. According to the reports, Release 3 resources targeted for
Release 4 are not becoming available as early as planned because of
delayed Release 3 development and SWIT activities. Unless resources from
an alternative source are forthcoming, this could impact Release 4
schedule commitments.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Increased concurrency and schedule delays resulted in Releases 1 through 4
costing more than planned.
The baseline cost estimate for Releases 1 and 2 was $86.1 million.
However, the actual cost was reported to be $109.4 million-an overrun of
$23.3 million (27 percent).
The baseline cost estimate for Release 3 and 4 was $149.5 million.
However, CBP now estimates the cost to be $185.7 million-an overrun of
about $36.2 million (24 percent).
According to program documentation, including cost variance analysis
reports, factors contributing to the cost overruns included
o resources to eliminate Release 1 and 2 defects; o overtime to
minimize Release 3 and 4 schedule delays;
o unavailability of testing and development environments;
o Release 3 delays caused by resources being used to support delayed
Release 2 activities;
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Observation 3
o added Release 3 and 4 scope as a result of requirements shift from
Release 2; and
o Release 3 receipt of unresolved defects from Release 2.
CBP reported that it funded the $23.3 million Release 1 and 2 overrun with
a $7.9 million forfeiture of eCP's incentive fees and a $15.4 million
reallocation of funds from Release 5. CBP also reported that it is
considering funding the $36.2 million Release 3 and 4 overrun out of its
accumulated management reserve.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Observation 4: Releases 1 and 2 are largely operating according to
adjusted requirements; defect density has stabilized and system is mostly
up and running when needed.
Two indicators of how well a system is operating are defect density (i.e.,
the volume and severity of unresolved problems) and system availability
(i.e., the time the system is operating satisfactorily, expressed as a
percentage of time that the system is required to be operational).
Releases 1 and 2 began operating on October 17, 2003. According to CBP,
these releases are providing support to 73 import accounts and 3300 users.
Further, these accounts represent (1) about seven percent of the 1100
accounts expected to be supported by the end of 2004 and (2) about 19
percent and 20 percent of annual import value and trade duties collected,
respectively.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Release 1 and 2 defect density has stabilized, but nontrivial defects
remain.
As indicated by the shaded area on the following graph representing the
period since Releases 1 and 2 began operating, the defect density
associated with both releases has stabilized. In particular, CBP defect
data show that as of October 17, 2003, these releases had no severe or
critical PTRs, and 44 moderate and 21 minor PTRs open. As of January 2004,
these data show improvement, with 1 severe, 34 moderate, and 17 minor PTRs
open, and documentation shows that the severe defect was resolved on
February 3, 2004.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
SIT SAT UAT and Pilot Period
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
For each of these outages, eCP documents a root cause analysis. Some of
the outages can be attributed to problems such as an expired secure socket
layer certificate, full log file for servers, and loss of database
connection between two subsystems. In addition to identifying the root
cause, eCP reports on the mitigation actions taken to prevent a
reoccurrence of the problem. For example, to address the problem with the
log file, eCP has opened a PTR for automated monitoring of file systems.
Until the PTR is resolved, eCP reported that it is performing manual
monitoring of the log file systems daily.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Observation 5: Following Release 1 problems, steps were taken to avoid
future problems, but success of actions is unclear.
Effective project management, including directing a contractor to perform
a particular task (e.g., developing and implementing a corrective action
plan), requires, among other things, a clear statement of objectives,
identification of necessary resources, and defined measures of progress
and results.
As a result of the Release 1 cost overruns and schedule delays, CBP
directed eCP, in February 2003, to develop and implement a plan to improve
program performance. According to the plan, which is dated April 2003, the
objective was to demonstrate serious, tangible, and measurable actions to
resolve core program issues and achieve fundamental program improvement.
The plan identified eight problem areas that eCP was to correct:
o Ineffective leadership
o Unclear roles and responsibilities
o Misaligned priorities
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Observation 5
o Weak system engineering
o Poor program planning
o Inaccurate and inconsistent financial data
o Unmet contract commitments
o Unstable system performance
However, the plan did not define the level of resources required to
execute it, and it did not specify how progress and improvement in each of
these areas would be determined, including any specific measures and
metrics.
In October 2003, program management review documents show that all the
corrective actions were reported to be completed. In January 2004, CBP
conducted an evaluation to assess eCP's performance. Although the
resulting January report was not available in time to be included in our
review, CBP officials told us that implementation of eCP's corrective
action plan is complete. They provided the following examples of
corrective actions that eCP has taken to address each of the eight problem
areas (see following slides).
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Problem areas CBP-provided examples of eCP actions to correct problems
Ineffective Increased the number and caliber of resources. For example,
eCP has brought on
leadership board an Enterprise Resource Planning1 expert, a chief
technology officer, a systems architect, and a workforce transformation
leader.
Unclear roles and Business process owners now manage functional areas
through the system life
responsibilities cycle. In the prior operating model, the business process
teams would only be responsible for planning and defining the
requirements, while the engineering teams would be responsible for
building, preparing, and delivering the support for accounts management.
In the new operating model, the business process owner would be
responsible for ensuring that accounts management in Release 1 and 2 is
planned and executed throughout all phases of the systems life cycle.
Misaligned priorities Assessed each release to identify the desired
business results that are being addressed so that there is visibility into
the benefits and capabilities that are being gained through ACE.
1 Enterprise resource planning is an integrated software solution used to
manage a company's resources.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Problem areas CBP-provided examples of eCP actions to correct problems
Weak system Improved testing and configuration management. For example,
the environment
engineering engineering group is now using a repeatable environment build
process, and the business process owners participate in the testing
activities and provide an internal check outside the software development
team.
Improved quality of products as a result of using the Ascendant Life Cycle
Methodology1 and tailoring it to the ACE program. For example, the
deliverables now require less rework than previously and are more aligned
with CBP's expectations.
Poor program Revised the program plan to include more detail on the
content of the various
planning releases, descriptions of the end-state architecture,
descriptions of testing and development needs integrated with cost and
schedule; and cost and schedule estimates based on historical performance.
Inaccurate and Improved financial controls and financial reporting as a
result of new tools and inconsistent automation, evident in the monthly
cost performance reviews and program financial data management reviews.
Unmet contract Improved quality of deliverables with higher standards of
acceptance evident with the commitments requirements, integrated baseline
reviews, and preparation for proposals.
Improved collaboration between eCP and the government with active
monitoring and control.
1 The Ascendant Life Cycle methodology is a specific development
methodology for use with the SAP Enterprise Resource Planning software
package.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Problem areas CBP-provided examples of eCP actions to correct problems
Unstable system Stable and controlled testing environments. For example,
for Release 3, more
performance defects were identified in early testing activities, while in
Release 2, the majority were identified in later tests, which is an
indication of infrastructure stability.
Source: GAO based on CBP data.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Despite the examples offered, program officials could not point to metrics
that demonstrate improved performance. Instead they stated that the
corrective actions have not been implemented long enough to see
substantial improvements and that the corrective actions will require time
to demonstrate results. They added, however, that they have continued to
see some improvement in the quality of deliverables.
More concrete measures of performance that are available do not suggest
adequate improvements. For example, as previously noted, Releases 3 and 4
have experienced 5 to 6 month delays and are over budget by $36.2 million.
Program officials attributed the problems that occurred to problems
experienced in Release 1 affecting Releases 2, 3, and 4, before the
corrective action plan was developed and implemented. They also stated
that contract task orders are generally completed on time and under
budget.
According to program officials, the effect of the corrective actions will
only be evident when eCP begins achieving milestones on schedule and
within budget.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
CBP expects these results to be seen with the delivery and approval of the
Global Business Blueprint1 in May 2004, which is the first major activity
to be initiated after the corrective actions were implemented.
Cost to develop and implement corrective action plan is unknown.
Effective project management, including directing a contractor to perform
a particular task (e.g., developing and implementing a corrective action
plan), includes estimating the associated cost and monitoring the activity
to ensure that cost estimates are met.
According to program officials, they do not know the costs that the
government incurred and the payments that the government made to eCP to
develop and implement its plan for correcting problems associated with its
delivery of proposed system capabilities on time and within budget.
Rather, they said that these costs are spread across multiple contract
tasks and are not identifiable.
Program officials told us that while CBP requested eCP to track the costs
of implementing the corrective actions, eCP representatives told them that
it would be impossible to track these costs separately because they would
be incurred as part of other ongoing tasks that eCP is required to perform
under executed task orders.
1 Global Business Blueprint is a planning project for future releases that
is to establish an integrated business framework and identify appropriate
technology enablers to support ACE cargo management.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Observation 6: Initial releases represent a small fraction of the total
ACE software.
A system's software size can be expressed in function points. Function
points are standard units of measure that describe software in terms of
end-user inputs and resulting outputs. They can be used to measure the
size of any software regardless of the language, development method, or
platform. Function point counts are derived from a variety of sources,
including requirements documentation, interviews with end users, and
business process descriptions. The International Function Point Users
Group (IFPUG) maintains the standards for identifying and quantifying
function points. The ACE system has been sized using function points with
IFPUG-certified counters.
All ACE releases collectively consist of 11,936 function points. Releases
1 and 2 together contain 618 function points, and Releases 3 and 4
together contain 1,016 function points. The remaining 10,302 function
points are contained in the three selectivity releases and Releases 5, 6,
and 7, with the majority (8400) contained in Releases 6 and 7.
The following chart summarizes the function point counts for Releases 1
and 2, Releases 3 and 4, and Releases 5 through 7 plus Selectivity 1
through 3.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Observation 6
Size of ACE Software by Release Groupings
Releases 1 and 2 5 percent
Releases 3 and 4 9 percent
Releases 5, 6, and 7 Selectivity 1, 2, and 3 86 percent
Source: GAO based on CBP data.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Releases 1, 2, 3, and 4 reportedly account for 1634 function points being
addressed by custom software and COTS, including SAP. In contrast,
Releases 5, 6, and 7 plus Selectivity 1, 2, and 3 constitute 10,302
function points, 5450 of which are to be addressed by custom software and
other COTS with the balance of 4852 to be addressed by the SAP product.
The graph on the following page provides this information by release.
Despite these relative sizes, the scheduled time for completing the last
six releases is about the same as the time anticipated to complete the
first four releases, even though the software size of the last six is over
three times that of the first four, and this does not include SAP
software. Specifically, as of January 2004, the total time projected for
design, development, and deployment of Releases 1 to 4 is approximately 3
years. According to program documentation, the total time projected for
design, development, and deployment of Releases 5, 6, and 7 and
Selectivity 1, 2, and 3 is approximately 3-1/2 years.
Program officials stated that they expect the Global Business Blueprint
activity that they are currently performing to better identify the extent
to which function points can be addressed by custom software, other COTS,
and SAP in Releases 5, 6, and 7 and Selectivity 1, 2, and 3.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Observation 7: Reliability of key cost estimates in expenditure plan is
unclear.
Cost estimates in expenditure plan varied from independent cost estimates.
CBP has established a process for developing cost estimates in expenditure
plans. This process calls for estimates from both eCP and from an
independent cost estimating contractor. The process also calls for
reconciling these estimates before cost estimates are incorporated into an
expenditure plan.
In formulating the fiscal year 2004 ACE expenditure plan, CBP obtained eCP
cost estimates in early October 2003, and most independent cost estimating
contractor estimates in late October 2003. However, CBP used the eCP
estimates in the expenditure plan without reconciling them to those of the
independent cost estimator. According to program officials, the
reconciliation step was not done because CBP did not have sufficient time
to do so before completing the plan and submitting it for DHS review in
mid-October 2003. However, CBP did task its support contractor with
reconciling the estimates, including analyzing the reasons for
differences. This reconciliation was completed in mid-November 2003, about
2 months before the expenditure plan was submitted to the appropriations
committees. The net difference between eCP's estimates in the plan and the
independent estimates is $3.4 million. However, the sum of the differences
for individual expenditure plan elements is $39.8 million, as shown on the
following table.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Expenditure plan cost estimates partially satisfy relevant guidance.
One indicator of the reliability of cost estimates is the effectiveness of
processes used to derive the estimates. SEI has defined guidelines for
assessing such processes and determining the reliability of estimates.1
To determine the reliability of the cost estimates in the expenditure
plan, the support contractor and eCP analyzed the estimates against SEI
guidance. The cost estimates analyzed pertained to the following plan
elements: Release 6 definition and design, Selectivity 2 definition and
design, foundation program management, architecture and engineering,
workforce transformation and training, and operations and maintenance. The
analysis shows that the expenditure plan estimates partially satisfy the
SEI's guidance. The results of this analysis follow.
1 SEI's institutional estimating guidelines are defined in Checklists and
Criteria for Evaluating the Cost and Schedule Estimating Capabilities of
Software Organizations, and project-specific estimating guidelines are
defined in A Manager's Checklist for Validating Software Cost and Schedule
Estimates.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
The fiscal year 2004 ACE expenditure plan, along with program
documentation and officials' commitments, largely satisfies the
legislative conditions imposed by Congress. Further, many of the
recommendations that we have previously made to strengthen ACE management
have been addressed, and CBP leadership has committed to addressing the
remaining recommendations. However, much remains to be done before these
open recommendations are fully implemented. In particular, progress on
overcoming human capital challenges has been slow.
To CBP's credit, the first two ACE releases are operating largely as
intended. Achieving this initial operating capability, however, has not
occurred without some difficulty, such as borrowing resources from future
releases to overcome problems on near-term releases; similar difficulties
that could impact cost, schedule, and capability commitments are being
experienced on ongoing releases. Prospectively, this pattern is not likely
to change unless the degree of concurrency among activities within and
between releases is better controlled, and the underlying reasons for
introducing this concurrency are addressed. While time and resources were
invested in management improvements over a year ago to address the root
causes, the absence of meaningful measures for determining whether these
investments will be successful, coupled with limited progress on some of
our previous recommendations, makes the chances of CBP delivering future
ACE release capabilities on time and within budget uncertain at best.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
To assist CBP in managing ACE and increasing the chances that future
releases will deliver promised capabilities on time and within budget, we
recommend that the DHS Secretary, through the Under Secretary for Border
and Transportation Security and the CBP Commissioner, direct the CBP CIO
to
o ensure the independence of its IV&V agent;
o ensure that future expenditure plans are based on cost estimates that
are reconciled with independent cost estimates;
o reconsider the ACE acquisition schedule and cost estimates in light of
early release problems, including these early releases' cascading effects
on future releases and their relatively small size compared to later
releases, and in light of the need to avoid the past levels of concurrency
among activities within and between releases;
o define measures, and collect and use associated metrics, for
determining whether prior and future program management improvements are
successful; and
o report quarterly to the House and Senate Appropriations Committees on
efforts to address the above, as well as our previously made but still
open recommendations.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
We provided this briefing to and discussed its contents with DHS, CBP, and
ACE program officials. These officials, including the DHS Chief
Information Officer and the CBP Acting Chief Information Officer,
generally agreed with our findings, conclusions, and recommendations, and
stated that the briefing was fair and balanced. They also provided some
technical comments, which we have incorporated into the briefing, as
appropriate.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Attachment 1
Scope and Methodology
Scope and Methodology
To accomplish our objectives, we analyzed the ACE fiscal year 2004
expenditure plan and supporting documentation, comparing them to relevant
federal requirements and guidance and to applicable best practices; and we
interviewed DHS and CBP officials and ACE program contractors. In
particular, we reviewed
o DHS and CBP investment management practices, using OMB A-11, part 7;
o DHS and CBP activities for assessing ACE compliance with the DHS
enterprise architecture;
o CBP acquisition management practices, using the Software Engineering
Institute's (SEI) Software Acquisition Capability Maturity Model
(SA-CMM);1
1 The Software Acquisition Capability Maturity Model(R) developed by
Carnegie Mellon University's Software Engineering Institute (SEI) defines
five levels of maturity that provide a roadmap for continuously improving
an organization's acquisition management processes.
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Attachment 1
Scope and Methodology
o CBP cost estimating program, using SEI's institutional and
project-specific estimating guidelines;1
o CBP human capital management activities, against its human capital
management strategy;
o ACE testing activities, system defect data, and system performance data
using industry best practices;2
o independent verification and validation (IV&V) activities using the
Institute of Electrical and Electronics Engineers Standard for Software
Verification and Validation;3
1 SEI's institutional estimating guidelines are defined in Checklists and
Criteria for Evaluating the Cost and Schedule Estimating Capabilities of
Software Organizations, and SEI's project-specific estimating guidelines
are defined in A Manager's Checklist for Validating Software Cost and
Schedule Estimates.
2 U.S. General Accounting Office, Year 2000 Computing Crisis: A Testing
Guide, GAO/AIMD-10.1.21 (Washington, D.C.: November 1998).
3Institute of Electrical and Electronics Engineers (IEEE) Standard for
Software Verification and Validation, IEEE Std 10121998 (New York: Mar. 9,
1998).
Appendix I
IT: Early Releases of Customs Trade System
Operating, but Pattern of Cost and Schedule
Problems Needs to Be Addressed
Attachment 1
Scope and Methodology
o cost and schedule data, and functional requirements from program
management documentation; and
o CBP and its contractor activities for improving the ACE program with
program improvement action plans.
We did not independently verify DHS-, CBP-, and contractor-provided data
or calculations nor do we attest to their accuracy.
For DHS-, CBP-, and contractor-provided data that we did not substantiate,
we have made appropriate attribution indicating the data's sources.
We conducted our work at CBP headquarters and contractor facilities in the
Washington, D.C., metropolitan area from January through March 2004 in
accordance with generally accepted government auditing standards.
Appendix II
Comments from the U.S. Department of Homeland Security
Appendix II
Comments from the U.S. Department of
Homeland Security
Appendix II
Comments from the U.S. Department of
Homeland Security
Appendix II
Comments from the U.S. Department of
Homeland Security
Appendix III
GAO Contact and Staff Acknowledgments
GAO Contact Mark Bird, (202) 512-6260
Staff In addition to the person named above, Barbara Collier, Bill Cook,
Neil Doherty, Scott Farrow, Michael P. Fruitman, Tamra Goldstein, Neela
Acknowledgments Lakhmani, and Freda Paintsil made key contributions to
this report.
GAO's Mission The General Accounting Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony
The fastest and easiest way to obtain copies of GAO documents at no cost
is through the Internet. GAO's Web site (www.gao.gov) contains abstracts
and fulltext files of current reports and testimony and an expanding
archive of older products. The Web site features a search engine to help
you locate documents using key words and phrases. You can print these
documents in their entirety, including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document files.
To have GAO e-mail this list to you every afternoon, go to www.gao.gov and
select "Subscribe to e-mail alerts" under the "Order GAO Products"
heading.
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:
U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C.
20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
To Report Fraud, Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htmWaste, and Abuse in E-mail:
[email protected] Federal Programs Automated answering system: (800)
424-5454 or (202) 512-7470
Public Affairs Jeff Nelligan, Managing Director, [email protected] (202)
512-4800 U.S. General Accounting Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
Presorted Standard
Postage & Fees Paid
GAO
Permit No. GI00
United States
General Accounting Office
Washington, D.C. 20548-0001
Official Business
Penalty for Private Use $300
Address Service Requested
*** End of document. ***