Information Technology: Customs Has Made Progress on Automated
Commercial Environment System, but It Faces Long-Standing
Management Challenges and New Risks (31-MAY-06, GAO-06-580).
The Department of Homeland Security (DHS) is conducting a
multiyear, multibillion-dollar acquisition of a new trade
processing system, planned to support the movement of legitimate
imports and exports and strengthen border security. By
congressional mandate, plans for expenditure of appropriated
funds on this system, the Automated Commercial Environment (ACE),
must meet certain conditions, including GAO review. This study
addresses whether the fiscal year 2006 plan satisfies these
conditions; it also describes the status of DHS's efforts to
implement prior GAO recommendations for improving ACE management,
and provides observations about the plan and DHS's management of
the program.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-06-580
ACCNO: A54904
TITLE: Information Technology: Customs Has Made Progress on
Automated Commercial Environment System, but It Faces
Long-Standing Management Challenges and New Risks
DATE: 05/31/2006
SUBJECT: Appropriated funds
Border security
Customs administration
Exporting
Federal procurement
Federal procurement policy
Importing
Performance measures
Procurement planning
Program evaluation
Program management
Information sharing
Customs Service Automated Commercial
Environment System
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-06-580
* Compliance with Legislative Conditions
* Status of Our Open Recommendations
* Observations about ACE Management
* Conclusions
* Recommendations for Executive Action
* Agency Comments
* GAO Contact
* Staff Acknowledgments
* GAO's Mission
* Obtaining Copies of GAO Reports and Testimony
* Order by Mail or Phone
* To Report Fraud, Waste, and Abuse in Federal Programs
* Congressional Relations
* Public Affairs
Report to Congressional Committees
United States Government Accountability Office
GAO
May 2006
INFORMATION TECHNOLOGY
Customs Has Made Progress on Automated Commercial Environment System, but
It Faces Long-Standing Management Challenges and New Risks
GAO-06-580
Contents
Letter 1
Compliance with Legislative Conditions 3
Status of Our Open Recommendations 3
Observations about ACE Management 7
Conclusions 9
Recommendations for Executive Action 11
Agency Comments 12
Appendix I Briefing to Subcommittees on Homeland Security, House and
Senate Committees on Appropriations 13
Appendix II Comments from the Department of Homeland Security 143
Appendix III Contact and Staff Acknowledgments 150
Abbreviations
ACE Automated Commercial Environment
CBP Customs and Border Protection
CBPMO Customs and Border Protection Modernization Office
CIO chief information officer
CMU Carnegie Mellon University
DHS Department of Homeland Security
EA enterprise architecture
EVM earned value management
IT information technology
IV&V independent verification and validation
OMB Office of Management and Budget
ORR operational readiness review
SA-CMM(R) Software Acquisition Capability Maturity Model
SEI Software Engineering Institute
US-VISIT United States Visitor and Immigrant Status Indicator Technology
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
United States Government Accountability Office
Washington, DC 20548
May 31, 2006
The Honorable Judd Gregg Chairman The Honorable Robert C. Byrd Ranking
Minority Member Subcommittee on Homeland Security Committee on
Appropriations United States Senate
The Honorable Harold Rogers Chairman The Honorable Martin Olav Sabo
Ranking Minority Member Subcommittee on Homeland Security Committee on
Appropriations House of Representatives
In February 2006, U.S. Customs and Border Protection (CBP), within the
Department of Homeland Security (DHS), submitted to the Congress its
fiscal year 2006 expenditure plan for the Automated Commercial Environment
(ACE) program. ACE is to be CBP's new import and export processing system.
The program's goals include
o supporting border security by enhancing analysis and
information sharing with other government agencies and
o streamlining time-consuming and labor-intensive tasks for CBP
personnel and the trade community through the provision of a
single Web-based interface.
DHS currently plans to acquire and deploy ACE in 11 increments,
referred to as "releases," in approximately 8.5 years. The first
three releases are deployed and operating, and the fourth release
is being deployed. Other releases are in various states of
definition and development. For the ACE life-cycle, the
risk-adjusted cost estimate is about $2.8 billion; through fiscal
year 2005, ACE-appropriated funding has been about $1.6 billion.
The Department of Homeland Security Appropriations Act of 2006 1
appropriates about $320 million for ACE. The act states that DHS
may not obligate any funds for ACE until it submits for approval
to the House and Senate Committees on Appropriations a plan for
expenditure that
1. meets the capital planning and investment control
review requirements established by the Office of
Management and Budget (OMB), including Circular A-11,
part 7;2
2. complies with DHS's enterprise architecture;
3. complies with the acquisition rules, requirements,
guidelines, and systems-acquisition management
practices of the federal government;
4. includes a certification by the DHS Chief
Information Officer that an independent verification
and validation agent is currently under contract;
5. is reviewed and approved by the DHS Investment
Review Board (IRB),3 the Secretary of Homeland
Security, and OMB; and
6. is reviewed by GAO.
As required by the act, we reviewed the ACE fiscal year 2006
expenditure plan. Our objectives were to (1) determine whether the
expenditure plan satisfies certain legislative conditions, (2)
determine the status of our open ACE recommendations, and (3)
provide any other observations about the expenditure plan and
DHS's management of the ACE program.
On March 10, 2006, we briefed your offices on the results of this
review. This report transmits the results of our work. The full
briefing, including our scope and methodology, can be found in
appendix I.
The legislative conditions that the Congress placed on the use of
fiscal year 2006 ACE appropriated funds have been either partially
or fully satisfied by the latest expenditure plan and related
program documentation and activities. However, more can be done to
better address several aspects of these conditions. For example:
o One legislative condition states that the plan should meet
OMB's capital planning and investment control review requirements,
which include addressing security and privacy issues. However, a
privacy impact assessment4 for ACE has been in draft for several
months and is not yet approved. Another capital planning and
investment control review requirement is that performance goals
and measures be provided in the business case for ACE. Although
CBP describes selected performance goals and measures, the goals
(i.e., targets) are not always realistic (we provide further
discussion of this issue later in this report).
o According to another legislative condition, the expenditure
plan must comply with DHS's enterprise architecture. However, DHS
does not have a documented methodology for evaluating programs for
compliance with its enterprise architecture, other than relying on
the professional expertise of its staff.
o According to a third legislative condition, the DHS Chief
Information Officer is to certify that an independent verification
and validation (IV&V) agent is under contract. Although DHS
satisfied this condition, the scope of the IV&V contractor's
activities is not consistent with the operative industry standard,
which states that IV&V should extend to key system products and
development processes.5
CBP has addressed some recommendations, while progress has been
slow on others. Each recommendation, along with the status of
actions to address it, is summarized below.
o Ensure that future expenditure plans are based on cost
estimates that are reconciled with independent cost estimates.
Complete.6 In October 2005, CBP, with contractor support, compared
the program plan cost estimate with the independent cost estimate.
According to the analysis performed, the two estimates are
consistent.
o Develop and implement a rigorous and analytically verifiable
cost estimating program that embodies the tenets of effective
estimating, as defined in the institutional and project-specific
estimating models developed by the Software Engineering Institute
(SEI).7
In progress. CBP has taken steps such as (1) hiring a contractor
to develop cost estimates (including contract task order cost
estimates) that are independent of CBP's estimates, and (2)
tasking a support contractor with evaluating both the independent
and CBP estimates against the criteria defined by SEI. According
to the results of the support contractor's evaluation, the
independent estimates satisfied the SEI criteria; CBP's estimates
largely satisfied the criteria.
However, according to the support contractor, CBP's cost
estimating had limitations. First, the CBP estimate did not
adequately consider past projects in its cost and schedule
estimates. In addition, the CBP estimate was an aggregation of
estimates developed separately for three ACE components, each
according to a different cost estimating methodology; the support
contractor advised against this approach, recommending that
component estimates be based on the same methodology.
o Immediately develop and implement a human capital management
strategy that provides both near- and long-term solutions to
program office human capital capacity limitations, and report
quarterly to the Appropriations Committees on the progress of
efforts to do so.
In progress. CBP has expanded its contractor and government
workforce dedicated to the ACE program by merging staff assigned
to trade-related legacy systems with the ACE program staff. In
addition, it is beginning to use subject matter experts from
existing field operations advisory boards to help program
officials define requirements for future releases. However, it
does not have a documented human capital strategy covering its ACE
program.
o Have future ACE expenditure plans specifically address any
proposals or plans, whether tentative or approved, for extending
and using ACE infrastructure to support other homeland security
applications, including any impact on ACE of such proposals and
plans.
In progress. The expenditure plan describes steps both planned and
under way to ensure that ACE infrastructure supports both ACE and
other homeland security applications. For example, it states that
both ACE and the United States Visitor and Immigrant Status
Indicator Technology8 (US-VISIT) program should conform to the DHS
enterprise architecture, which is to define standard shared
services that the two systems can request. Such a services
oriented architecture is intended to promote reuse, as well as
reducing overlap and duplication.
o Define measures, and collect and use associated metrics, for
determining whether prior and future program management
improvements are successful.
In progress. CBP continues to make changes that are intended to
improve overall program management, but it has not consistently
defined measures to determine whether the changes are successful.
For example, CBP has reorganized its Office of Information
Technology; this reorganization is intended to improve program
management by providing (1) enhanced government oversight of ACE
development, (2) better definition of requirements for future ACE
releases, and (3) faster and cheaper delivery of ACE capabilities.
However, program officials told us that they have not established
measures or targets for determining whether the reorganization is
providing these benefits.
o Define and implement an ACE accountability framework that
fulfills several conditions:
1. The framework should cover all program commitment
areas, including key expected or estimated system (a)
capabilities, use, and quality; (b) benefits and
mission value; (c) costs; and (d) milestones and
schedules.
In progress. CBP has prepared an initial version of
an accountability framework that it intends to
improve as it proceeds. The framework is built around
measuring progress against costs, milestones,
schedules, and risks for select releases; however,
the benefit measurement has not been well defined,
and the performance targets are not always realistic.
2. The framework should ensure currency,
relevance, and completeness of all
program commitments made to the Congress
in expenditure plans.
In progress. The fiscal year 2006
expenditure plan includes inaccurate,
dated, and incomplete information and
omits other relevant information. For
example, the plan did not include
information regarding CBP's decision to
eliminate the dependencies among the
screening and targeting releases and the
cargo releases, and to take advantage of
the capabilities of its existing
Automated Targeting System.9
3. The framework should ensure reliable
data that are relevant to measuring
progress against commitments.
In progress. The data that CBP uses to
measure progress against commitments are
not consistently reliable. For example,
data in the defect tracking system show
that defects in Release 4 (which is now
operational) have not been closed;
however, program officials told us that
many of these defects have been
resolved.
4. The framework should ensure that
future expenditure plans report progress
against commitments contained in prior
expenditure plans.
In progress. The current expenditure
plan does not adequately describe
progress against commitments made in
previous plans. For example, the plan
provides a summary of the funding
requested in each of the previous six
expenditure plans, but it does not
provide information on whether these
funding amounts were actually expended
or obligated as planned.
5. The framework should ensure that
criteria for exiting key readiness
milestones adequately consider
indicators of system maturity, such as
the severity of open defects.
In progress. ACE milestone exit criteria
provide for addressing the risk
associated with severe defects that are
unresolved. Using these criteria, CBP
passed several release milestones with
severe defects still open. However, CBP
officials were unable to provide us with
any documentation on how they assessed
the inherent risks of passing these
milestones with open severe defects.
6. The framework should ensure clear and
unambiguous delineation of the
respective roles and responsibilities of
the government and the prime contractor.
Complete. The current ACE program plan
describes general roles and
responsibilities for the government and
the prime contractor. More detailed
roles have been documented in a roles
and responsibilities matrix that assigns
primary responsibility for each
activity.
o Report quarterly to the House and Senate Appropriations
Committees on efforts to address our open recommendations.
In progress. CBP submitted quarterly reports to both Committees on
its efforts to address our open recommendations; however, progress
in addressing our recommendations was not always reported
accurately.
We have several observations about the development of ACE
releases, as well as several more concerning the performance of
ACE releases that are deployed and operating.
o ACE development: Steps have been taken to address a past
pattern of ACE release shortfalls, but new release management
weaknesses are emerging.
As we have previously observed, CBP established a pattern of
borrowing resources from future releases to address problems with
the quality of earlier releases; this led to schedule delays and
cost overruns. This pattern has continued with the most recently
deployed cargo release, which developed problems that caused
delays with a subsequent screening and targeting release.10 CBP
took steps to mitigate this problem by eliminating the
dependencies between the cargo releases and the screening and
targeting releases.
However, CBP's planned schedule for developing additional releases
includes a significant level of concurrence, because of CBP's
interest in delivering ACE functionality sooner. Such concurrence
between ACE release activities has led to cost overruns and
schedule delays in the past. Thus, the revised ACE plans and
actions are potentially reintroducing the same problems that
produced past shortfalls.
We made several specific observations related to these weaknesses,
including the following:
o On two recent releases, key milestones were passed
despite unresolved severe defects. Officials told us
that the risk of proceeding did not outweigh the need
to get the releases to users, and thereby gaining
user acceptance and feedback. However, the risks were
not documented and formally managed.
o Concurrence in developing early ACE releases
caused schedule slips and cost overruns. Despite
these experiences, CBP has established a risky plan
that involves considerable overlap across the
development schedules for three future releases.
o Although the use of earned value management (EVM)
is an OMB requirement, it was not being used to
manage the development of two recent releases.11 For
example, CBP discontinued use of EVM on one release
because this method was not familiar to staff who
were transferred to work on the program.
o ACE operations: Operational performance has been mixed, and
mission impact is unclear.
ACE releases one through four are in operation. To date, these
releases' operational performance has been uneven. For example,
ACE has largely been meeting its goals for being available and
responsive in processing virtually all daily transactions, and has
decreased truck processing times at some ports. However, ACE is
not being used by as many CBP and trade personnel as was expected,
and truck processing times at other ports have increased.
Moreover, overall user satisfaction has been low.
In addition, ACE goals, expected mission benefits, and performance
measures are not fully defined and adequately aligned with each
other. For example, not every goal has defined benefits, every
benefit is defined only in terms of efficiency gains, not every
benefit has an associated business result, and not every benefit
and business result has associated performance measures.
Further, where performance measures have been defined, the
associated targets are not always realistic. For example, the
performance target in fiscal year 2005 for ACE usage was that 11
percent of all CBP employees would use ACE. However, that many CBP
employees will never need to use the system. This performance
target does not reflect that.
Because performance measures are not always realistic or aligned
with program goals and benefits, it is unclear whether ACE has
realized-or will realize-the mission value that it was intended to
bring to CBP's and other agencies' trade- and border
security-related operations.
The legislative conditions that the Congress placed on the use of
fiscal year 2006 ACE appropriated funds have been either partially
or fully satisfied by the latest expenditure plan and related
program documentation and activities. Nevertheless, more can be
done to better address several aspects of these conditions, such
as ensuring that the program's privacy impact assessment is
approved, measuring ACE performance and results, ensuring
architectural alignment, and employing effective IV&V practices.
Given that the legislative conditions are collectively intended to
promote accountability and increase the chances of program
success, it is important that each receives DHS's full attention.
Also important to ACE's success is the swift and complete
implementation of the recommendations that we have previously made
to complement the legislative conditions and improve program
management, performance, and accountability. In this regard, some
recommendations have been addressed, while progress has been slow
on others, such as
o accurately reporting to the Appropriations Committees on CBP's
progress in implementing our prior recommendations;
o developing and implementing a strategic approach to meeting the
program's human capital needs;
o using criteria for exiting key milestones that adequately
consider indicators of system maturity, such as severity of open
defects and the associated risks; and
o developing and implementing a performance and accountability
framework for ensuring that promised capabilities and benefits are
delivered on time and within budget.
To its credit, CBP has taken several steps to stem the pattern of
cost, schedule, and performance shortfalls that it experienced on
early ACE releases. However, future releases are unlikely to
realize the impact of these steps because revised ACE plans and
actions are reintroducing the same pattern that led to early
release shortfalls. This pattern includes not formally and
transparently considering, and proactively addressing, the risks
associated with passing key release milestones with known severe
defects; building considerable overlap and concurrence in the
development schedules of releases that will contend for the same
resources; and not performing EVM on all releases. If this pattern
continues, the prospects for a successful program will be
diminished.
Although availability and responsiveness targets are largely being
met and long-standing help desk limitations are being addressed,
the prospects for a successful program nevertheless remain
unclear. The true measure of ACE's success is arguably the mission
value that it brings to CBP's and other agencies' trade- and
border security-related operations and users. Such value depends
both on the operational performance of ACE and on CBP's ability to
demonstrate that this performance is achieving program goals,
delivering expected benefits, and producing desired business
results. At this juncture, however, neither the system's
performance nor its value is clear because of several factors: the
operational performance of deployed releases has been mixed;
users' satisfaction has been low; the relationships among goals,
benefits, and desired business outcomes are not evident; and the
range of measures needed to create a complete and realistic
picture of ACE's performance is missing.
In summary, a number of ACE activities have been and are being
done well; these have contributed to the program's progress to
date and will go a long way in determining the program's ultimate
success. However, it will be important for CBP to effectively
address long-standing ACE management challenges along with
emerging problems. Until it does so, ACE will remain a risky
program.
To assist CBP in managing ACE-and increasing the chances that it
will deliver required capabilities on time and within budget,
demonstrating promised mission benefits and results-we recommend
that the Secretary of Homeland Security direct the appropriate
departmental officials to fully address those legislative
conditions associated with having an approved privacy impact
assessment and ensuring architectural alignment.
We also recommend that the Secretary, through CBP's Acting
Commissioner, direct the Assistant Commissioner for Information
and Technology to
o fully address those legislative conditions associated with
measuring ACE performance and results and employing effective IV&V
practices;
o accurately report to the appropriations committees on CBP's
progress in implementing our prior recommendations;
o include in the June 30, 2006, quarterly update report to the
appropriations committees a strategy for managing ACE human
capital needs and the ACE framework for managing performance and
ensuring accountability;
o document key milestone decisions in a way that reflects the
risks associated with proceeding with unresolved severe defects
and provides for mitigating these risks;
o minimize the degree of overlap and concurrence across ongoing
and future ACE releases, and capture and mitigate the associated
risks of any residual concurrence;
o use EVM in the development of all existing and future releases;
o develop the range of realistic ACE performance measures and
targets needed to support an outcome-based, results-oriented
accountability framework, including user satisfaction with ACE;
and
o explicitly align ACE program goals, benefits, desired business
outcomes, and performance measures.
In written comments on a draft of this report signed by the
Director, Departmental GAO/OIG Liaison, DHS agreed with our
findings concerning progress in addressing our prior
recommendations, and it agreed with the recommendations in this
report. DHS also described actions that it has under way and
planned to address the recommendations. The department's comments
are reprinted in appendix II.
We are sending copies of this report to the Chairmen and Ranking
Minority Members of other Senate and House committees and
subcommittees that have authorization and oversight
responsibilities for homeland security. We are also sending copies
to the DHS Secretary, the CBP Commissioner and, upon their
request, to other interested parties. In addition, the report will
be available at no charge on the GAO Web site at
http://www.gao.gov.
Should you or your offices have any questions on matters discussed
in this report, please contact me at (202) 512-3459 or at
[email protected]. Contact points for our Offices of Congressional
Relations and Public Affairs may be found on the last page of this
report. Other contacts and key contributors to this report are
listed in appendix III.
Randolph C. Hite Director, Information Technology Architecture and
Systems Issues
Randolph C. Hite, (202) 512-3459
In addition to the person named above, Justin Booth, Barbara
Collier, William Cook, Neil Doherty, Michael Marshlick, Shannin
O'Neill, Tomas Ramirez, and Jennifer Vitalbo made key
contributions to this report.
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in
meeting its constitutional responsibilities and to help improve
the performance and accountability of the federal government for
the American people. GAO examines the use of public funds;
evaluates federal programs and policies; and provides analyses,
recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
The fastest and easiest way to obtain copies of GAO documents at
no cost is through GAO's Web site ( www.gao.gov ). Each weekday,
GAO posts newly released reports, testimony, and correspondence on
its Web site. To have GAO e-mail you a list of newly posted
products every afternoon, go to www.gao.gov and select "Subscribe
to Updates."
The first copy of each printed report is free. Additional copies
are $2 each. A check or money order should be made out to the
Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are
discounted 25 percent. Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM
Washington, D.C. 20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax:
(202) 512-6061
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail:
[email protected] Automated answering system: (800) 424-5454 or
(202) 512-7470
Gloria Jarmon, Managing Director, [email protected] (202) 512-4400
U.S. Government Accountability Office, 441 G Street NW, Room 7125
Washington, D.C. 20548
Paul Anderson, Managing Director, [email protected] (202)
512-4800 U.S. Government Accountability Office, 441 G Street NW,
Room 7149 Washington, D.C. 20548
1Pub. L. 109-90 (Oct. 18, 2005).
2OMB Circular A-11 establishes policy for planning, budgeting,
acquisition, and management of federal capital assets.
3The purpose of the Investment Review Board is to integrate capital
planning and investment control, along with the budgeting, acquisition,
and management of investments. It is also to ensure that spending on
investments directly supports and furthers the mission, and that this
spending provides optimal benefits and capabilities to stakeholders and
customers.
Compliance with Legislative Conditions
Status of Our Open Recommendations
4The purpose of a privacy impact assessment is to ensure that there is no
collection, storage, access, use, or dissemination of identifiable
personal or business information that is not both needed and permitted.
5Institute of Electrical and Electronics Engineers Computer Society,
Standard for Software Verification and Validation 1012-1998 (June 8,
2005).
6CBP's implementation of this recommendation is complete with respect to
the fiscal year 2006 expenditure plan.
7SEI's institutional and project-specific estimating guidelines are
defined respectively in Robert E. Park, Checklists and Criteria for
Evaluating the Cost and Schedule Estimating Capabilities of Software
Organizations, CMU/SEI-95-SR-005, and A Manager's Checklist for Validating
Software Cost and Schedule Estimates, CMU/SEI-95-SR-004 (Pittsburgh, Pa.:
Carnegie Mellon University Software Engineering Institute, 1995).
8US-VISIT is a governmentwide program to collect, maintain, and share
information on foreign nationals for enhancing national security and
facilitating legitimate trade and travel, while adhering to U.S. privacy
laws and policies.
9The Automated Targeting System is a DHS system that targets containers
for inspection based on a perceived level of risk.
Observations about ACE Management
10Screening is the method of determining high-risk people or shipments
before their arrival at a port. Targeting is the risk-based determination
of whether a shipment should undergo additional documentary review or
physical inspection.
11Earned value management is a tool for measuring program progress by
comparing the value of work accomplished during a given period with that
of the work expected in that period; this comparison permits performance
to be evaluated based on calculated variances from the planned cost and
schedule.
Conclusions
Recommendations for Executive Action
Agency Comments
Appendix I: Briefing to Subcommittees on Homeland Security, House and
Senate Committees on Appropriations Appendix I: Briefing to Subcommittees
on Homeland Security, House and Senate Committees on Appropriations