Year 2000 Computing Crisis: Customs Has Established Effective Year 2000
Program Controls (Letter Report, 03/29/99, GAO/AIMD-99-37).

Pursuant to a congressional request, GAO evaluated the Customs Service's
efforts to address its year 2000 computing problem, focusing on whether
Customs has established effective management structures and processes
for managing and reporting on key aspects of its Year 2000 program.

GAO noted that: (1) Customs has established effective Year 2000 program
management controls, including structures and processes for year 2000
testing, contingency planning, and year 2000 status reporting; (2) as a
result, the agency's latest status reports to the Department of the
Treasury show good progress in converting its systems and mitigating
century date change risks to its core business operations; (3)
specifically, as of January 1999, Customs had met milestones recommended
by the Office of Management and Budget for renovating and validating
most of its mission-critical systems; (4) also, Customs has actions
under way, and plans and management controls in place, to help ensure
that it completes remaining validation and implementation activities for
all its mission-critical systems by June 1999; (5) very important tasks
remain to be accomplished, such as completing end-to-end tests and
validating contingency plans for ensuring continuity of core business
functions, and serious risks outside of Customs' control remain, such as
year-2000-induced failures of both public infrastructure and business
partner systems; (6) Customs has plans in place for completing key tasks
and addressing external risks, and it has the management controls in
place to ensure that they are accomplished; and (7) while these controls
do not guarantee that year-2000-induced system failures will not occur,
if Customs follows through on its plans and continues to implement its
management controls as it has to date, its risk of year-2000-induced
business failures will be effectively reduced.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  AIMD-99-37
     TITLE:  Year 2000 Computing Crisis: Customs Has Established 
             Effective Year 2000 Program Controls
      DATE:  03/29/99
   SUBJECT:  Y2K
             Computer software verification and validation
             Systems conversions
             Strategic information systems planning
             Information resources management
             Data integrity
             Computer software
             Customs administration
IDENTIFIER:  Y2K
             Customs Service Year 2000 Program
             
******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  This text was extracted from a PDF file.        **
** Delineations within the text indicating chapter titles,      **
** headings, and bullets have not been preserved, and in some   **
** cases heading text has been incorrectly merged into          **
** body text in the adjacent column.  Graphic images have       **
** not been reproduced, but figure captions are included.       **
** Tables are included, but column deliniations have not been   **
** preserved.                                                   **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                                            **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************
ai99037.PDF GAO United States General Accounting Office

Report to Congressional Requesters

March 1999 YEAR 2000 COMPUTING CRISIS

Customs Has Established Effective Year 2000 Program Controls




GAO/AIMD-99-37

  GAO/AIMD-99-37

United States General Accounting Office Washington, D. C. 20548
Letter

Page 1 GAO/AIMD-99-37 Customs' Year 2000 Program Controls

GAO

Accounting and Information Management Division

B-281520 Letter March 29, 1999 The Honorable Amo Houghton
Chairman, Subcommittee on Oversight Committee on Ways and Means
United States House of Representatives

The Honorable Philip M. Crane Chairman, Subcommittee on Trade
Committee on Ways and Means United States House of Representatives

This report responds to your requests that we evaluate the United
States Customs Service's efforts to address its Year 2000
computing problem. 1 If key automated systems affecting trade
between the United States and other countries (valued at over a
trillion dollars a year) malfunction, trade processing could be
delayed, trade revenue lost, and illegal activities, such as
narcotics smuggling, money laundering, and commercial fraud, could
increase.

The objective of our review was to determine whether Customs has
established effective management structures and processes for
managing and reporting on key aspects of its Year 2000 program. We
performed our work at Customs headquarters in Washington, D. C.,
and the Newington Data Center in Newington, Virginia, from July
1998 through January 1999 in accordance with generally accepted
government auditing standards. In brief, we analyzed Customs' Year
2000 program management against our Year 2000 guidance 2 to
determine whether key management controls were in place and
functioning, and we traced the reported status of selected system
components back to Customs' documentation to verify the status
information's accuracy. Appendix I provides details on our
objective,

1 The problem is rooted in the way dates are recorded and computed
in automated information systems. For the past several decades,
systems have typically used two digits to represent the year, such
as "97" representing 1997, in order to conserve electronic data
storage and reduce operating costs. With this two- digit format,
however, the year 2000 is indistinguishable from 1900, or 2001
from 1901. As a result of this ambiguity, system and application
programs that use dates to perform calculations, comparisons, or
sorting may generate incorrect results.

2 Year 2000 Computing Crisis: An Assessment Guide (GAO/ AIMD-
10.1.14, issued as an exposure draft in February 1997, issued
final in September 1997); Year 2000 Computing Crisis: Business
Continuity and Contingency Planning (GAO/ AIMD- 10.1.19, issued as
an exposure draft in March 1998; issued final in August 1998); and
Year 2000 Computing Crisis: A Testing Guide (GAO/ AIMD- 10.1.21,
issued as an exposure draft in June 1998; issued final in November
1998).

B-281520 Page 2 GAO/AIMD-99-37 Customs' Year 2000 Program Controls

scope, and methodology. We requested comments on a draft of this
report from the Commissioner of Customs or his designee. The
Commissioner provided us with written comments, which are
discussed in the Agency Comments section of this report.

Results in Brief Customs has established effective Year 2000
program management controls, including structures and processes
for Year 2000 testing,

contingency planning, and Year 2000 status reporting. As a result,
the agency's latest status reports to the Department of the
Treasury show good progress in converting its systems and
mitigating century date change risks to its core business
operations. Specifically, as of January 1999, Customs has met
milestones recommended by the Office of Management and Budget
(OMB) for renovating and validating most of its mission- critical
systems. 3 Also, Customs has actions underway, and plans and
management controls

in place, to help ensure that it completes remaining validation
and implementation activities for all its mission- critical
systems by June 1999.

Very important tasks remain to be accomplished, such as completing
endto- end tests and validating contingency plans for ensuring
continuity of core business functions, and serious risks outside
of Customs' control remain, such as Year 2000- induced failures of
both public infrastructure and business partner systems. Customs
has plans in place for completing key tasks and addressing
external risks, and it has the management controls in place to
ensure that they are accomplished. While these controls do not
guarantee that Year 2000- induced system failures will not occur,
if Customs follows through on its plans and continues to implement
its management controls as it has to date, its risk of Year 2000-
induced business failures will be effectively reduced.

Background Customs' mission is to ensure that all goods and
persons entering and exiting the United States do so in compliance
with all U. S. laws and

regulations. It does this by (1) enforcing the laws governing the
flow of goods and persons across the borders of the United States
and (2) assessing and collecting duties, taxes, and fees on
imported merchandise. To accomplish these goals, Customs has
identified and organized its

3 OMB requires that agencies complete renovation of their systems
by September 1998, validation by January 1999, and implementation
by March 1999.

B-281520 Page 3 GAO/AIMD-99-37 Customs' Year 2000 Program Controls

operations around six core business missions-- trade compliance,
passenger, outbound, finance, human resources, and enforcement/
investigations. See appendix II for a description of these core
business missions.

To carry out its responsibilities, Customs relies extensively on
information technology. For example, Customs depends upon the
Automated Commercial System as its primary vehicle for tracking,
controlling, and processing all commercial goods imported into the
United States. In addition, Customs uses cargo inspection systems
and x- ray systems to screen and inspect cargo for narcotics and
other contraband. Since every computer- controlled system may be
vulnerable to Year 2000 failure, Customs' Year 2000 problem
extends to all of its systems.

Customs' Mission- Critical Systems: A Brief Description

In managing its Year 2000 program, Customs divided its mission-
critical systems into information technology (IT) and non- IT
systems (e. g., office equipment, facilities, security systems,
and vehicles). Customs has five IT systems that run mission-
critical applications: the Automated Commercial System (ACS),
Treasury Enforcement Communications System (TECS), Administrative
System (ADMIN), Automated Export System (AES), and Seized Asset
and Case Tracking System (SEACATS). These five systems are
mainframe- based and are accessed by users around the country from
terminals or personal computers (PC) emulating terminals. 4
Customs' IT systems also include its telecommunication systems.
See appendix III for a description of Customs' IT and non- IT
systems.

Customs Reports That Y2K Remediation of Most Mission- Critical
Systems Is on Schedule

OMB's guidance, as amended in January 1998, requires that agencies
complete renovation of their systems by September 1998, validation
by January 1999, and implementation by March 1999. Customs'
reported status of its mission- critical IT and non- IT systems
are described below. 5 As of January 1999, Customs reported that
it had renovated and validated

its mission- critical application software components and was
running the application software components in a production
environment. Customs

4 A program that enables a microcomputer to appear to be a
mainframe terminal by using procedures and codes compatible with
the mainframe computer.

5 We did not independently validate all of Customs' reported
system status information.

B-281520 Page 4 GAO/AIMD-99-37 Customs' Year 2000 Program Controls

also reported that it has completed systems acceptance testing of
its five mission- critical systems (i. e., the full complement of
application software running on the target hardware and systems
software infrastructure). Endto- end testing for these systems and
associated telecommunications systems is scheduled to be completed
by March 1999. (See sections on system acceptance testing and end-
to- end testing for further explanation.)

Customs' telecommunications systems include (1) equipment at
Customs' national data center where its mainframe- based
applications are operated and maintained, 6 (2) local equipment in
Customs' field offices that supports intra- office communications
and connects these offices to the data center, and (3)
headquarters and field office voice communications, including
telephone and voice mail systems. As of January 1999, Customs
reported that:

 National data center- related telecommunication systems were 100
percent assessed, and 4 percent of this inventory required
renovation or replacement. Of this 4 percent, Customs reports that
92 percent has been renovated, validated, and implemented. Customs
plans for the remaining 8 percent to be fully implemented by March
1999.  Field offices' telecommunications equipment was 100 percent
assessed

and all of the equipment required renovation or replacement.
Customs also reported that 68 percent of the equipment had been
renovated, validated, and implemented and that Customs plans to
complete implementation of the remaining 32 percent by May 1999.
Voice communications were 100 percent assessed, and 50 percent

required renovation or replacement. Of this 50 percent, Customs
reported that 40 percent had been renovated or replaced,
validated, and implemented. Customs plans to complete
implementation of the remaining 60 percent of its voice
telecommunications by June 1999.

Customs is still assessing some of its non- IT equipment. As of
January 1999, Customs reported that 82 percent of the mission-
critical non- IT products had been assessed, and that of this 82
percent, 95 percent of the products were compliant, 4 percent
require renovation or replacement, and 1 percent is being retired.
Customs plans to complete all assessment, validation, and
implementation activities by May 1999.

6 This equipment includes front- end processors, external dial- in
rotaries for the trade community, and network connection devices,
such as nodes, pads, and switches.

B-281520 Page 5 GAO/AIMD-99-37 Customs' Year 2000 Program Controls

Customs Has Implemented an Effective Year 2000 Management
Structure

According to our Year 2000 assessment guide, a successful Year
2000 program begins with establishing an effective Year 2000
program management structure. Such a structure includes an
empowered and accountable program office and program manager. It
also includes an active committee of agency executives to guide,
direct, and facilitate program office efforts.

Our guidance also states that, among other things, Year 2000
program management organizations should develop comprehensive Year
2000 strategies and plans that (1) identify organizational roles
and responsibilities and define tasks, (2) establish schedules,
(3) establish reporting requirements, (4) define performance
measures, and (5) estimate and allocate resources. Program
management organizations should also develop and implement Year
2000 guidance and standards and should establish processes within
their Year 2000 management structure for ensuring that guidance is
understood and for tracking progress against plans.

Customs Has a Well- Defined Year 2000 Program Management Structure

Customs established its Year 2000 Program Office and designated a
Year 2000 Program Manager in May 1997. Customs officially
chartered the Program Office in October 1997, giving it authority
over and responsibility for agencywide Year 2000 efforts,
including such functional areas as Year 2000 contracting,
budgeting and planning, technical support to project teams,
quality assurance, auditing, and reporting. Team leaders were
established within the Program Office to manage these functional
efforts. Also, the program was structured around IT (mainframe
systems and telecommunications systems) and non- IT systems, and
managers for each area were designated and empowered. These
functional team leaders and area managers are accountable to the
Year 2000 Program Manager, who is accountable to the chief
information officer (CIO).

In addition to chartering the Year 2000 Program Office, Customs
also engaged its senior executives by charging the agency's
Executive Council with approving and overseeing the implementation
of the Year 2000 strategy and resolving such issues as
institutional Year 2000 priorities. The Council is co- chaired by
the CIO and the chief financial officer and includes the Year 2000
project managers as members. The Executive Council reports to
Customs' Investment Review Board.

B-281520 Page 6 GAO/AIMD-99-37 Customs' Year 2000 Program Controls

Customs Has Defined Year 2000 Strategies, Plans, Guidance, and
Standards

The Year 2000 Program Office, in collaboration with the Executive
Council, issued Customs' Year 2000 Strategic Program Management
Plan and Operational Program Management Plan in June 1998. 7
Consistent with our

Year 2000 guidance, these plans (1) identify organizational roles
and responsibilities and define tasks; (2) establish schedules for
completing each program phase (i. e., awareness, assessment,
renovation, validation, and implementation) and describe the tasks
to be completed under each phase; (3) establish reporting
requirements to track progress in the various phases; (4) define
performance measures; and (5) estimate and allocate resources for
the tasks and system activities within these phases.

In addition to the strategic and operational plans, the Year 2000
Program Office has issued policies, guidelines, and procedures for
managing and implementing the Year 2000 program in accordance with
our guidance. For example, the Program Office has issued Year 2000
quality assurance, configuration management, and testing guidance
to use in managing the conversion of its IT systems, as well as
Year 2000 business continuity and contingency planning guidelines
to be used by business owners, field offices, and project
managers.

To ensure that the plans, policies, and guidelines are being
implemented, the Year 2000 Program Manager is (1) holding weekly
status meetings with the Year 2000 Program Office staff and the
project teams; (2) tracking, prioritizing, and managing the risks
associated with the IT and non- IT system conversion efforts; (3)
overseeing and managing budget- related issues; and (4) conducting
internal audit reviews to monitor and assess the implementation of
established Year 2000 procedures. For example, in June 1998 the
Program Office assessed the implementation of configuration
management and testing procedures for mission- critical IT
systems, identified weaknesses in the procedures, recommended
solutions, and followed up on the findings and recommendations to
ensure that configuration management and testing procedures are
consistently followed across Year 2000 projects. Program Office
staff are also tracking the development and providing quality
reviews of the contingency plans for continuity of operations.

The Customs' Year 2000 Program Office has also developed a central
database for tracking progress against plans and for identifying
issues that

7 Customs initially developed a draft management plan in July
1997.

B-281520 Page 7 GAO/AIMD-99-37 Customs' Year 2000 Program Controls

may affect the strategy. The database includes information on the
status of systems conversion (e. g., schedule and risks) and
external interface information. The database provides the
information required for Customs' monthly and quarterly reporting
to Treasury.

The Year 2000 Program Office has also developed estimates of
resource needs and has obtained and allocated funding to support
these needs. Specifically, Customs' Year 2000 cost estimate from
fiscal year 1997 through fiscal year 2000 is $118 million. This
includes the costs to renovate over 20 million lines of COBOL in
mainframe- based applications, telecommunications equipment,
personal computers, commercial off- theshelf software (COTS), and
non- IT systems. It also includes Year 2000 Program Office support
costs. As of January 1999, Customs reported that it had spent $83
million on Year 2000 efforts.

Customs Has Adopted a Structured and Disciplined Approach to
Managing Year 2000 Testing Activities

Complete and thorough testing is essential to provide reasonable
assurance that new or modified systems process dates correctly and
will not jeopardize an organization's ability to perform core
business operations during and after the transition to a Year 2000
computing environment. Our Year 2000 test guide describes a
structured and disciplined approach to Year 2000 test activities.
This five- phased approach begins with establishing an
organizational testing infrastructure, followed by designing,
conducting, and reporting on four incremental levels of system-
related testing (software unit testing, software integration
testing, system acceptance testing, and end- to- end testing).

Customs Has Established an Effective Testing Infrastructure

The purpose of establishing an effective testing infrastructure is
to put in place a structured and disciplined framework for
managing (i. e., planning, directing, controlling, overseeing, and
reporting) each of the next four phases of testing (i. e.,
software unit testing, software integration testing, system
acceptance testing, and end- to- end testing). Our test guide
defines 11 key processes associated with establishing an effective
testing infrastructure, including (1) assigning Year 2000 test
management authority and responsibility, (2) establishing an
agencywide definition of Year 2000 compliance, (3) engaging
independent quality assurance and verification and validation
agents, (4) providing for Year 2000 compliance of vendorsupported
products, and (5) establishing a Year 2000 test environment.

Customs has satisfied each of the 11 key processes. For example,
Customs (1) designated a Year 2000 test manager for mission-
critical IT systems and

B-281520 Page 8 GAO/AIMD-99-37 Customs' Year 2000 Program Controls

assigned this manager authority and responsibility for key test
activities (e. g., defining exit criteria, designing and planning
the tests, executing the tests); (2) established in its Year 2000
Application Testing Strategy and Plan an agencywide definition of
Year 2000 compliance (i. e., proper handling of

system date changes, correct manipulation of date- related data,
proper storage of dates with four- digit years, and accurate
inference of century values given two- digit years); (3) engaged
an independent verification and validation (IV& V) agent to ensure
that process standards have been followed and that software
products perform as intended; 8 (4) provided for testing of
vendor- supported IT and non- IT products (e. g., vendor-
certified mainframe operating systems and utilities, and PC
equipment and COTS software) to ensure that they are Year 2000
compliant; and (5) established a Year 2000 test environment
consisting of a Year 2000 logical partition 9 on Customs'
mainframe computer along with Year 2000 compliant
telecommunications components that replicate the organization's
operations in a future- date environment. By implementing these
key processes, Customs has established an effective framework for
managing Year 2000 testing.

Customs Has Completed Software Unit and Integration Testing for
Its Mission- Critical IT Systems

The purpose of software unit and integration testing is to verify
that units of software, both individually and combined, work as
intended. According to our test guide, effective unit and
integration testing includes, among other things, (1) developing
unit and integration test plans, (2) preparing test procedures and
data, (3) documenting test results, (4) correcting defects, and
(5) ensuring that test exit criteria are met.

Customs has completed software unit and integration testing for
its mission- critical IT systems. For the system components that
we reviewed, Customs satisfied all of the unit and integration
test phase key processes. For example, Customs (1) developed
combined unit and integration test plans that specified the Year
2000 compliance criteria defined in the Year 2000 Application
Testing Strategy and Plan; (2) prepared test procedures

and data that included Year 2000 date conditions specified in test
guidance 8 The agent is in the process of using an automated tool
to, for example, independently analyze renovated and tested
mainframe- based code for Year 2000 problems, such as conflicting
date formats and deviations from date windowing standards. As of
January 1999, the agent had analyzed 84 percent of Customs'
mainframe- based code and found .7 errors per 10,000 lines of
code.

9 A logical partition is a distinct portion of memory that
functions as though it were a physically separate unit.

B-281520 Page 9 GAO/AIMD-99-37 Customs' Year 2000 Program Controls

and specified, among other things, test results documentation
requirements and test exit criteria; (3) documented test results,
including problems; (4) corrected documented problems; and (5)
ensured that exit criteria were met by requiring user and tester
sign- off approval before software was moved into production and
the Year 2000 logical partition for subsequent system acceptance
testing. By implementing these key processes, Customs should be
well positioned to begin the next phases of testing.

Customs Has Completed Systems Acceptance Testing

The purpose of system acceptance testing is to verify in an
operational environment (either simulated or actual production) 10
that the complete system (i. e., the full complement of
application software running on the target hardware and systems
software infrastructure) satisfies specified requirements (e. g.,
functional, performance, and security) and is acceptable to end
users. According to our test guide, effective system acceptance
testing includes, among other things (1) developing systems
acceptance test plans that specify such things as the type of
tests to occur and whether users will actively participate in the
test, (2) confirming compliance of vendor- supported system
components, (3) executing the system acceptance tests, and (4)
ensuring that system acceptance test exit criteria are met.

As of January 1999, Customs reported that it had completed system
acceptance testing of all its mission- critical IT systems. In
doing so, Customs satisfied our system acceptance test phase key
processes. For example, Customs (1) developed detailed system
acceptance test plans for the five mission- critical IT systems
that specify the types of acceptance tests to be performed (e. g.,
functional, security, performance, and stress testing), (2)
confirmed that vendor- supported components are Year 2000
compliant, (3) executed acceptance tests in accordance with plans
and procedures, which included the participation of user
representatives and configuration management staff, and (4)
required users to attest to the systems' Year 2000 compliance
according to the exit criteria defined for the systems. By
implementing these key processes, Customs should have reasonable
assurance that individual systems perform as intended.

10 Risks of testing in the production environment must be
thoroughly analyzed and precautions taken to preclude damage to
systems and data.

B-281520 Page 10 GAO/AIMD-99-37 Customs' Year 2000 Program
Controls

End- to- End Testing Underway

The purpose of end- to- end testing is to verify that a defined
set of interrelated systems, which collectively support an
organizational core business area or function, interoperate as
intended in an operational environment, 11 either actual or
simulated. According to our test guide, effective end- to- end
testing includes, among other things, (1) defining the system
boundaries of the end- to- end tests, (2) securing the commitment
of key data exchange partners, (3) confirming the Year 2000
compliance of vendor- supported telecommunications and other
infrastructures, and (4) executing end- to- end tests and
documenting results.

Customs has either performed or has plans to perform all of our
end- to- end key processes. For example, Customs (1) defined the
system boundaries of its end- to- end tests to include field site
configurations, external partners, and telecommunications systems,
(2) secured commitments from key data exchange partners, and (3)
confirmed the availability of compliant telecommunications
products and services. Also, while Customs has not yet completed
end- to- end tests, it has developed test plans and procedures
that assign responsibilities for executing tests and documenting
test results.

Additionally, to validate that it can exchange data with some of
its commercial trade partners, Customs conducted a preliminary
end- to- end test in May 1998. The test demonstrated successful
data interchange between ACS and external partners represented by
four leading software vendors. 12 By completing these end- to- end
test key processes and activities, Customs will have greater
assurance that its systems and the systems of its business
partners interoperate as intended.

11 Risks of testing in the production environment must be
thoroughly analyzed and precautions taken to preclude damage to
systems and data.

12 These vendors provide software to 32 percent of the import
community with whom Customs exchanges data.

B-281520 Page 11 GAO/AIMD-99-37 Customs' Year 2000 Program
Controls

Customs Is Developing Contingency Plans to Ensure Continuity of
Core Business Operations

Despite organizations' best efforts to remediate their mission-
critical systems, core business processes may still be disrupted
by Year 2000- induced system failures and errors in internal
systems, business partners' systems, or public infrastructure
systems, such as power, water, transportation, and
telecommunications systems. Business continuity and contingency
plans help mitigate the risks associated with unexpected internal
and uncontrollable external system failures. Our business
continuity and contingency planning guide provides a four- phased
structured approach for business continuity planning-- initiation,
business impact analysis, contingency planning, and testing.

Customs Has Completed the Initiation Phase of Contingency Planning

According to our contingency planning guide, effective initiation
of a contingency planning effort includes, among other things, (1)
establishing a business continuity project work group, (2)
developing and documenting a high- level business continuity
planning strategy, (3) developing a master schedule and
milestones, (4) implementing a risk management process and
establishing a reporting system, and (5) implementing quality
assurance reviews.

Customs has implemented all of the initiation phase key processes.
For example, Customs (1) formed a business continuity project work
group in January 1998 to lead, manage, and oversee the continuity
planning effort for its core business processes; (2) developed and
documented its Contingency Management Strategy in February 1998,
which defines roles and responsibilities for business process
owners and systems owners, defines the continuity project
structure, and specifies that plans are to be developed for each
of Customs' six core business processes; (3) defined, in its
Contingency Management Strategy, a master schedule for the
planning effort along with milestones for the delivery of interim
products; (4) implemented a risk management process, i. e., steps
for identifying and ranking internal and external risks and
developing risk mitigation plans, and defined progress reporting
requirements to manage the business continuity planning tasks and
assist business units in developing individual contingency plans;
and (5) implemented a quality review process to verify that the
continuity of operations and contingency plans satisfy information
requirements (e. g., on September 8, 1998, the quality review team
reported its findings regarding missing and/ or incomplete
information to the preparers for correction or clarification). By
satisfying these key processes, Customs has established an
effective structure for managing its contingency planning efforts.

B-281520 Page 12 GAO/AIMD-99-37 Customs' Year 2000 Program
Controls

Customs Has Assessed the Impact of Mission- Critical System
Failures on Core Business Processes

The principal objective of this phase is to determine the effect
of missioncritical information systems' failures on the viability
and effectiveness of agency core business processes. According to
our contingency planning guide, effective business impact analysis
includes, among other things, (1) defining and documenting Year
2000 failure scenarios, (2) performing risk and impact analyses of
each core business process, and (3) assessing and documenting
infrastructure risks.

Customs has performed all of the business impact analysis phase
key processes. For example, Customs has (1) defined and documented
potential disruption scenarios (e. g., the cause and nature of the
disruption, the duration of the disruption, the business processes
and supporting systems affected by the disruption); (2) performed
risk and impact analyses for its six major business processes that
include an evaluation of business, legal, and regulatory impacts;
and (3) determined if existing disaster recovery plans address
potential disruption scenarios, and, if not, Customs is expanding
the plans. By performing these key processes, Customs acquired the
information needed to develop effective contingency plans for
continuity of operations.

Customs Is Developing Contingency Plans

The purpose of the contingency planning phase is to integrate and
act on the business impact analysis results. According to our
contingency planning guide, effective contingency planning
includes, among other things, (1) assessing the costs and benefits
of identified alternatives and selecting the best contingency
strategy for each core business process, (2) defining and
documenting triggers for activating contingency plans, and (3)
developing and documenting "zero- day" strategy and procedures. 13
Customs has performed all of the key processes of the contingency

planning phase. For example, Customs (1) used its business impact
and disaster recovery analysis 14 to assess the costs and benefits
of alternative contingency plans for each core business process
and to select business continuity strategies and (2) defined
triggers for activating contingency plans and designated
responsible individuals to ensure that the plans are

13 A "zero- day" strategy is a risk reduction strategy for the
period between Thursday, December 30, 1999, and Monday, January 3,
2000. The strategy may include an agencywide shutdown of all
information systems on December 31, 1999 and a staged power- up on
January 1, 2000.

14 Business Impact and Disaster Recovery Requirements Analysis,
(DynCorp, June 17, 1996, Draft).

B-281520 Page 13 GAO/AIMD-99-37 Customs' Year 2000 Program
Controls

executed if necessary. Additionally, Customs considered developing
zeroday procedures, and concluded that it was not necessary
because its business continuity and contingency plans address
potential Year 2000- induced failures, including the period
between December 31, 1999, and January 3, 2000. Customs has
completed development of most contingency plans and it has plans
to complete development of all contingency planning by March 1999.

Customs Is Testing Its Contingency Plans

The objective of the testing phase is to verify that, when
implemented, contingency plans provide the required levels of
business performance. According to our contingency planning guide,
effective testing includes, among other things, (1) developing and
documenting contingency test plans, (2) updating disaster recovery
plans and procedures, and (3) updating continuity plans based upon
lessons learned from the tests and retesting the plans if
necessary.

Customs has either implemented or has plans for implementing the
key processes for this phase. For example, Customs (1) developed
contingency test plans for each business process that specify such
things as the test approach, required facilities and resources,
and schedules and locations, and conducted preliminary tests to
validate certain test procedures and (2) assessed its disaster
recovery plans and procedures 15 and updated them to address
failures of facility power, telecommunications, and networks.
Customs is currently testing contingency plans and it has plans to
complete contingency plan testing, including plans for non- IT
systems, by June 1999. In addition, Customs plans to update the
plans to reflect the test results and to retest these plans as
necessary.

Customs Has Established Processes to Help Ensure the Reliability
of Y2K Status Reporting

To effectively manage and oversee Year 2000 programs, managers and
executive decisionmakers need reliable information about the
nature and status of Year 2000 conversion efforts. Our Year 2000
guides recognize the importance of such information. Accordingly,
the guides provide for establishing formal reporting mechanisms
early in the Year 2000 program life cycle and using the
information reported to oversee and control program efforts.
Additionally, the guides describe the need to specify the content
and format of the reports and the reporting frequency and to

15 Information Technology Infrastructure Recovery Assessment,
Customs' Office of Information and Technology (September 23,
1998).

B-281520 Page 14 GAO/AIMD-99-37 Customs' Year 2000 Program
Controls

establish management controls (e. g., the use of quality assurance
and IV& V groups) to ensure that the information being reported is
reliable.

Customs has established formal reporting mechanisms for both its
IT and non- IT conversion efforts, and it has clearly defined the
content and format of the reports and the reporting frequency. For
example, Year 2000 project managers are required to report weekly
on the number of system components assessed, renovated, validated,
and implemented as well as actual expenditures according to
specified cost categories. Project managers must also submit more
detailed weekly reports on validationspecific activities and
results (e. g., the number of lines of code renovated, waiting to
be tested, and behind or ahead of schedule) and business- owner
contingency planning- specific activities (e. g., the status of
specific contingency planning tasks for each mission- critical
system component). These activity and progress reports are entered
into Customs' Year 2000 central database and are used to control
and oversee the program, as well as to prepare Customs' monthly
and quarterly status reports to Treasury. To ensure that the
information reported to Customs' executives and Treasury is
reliable, a quality review team reviews the information submitted
by project managers for (1) consistency (by comparing it to
previously reported information), (2) completeness (by comparing
it to reporting standards), and (3) accuracy (by validating it
through either observation, inquiry, or review of supporting
documentation). For example, in June 1998, a quality review team
compared the test data reported by project managers to the actual
test results for both high- risk and randomly- selected components
and found no discrepancies between the reported data and the
actual test activities and results.

To further determine the reliability of the information included
in Customs' monthly reports to Treasury, we independently traced
the reported status for selected components of three systems back
to supporting source documentation (i. e., project- level status
reports and documented test results) and found no discrepancies.
16 Because we did not statistically sample Customs' IT and non- IT
components and verify the reliability of the reported information
in the sample, we cannot conclude that the Year 2000 status
information that Customs is reporting is reliable. However, in
light of the results of our work on the reliability of Customs'
reporting, we decided not to perform additional tracing.

16 The selected components were ACS Quota, TECS Pre- clearance
Alert, and QIK Sun/ Solaris.

B-281520 Page 15 GAO/AIMD-99-37 Customs' Year 2000 Program
Controls

Conclusions Customs reports, with a few system exceptions, that it
has met OMB Year 2000 milestones, and Customs' plans provide for
completing all remaining

Year 2000 efforts well in advance of January 1, 2000. Customs'
good progress to date is attributable to the effective Year 2000
management structures and processes that Customs has established.
Clearly, Customs still has much to accomplish before it is ready
for the century date change, including completing conversion
efforts for its internal systems and preparing for the possibility
of external system failures. However, Customs has plans in place
for completing key tasks, and it has the management structures and
processes in place to ensure that they are accomplished. While
these structures, processes, and plans do not guarantee that Year
2000- induced system failures will not occur, if Customs
implements its plans and follows its policies and procedures, it
will have effectively reduced its risk of significant Year 2000-
induced business failure.

Agency Comments In commenting on a draft of this report, Customs
agreed with our analysis and conclusions. In addition, Customs
stated that our Year 2000 guidance

has been of great value to Customs in establishing a sound
foundation for Customs' Year 2000 effort.

We are making copies of this letter available to Representative
Bill Archer, Chairman, and Representative Charles B. Rangel,
Ranking Minority Member, House Committee on Ways and Means;
Senator Robert F. Bennett, Chairman, and Senator Christopher J.
Dodd, Vice Chairman, Senate Special Committee on the Year 2000
Technology Problem; Representative Stephen Horn, Chairman, and
Representative Jim Turner, Ranking Minority Member, Subcommittee
on Government Management, Information and Technology, House
Committee on Government Reform; Representative Constance A.
Morella, Chairwoman, and Representative James A. Borcia, Ranking
Minority Member, Subcommittee on Technology, House Committee on
Science. We are also sending copies to the Honorable Robert E.
Rubin, Secretary of the Treasury; Raymond W. Kelly, Commissioner
of Customs; and the Honorable Jacob J. Lew, Director of the Office
of Management and Budget. Copies will also be made available to
others upon request.

B-281520 Page 16 GAO/AIMD-99-37 Customs' Year 2000 Program
Controls

If you have any questions about this report, please contact me by
phone at (202) 512- 6240, or by email at hiter. aimd@ gao. gov.
Major contributors to this report are listed in appendix IV.

Randolph C. Hite Associate Director, Governmentwide

and Defense Information Systems

Page 17 GAO/AIMD-99-37 Customs' Year 2000 Program Controls

Page 18 GAO/AIMD-99-37 Customs' Year 2000 Program Controls

Contents Letter 1 Appendix I Objective, Scope, and Methodology

20 Appendix II Customs' Six Core Business Missions

22 Appendix III Customs' MissionCritical Systems

24 Appendix IV Comments From the U. S. Customs Service

26 Appendix V Major Contributors to This Report

27

Contents Page 19 GAO/AIMD-99-37 Customs' Year 2000 Program
Controls Abbreviations

ACS Automated Commercial System ADMIN Administrative System AES
Automated Export System CIO chief information officer COTS
commercial off- the- shelf IBIS Interagency Border Inspection
System IT information technology IV& V independent verification
and validation OMB Office of Management and Budget PC personal
computer SEACATS Seized Asset and Case Tracking System TECS
Treasury Enforcement Communications System

Page 20 GAO/AIMD-99-37 Customs' Year 2000 Program Controls

Appendix I Objective, Scope, and Methodology Append i x I

Our objective was to determine whether Customs has established
effective management structures and processes for managing and
reporting on key aspects of its Year 2000 program. We accomplished
this objective by identifying Customs' Year 2000 program
management controls and comparing these to the controls (i. e.,
key processes) described in our Year 2000 guides. 1 Additionally,
for selected systems and systems' components, we reviewed
supporting documentation to verify that the management controls
were functioning as intended.

More specifically, we reviewed Customs' Year 2000 program
management plans, guidance, procedures, and organizational
structures relating to Year 2000 conversion, testing, contingency
planning, reporting, quality assurance/ independent verification
and validation, and risk management. In particular, we analyzed
the following: Customs' Year 2000 Strategic and Operational
Program Management Plans, Customs' Year 2000 Application Testing
Strategy and Plan, Customs' Telecommunications Year 2000 Program
Plan, Customs' Year 2000 Contingency Management Strategy and Plan,
Customs' Year 2000 Quality Assurance Plan, Customs' Year 2000

program office charter defining roles, responsibilities, and
authority, Customs' progress reporting and tracking procedures,
and correspondence between Customs' Executive Council and the Year
2000 program office. We then reviewed documentation associated
with individual systems and systems' components to determine
whether these structures and processes were being implemented. For
example, we reviewed system component unit and integration test
plans, system acceptance and end- to- end test plans,
telecommunications criticality analyses, weekly status meeting
minutes, progress and status reports, business impact analyses for
core business processes, IT infrastructure risk assessments, and
existing disaster recovery plans.

We then selected software components from two mission- critical
systems (ACS' Quota and TECS' Pre- clearance Alert) and one
telecommunications system (QIK Sun/ Solaris, which consists of
five components) to determine whether conversion activities
required in Customs' plans, procedures, and guidance were being
executed. For these systems and components, we reviewed supporting
documentation relating to conversion and testing,

1 Year 2000 Computing Crisis: An Assessment Guide (GAO/ AIMD-
10.1.14, issued as an exposure draft in February 1997, issued in
final in September 1997); Year 2000 Computing Crisis: Business
Continuity and Contingency Planning (GAO/ AIMD- 10.1.19, issued as
an exposure draft in March 1998, issued in final in

August 1998); and Year 2000 Computing Crisis: A Testing Guide
(GAO/ AIMD- 10.1.21, issued as an exposure draft June 1998, issued
in final in November 1998).

Appendix I Objective, Scope, and Methodology

Page 21 GAO/AIMD-99-37 Customs' Year 2000 Program Controls

such as project schedule risk mitigation strategies, print- outs
of before- and- after screen changes and database field changes,
print- outs showing data entry of new date formats and correct
processing of dates, configuration management documentation
showing the movement of programs into source control libraries
and, subsequently, into production, sign- off sheets showing
quality assurance and user approval to move changed programs into
production, results of quality assurance reviews of reporting
accuracy, test results for Customs' mainframe operating system,
utilities, and environmental software along with vendor
information regarding the compliancy of Customs' lab environment,
and plans for testing contingency plans.

For the selected systems and system components, we also traced the
status information that Customs was reporting back to supporting
source documentation (e. g., project level status reports and test
results). We did not statistically sample Customs IT and non- IT
components.

To supplement our analysis of documentation, we interviewed key
Year 2000 program officials, such as the Year 2000 Program
Director, the Year 2000 Test Manager, individual project managers,
quality assurance officials, business process owners, the
Contingency Planning Team's members, and support contractor
representatives. We conducted our work in collaboration with the
Treasury Inspector General and in accordance with generally
accepted government auditing standards from July 1998 through
January 1999.

Page 22 GAO/AIMD-99-37 Customs' Year 2000 Program Controls

Appendix II Customs' Six Core Business Missions Append i x I I

To accomplish its mission, Customs is organized into six business
areas-- trade compliance, outbound, passenger, finance, human
resources, and investigations. Each business area is described
below.

 Trade compliance includes enforcement of laws and regulations
associated with the importation of goods into the United States.
To accomplish its trade compliance mission, Customs (1) works with
the trade community to promote understanding of applicable laws
and regulations, (2) selectively examines cargo to ensure that
only eligible goods enter the country, (3) reviews documentation
associated with cargo entries to ensure that they are properly
valued and classified, (4) collects billions of dollars annually
in duties, taxes, and fees associated with imported cargo, (5)
assesses fines and penalties for noncompliance with trade laws and
regulation, (6) seizes and accounts for illegal cargo, and (7)
manages the collection of these moneys to ensure that all trade-
related debts due to Customs are paid and properly accounted for.
Outbound includes Customs' enforcement of laws and regulations

associated with the movement of merchandise and conveyances from
the United States. To accomplish its mission in the outbound area,
Customs (1) selectively inspects cargo at U. S. ports to guard
against the exportation of illegal goods, such as protected
technologies, stolen vehicles, and illegal currency, (2) collects,
disseminates, and uses intelligence to identify high- risk cargo
and passengers, (3) assesses and collects fines and penalties
associated with the exportation of illegal cargo, and (4)
physically examines baggage and cargo at airport facilities for
explosive and nuclear materials. In addition, the outbound
business includes collecting and disseminating trade data within
the federal government. Accurate trade data are crucial to
establishing accurate trade statistics on which to base trade
policy decisions and negotiate trade agreements with other
countries. By the year 2000, Customs estimates that exports will
be valued at $1.2 trillion, as compared to a reported $696 billion
in 1994.  Passenger includes processing all passengers and crew of
arriving and

departing air, sea, and land conveyances and pedestrians. In
fiscal year 1997, Customs reported that it processed nearly 450
million travelers, and by the year 2000, expects almost 500
million passengers to arrive in the United States annually. Many
of Customs' passenger activities are coordinated with other
federal agencies, such as the Immigration and Naturalization
Service and the Department of Agriculture's Animal and Plant
Health Inspection Service. Activities include targeting high- risk
passengers, which requires prompt and accurate information, and

Appendix II Customs' Six Core Business Missions

Page 23 GAO/AIMD-99-37 Customs' Year 2000 Program Controls

physically inspecting selected passengers, baggage, and vehicles
to determine compliance with laws and regulations.  Finance
includes asset and revenue management activities. Asset

management consists of activities to (1) formulate Customs'
budget, (2) properly allocate and distribute funds, and (3)
acquire, manage, and account for personnel, goods, and services.
Revenue management encompasses all Customs activities to identify
and establish amounts owed Customs, collect these amounts, and
accurately report the status of revenue from all sources. Sources
of revenue include duties, fees, taxes, other user fees, and
forfeited currency and property. The revenue management activities
interrelate closely with the revenue collection activities in the
trade compliance, outbound, and passenger business areas.  Human
resources is responsible for filling positions, providing employee

benefits and services, training employees, facilitating workforce
effectiveness, and processing personnel actions for Customs'
18,000 employees and managers.  Investigations includes activities
to detect and eliminate narcotics and

money laundering operations. Customs works with other agencies and
foreign governments to reduce drug- related activity by
interdicting (seizing and destroying) narcotics, investigating
organizations involved in drug smuggling, and deterring smuggling
efforts through various other methods. Customs also develops and
provides information to the trade and carrier communities to
assist them in their efforts to prevent smuggling organizations
from using cargo containers and commercial conveyances to
introduce narcotics into the United States.

Page 24 GAO/AIMD-99-37 Customs' Year 2000 Program Controls

Appendix III Customs' Mission- Critical Systems Append i x I I I

To carry out its responsibilities, Customs relies on a variety of
information systems and processes to assist its staff in (1)
documenting, inspecting, and accounting for the movement and
disposition of imported goods and (2) collecting and accounting
for related revenues.

In managing its Year 2000 program, Customs divided its mission-
critical systems into information technology (IT) and non- IT
systems (e. g., office equipment, facilities, and security
systems). Customs has five IT systems that run mission- critical
applications: Automated Commercial System (ACS), Treasury
Enforcement Communications System (TECS), Administrative System
(ADMIN), Automated Export System (AES), and Seized Asset and Case
Tracking System (SEACATS). These are described below.

 ACS supports the trade compliance core business process by
tracking, controlling, and processing all commercial goods
imported into the United States. Over 97 percent of the data filed
for imported cargo entries are transmitted to Customs
electronically through ACS. ACS has been operational since 1984
and is accessed by over 15,000 trade and other government agency
users.  TECS supports the enforcement core business process and
provides

support to federal law enforcement missions. Consequently, TECS
interfaces with a number of law enforcement systems, including the
Federal Bureau of Investigations' National Crime Information
Center system, and is the major automation component of the
Interagency Border Inspection System (IBIS). IBIS serves as the
"clearinghouse" for law enforcement data and provides border
inspection support software and communications. In addition, TECS
supports Customs' Informed Compliance, Targeting, Identifying, and
Examining strategies as well as Investigations (Narcotics and
Money Laundering). Approximately 27,000 users access TECS,
including Customs, the Immigration and Naturalization Service, the
Internal Revenue Service, the Bureau of Alcohol, Tobacco, and
Firearms, and the State Department.  ADMIN provides information
technology support for the financial and

human resources core business processes. It is comprised of 40
separate systems which interface with each other and with ACS,
AES, and TECS. ADMIN is accessed by almost 19,000 users,
predominantly from Customs' Office of Finance and Human Resources
Management.  AES supports the trade compliance and outbound core
business

processes. It electronically gathers export- related information
from exporters and carriers and is used to help Customs' target
export violators. AES is accessed by over 28,000 users nationwide.

Appendix III Customs' Mission- Critical Systems

Page 25 GAO/AIMD-99-37 Customs' Year 2000 Program Controls

 SEACATS supports the enforcement, finance, and trade compliance
core business processes and tracks activity associated with
seizures from the initial enforcement interest in the property
until its final disposition. SEACATS is accessed by over 16,000
Customs' users and interfaces with the Justice Department and
Internal Revenue Service systems.

Customs' IT systems also include its telecommunications systems.
Customs' telecommunications devices and software components
include (1) the interface between the mainframe systems and the
Treasurymanaged TCS equipment, (2) voice telecommunications
systems installed throughout Customs' locations, and (3) the local
area network equipment which connects Customs' field locations to
the mainframe systems.

Customs' mission- critical non- IT assets include office equipment
such as check- writers, scanners, and optical readers; facilities,
such as heating and air, lights, and fire alarms; security
systems, including badge readers, cameras, secured doors, and
safes; planes; and automobiles. These assets are installed in over
900 facilities.

Page 26 GAO/AIMD-99-37 Customs' Year 2000 Program Controls

Appendix IV Comments From the U. S. Customs Service Append i x I V

Page 27 GAO/AIMD-99-37 Customs' Year 2000 Program Controls

Appendix V Major Contributors to This Report Append i x V

Accounting and Information Management Division, Washington, D. C.

Jack L. Brock, Jr., Director, Governmentwide and Defense
Information Systems Dr. Rona B. Stillman, Chief Scientist for
Computers and Telecommunications Deborah A. Davis, Assistant
Director Garry D. Durfey, Senior Information Systems Analyst

Atlanta Field Office Teresa F. Tucker, Senior Information Systems
Analyst John M. Ortiz, Senior Information Systems Analyst

(511127) Le t t e r

Ordering Information The first copy of each GAO report and
testimony is free. Additional copies are $2 each. Orders should be
sent to the following address, accompanied by a check or money
order made out to the Superintendent of Documents, when necessary,
VISA and MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address are
discounted 25 percent.

Orders by mail: U. S. General Accounting Office P. O. Box 37050
Washington, DC 20013

or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW)
U. S. General Accounting Office Washington, DC

Orders may also be placed by calling (202) 512- 6000 or by using
fax number (202) 512- 6061, or TDD (202) 512- 2537.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512- 6000 using a
touchtone phone. A recorded menu will provide information on how
to obtain these lists.

For information on how to access GAO reports on the INTERNET, send
an e- mail message with info in the body to:

info@ www. gao. gov or visit GAO's World Wide Web Home Page at:
http:// www. gao. gov

United States General Accounting Office Washington, D. C. 20548-
0001

Official Business Penalty for Private Use $300

Address Correction Requested Bulk Rate

Postage & Fees Paid GAO Permit No. GI00

*** End of document. ***