Electronic Records: Clinton Administration's Management of
Executive Office of the President E-Mail System (30-APR-01,
GAO-01-446).
This report discusses the facts surrounding key events affecting
the ability of the Clinton administration's Executive Office of
the President (EOP) to preserve certain electronic mail (e-mail)
deemed official government records. GAO found that two
malfunctions occurred in the EOP e-mail system that prevented
official records from being properly recorded in the Automated
Records Management System (ARMS). The first problem involved an
anomaly with incoming Internet e-mail users of the "Mail2" e-mail
server. The second problem surfaced while the contractor was
trying the resolve the first problem. During its efforts to
determine the cause of the Mail2 e-mail errors, the letter D was
deleted from a key piece of software, causing the ARMS scanner to
skip e-mail accounts of users with first names beginning with the
letter D. The EOP initiated a costly and time-consuming project
to recover e-mail records that had not been effectively managed.
Several factors contributed to the cost and scope of the tape
restoration project, including the contractor's performance of
tape management and systems maintenance and legal scrutiny of
e-mail malfunctions by external authorities.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-01-446
ACCNO: A00912
TITLE: Electronic Records: Clinton Administration's Management
of Executive Office of the President E-Mail System
DATE: 04/30/2001
SUBJECT: Contract performance
Electronic mail
Records management
Computer software verification and
validation
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Testimony. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-01-446
Report to the Chairman, Committee on Government Reform, House of
Representatives
United States General Accounting Office
GAO
April 2001 ELECTRONIC RECORDS
Clinton Administration?s Management of Executive Office of the President?s
E- Mail System
GAO- 01- 446
Page i GAO- 01- 446 EOP E- mail System Letter 1
Appendix I Scope and Methodology 19
Appendix II Chronology of E- mail Malfunctions 21
Appendix III Key Officials Involved in Primary Mail2 and Letter D Events 39
Appendix IV Time Line of Key Events in the Mail2 and Letter D Malfunctions
41
Appendix V Itemization of Budgeted Cost for the Tape Restoration Project 42
Appendix VI Comments From the Office of the Former Vice President 43
Tables
Table 1: Summary of Budgeted Tape Restoration Costs. 9 Table 2: Additional
Project Activities and Related Costs. 15 Contents
Page 1 GAO- 01- 446 EOP E- mail System
April 30, 2001 The Honorable Dan Burton Chairman, Committee on Government
Reform House of Representatives
Dear Mr. Chairman: In response to your requests of May and August 2000, this
report discusses our examination of the facts surrounding key events
affecting the ability of the Clinton Administration?s Executive Office of
the President (EOP) to preserve certain e- mail messages deemed official
government records. As you know, the Presidential Records Act (PRA) and the
Federal Records Act (FRA) require the EOP to preserve official records,
including e- mail. Adequate controls must be in place to ensure that these
records, whether manual or electronic, are complete and maintained in
accordance with applicable requirements. However, e- mail system
malfunctions and management weaknesses prevented archiving of some e- mail
records for EOP components, including the Office of the Vice President
(OVP).
In our review of these events surrounding the EOP?s e- mail systems, you
asked us to:
develop a chronology of the e- mail malfunctions that included a
description of EOP actions taken in discovering and repairing the
malfunctions and recovering missing e- mail messages;
identify officials and contractors responsible for maintaining the e- mail
system and correcting the malfunctions;
determine whether the OVP implemented adequate practices in accordance
with applicable laws, regulations, and generally accepted industry practices
for management of e- mail records;
identify elements contributing to the estimated cost to repair the system
and restore lost messages; and
determine whether EOP officials notified government officials- including
the Congress, the Justice Department, and the Office of Independent Counsel-
when the malfunctions were discovered.
To address these objectives, we reviewed documents submitted to your
Committee by the EOP. In addition, we observed July through October 2000 U.
S. District Court hearings pertaining to EOP e- mail. We also submitted
written questions to the EOP, and reviewed the written
United States General Accounting Office Washington, DC 20548
Page 2 GAO- 01- 446 EOP E- mail System
responses and additional documentation provided to us. We performed our work
from July 2000 through January 2001, in accordance with generally accepted
government auditing standards; however, our review was limited by the
unavailability of complete information pertaining to the OVP?s retention of
hard copy e- mail records, the EOP?s assessment of contractor performance,
and the EOP?s notification of government officials. A detailed discussion of
the scope and methodology for conducting our work is presented in appendix
I.
Two malfunctions that occurred in the EOP e- mail system prevented official
records from being properly recorded in the Automated Records Management
System (ARMS), which is a searchable database of e- mail records that was
implemented by the EOP in 1994. The first- an anomaly with incoming Internet
e- mail to affected users of the ?Mail2? e- mail server- was identified in
late January 1998. In November 1998, in addressing the Mail2 repair, an EOP
contractor introduced a second problem that prevented incoming e- mail to
users with first names starting with the letter D from being captured by
ARMS. Although these malfunctions prevented certain e- mail records from
being archived in ARMS, copies of these records were retained in the system
backup tapes that were saved between November 1998 and June 1999 to ensure
that the system could be restored after an interruption in operation. To
ensure that official government records were preserved in a searchable
format, the EOP initiated a tape restoration project in March 2000 to
retrieve e- mail records from available backup tapes and add these records
to ARMS. A detailed chronology of these events is presented in appendix II.
Appendix III contains a list of federal officials and contractors involved
in key events surrounding maintenance of the e- mail systems and repair of
the Mail2 and Letter D malfunctions.
The OVP did not implement adequate records management practices to ensure
that all e- mail records generated or received were preserved in accordance
with applicable law and best practices. The OVP initially followed a dual
approach to managing its e- mail records pursuant to the EOP policy-
maintaining paper copies of records as well as retaining backup e- mail
tapes. The OVP discontinued this approach at some point after May 1993- in
the mistaken belief that the e- mail records were being archived in ARMS and
that EOP?s Office of Administration was managing the backup tapes. The OVP
could not demonstrate that all e- mail records had been preserved by
acceptable methods until May 2000, when ARMS began capturing all OVP e- mail
records. As a result, about 600 system Results in Brief
Page 3 GAO- 01- 446 EOP E- mail System
backup tapes needed restoration to determine if any non- archived e- mail
records existed on the tapes.
Several factors contributed to the expected cost to restore omitted EOP e-
mail records to ARMS, estimated by an EOP contractor to be $11.7 million:
An EOP support contractor?s performance of tape management and systems
maintenance and documentation activities contributed to the increased size
and scope of the tape restoration project. Specifically, the EOP?s
assessments of contractor performance revealed that management of e- mail
backup tapes was weak and that maintenance of e- mail systems and user
accounts needed improvement. In addition, the automated link between the
EOP?s e- mail system and ARMS was not documented as required. As a result,
the EOP could not identify the specific tapes within its entire tape
population that needed restoration and could not readily understand and
optimize records management controls.
The EOP did not effectively monitor management of e- mail records. It did
not include critical elements in its monitoring program, such as evaluations
of the creation of records, maintenance and use of records, and records
disposition. This increased the time needed to detect the Mail2 and Letter D
malfunctions and added to the accumulation of backup tapes to be restored.
Scrutiny of the e- mail malfunctions and the EOP?s tape restoration
practices by external authorities introduced additional project tasks,
including contracts for independent verification and validation of the
restoration process and for additional security over the project.
Although the EOP knew of the malfunctions that prevented e- mail from being
captured by ARMS from October 1996 through mid- May 1999, officials stated
that they did not understand the breadth and scope of the impact of these
malfunctions on the EOP?s response to subpoenas and production of other
documents until February 2000. The EOP claims that it notified appropriate
investigative bodies at that time. However, we were unable to obtain
sufficient evidence to confirm that these notifications were made.
We requested comments on a draft of this report from the EOP and the
personal representatives of former President Clinton and former Vice
President Gore. The Deputy General Counsel, Office of Administration, EOP,
orally provided a technical clarification, which was incorporated. The
Counsel to former Vice President Gore provided written comments stating that
(1) the report did not explicitly state that we found no
Page 4 GAO- 01- 446 EOP E- mail System
evidence of attempts by OVP staff to deliberately fail to preserve records,
and (2) ARMS was a unique system for which best practices did not apply and
additional information on the creation and use of this system was provided.
In regard to the statements regarding the intent of OVP staff, our review-
as requested- focused on the OVP?s management of e- mail records. We did not
assess the intent of individuals responsible for managing these records and,
as such, made no conclusions on this subject. Regarding ARMS, we disagree
with the Office of the former Vice President?s characterization of the
applicability of best practices to ARMS. The best practices cited in this
report currently exist in the form of National Archives and Records
Administration guidance and apply to records management programs,
irrespective of the type of system used. However, we have revised the report
to include the additional information on ARMS that was provided. The Office
of the former Vice President also provided technical comments that we have
incorporated as appropriate. The representative of former President Clinton
provided a letter, which stated that he did not have access to the records
necessary to properly review and comment on its accuracy.
PRA, set forth in title 44 of United States Code, chapter 22, requires the
President and Vice President to adequately record their official acts,
maintain certain official records, and transfer custody of such records to
the Archivist of the United States upon termination of their terms of
office. Pursuant to the PRA, both the Office of the President and the Office
of the Vice President are to implement records management controls and other
necessary actions to ensure that presidential and vice presidential
activities, deliberations, decisions, and policies are adequately documented
and maintained. It also provides that presidential records shall be made
available pursuant to subpoena or other judicial process and to either House
of Congress, and prohibits destruction of records without prior concurrence
of the Archivist and notification of Congress 60 days prior to disposal. The
Archivist promulgates standards and guidance for implementation of PRA;
these are contained in title 36 of the Code of Federal Regulations (CFR).
This regulation defines criteria for access to and disposition of
presidential records.
Federal records are subject to the provisions of FRA, set forth in title 44
U. S. C., chapters 29, 31, and 33. FRA requires the heads of federal
agencies to make and preserve records documenting the official activities of
the agency. FRA directs federal agencies to establish (1) a program for the
management of agency records, (2) effective controls over the creation,
maintenance, and use of records, and (3) safeguards against the removal
Background
Page 5 GAO- 01- 446 EOP E- mail System
or loss of records. The Archivist promulgates standards and guidance for
implementation of FRA, which are contained in title 36 CFR. Requirements for
federal records are more extensive than are those for presidential records,
and specify that federal agencies
preserve records documenting the official activities of federal agencies
to protect the legal rights of the government and of persons directly
affected by the agency?s activities;
periodically monitor staff determinations of the record status of
documentary materials;
implement a records maintenance program so that complete records are filed
or otherwise identified and preserved and that records can be found when
needed;
implement an agencywide program for the management of all federal records
created, received, and stored on electronic media;
ensure compliance with federal criteria and review electronic information
systems periodically for conformance with established agency procedures,
standards, and policies;
develop and maintain up- to- date documentation about all electronic
information systems adequate to specify technical characteristics,
understand the purpose and function of the systems, and ensure timely,
authorized disposition of records;
consider specific criteria when developing procedures for maintenance of
electronic mail records in recordkeeping systems, including copying records
from the e- mail system to a separate recordkeeping system; 1 and
retain records from e- mail systems in an off- line storage format.
Agencies that use electronic formats (such as optical disk or magnetic tape)
must maintain the ability to convert the records to the National Archives
and Records Administration?s required format and medium at the time of
transfer. Agencies that maintain paper files as their recordkeeping systems
are required to print their electronic mail records and the related
transmission and receipt data.
To ensure the adequacy of management and control of federal information
systems, the Office of Management and Budget (OMB) issued circulars A123,
Management Accountability and Control, and A- 130, Management of Federal
Information Resources. Circular A- 123 requires that federal
1 A recordkeeping system is a manual or automated system in which records
are collected, organized, and categorized to facilitate their preservation,
retrieval, use, and disposition. Because they do not have the features
specified by the National Archives and Records Administration, system backup
tapes are not to be used for recordkeeping purposes.
Page 6 GAO- 01- 446 EOP E- mail System
agencies implement adequate internal controls and management structures to
ensure the effective operation of federal programs. Appendix III of circular
A- 130, 2 Security of Federal Automated Information Resources, requires that
federal agencies periodically test system controls meant to ensure the
integrity, confidentiality, and availability of information resources.
The Administration issued guidance to clarify the responsibilities of each
EOP component regarding presidential, federal, and personal records in order
to implement the provisions of both PRA and FRA. A detailed policy and
procedures document issued by the EOP in May 1993 required that no federal
or presidential e- mail records could be deleted unless maintained in an
electronic recordkeeping system or printed and placed in a file. In
addition, it required that the White House Records Management Office monitor
electronic systems to ensure that correct record status determinations had
been made for presidential and federal e- mail records. This policy was
applicable to all EOP components, 3 including the OVP.
In May 1997, the White House replaced the PRA section of the 1993 policy
with one that provided a summary of White House policies and guidance
regarding PRA. It required preservation of all original presidential records
and all materials covered by subpoena or other such requests. It also
directed each staff member to determine the record status of outgoing email
messages and to maintain hard copy records in organized files. According to
EOP officials, the policy issued in 1997 was substantively the same as the
1993 policy, and it served as a reminder to staff of the policy originally
articulated in the 1993 policy regarding compliance with PRA.
A series of events occurred between July 1994, when ARMS was implemented,
and March 2000, when the tape restoration project was initiated. These
events are detailed in appendix II. Various EOP federal and contractor staff
were involved in the maintenance and repair of the
2 OMB circular A- 130 applies to EOP?s Office of Administration component,
which had responsibility for electronic systems, including e- mail and ARMS.
3 The EOP consists of the White House Office, the OVP, and other components,
such as OMB and the Council on Environmental Quality. EOP components that
create and receive presidential records included the White House Office, the
OVP, the National Security Council, and the Office of Policy Development.
The EOP Policies
Required Management of Records
History Surrounding the EOP E- mail Systems
Page 7 GAO- 01- 446 EOP E- mail System
e- mail systems between October 1996 and June 1999; they are listed in
appendix III.
According to the EOP, in order to comply with the Armstrong ruling, 4 it
contracted with Information Management Consultants, Incorporated, to develop
the ARMS recordkeeping system. ARMS began archiving e- mail records in 1994.
5
Since 1996, the EOP had used Lotus Notes� software as its e- mail
application. During the former administration, the Office of Administration
within the EOP maintained four Lotus Notes e- mail servers, one remote
server, and one ARMS interface server, which transferred e- mail records
from the e- mail system to ARMS.
E- mail messages designated as records subject to either PRA or FRA were
passed to ARMS through the interface between the two systems. The Notes- to-
ARMS interface program automatically identified unrecorded e- mail messages
when it periodically scanned for new messages in groupings of user accounts
called ?views.? User accounts were distributed to a view depending on the
first letter of the account name. Then, the ARMS interface copied these new
messages for processing to ARMS and marked the messages as ?recorded.? In
its comments on a draft of this report, the Office of the former Vice
President stated that the ARMS database maintained e- mail records in a
searchable format, which came to facilitate the EOP?s responses to document
requests and subpoenas sometime much later in the administration. A key
feature included in ARMS was the ability to perform a computerized search of
e- mail using keyword terms to search for responsive materials.
From October 1996 through May 1999, two malfunctions occurred in the EOP e-
mail system that prevented incoming Internet e- mail from being properly
archived in ARMS. In late January 1998, an EOP employee initially identified
the first of these malfunctions- an anomaly with incoming Internet e- mail
to affected users of the Mail2 e- mail system server- and documented this in
an incident report. According to EOP
4 Armstrong v. EOP, 810 F. Supp. 335 (1993). The court in the Armstrong case
held that the thencurrent EOP records management practice did not save all
information from an electronic record, and thus violated FRA.
5 The EOP initially used ARMS with the e- mail system that preceded use of
the Lotus Notes� e- mail system.
Page 8 GAO- 01- 446 EOP E- mail System
officials, efforts were undertaken then to analyze this anomaly, but it was
thought to be an isolated event. According to these officials, the precise
cause of the Mail2 malfunction was determined in June 1998 by Northrop
Grumman, the EOP?s information technology services support contractor since
October 1997, when performing other work on the Lotus Notes email system
interface to the ARMS records management database. The malfunction was
caused by improper user account configuration in which the server name
?Mail2? was spelled ?MAIL2.? Because the ARMS interface program did not
recognize the upper case spelling of the mail server name, it was unable to
locate and capture new incoming e- mail messages for these user accounts.
This malfunction was subsequently repaired in November 1998.
Also in November 1998, in addressing the Mail2 repair, a second problem was
introduced by Northrop Grumman staff that prevented incoming Internet e-
mail to users with first names starting with the letter D from being
captured by ARMS. It involved omission of the letter D from the Lotus Notes
view (described above), causing the ARMS scanner to skip those accounts
beginning with the letter D and not capture incoming email to the affected
user accounts. This malfunction was discovered in April 1999 and repaired in
May 1999. Appendix IV presents a time line of key events pertaining to the
discovery and repair of the Mail2 and Letter D malfunctions.
The EOP e- mail systems were periodically backed up, or copied, to tape
media for use in the event of system failure and subsequent need to restore
the system to full operation. In addition, Northrop Grumman followed a
standard industry practice of economical tape recycling. This involved
retaining the tapes for 3 weeks and then recycling them- resulting in either
overwriting the data on the tapes or destroying the used tape and replacing
it with a new tape. Because the backup tapes contained copies of e- mail not
processed to the ARMS records management database, the EOP stopped its
normal practice of tape recycling from June 1998, when the cause of the
Mail2 malfunction was documented, until June 1999, after the Letter D
malfunction was repaired.
In March 2000, the EOP implemented the tape restoration project, for which
it hired Enterprise Computing Solutions Technology, Incorporated (ECS) to
recover the lost e- mail messages and restore the messages to the ARMS
records management database. The EOP hired another contractor, Vistronix, to
oversee the work of ECS. In July 2000, Vistronix estimated that the project
would cost about $11. 7 million for restoration of about
Page 9 GAO- 01- 446 EOP E- mail System
6,000 tapes. Table 1 presents a breakdown of the $11.7 million budgeted
cost. Appendix V presents further details on this.
Table 1: Summary of Budgeted Tape Restoration Costs. Budgeted cost element
Description of cost element Estimated
cost
ECS contract Restoration of e- mail messages not recorded in ARMS $5,125,829
Vistronix contract Independent validation and verification of the
tape restoration project 1, 993,819 AAC Associates contract Advice on
recovery of OVP e- mail
900,000 Special services Forensic analysis and recovery of broken or
unreadable tapes 2,092,400 Direct costs Additional hardware, software
licenses, and
12,000 tapes for two sets of tape copies 498,480 Other Project overrun
contingency 1,098,891
Total $11,709,419
Source: Tape restoration project cost estimate report prepared by Vistronix,
Incorporated, for the EOP in July 2000.
According to the EOP, although the 1993 EOP records management policy
applied to the OVP, the OVP did not implement adequate practices to ensure
continual preservation of its e- mail records in accordance with PRA, which
required implementation of records management controls. The OVP?s internal
policy from 1993 until some time in 1998 was twofold- to rely on its staff
to retain its e- mail server backup tapes and to maintain files of hard copy
e- mail records. 6 According to the EOP, however, from May 1993 until May
2000, the OVP could not demonstrate that its practices for managing
presidential e- mail records provided continual preservation of records
during the former administration.
During a portion of the administration, the OVP was responsible for
operation, maintenance, and records management of its own e- mail system
separate from that used by the EOP. In March 1998, responsibility for
operation and maintenance of the system transferred to the Office of
6 According to EOP, a December 21, 2000, letter from EOP to the National
Archives and Records Administration documented agreement reached regarding
OVP?s records management approach. However, our review of this letter,
obtained from the National Archives and Records Administration, found the
letter to discuss disposition of e- mail records upon the presidential
transition and it did not mention the OVP records management approach. The
OVP Did Not
Ensure Adequate Preservation of E- mail Records
Page 10 GAO- 01- 446 EOP E- mail System
Administration. However, the OVP did not communicate its requirements for
management of e- mail records and assumed that records management
responsibility also transitioned to the Office of Administration. According
to EOP, the Office of Administration assumed that the OVP had retained
records management responsibilities and did not take steps to ensure that
OVP e- mail records were appropriately preserved.
According to the EOP, ?at some subsequent point? since May 1993, the OVP
discontinued retention of paper copies of e- mail records because it
believed its e- mail to be captured by ARMS. This mistaken belief was based
on the precedent that ARMS managed e- mail records for all other EOP
components since its implementation in 1994. In its comments on a draft of
this report, the Office of the former Vice President stated that this belief
was also based on the fact that searches of ARMS produced OVP email records
and on the observation that some outgoing OVP e- mails were in fact being
managed by ARMS (as discussed below). The EOP estimated that ?by 1998,? the
OVP discontinued retention of paper copies of e- mail records not captured
by ARMS (but subject to PRA). Therefore, paper copies of e- mail records did
not exist for a period of at least 2 years (1998 and 1999) and up to 7 years
(1993 through 1999). In its comments on a draft of this report, the Office
of the former Vice President stated that these records are being restored as
part of the tape restoration project. We requested access to files of any
hard copy e- mail records covering this period, but it was not provided.
Although the OVP did not continually retain paper copies of e- mail records
in accordance with the EOP policy, it did retain the backup tapes created
through late March 1998 (except for a backup malfunction from midFebruary to
mid- March 1994). According to the EOP, the OVP backup tapes were for system
recovery and records archival, but they were not in a text- searchable
format. 7
After transition of the e- mail system in March 1998, the Office of
Administration maintained the OVP e- mail system in accordance with its
established operating practices for other EOP systems. One practice followed
was to recycle system backup tapes every 3 weeks. Application of this
practice to the OVP system resulted in irretrievable loss of the
7 According to the EOP in June 2000, the OVP tapes were saved for compliance
with PRA. In January 2001, the EOP stated that the OVP did not routinely
search its backup tapes in response to subpoenas and that only through the
tape restoration project has the OVP demonstrated that the tapes are
recoverable.
Page 11 GAO- 01- 446 EOP E- mail System
electronic form of e- mail records while tapes were recycled between March
1998 and March 2000. Another practice was to use the standard EOP ARMS e-
mail template 8 to create new OVP e- mail user accounts between March 1998
and March 2000, which inadvertently caused ARMS to capture outgoing e- mail
records sent by these users. The result of this practice was that ARMS
captured outgoing e- mail for those OVP users that had the EOP e- mail
template (133 of 157 OVP users). When it learned in March 2000 that ARMS did
not fully preserve OVP e- mail records, the EOP ordered that tape recycling
be stopped and initiated corrective action. According to the EOP, all OVP e-
mail users, except Senate OVP staff, 9 became fully recorded in ARMS as of
May 8, 2000. However, because the OVP did not ensure the preservation of e-
mail records, in either paper or electronic form, presidential records may
have been irretrievably lost. In addition, about 600 backup tapes were added
to the restoration project to recover as many e- mail records as possible.
Several factors affected the size and scope of the tape restoration project
and the related estimated cost of $11.7 million to restore omitted e- mail
records to ARMS. First, an EOP support contractor?s performance led to
increases in the number of tapes for restoration and adding to the passage
of time necessary to diagnose system anomalies and repair malfunctions.
Second, the EOP?s monitoring of e- mail records did not provide reasonable
assurance that ARMS captured all required records, which increased the time
needed to detect the Mail2 and Letter D malfunctions and added to the
accumulation of backup tapes for the restoration effort. Finally, tasks were
added to the project due to legal scrutiny of the e- mail malfunctions and
the EOP?s tape restoration practices.
In supporting its information technology functions, the EOP had a primary
support services contract, which required the contractor to provide common
services to all users of the EOP data center. The data center provides
general computer operations, server and network support, and software
support. Under its contract, Northrop Grumman was required to
8 The ARMS e- mail template was configured to automatically pass to ARMS a
blind copy of all e- mail records sent by an EOP e- mail user. 9 Because the
Vice President also serves as the President of the Senate, some OVP staff
support the Vice President?s Senate duties. Some Senate OVP staff used a
separate e- mail system controlled by the Senate. Several Factors Have
Driven Cost of E- mail Restoration
The EOP?s Support Contractor Did Not Effectively Perform Tasks Related to E-
mail Systems
Page 12 GAO- 01- 446 EOP E- mail System
?manage cartridge tapes, monitor catalog assignments, maintain appropriate
tape management systems (automated on IBMs and VAXs, manual on all other
platforms), 10 [and] maintain back up system tapes in accordance with
Government standards/ procedures? as part of its management of the data
center tape library;
develop and support Notes e- mail, administer Notes systems including
server hardware and software configuration, maintain mail backup systems,
resolve hardware and software problems for the ARMS interface, provide
technical support for operation and maintenance of the interface, and
develop and implement quality assurance programs for this work; and
develop system documentation to capture the functions, interface, and
internal control requirements, and document requirements of existing systems
that need updating or were not previously documented.
The contract also included provisions for EOP oversight and monitoring of
contractor performance. The EOP established contract performance measures
and an associated performance evaluation plan, which provided a mechanism
for formal measurement and documentation of contractor performance and
allocating award fees and penalties based on performance. The EOP conducted
formal award fee assessments of contractor performance twice per year, in
accordance with the contract.
In monitoring the contractor, the EOP found weaknesses in the contractor?s
performance of data center tape management tasks. The EOP?s award fee
assessment report from October 1998 to mid- April 1999 noted that tape
management, in general, had been a weak point in the contractor?s
performance. Specifically, the report stated that the contractor had not
created an inventory of the e- mail system backup tapes, and that these
tapes continued to mount in an uncontrolled manner. As a result, the EOP
rated the contractor?s performance as marginal and withheld about $10,400 in
award fees for this and other data center tasks.
The EOP?s subsequent award fee assessment report for the period of midApril
to September 1999 stated that there continued to be a lack of effective tape
management in the data center and rated the contractor?s performance as
unsatisfactory. We were not able to determine what, if
10 According to FRA, manual external labels or automated tape management
systems for tapes storing electronic records must provide unique
identification, including the name of the unit responsible for the data and
system title. Other identifying information such as file title( s), dates of
creation, volume serial number, and software dependency must be maintained
but not necessarily attached to the tape.
Page 13 GAO- 01- 446 EOP E- mail System
any, award fee was withheld because the EOP did not provide us with complete
award fee documentation. The problems with tape management were not
subsequently resolved. In March 2000, the EOP tape restoration project
manager stated in a memorandum that e- mail backup tapes needing restoration
could not be identified and that all tapes that had been retained would need
to be examined to determine their contents.
The EOP also documented deficiencies in the contractor?s performance of e-
mail system and user account maintenance. According to the EOP, the
contractor had not implemented satisfactory quality controls to prevent or
detect mistakes in user account and system configurations. As a result, the
EOP did not detect the Letter D malfunction for 5 months and the Mail2
malfunction for almost 2 years. For example:
The EOP?s award fee assessment report for October 1998 through midApril
1999 stated that the configuration error that caused the Letter D
malfunction ?should not have occurred and [it] could have been fixed on the
spot had [the contractor implemented] proper testing procedures and quality
assurance measures been in place.? As a result, the EOP rated the
contractor?s performance in this area as unsatisfactory and withheld $20,832
in award fees for this and other system life cycle tasks.
The error in configuration of user accounts that caused the Mail2
malfunction- entering ?MAIL2? instead of ?Mail2?- continued even after it
was discovered in June 1998. In April 1999, the EOP found that this
configuration error had not only occurred continually on the Mail2 server
since October 1996, but also on another EOP e- mail server. The EOP
initiated a review of controls to detect such configuration errors in May
1999.
Finally, up- to- date system documentation for the e- mail system interface
to ARMS was not maintained, as required by the contract and under FRA.
Accurate system documentation and availability of source code facilitate an
understanding of how a system functions, which enables the system owner and
system administrator to diagnose system anomalies- such as those experienced
with the Mail2 and Letter D malfunctions- and optimize system controls.
However, although a subcontractor to a former EOP contractor documented the
system when it developed the interface between the Lotus Notes and ARMS
systems, the Office of Administration staff found that both the system
documentation and the programming source code were missing in 1997. Until
the source code was located in 1998 and the EOP contractor documented the
system functions in 1999, the EOP was unable to implement changes to the
code or develop automated programs to monitor the effective functioning of
the interface.
Page 14 GAO- 01- 446 EOP E- mail System
These shortcomings in tape management, system and account maintenance, and
system documentation contributed to the time necessary for the EOP to detect
the malfunctions and ultimately hindered the EOP?s identification of the e-
mail backup tapes requiring restoration, increasing the quantity of tapes
that would have to be examined in the project. From the time between
identification and repair of the malfunctions- for Mail2, June to November
1998; for Letter D, April to May 1999- and restoration of the tapes that
began in March 2000, the EOP continued to accumulate backup tapes. This
resulted in raising the total number of tapes to be restored from 788 in
December 1998 to about 4, 500 in January 2001 (comprised of 3,900 Mail2 and
Letter D tapes and 600 OVP tapes).
While the EOP?s 1993 policy required that the White House Records Management
Office monitor electronic systems containing presidential and federal e-
mail records, the EOP?s records management program did not provide
reasonable assurance that adequate controls were in place and functioning to
monitor the preservation of such records. Such a monitoring program should
include evaluations of the creation of records, maintenance and use of
records, and records disposition to determine compliance with established
agency and federal recordkeeping requirements.
Pursuant to its policy, the EOP implemented a monitoring program in July
1994 to identify systemic problems with records determinations made by EOP
e- mail users. The EOP?s monitoring program applied, however, only to
federal records and did not cover monitoring of presidential records as
required by the EOP policy. In addition, the monitoring program did not
include tests to ensure the completeness of federal and presidential e- mail
records captured by ARMS that are created by incoming e- mail messages to
the EOP e- mail users. Controls were specifically needed over these incoming
messages because the e- mail system allowed the EOP users to delete incoming
messages before they were automatically copied to ARMS. This ability,
coupled with the EOP?s normal practice of recycling backup tapes, may have
resulted in the irretrievable loss of the electronic form of e- mail
records. However, the EOP did not document or regularly test the adequacy of
records management controls until May 1999 after repair of the Letter D
malfunction, when the EOP implemented an automated e- mail monitoring
program. Because controls were not in place until May 1999, the Letter D
malfunction went undetected for 5 months, adding additional backup tapes to
the size of the restoration project. Monitoring Program
Did Not Ensure Effective Management of E- mail Records
Page 15 GAO- 01- 446 EOP E- mail System
Scrutiny over the tape restoration project by the U. S. District Court, 11
Office of the Independent Counsel, and the Congress, began in March 2000,
introducing additional tasks. Examples of such tasks include use of law
enforcement imaging software (required by the Office of the Independent
Counsel) and development of an interim searchable database and hiring of an
independent validation and verification contractor and security guard
support to enhance integrity of tape processing (required by the U. S.
District Court and the Congress).
According to the EOP, the restoration project was initiated in March 2000-
not in 1998 after repair of Mail2 or in 1999 after repair of Letter D-
because the EOP?s focus in 1999 was on remediation of its systems to ensure
continued processing after the year 2000 and leap day (February 29, 2000)
rollovers. As a result, the quantity of e- mail messages requiring
restoration escalated, drawing the attention of authorities interested in
ascertaining whether the EOP had implemented adequate controls. Table 2
lists restoration project activities estimated at about $3.5 million that
were introduced because of this scrutiny.
Table 2: Additional Project Activities and Related Costs. Project activity
Cost (estimated)
Independent verification and validation contractor $1,993,819 Security (two
contracts): General Services Administration TW & Company
(actual) 97, 920 54,894 Development of interim searchable database 963,400
Tape inventory and catalogue (list) tape contents 455,000
Total $3,565,033
Source: Tape restoration project cost estimate report prepared by Vistronix,
Incorporated, for the EOP in July 2000, the EOP summary of actual costs
through October 29, 2000, and discussions with EOP officials.
11 The tape restoration project was being monitored by Judge Lamberth of the
U. S. District Court incident to the case of Alexander v. FBI (CA 96- 2125,
D. D. C.). The plaintiffs in that case had submitted document requests to
the EOP in an attempt to discover relevant information in EOP?s email.
Legal Scrutiny Added Restoration Tasks
Page 16 GAO- 01- 446 EOP E- mail System
The EOP claimed to have notified investigative bodies as soon as it
understood that document production might have been affected by the email
malfunctions. Although the cause of the Mail2 e- mail malfunction was known
as early as June1998 and Letter D in April 1999, the White House Counsel?s
Office stated that it did not discover until February 2000 that the e- mail
malfunctions ?had affected the integrity of White House document
productions.? Thus, ?the Counsel?s Office did not perceive a need to notify
investigative bodies? prior to that time.
According to EOP officials, when the extent of the malfunctions became clear
to the White House Counsel?s Office in February 2000, that office
?had both written and oral communications with various investigative bodies
regarding the Mail2 issue.? The EOP provided us with copies of its
correspondence to the House Committee on Government Reform that range in
time from March 17 to October 30, 2000. On March 17, 2000, the EOP provided
the Committee with a description of the e- mail system, the malfunctions,
and their effect on document production. A letter to the Committee in June
2000 updated the status of e- mail records management for the Office of the
Vice President. Finally, an October 6, 2000, letter from the EOP to the
Committee referred to 30 letters from the EOP to various investigative
bodies, between March 15 and October 3, 2000, including the Offices of
Independent Counsel, congressional committees, and the Department of
Justice.
We requested copies of these 30 letters, but we were not given copies of any
of them. Without complete documentation, we were unable to confirm the EOP?s
claims to have notified officials concerning the effect of the email
malfunctions on respective document productions to investigative bodies
other than the Committee. In its comments on a draft of this report, the
Office of the former Vice President stated that as a matter of policy,
?the EOP did not cross- pollinate its communications with various
investigative bodies as a means of maintaining the confidentiality of the
legitimate interests and investigative priorities of those investigative
bodies.?
Computer malfunctions, ineffective systems and records management practices,
and miscommunication between EOP components led to e- mail records not being
preserved by ARMS. As a result, the EOP initiated a time- consuming and
expensive effort to recover e- mail records that had not been effectively
managed. Limited Access to
Information Prevented Confirmation of the EOP?s Notification of Appropriate
Officials
Conclusions
Page 17 GAO- 01- 446 EOP E- mail System
The management control weaknesses have implications for the current
administration that inherited the core e- mail systems and records
management structures discussed in this report. Accordingly, we met with the
current Special Assistant to the President and Director of the Office of
Administration and interested EOP staff, to provide suggestions for
establishing effective controls in the records management and e- mail system
support functions. These officials stated that actions are underway to
improve the management of electronic records within EOP. These efforts
included developing and updating policies for federal and presidential
records, defining procedures for electronic records, and establishing a
chief information officer position.
In written comments on a draft of this report, which are reprinted in
appendix VI, the Counsel to former Vice President Gore had two general
points. First, he stated that the report did not explicitly state that we
found no evidence of attempts by OVP staff to deliberately fail to preserve
records. In our report, our objective was to examine the facts surrounding
the e- mail systems. Specifically, we were requested to assess the adequacy
of management controls over EOP?s records management system, particularly as
they related to the numerous e- mail system malfunctions experienced by EOP
since 1996. That assessment entailed the examination of management policies,
programs, and practices as they related to effective records management
controls and to the requirements of the PRA and FRA. Our review was not
directed at assessing the intent of individuals responsible for preserving
records and providing responsive records to investigative bodies. As such,
we can offer no conclusions on this matter.
Second, the Office of the former Vice President stated that ARMS was a
unique system for which best practices did not apply and provided additional
information on the creation and use of this system that was not reflected in
the draft report. In response, we have revised the report to reflect this
additional information. However, we disagree with the Office of the former
Vice President?s characterization of the applicability of best practices to
ARMS. Best practices for management of records currently exist in the form
of National Archives and Records Administration guidance contained in title
36 CFR and apply to records management programs irrespective of the type of
system involved. The majority of this guidance is prescriptive only for
federal records; however, this guidance applies to ARMS because ARMS
archives federal records. Also, the guidance does not explicitly preclude
application of these practices in the management of presidential/ vice
presidential records. Although the Agency Comments
and Our Evaluation
Page 18 GAO- 01- 446 EOP E- mail System
evolution of ARMS may continue, the stability of generally accepted records
management best practices establishes a control foundation for any system,
whether manual or automated.
Comments from the Office of the former Vice President are reprinted in
appendix VI. In addition, this appendix contains our detailed responses to
numerous technical comments that were provided.
In his comments on the draft report, the representative of former President
Clinton stated that he had reviewed the draft but did not have access to the
records necessary to properly review and comment on its accuracy. He urged
us to incorporate comments that we received from EOP during the course of
our review, which we have done as appropriate.
The Deputy General Counsel, Office of Administration, EOP, orally provided a
technical clarification, stating that the May 1997 policy that replaced the
PRA section of the 1993 policy was issued by the White House and not EOP,
which we incorporated as appropriate.
We are sending copies of this report to the Honorable Henry Waxman, Ranking
Minority Member; the Honorable Hector Irastorza, the Deputy Assistant to the
President for Management and Administration, the White House; Bruce R.
Lindsey, the personal representative of former President Clinton; Andrew M.
Wright, Counsel to former Vice President Gore; and interested congressional
committees. Copies will then be available on our Web site at www. gao. gov.
If you or your office have any questions about this report, please contact
me at (202) 512- 6240 or by e- mail at KoontzL@ gao. gov. Nancy DeFrancesco,
Michael Fruitman, Linda Lambert, and Charles Roney were major contributors
to this report.
Sincerely yours, Linda D. Koontz Director, Information Management Issues
Enclosures
Appendix I: Scope and Methodology Page 19 GAO- 01- 446 EOP E- mail System
To obtain information regarding events and to identify individuals key to
the Executive Office of the President?s (EOP) e- mail systems and the
malfunctions experienced, we
reviewed about 10, 000 pages of documentation submitted to the House
Committee on Government Reform by the EOP and its contractor Northrop
Grumman;
reviewed 159 hearing exhibits prepared by the Committee from the above
documentation, and pertinent pieces of correspondence and other related
documentation provided by the Committee;
obtained copies of the EOP?s task order OA8004 dated September 30, 1997,
with Northrop Grumman, and letter contract OA20C002 dated March 29, 2000,
with Enterprise Computing Solutions Technologies, Incorporated (ECS);
obtained transcripts of a hearing pertaining to the timing of the
restoration and production of EOP e- mail, held by the U. S. District Court
beginning July 13, 2000, and observed the hearing sessions between July 31
and October 3, 2000; and
obtained transcripts of hearings pertaining the EOP e- mail systems, held
by the House Committee on Government Reform on March 23, March 30, May 3,
and May 4, 2000.
To obtain additional information on the e- mail malfunctions and Office of
the Vice President?s e- mail records procedures that were not addressed by
the Committee documents or hearing transcripts, the EOP stipulated that we
submit written questions. We submitted seven sets of questions and reviewed
the written responses and additional documentation provided, including two
additional tape restoration project contracts: Vistronix, Incorporated,
letter contract CI20C03 dated May 8, 2000; and TW and Company contract
DC21C01 dated November 9, 2000. We also met with EOP officials to discuss
specific issues pertaining to the restoration of the e- mail.
To determine costs for the project, we obtained copies of paid ECS invoices
covering the period May 5 through September 5, 2000. We also obtained a copy
of the July 28, 2000, Vistronix report that estimates project costs and
compared this estimate to updated information regarding project status and
actual costs incurred for the project through October 29, 2000, that we
obtained from the EOP.
Our review of cost issues was limited by the unavailability of complete and
reliable information. Specifically, we were not given access to OVP files of
hard copy e- mail records, which impaired a full assessment of the OVP?s
Appendix I: Scope and Methodology
Appendix I: Scope and Methodology Page 20 GAO- 01- 446 EOP E- mail System
practices to preserve e- mail records. In addition, we were not provided
complete documentation of contract performance assessments and related
consequences for fiscal years 1998, 1999, and 2000; thus, we were unable to
determine the extent to which the EOP documented contractor performance
weaknesses and withheld award fees. Finally, we were not provided with
copies of letters from the EOP to investigative bodies necessary for us to
confirm that the EOP notified these entities of the email malfunctions.
We performed our work from July 2000 through January 2001, in accordance
with generally accepted government auditing standards.
Appendix II: Chronology of E- mail Malfunctions
Page 21 GAO- 01- 446 EOP E- mail System
Reference a documents Date Description of event a
Cross- reference to other documents listed in this table a Comments a
E5346 through E5347, E5353 through E5354
07/ 94 and 10/ 94 E- mail from Nell Doering to various staff
including Daniel Barry, James Wright, and others establishing an e- mail
monitoring program that was implemented 07/ 25/ 94. The
Presidential and Office of the Vice President
(OVP) Records Monitoring Plan dated 10/ 28/ 94 that was attached provided a
quarterly monitoring schedule through fiscal year (FY) 1995.
This monitoring program required that EOP agencies review a sample of record
and nonrecord e- mail messages to ensure that proper record designation was
applied by e- mail users.
E0595 through E0608 03/ 07/ 95 A document entitled Automated Records
Management System (ARMS) for Lotus Notes Mail provided a technical
description of the Notes/ ARMS interface functionality. The document stated
that the interface copies all internal and external e- mail and receipts to
the ARMS records management database. White House Executive Office of the
President (EOP) e- mail users may elect to remove the record designation of
an outgoing mail message, which they created, by selecting a nonrecord
command button. This option causes the message to not be captured as an
official federal or presidential record to be archived in the ARMS records
management database.
E5346 through E5347 and E5353 through E5354
Nonrecords were also captured temporarily by ARMS for sampling under EOP?s
records monitoring program to detect and correct improper designation of e-
mail records.
E1115 08/ 29/ 96 The Mail2 server was created with the server name of Mail2
and User ID of Mail2/ EOP. E0564 E0564 through E0566 10/ 29/ 96 E- mail
stating that a Planning Research
Corporation (PRC) contract employee installed the Mail2 server, as well as
performed initial configuration of the Notes server. The Notes/ ARMS
interface Name and Address Book (NAB) was configured with five views.
Exhibit 125 and document E1115 PRC was the predecessor
contractor to Northrop Grumman (NG) at the White House.
E2309 through E2310 02/ 25/ 97
and 02/ 26/ 97
Series of e- mail between Daniel Barry and Laura Crabtree regarding ??
?problem
attachment? problem? whereby messages were deleted manually if they caused a
problem with the Notes/ ARMS interface.
Based on available documentation, this problem was not related to the Mail2
or Letter D malfunctions. Exhibit 17 03/ 04/ 97 E- mail from Daniel Barry to
John W. McGinnis
asking him to write a paragraph regarding efforts performed to restore
10,000+ lost E- mail records. Mr. McGinnis? contribution would finish up Mr.
Barry?s ?memo to the record.?
The 3/ 23/ 00 hearing witness list lists Mr. Barry as Daniel ?Tony?
Barry. He was referred to as Tony in many emails and other documents.
Based on available documentation, this problem was not related to the Mail2
or Letter D malfunctions.
Appendix II: Chronology of E- mail Malfunctions
Appendix II: Chronology of E- mail Malfunctions
Page 22 GAO- 01- 446 EOP E- mail System
Reference a documents Date Description of event a
Cross- reference to other documents listed in this table a Comments a
E2316 through E2323 03/ 06/ 97 Series of e- mails from Daniel Barry to James
Wright and others regarding ?missing Notes records.? The problem occurred in
September 1996 and caused some e- mail to be sent to the ARMS records
management database without the message header. Other e- mails would be sent
to the ARMS records management database without the message text. Mr. Barry
stated he had received no training on Notes. He reported that to his
knowledge, efforts to recover these missing e- mail messages in the
beginning of 1997 were unsuccessful. According to this e- mail, between
September 11 and 16, 1996 23, 436 e- mail messages were missing headers
(that is, ?To,? ?Subject,? etc.) and 10, 138 messages were missing text.
Exhibit 17 Based on available documentation, this problem was not related to
the Mail2 or Letter D malfunctions.
E6001 through E6004 03/ 17/ 97 Minutes of a staff meeting involving Jim
Wright,
Laura Crabtree, Karl Heissner, and others. Backup of EOP Information Systems
and Technology Division (IS& T) servers was discussed. The minutes stated
that
?[ r] equirements haven?t been documented but approaches are being
developed. An inventory does exist of what does need to be backed up.? The
minutes also question what the retention period should be, and stated that
all types of server platforms ?are being reviewed: VAX, client server, and
mainframe.? E6547 through E6558 03/ 21/ 97 E- mail to Ada Posey and others
regarding
restricted 1997 appropriation funding and that funds were needed for
additional e- mail servers to relieve network strain. E- mail records
management traffic was projected to increase 90 percent in FY 1998 from FY
1995?s traffic rates. The e- mail noted that EOP?s migration from Novell
Netware- based network to Microsoft Windows NT began in May 1996. It also
stated that the ?? client/ server backup system used to preserve data
integrity and facilitate the archival of Presidential and Federal records
has exceeded its life cycle [and can only] backup 50% of our file servers.
However, due to hardware and software failures, less than 30% of our file
servers have current backups. Currently, the only copy of data that is not
backed up resides on each file server?s physical hard drives. If any of
those hard drives fail, data will be lost.?
E1022, E1394, Exhibit 9
Appendix II: Chronology of E- mail Malfunctions
Page 23 GAO- 01- 446 EOP E- mail System
Reference a documents Date Description of event a
Cross- reference to other documents listed in this table a Comments a
E6005 through E6009 03/ 24/ 97 Minutes of a staff meeting involving Laura
Crabtree, Nell Doering, Karl Heissner, Jim Wright, and others. The minutes
discussed external e- mail, stating ?Laura will have a report on the
problems we?ve been having for Ada [Posey] tomorrow.?
Based on available documentation, these problems were not related to the
Mail2 or Letter D malfunctions.
E2379 through E2380 07/ 17/ 97 E- mail from Bruce Overton to Laura Crabtree
and others with attached e- mail notice to all EOP mail users regarding
implementation of Notes janitor agent. The janitor agent was set to move up
to 300 e- mail messages that were greater than 45 days old from the ?inbox?
folder to the ?trash? folder in the user?s mail file. The messages were then
automatically deleted 1 week later. The process was scheduled to run weekly
beginning July 18, 1997. Mr. Overton asked whether the records management
team had reviewed the janitor policy.
A 9/ 29/ 00 response to our question on this matter stated that ??. EOP does
not believe that this
?janitor agent? was ever implemented.? In its comments on a draft of this
report, EOP stated that ?? legal and records management staff never approved
use of the agent.? E7209 through E7211 08/ 11/ 97 E- mail to Daniel Barry,
Nellie Doering, Karl
Heissner, Laura Crabtree, Ada Posey, and others containing minutes of a
staff meeting. The minutes state that ?Lotus Notes staff will be [coming] in
to address the lost mail problem (we need a ruling from [counsel] as to
issues with mail and as to what may be purchased).?
Based on available documentation, this problem was not related to the Mail2
or Letter D malfunctions.
Exhibit 49 01/ 30/ 98 A document created by Daniel Barry said that he
noticed a potential problem with the capture of e- mail traffic coming into
the EOP Lotus Notes system from the Internet. John Spriggs, the NG e- mail
administrator, was asked to print out the firewall log to determine whether
incoming Internet e- mail messages were received. The document stated ?We
are not sure where the problem lies at this point.?
Exhibit 57 A computer directory location and date indicating 1/ 30/ 98 were
handwritten on the document.
Exhibit 10 No date A document containing May 16 meeting minutes discusses a
problem with the Notes/ ARMS interface scan process randomly skipping e-
mail boxes on the Mail2 server. A random sample of e- mail boxes was
analyzed and it was discovered that e- mails dating back to 1996 were not
subject to records management, or captured by the ARMS records management
database. The course of action was to further assess the problem.
Exhibit 12
Appendix II: Chronology of E- mail Malfunctions
Page 24 GAO- 01- 446 EOP E- mail System
Reference a documents Date Description of event a
Cross- reference to other documents listed in this table a Comments a
Exhibit 16 06/ 01/ 98 E- mail from Virginia Apuzzo to all EOP staff
requesting that users delete unneeded e- mail on e- mail servers or save e-
mail to individual workstations to prevent system failure. This message was
repeated and re- sent to all EOP staff on 03/ 24/ 99 (Exhibit 16, page
E0545).
E7202 through E7203
Exhibit 12 06/ 12/ 98 E- mail from Robert Haas, NG contract employee, to
Betty Lambuth, NG subcontractor, mentioning ?a design flow in the Record
Management design in Notes that would allow for certain types of mail
document[ s] to bypass record management.?
Exhibit 10 The term ?design flow? may be a typo of ?design
flaw?. Exhibit 50 06/ 18/ 98 A draft document entitled Lotus Notes to ARMS
Interface Anomaly that was sent from the fax machine shared by the Office of
Administration (OA) General Counsel and the OA Director to an unknown
recipient. The draft described the Mail2 server malfunction and stated that
the root of the problem stems from a discrepancy between the capitalization
of the letters in the name of the affected Lotus Notes server. The problem
was introduced by human error in capitalization of the server name letters
in the affected users? e- mail accounts as ?MAIL2?
instead of ?Mail2.? The scope of the problem was that 526 users of Mail2
were affected. Exhibits 1 through 5 06/ 19/ 98 Memo from Virginia M. Apuzzo
to John D.
Podesta, former Deputy Chief of Staff, entitled Technical Anomaly in
Automated E- Mail Records Management System, which describes the
capitalization of the server name letters in the affected users? e- mail
accounts as ?MAIL2?
instead of ?Mail2? as the cause of the Mail2 malfunction. Exhibits 1 and 3
contained a handwritten note saying, ?Chuck- I sent this memo to John this
afternoon. Ginny? dated 6/ 19. Exhibits 4 and 5 contained a handwritten note
saying, ?Ginny Please ask Mark to brief me on this. Thanks, John.?
Exhibits 1 through 5 were variations of the same memo. U. S. District Court
testimony by Charles Ruff on 08/ 28/ 00 confirmed
?Chuck? to be Charles Ruff, ?Ginny? to be Virginia Apuzzo, and
?John? to be John Podesta.
E0567 through E0569, E0573 through E0578, E0589 through E0594
07/ 09/ 98 Draft plan dated July 1998 and the undated finalized versions of
the same plan to first process data records from Notes e- mail files to the
VAX subsystem and then to restore Notes/ ARMS interface processing on Mail2.
The plan was to first turn the Notes/ ARMS interface monitor task off, copy
each affected user?s existing mail file (containing the unrecorded mail) to
the Notes/ ARMS interface server, and then turn the Notes/ ARMS interface
Exhibit 19
Appendix II: Chronology of E- mail Malfunctions
Page 25 GAO- 01- 446 EOP E- mail System
Reference a documents Date Description of event a
Cross- reference to other documents listed in this table a Comments a
monitor back on for processing of unrecorded messages into the ARMS records
management database. Once transferred, all documents would be marked as
recorded and the syntax [capitalization of the server name letters in the
affected users? e- mail accounts as ?MAIL2?
instead of ?Mail2?] of the server name for the user?s account would be
corrected. Day- forward processing would resume on Mail2. The plan also
noted that a ?critical issue? existed for the processing of affected records
from Notes to the ARMS records management database, which resides on the VAX
subsystem. It stated that ?preliminary investigation has shown that the date
used to create the ARMS header files is the date that the messages are
written to the Records Management mail- in database. This date doesn?t
reflect the true sent mail date in the original message.? Exhibit 19 07/ 24/
98 E- mail from Daniel Barry to James Wright
noting under Additional activities that he continues to be involved in
discussions regarding the Mail2 problem but there has been no movement thus
far in correcting the problem or getting the data over to the ARMS records
management database. The plan for fixing the problem had been submitted to
EOP by NG. Exhibit 22 07/ 30/ 98 E- mail from Daniel Barry to James Wright
with
language similar to Exhibit 19. Exhibit 23 08/ 13/ 98 E- mail from Daniel
Barry to James Wright
expressing ?concerns? about ?the mail2 problem? or project X. He stated that
there was no movement underway to fix the problem and recover the lost
records from the backup tapes.
Exhibit 24 Exhibit 24 08/ 13/ 98 E- mail from James Wright to Daniel Barry
responding to his ?concerns? e- mail saying that there has been some
movement to get back on track and NG can develop a plan to get this effort
going. ?Certainly the Data Center and the Records Team has been left out of
this matter and the result could be a great deal of work put upon us later.?
This exhibit stated that Kathleen Gallant informed Mr. Wright that a
Paulette [Cichon] briefed Jim Welsh of NG to proceed with developing the
plan.
Exhibit 23
Appendix II: Chronology of E- mail Malfunctions
Page 26 GAO- 01- 446 EOP E- mail System
Reference a documents Date Description of event a
Cross- reference to other documents listed in this table a Comments a
Exhibit 40 08/ 13/ 98 E- mail from Kathleen Gallant to James Wright with a
copy to Daniel Barry. It stated that Jim Webster was taking Betty Lambuth?s
place and a meeting was held authorizing that it was okay to discuss the
Mail2 project in detail with Mr. Webster. The e- mail stated that ?I also
agree with Tony [Daniel Barry] about the new searches that will have to be
done. We need direction from OA counsel on that front.? Exhibit 25 09/ 01/
98 E- mail from Daniel Barry to James Webster,
with a copy to James Wright, asking for a meeting to discuss the plan/
approach on the Mail2 problem. An attached e- mail ?Weekly?
report dated 9/ 4/ 98 (E2016) says that Mr. Barry had a discussion with Jim
Webster, NG,
?regarding OVP E- mail and records management.? Mr. Barry referred Mr.
Webster to Kathleen Gallant for a decision to have all OVP users subject to
records management through the ARMS records management database. The e- mail
noted that a goods and services authorization request memo was prepared for
64GB disks for the Notes Mail2 recovery project. This will go to Financial
Management Division (FMD) for signature and then make its way to the OA
counsel. Exhibit 120 09/ 01/ 98,
09/ 15/ 98 E- mail from Daniel Barry to James Webster, NG, requesting
meeting with him, Sandra
Golas, Robert Haas, and John Spriggs to discuss the Mail2 problem and the
plan/ approach for proceeding. Webster forwarded Barry?s e- mail to Spriggs,
Golas, Robert Haas, and Yiman Salim, saying Barry wants ?to set up a meeting
to discuss our favorite issue.? I will tell Tony [Daniel Barry] that I will
have to delay any meeting until we resolve some internal issues.? On page
E4015 of this exhibit, an e- mail dated 9/ 15/ 98 from Daniel Barry to James
Webster about the Mail2 project meeting asks about Webster?s plan for
?righting the wrong? phase of this project. Barry wants to know when the
meeting will occur. E5874 09/ 10/ 98 E- mail from Mark Bartholomew to Adam
Greenstone [copy to Daniel Barry, Karl Heissner, Kathleen Gallant] regarding
processing of incoming Internet e- mail to the President, Vice President,
and the First Lady, referred to as White House ?principals.?
Incoming e- mail to these principals was E6020 through
E6022 The bulk e- mail ?package?
was captured in the ARMS records management database.
Appendix II: Chronology of E- mail Malfunctions
Page 27 GAO- 01- 446 EOP E- mail System
Reference a documents Date Description of event a
Cross- reference to other documents listed in this table a Comments a
?packaged? into another e- mail to the correspondence office for each
respective principal. These ?packaged? e- mails, referred to as ?bulk mail,?
were not subject to records management in the ARMS records management
database message by message or as e- mail to the principal, but rather as
bulk e- mail to the appropriate representative of the principal. Exhibit 26
09/ 10/ 98 E- mail from Daniel Barry to Kathleen Gallant
and James Wright expressing concern about the lack of movement on the Mail2
problem.
?We have known about this problem for 4 months now and not a single record
[of the affected users] has been passed to ARMS... even worse, the root
problem has not been fixed.? Exhibit 64 09/ 14/ 98 Letter from Joseph
Lucente noting ?that in late
May of this year, a dysfunction in the EOP email system was detected by an
employee...? of NG. Exhibit 27 09/ 25/ 98 E- mail from Daniel Barry to James
Wright
stating there was still no movement in the Mail2 problem and asking what his
role was supposed to be in the project. Wright responds the same day saying
that IS& T needs to start by providing NG with direction. Exhibit 47 09/ 25/
98 E- mail from Kathleen Gallant to Daniel Barry
stating that the Notes Anomaly Team has met and come up with a strategy. The
team will back up Mail2 using a DLT [digital linear tape] drive that can
verify as it writes and execute
?the Notes agent changes to change all the [IDs] looking for Mail2 to
MAIL2.?? A second phase deals with reconstruction of the nonrecords- managed
files from the Mail2 server. She said, ?Contracts is aware of the whole
mess, and supports the creation of the IWO [Internal Work Order]?.?
E1235 and E1236 Agents are filtering instructions that automate operations
on documents in a Notes database.
E1363 through E1375 09/ 30/ 98 A NG document entitled Final Y2K Status
Report for OA/ IS& T stated that the software for the ARMS records
management database was not fully documented and controlled, and that it was
?born very quickly in 1994 to meet an immediate need for the recording of
message traffic.? The C- language programs in the interface were not Year
2000- compliant.
E0766 and E1015 through E1021 Year 2000 readiness of
information systems was a concurrent effort to repair e- mail malfunctions.
Appendix II: Chronology of E- mail Malfunctions
Page 28 GAO- 01- 446 EOP E- mail System
Reference a documents Date Description of event a
Cross- reference to other documents listed in this table a Comments a
Exhibit 28 10/ 13/ 98 E- mail from Daniel Barry to Christina VanFossan and
Joseph Kouba stating, ?Per our meeting last week... here are 4 projects that
I can see happening in the near term that are not currently budgeted for.?
The third project was listed as Project X (Mail2) Reconstruction, with an
estimated cost of $250,000.
E0766 through E0768 11/ 06/ 98,
11/ 12/ 98 E- mail created by Marvin Miller on 11/ 06/ 98 and revised 11/
12/ 98 by Yiman Salim regarding
an ARMS IWO feasibility and requirements study. The document stated that
Notes and Notes Application Programming Interface (API) version 4.6 software
was used in the Notes/ ARMS interface. To perform maintenance on the ARMS
software, the skill set required a C++ software engineer, a Notes developer
with C++ and VAX knowledge, and a system administrator with Windows NT and
Lotus Notes skills. The tasks listed in the emails included Year 2000 code
modification and implementation of calendaring and scheduling modifications.
However, in the 11/ 12/ 00 revision, Salim removed two tasks from Miller?s
list, including ?Mail server name case sensitivity.?
E1363 through E1375 ?Name case sensitivity?
refers to the capitalization of the server name letters in the affected
users? email accounts as ?MAIL2?
instead of ?Mail2.?
In its comments on a draft of this report, EOP stated that the Mail2 case
sensitivity was not related to Year 2000 efforts addressed by the ?ARMS
IWO.? Exhibit 30 11/ 12/ 98 E- mail from Daniel Barry to DeVere Patton
saying that NG needs technical guidance on several items, including a
response to the Mail2 IWO, which had not yet been finalized.
Exhibits 31 and 46
Exhibit 31 11/ 13/ 98 E- mail from Daniel Barry to James Wright mentioning
the response on the Mail2 IWO. Exhibits 30 and
46 Exhibit 121 11/ 16/ 98 E- mail from Daniel Barry to Sheryl Hall, John
Spriggs, and Joseph Vasta requesting a meeting to discuss a course of action
on moving the Mail2 reconstruction project forward. Exhibit 103 11/ 20/ 98
E- mail from Daniel Barry stating that Karl
Heissner was the project manager for phases two and three of the Mail2
reconstruction work. The e- mail mentioned that phase one was the
??. fix (Stop the bleeding) on the mail 2 server.? Phases two and three were
not described in this e- mail.
Appendix II: Chronology of E- mail Malfunctions
Page 29 GAO- 01- 446 EOP E- mail System
Reference a documents Date Description of event a
Cross- reference to other documents listed in this table a Comments a
Exhibit 70 12/ 02/ 98 Weekly Contracting Officer?s Technical Representative
(COTR) meeting minutes for 11/ 20/ 98. NG commented that the Rough Order of
Magnitude (ROM) ?? could be plus or minus 20 percent?
Exhibits 63, 67, 72, 73, 75, and 122
Exhibit 72 12/ 02/ 98 NG ROM for cost of IWO was $602,492. Exhibits 63, 67,
70, 73, 75, 122, E1016 Exhibit 122 12/ 02/ 98 E- mail from Daniel Barry to
Karl Heissner
stating that NG?s ROM did not include the cost to do the restoration.
Exhibits 63, 67, 70, 72, 73, and 75
E5783 12/ 04/ 98 E- mail from Moe Vela to ?All Staff? requesting staff to
delete unnecessary e- mail because
?Lotus Notes is within 5 [percent] of maximum capacity and it will shut down
at that time.? Exhibit 63 12/ 11/ 98 Memo from Joseph Vasta to DeVere
Patton,
mentioning, in part, the Mail2 E- mail Reconstruction. Said NG would
evaluate at least one tape.
Exhibits 67, 70, 72, 73, 75, and 122
Exhibit 73 12/ 18/ 98 Weekly COTR meeting minutes for 12/ 16/ 98.
?Government emphasized that before any work be undertaken, it is imperative
that a tape inventory be done.? NG insisted that an inventory in the absence
of an automated tape management system would be very laborintensive (over 1
hour per tape) and ?would far exceed the services in the Base Services IWO.?
Exhibits 63, 67, 70, 72, 75, and 122
Exhibit 67 12/ 23/ 98 Weekly COTR meeting minutes for 12/ 23/ 98. NG
reported that a recent tape was restored, and the universe of tapes was
identified as 788.
Exhibits 63, 70, 72, 73, and 122
Exhibit 32 12/ 24/ 98 E- mail from Daniel Barry to Joseph Kouba. It lists
three tiers of projects related to the decisions resulting from the
Armstrong court case. The third tier includes projects that, in his opinion,
could be done but were not vital. The Mail2 reconstruction (Project X) was
listed in the third tier, showing an FY 1999 cost of $650,000 and an FY 2000
cost of $1 million.
The Armstrong court held that EOP components must capture electronic records
subject to the Federal Records Act.
Exhibit 129 01/ 06/ 99 Weekly report from Daniel Barry in which he stressed
that OA Counsel needed to be in the decision- making process regarding
projects that should move forward, such as Project X. E6020 through E6022
01/ 07/ 99 Meeting agenda and handwritten meeting notes
regarding ?records disposition for internet email.? It stated that for e-
mail to ?? ?President
@whitehouse. gov, ? all emails [were] combined into one and sent to [the
President?s bulk e- mail
E5874
Appendix II: Chronology of E- mail Malfunctions
Page 30 GAO- 01- 446 EOP E- mail System
Reference a documents Date Description of event a
Cross- reference to other documents listed in this table a Comments a
representative].? It was packaged and sent on to the ARMS records management
database where it was retained as e- mail to the President?s bulk e- mail
representative. E5784 01/ 13/ 99 E- mail from Moe Vela to ?All Staff?
requesting
staff to delete unnecessary e- mail and databases in Lotus Notes because the
IS& T anticipated running out of disk space in about 1 hour. E0772 02/ 01/
99 E- mail from Daniel Barry about ?the effort
underway to locate and document the [Notes/ ARMS] interface code so that
changes can be made.? E0964 02/ 01/ 99 E- mail from Daniel Barry to Raul
Cavazos
stating that ??. the ARMS processing requested a tape from March 1997 and
Ops [Operations] was unable to find the tape.? Concludes, ?there needs to be
an inventory of the ARMS tapes that are stored in the boxes in the back of
the DC [data center], to make it possible to retrieve any one of the tapes.?
Exhibit 75 02/ 02/ 99 Weekly COTR meeting minutes for 01/ 27/ 99.
The EOP asked about NG?s tape inventory and tape tracking methodology. The
EOP requested a tape inventory (that included tape ID, volume, dataset ID,
etc.) from October 1997, the point when NG was awarded contract.
Exhibits 63, 67, 70, 72, 73, and 122
Exhibit 81 02/ 05/ 99 Memo from Karl Heissner explaining the Mail2 server
problem. He stated that ?the problem, that some e- mail messages were not
records managed properly, was discovered in February 1998, and Mr. Barry
immediately notified the Lotus Notes Group (NG- Spriggs/ Golas) as well as
IS& T Management of the problem.?
Exhibits 82 and 83
Exhibit 43 03/ 12/ 99 E- mail from Nellie Doering to Dorothy Cleal, copy to
Daniel Barry, suggesting changes to Mr. Barry?s proposal. Ms. Doering stated
that Mr. Barry agreed to ensure the ?Server 2? tapes were properly
inventoried and documented by NG. Exhibit 37 03/ 18/ 99 E- mail from Daniel
Barry to DeVere Patton, Karl
Heissner, and Nellie Doering about the Mail2 records management problem.
Barry stated that: ?It has come to my attention that when the
?bleeding? was stopped on MAIL2 in November 1998, ALL the bleeding may NOT
have been stopped. I have spoken with John Spriggs and it appears as though
at least one account,
Appendix II: Chronology of E- mail Malfunctions
Page 31 GAO- 01- 446 EOP E- mail System
Reference a documents Date Description of event a
Cross- reference to other documents listed in this table a Comments a
MILLENNIUM, may still have the problem. I believe NG should be instructed to
investigate and report back on exactly what the situation is with regard to
the MAIL2 problem.? Exhibit 41 03/ 18/ 99 A series of e- mails noting that
the Mail2
problem was not allowing EOP staff to recycle backup tapes. Nellie Doering
stated that the tapes need to be inventoried by NG and ??. ?secured? so that
they are easily identified as the Server 2 Backup Tapes that were not
[subject to records management.]? DeVere Patton stated that ?? it is
something that NG should be doing under the base contract [in response to
Dorothy Cleal?s inquiry regarding contract scope]. Exhibit 42 03/ 18/ 99 E-
mail from Joseph Kouba to Daniel Barry and
others stating, ?looks like MAIL 2 reconstruction is back on hold until some
additional confirmation is received.? Exhibit 124 03/ 19/ 99 E- mail from
Karl Heissner to Nellie Doering,
Daniel Barry, and Mark Bartholomew. Meeting minutes to discuss the Mail2
reconstruction. It stated that the project was on hold awaiting approval of
the OA counsel. E0812 03/ 24/ 99 E- mail from Michael Ritter to Joseph Vasta
regarding tape inventory. It stated that ?there
are between 700 and 800 tapes [in] mostly 8- mm [millimeter] format and do
not have a
?header? catalogue (meaning that the entire tape must be read to develop a
directory list).? Also, ?there is a short period of a few months where we do
not have backup tapes due to changes in the governments requirements.? Mr.
Ritter estimates that under ?perfect conditions? it would take 1.5 to 3
hours per tape to read the tape and print the catalogue information;
however, he stated, the tapes were old and were produced under various
hardware and software configurations. He stated that ?it is expected that
there will be tapes that cannot even be read with the current hardware on
site.? E1255 through E1256 03/ 25/ 99 NG memo from Joe Vasta to DeVere
Patton
regarding Mail2 backup tapes. It stated that NG will secure Mail2 backups
for October 1996 through November 1998 and provide minimal inventory
information. The memo identified that during this period, NG was instructed
by ?the
government? to reuse backup tapes causing
Appendix II: Chronology of E- mail Malfunctions
Page 32 GAO- 01- 446 EOP E- mail System
Reference a documents Date Description of event a
Cross- reference to other documents listed in this table a Comments a
some to be overwritten. The memo stated that a report detailing a
description of the system failure was beyond the scope of the base contract.
Exhibit 117 04/ 01/ 99 E- mail from Michael Ritter to Yiman Salim,
Robert Haas, John Spriggs, and Sandra Golas asking them a series of
questions about when the Mail2 server problem started and when it was
resolved. He requested responses from each by 4/ 2/ 99.
E0833 and E0834 E1194 04/ 01/ 99 E- mail from Benjamin Kirby to Adam
Greenstone, Daniel Barry, others, stating the millennium account was not
subject to records management. Exhibit 66 04/ 02/ 99 E- mail from Yiman
Salim to Michael Ritter
about the Mail2 issue. She stated that NG personnel became aware of the
Mail2 problem on June 12, 1998, and that the cause of the problem, writing
Mail2 as MAIL2, was identified a couple of days later by John Spriggs. She
also noted that the problem was inherent in the ARMS process because it ?is
currently enforcing server name case sensitivity.? The message also stated
that in December 1998 NG initiated a task to baseline the Notes/ ARMS
interface program. Completion was scheduled for April/ May 1999, after which
programming changes could be made as needed.
Exhibit 117 Exhibit 71 04/ 02/ 99 E- mail from Michael Ritter to Joseph
Vasta.
This e- mail explains the Mail2 server problem in detail. It also mentions
that NG staff have identified a similar problem on MAIL1 that was also
caused by human error. The message stated that an employee of the contractor
PRC designed a program to migrate user accounts to the Notes server, and
this program entered the users with the improper server name syntax. Exhibit
44 04/ 09/ 99 Meeting notice from Karl Heissner about
records management of all Mail2 and MAIL1 server accounts. This notice
stated that Mail2 and MAIL1 servers still contain
?unrecordsmanaged? accounts, preventing the recycling of backup tapes. The
objective was to make necessary corrections by April 18.
Appendix II: Chronology of E- mail Malfunctions
Page 33 GAO- 01- 446 EOP E- mail System
Reference a documents Date Description of event a
Cross- reference to other documents listed in this table a Comments a
Exhibit 125 04/ 09/ 99 Series of e- mails describing Letter D malfunction.
Marvin Miller e- mails to Robert Whiteman and Yiman Salim that the Letter D
malfunction was caused in 11/ 98 when NG was correcting a Lotus Notes buffer
overflow system failure. The user accounts (sorted by first initial/ last
name) were distributed among five Notes/ ARMS interface e- mail server
views, so that all 26 letters of the alphabet were assigned to one of the 5
views. ?The letter ?D? was inadvertently omitted and the letter ?J? was
added twice.? This caused all users on all email servers whose first names
began with D to not be subject to records management.
Exhibits 13, 79, documents E3218, E3219 through E3221, and E3222 through
E3229
E3219 through E3221 04/ 15/ 99 Memo from Robert Whiteman (NG) to Dorothy
Cleal describing events regarding the Letter D malfunction. It stated that a
Notes/ ARMS interface e- mail server system failure occurred in November
1998, which was caused by a configuration error. In correcting the failure,
NG reconfigured the Notes/ ARMS interface views on Lotus Notes e- mail
servers and
?inadvertently omitted? the letter D from the views, causing incoming non-
Notes e- mail for users with first names beginning with D not to be subject
to records management. According to the memo, the Lotus Notes team reported
this to NG management on 04/ 09/ 99. Mr. Whiteman recommends not recycling
tapes until the problem was resolved and to include the saved tapes in the
ongoing e- mail recovery effort.
Exhibit 79, documents E3218, and E3222 through E3229
E3218 04/ 15/ 99 Memo from Dorothy Cleal, through Michael Lyle, to Mark
Lindsay regarding the Letter D. The memo stated that 191 users were affected
and who have not had e- mail records- managed since November 1998. Cautions
that ?? records management problem may not be limited to this particular
error.?
Exhibits 13 and 79; documents E3218, E3219 through E3221, and E3222 through
E3229
Exhibit 79, E3222 04/ 15/ 99 Cover memo from Eric Ritter to Dorothy Cleal
with attached listing that shows the 191 users impacted by the Letter D
malfunction- 54 on MAIL1, 77 on Mail2, 29 on MAIL3, 30 on MAIL5, and 1 on
RDS1.
Exhibits 13, 125; documents E3218, E3219 through 3221, and E3222 through
E3229 Exhibit 99 04/ 20/ 99 This e- mail from Joseph Kouba stated that a
?legal determination [was needed] that reconstruction is required by the
court case and that this is a legitimate use of Armstrong funds.?
Appendix II: Chronology of E- mail Malfunctions
Page 34 GAO- 01- 446 EOP E- mail System
Reference a documents Date Description of event a
Cross- reference to other documents listed in this table a Comments a
E7265 through E7268 04/ 28/ 99 E- mail from Robert Whiteman to Marvin
Miller,
Robert Haas, Yiman Salim and others requesting review of Lotus Notes self
evaluation for 10/ 98 to 04/ 99. It stated that Notes staff began analysis
phase of the Notes/ ARMS interface task. During this analysis, staff
identified a records management problem (Letter D malfunction) and were
researching viable solutions. E1235 and E1236 05/ 17/ 99 Report explaining
e- mail audit agents that run
on Notes servers and search for unrecorded mail and incorrect mail
templates. It recommends running audit agents weekly and providing logs to
Mr. Barry. It also recommends resuming tape recycling. It stated that ?it
would not be possible to recycle any back- up tapes that were created prior
to the (managed date) completion of the correction and verification of the
system??.
Exhibit 47 Exhibit 38 05/ 21/ 99 E- mail from Michael Ritter to Albert
Leister
regaining the Mail2 Statement of Work (SOW).
?This is actually from September ?98. Still no action on? or response from?
the gov?t.? Exhibit 118 05/ 21/ 99 E- mail from Albert Leister Jr. to
Dorothy Cleal
stating that the ?? backup tapes are in the DC [data center].? E0793 05/ 21/
99 E- mail from Marvin Miller showing a schedule
to run various Notes agents. One scheduled for 05/ 25/ 99 was to ?fix letter
?D? and letter case problem. Records will be managed from this point on.?
E4483 06/ 04/ 99 E- mail from Albert Leister to Dorothy Cleal
reporting that the Notes audit agents indicate all documents requiring
records management were managed properly and the agents run weekly ??. to
insure no new errors surface.? The e- mail noted that audit logs were given
to Karl Heissner and requests personal direction to resume tape recycling.
E0798 06/ 09/ 99 E- mail from Yiman Salim stating that ?? the
[Notes/ ARMS interface] views, in the Name and Address Book, need to be
modified to make the selection criteria non- case sensitive.? The Notes/
ARMS interface scanning process had skipped a user account (that was not in
use) because the name started with lower case. ?A
CMC [configuration management change] has been submitted to implement this
modification.?
Appendix II: Chronology of E- mail Malfunctions
Page 35 GAO- 01- 446 EOP E- mail System
Reference a documents Date Description of event a
Cross- reference to other documents listed in this table a Comments a
Exhibit 6 01/ 13/ 00 Briefing materials for 1/ 19/ 00 meeting with the
National Archives and Records Administration regarding e- mail issues. It
mentions Mail2 and Letter D configuration issues. It stated that Mail2
affected 526 users and Letter D affected about 200 users. Exhibit 130 01/
21/ 00 Chronology of Mail2 server events and impact,
labeled ?Tony? in upper right- hand corner. It discusses the problem and
plan of action. It also identified the magnitude of e- mails being
researched as encompassing 850 tapes, and that some tapes contain classified
data. The plan of action says that Enterprise Computing Solutions
Technology, Incorporated (ECS), an 8a firm, has prior experience and was
currently on- site for the Armstrong effort. E4337 through E4338 02/ 01/ 00
E- mail from Daniel Barry to Michael Sullivan
with comments on an attached draft request for proposals (RFP) for NG to bid
on recovery of lost e- mails. Barry comments that a subcontractor to
Information Management Consultants, Incorporated (IMC), had developed the
original Notes/ ARMS interface code in 1996. The e- mail stated that the
Notes/ ARMS interface transitioned to NG in 1997/ 1998. The system broke in
late 1998 because it could not handle increased user population when
migrating from All- in- 1 [the previous e- mail system] to Notes, causing
the Letter D problem. The RFP requests a proposal from NG by 02/ 15/ 00, and
stated that NG caused the problem and that ?? corrective action should be at
no- cost to the Government.? Barry asks if it was ?fair and equitable? to
hold NG accountable for fixing bugs in the software developed by the IMC
subcontractor.
E5978 through E5979
E1393 02/ 25/ 00 Handwritten notes stating that some tapes included in the
tape recovery effort were classified as confidential or secret. E1394 02/
25/ 00 Handwritten notes of an interview about ?Bob?s
- Mail issue,? which state that backup problems were experienced due to
going from a File Allocation Table (FAT) system to a Windows NT File System
(NTFS). An accompanying timeline shows the backup problem existed from the
beginning of the Mail2 problem (10/ 96) until about 11/ 97.
Exhibit 9, E1022, and E6552
Appendix II: Chronology of E- mail Malfunctions
Page 36 GAO- 01- 446 EOP E- mail System
Reference a documents Date Description of event a
Cross- reference to other documents listed in this table a Comments a
E3237 No date Document entitled Tape Reconstruction Procedure Outline stated
that 850 tapes were involved in the recovery effort. E4349 through E4354 02/
29/ 00 Ontrack Computer Evidence Services?
Statement of Work and Proposal for Services
proposes a cost of $436, 000 plus expenses and 55 days to copy and catalog
800 tapes at the EOP, which would require acquisition of 11 computers. It
would cost $348,000 and 55 days to perform the work at Ontrack?s facility,
which was fully equipped. Pricing for data recovery varied depending on four
categories of optional services.
E3247 through E3248 03/ 01/ 00 E- mails between John Spriggs, Albert
Leister,
Terrence Misich, and William Burkey regarding backup tapes. Magnitude of
tapes revised by Burkey to 3,391, including 283 ?miscellaneous
tapes.? Spriggs stated that ?we really do not have any good evidence of what
backup [device] was actually backing up Mail2 on any given date.? Leister
identified three boxes of tapes containing 1,006 8- mm tapes and 51 DLTtapes
as definite to search and identified another three boxes containing 503 8-
mm tapes as ?possible,? for a total of 1,560 tapes. The tapes consist of 4-
mm, 8- mm, and DLTtape media spanning intermittent periods from 1/ 96 to 6/
99. The tapes reportedly contained backup of Office of Management and Budget
(OMB), White House Office, and other EOP agency servers, including Mail2.
Exhibit 46 No date Timeline for ARMS- related activities
(Development Tasks, Problem Discovery and Problem Fixing). It lists the
Mail2 problem as discovered on 6/ 12/ 98 and fixed on 11/ 22/ 98. The Letter
?D? was problem discovered on 4/ 9/ 99 and fixed on 6/ 1/ 99. Exhibit 9 No
date Draft Talking Points on the Mail2 Server
Anomaly stated that incoming e- mails exist on backup tapes, ?? except for a
small window of time from 8/ 96- 11/ 97 when, due to a glitch in the backup
software, the mail files may not have been backed up on a consistent basis.?
E1022, E1394, and E6552
E1040 through E1041 03/ 10/ 00 Memo from Terrence Misich to Jack Young,
General Counsel/ OA. It stated that he ?visually E1015 through E1021
Appendix II: Chronology of E- mail Malfunctions
Page 37 GAO- 01- 446 EOP E- mail System
Reference a documents Date Description of event a
Cross- reference to other documents listed in this table a Comments a
scanned 14 boxes of magnetic media, containing 3,391 magnetic tapes of
various types? and that a number of tapes do not have external labels and
some may have erroneous external labels. Mr. Misich recommends the
performance of the first phase of the Tape Restoration Plan. Exhibit 13 03/
14/ 00 Cover note from Yiman Salim with attached
audit log files from May 8 and 9, 1999, for EOP mail servers Mail2 and MAIL1
showing unrecorded documents. According to the logs, in May 1999, both Mail2
and MAIL1 had four user accounts each with server name syntax errors causing
unrecorded mail dating back to 1997. The logs also show 126 users affected
by the ?Letter D? malfunction (70 on Mail2 and 56 on MAIL1).
Exhibits 37, 44, 71, 79 and 125. Logs show that about 5
months after the Mail2 fix was implemented, Mail2 still had four user
accounts with syntax errors and unrecorded mail dating back to 1997- 1998.
Also, MAIL1 had four user accounts with syntax errors in this same time
frame. E1015 through E1021 03/ 15/ 00 A document entitled Draft Tape
Restoration
Plan revised to include new data from ECS. It stated that 3,391 tapes were
involved in the recovery effort and attributes the Letter D malfunction to a
?? software failure.? It also attributes the Mail2 malfunction to case
sensitivity. It stated that ?work associated with this data recovery began
almost as soon as the problem was discovered.? The draft plan mentioned that
the work was not begun in October 1998 when the SOW was developed for NG
because of Year 2000 efforts, so the recovery was postponed to March 2000.
In February 2000, OA ?? began exploring alternatives,? and received an
estimate from one company of $436,000 and 55 days to extract mail from 850
tapes. The draft plan extended these estimates to $1, 744, 000 and 220 days
for 3,391 tapes. The draft plan also stated that another vendor, familiar
with ARMS, estimated $3, 286, 596 and 1.7 years.
Exhibit 72, E1363 through E1375, and E0776
E1022 No date Hand- drawn timeline of e- mail malfunctions. Shows October
1996 to October 1997 ?backup
tapes missing, FAT to NSF file... Fixed.? August 1998 ?ARMS research to fix
Mail2 - discover they cannot.? November 20, 1998, Mail2 stop bleed, tape
recycling starts November 21. April 9, 1999, D problem identified, stopped
tape recycle. May 21, 1999, ?stopped D- bleed?.
E1394, E6552, and Exhibit 9 In its comments on a draft
of this report, EOP stated that ??. tapes were not recycled from June of
1998 until June of 1999.?
Appendix II: Chronology of E- mail Malfunctions
Page 38 GAO- 01- 446 EOP E- mail System
Reference a documents Date Description of event a
Cross- reference to other documents listed in this table a Comments a
Exhibit 57 03/ 17/ 00 Fax from Beth Nolan, White House Counsel, to The
Honorable Dan Burton, stated, ?In January 1998, Daniel Barry.? noticed a
possible anomaly within ARMS... that some incoming emails might be missing.?
Mr. Barry notified his superiors and documented his finding... The full
extent of the error causing the anomaly Mr. Barry noted was not discovered
until June 1998, when NG employees discovered?. certain incoming e- mail
messages that were coded as
?unrecorded?.? The contractor notified IS& T personnel?. By the fall of
1998, NG technical personnel working with IS& T staff discovered that the
problem was due to miscoding ?Mail2?
as ?MAIL2. ? They further determined that the miscoding affected 526 ARMS-
managed accounts.... By November 1998, the NG and IS& T personnel had
corrected the problem....? prospectively
?... incoming e- mail to ARMS- managed accounts with the first names
beginning with the letter ?D? had not been recorded by ARMS since November
1998. It appears that this error remained undetected until April 1999....?
Approximately 200 accounts within EOP were affected.
?In the course of gathering these preliminary facts concerning these
configuration errors, we were informed this week that e- mails on the server
of the Office of the Vice President (OVP) have not been fully managed by
ARMS.?
Exhibit 50 a References are to the House Committee on Government Reform
exhibits used at the March 2000 hearings; additional documents provided to
the Committee are identified by an ?Exxxx? scheme. Bracketed items within
the ?Description of Event? column and the ?Comments? column have been added
by us for readability and clarification.
Appendix III: Key Officials Involved in Primary Mail2 and Letter D Events
Page 39 GAO- 01- 446 EOP E- mail System
Virginia Apuzzo, former Assistant to the President for Management and
Administration
Daniel A. ?Tony? Barry, Computer Specialist, Office of Administration (OA)/
Information Systems and Technology Division (IS& T)
Mark Bartholomew, Web Development, OA/ IS& T Paulette Cichon, former Deputy
Director, Information Management, OA Dorothy Cleal, former Associate
Director, OA/ IS& T Laura Crabtree, former Desktop Systems Branch Chief, OA/
IS& T Nellie Doering, Records Management, OA/ IS& T Kathleen Gallant, former
Associate Director, OA/ IS& T Sheryl Hall, former Computer Specialist, OA/
IS& T Karl Heissner, Systems Integration and Development Branch Chief, OA/
IS& T
Joseph Kouba, Budget Analyst, Financial Management Division Mark Lindsay,
Assistant to the President for Management and Administration, former General
Counsel, OA, and former Director, OA
Michael Lyle, Director, OA, and former General Counsel, OA Cheryl Mills,
former Deputy Counsel to the President Terrence Misich, Chief Warrant
Officer 5, U. S. Army, on detail from White House Communications Agency to
OA/ IS& T, Tape Restoration Project Manager
Beth Nolan, Counsel to the President Bruce Overton, Deputy General Counsel,
OA DeVere Patton, Contracting Officer?s Technical Representative (COTR) for
the Northrop Grumman Contract Appendix III: Key Officials Involved in
Primary Mail2 and Letter D Events Executive Office of the President Federal
Staff, October 1996 Through June 1999:
Appendix III: Key Officials Involved in Primary Mail2 and Letter D Events
Page 40 GAO- 01- 446 EOP E- mail System
John Podesta, Chief of Staff Ada Posey, former Director, OA Charles Ruff,
former Counsel to the President Michael Sullivan, Deputy Associate Director
for General Services Division, OA
Christina VanFossan, Director, Financial Management Division James Wright,
former COTR for the Northrop Grumman contract, and Data Center Branch Chief,
OA/ IS& T
Sandra Golas, VAX System Administrator Robert Haas, Lotus Notes E- mail
System Administrator, Lotus Notes team Steve Hawkins, former Project Manager
Betty Lambuth, CEXEC (a subcontractor to Northrop Grumman), former Lotus
Notes team manager
Joseph Lucente, Director of Contracts and Subcontracts Yiman Salim, Lotus
Notes Developer, Lotus Notes team John Spriggs, Senior Engineer Joseph
Vasta, former Program Manager Northrop Grumman
Contract Staff From Contract Award in October 1997 Through June 1999:
Appendix IV: Time Line of Key Events in the Mail2 and Letter D Malfunctions
Page 41 GAO- 01- 446 EOP E- mail System
Source: GAO compilation of documents provided by EOP to the House Committee
on Government Reform.
Appendix IV: Time Line of Key Events in the Mail2 and Letter D Malfunctions
Appendix V: Itemization of Budgeted Cost for the Tape Restoration Project
Page 42 GAO- 01- 446 EOP E- mail System
Cost element Description Estimated cost
ECS contract: Restoration of e- mail messages not recorded in ARMS Phase I
Task 1: Develop a detailed project plan $ 836,600
Task 2: Define, configure, acquire, and install required equipment 862,300
Task 3: Develop a quality assurance and quality control program for the
project 217,600 Task 4: Develop, test, and refine tape recovery software
742,600 Task 5: Test and validate recovery plan and software 128,400 Phase
II Task 6: Receive and inventory all tapes 118,000
Task 7: Make two copies of each tape 364,000 Task 8: Place original tapes in
secure custody 31,000 Task 9: Scan duplicated tapes into message recovery
system 337,000 Phase III Task 10: Process e- mail into searchable database
92,400
Task 11: Remove duplicate e- mail messages and prepare for transfer to ARMS
93,000 Task 12: Place duplicate e- mail messages into a searchable database
105,400 Task 13: Process classified tapes as above 208,000 Task 14: Deliver
final products 80,000 General and administrative expenses and other direct
costs 909,529
ECS total: $5,125,829
Vistronix contract: Independent validation and verification of tape
restoration Phase I Tasks 1 through 5 (see above) $ 465,483 Phase II Tasks 6
through 9 (see above) 1,078,233 Phase III Tasks 10 through 14 (see above)
450,103
Vistronix total: $1,993,819
Other costs: Other restoration costs AAC Associates contract Advice on
recovery of OVP e- mails $900,000 Vendor not determined Forensic analysis
and recovery of broken or unreadable tapes 2,092,400 Vendor not specified
Additional hardware, software licenses, and 12,000 tapes for two sets of
tape copies 498,480 Not applicable Project overrun contingency 1,098,891
Other costs totals $4,589,771 Total $11,709,419
Source: Tape restoration project cost estimate report prepared by Vistronix,
Incorporated, for the EOP in July 2000.
Appendix V: Itemization of Budgeted Cost for the Tape Restoration Project
Appendix VI: Comments From the Office of the Former Vice President
Page 43 GAO- 01- 446 EOP E- mail System
Appendix VI: Comments From the Office of the Former Vice President
Note: GAO comments supplementing those in the report text appear at the end
of this appendix.
Appendix VI: Comments From the Office of the Former Vice President
Page 44 GAO- 01- 446 EOP E- mail System
Now on p. 2. See comment 3. Now on p. 1.
See comment 1. Now on p. 2. See comment 4.
Now on p. 2. See comment 5. Now on p. 1.
See comment 2. Now on p. 2. See comment 6.
Appendix VI: Comments From the Office of the Former Vice President
Page 45 GAO- 01- 446 EOP E- mail System
Now on p. 10. See comment 16. Now on p. 7.
See comment 13. Now on p. 5.
See comment 11. Now on p. 4.
See comment 10. See comment 9.
Now on pp. 2 to 3. See comment 8.
Now on p. 10. See comment 15.
Now on p. 2. See comment 7.
Now on p. 9. See comment 14.
Now on pp. 5 to 6. See comment 12.
Appendix VI: Comments From the Office of the Former Vice President
Page 46 GAO- 01- 446 EOP E- mail System
Now on p. 16. See comment 25. Now on p. 14.
See comment 24. Now on p. 11.
See comment 28. Now on p. 11.
See comment 22. Now on p. 10.
See comment 21. Now on p. 10.
See comment 20. Now on p. 10.
See comment 19. Now on p. 10.
See comment 18. Now on p. 10.
See comment 17.
Appendix VI: Comments From the Office of the Former Vice President
Page 47 GAO- 01- 446 EOP E- mail System
Now on p. 16. See comment 26.
Appendix VI: Comments From the Office of the Former Vice President
Page 48 GAO- 01- 446 EOP E- mail System
The following are GAO?s comments on the office of the former Vice
President?s letter, dated April 11, 2001.
1. We revised the report to include the word ?certain,? but did not
incorporate the remaining text as suggested by the Office of the former Vice
President. The ?certain? records that the report states are
?deemed official government records? are those incoming e- mail messages not
captured by the Automated Records Management System (ARMS) due to the
malfunctions. These records consisted of both presidential/ vice
presidential and federal records.
2. We revised the report to reflect the technical clarification provided. 3.
We submitted written questions to the Office of the Vice President
(OVP), through the Executive Office of the President (EOP), on September 14,
2000, in which we asked if, and how, the OVP implemented the portion of the
1997 EOP policy which stated that staff
?[ m] aintain Presidential Records in organized files.? In its November 14
response, the OVP stated that OVP staff implemented the requirements of the
statute by maintaining presidential records ?in organized files.? In follow-
up, we submitted another set of written questions to the OVP, through the
EOP, on November 30. At this time, our request was to
?[ p] lease provide a list of all OVP staff who maintain the organized files
of hard copy OVP records as mentioned in the EOP response. We would like to
physically observe the files of randomly selected OVP staff and would like
to arrange for this observation during the week of December 11, 2000.? Our
purpose in reviewing the files of selected staff was to respond to the House
Committee on Government Reform?s request that we determine whether OVP
implemented adequate practices in accordance with applicable criteria for
management of email records. The intent of our request was to confirm the
existence of e- mail records in such files and not to examine the text of
the records contained therein, as explained verbally to the EOP Office of
Administration General Counsel on numerous occasions during December 2000
and January 2001. We were not provided a contact within OVP with whom to
discuss the request. The OVP did not respond to our request in writing until
6: 49 p. m. on January 19, 2001, at which time OVP?s response was that the
OVP did not maintain a centralized filing system. In its response, OVP may
have confused the word ?organized? in our request with ?centralized.?
Several hours after we were provided OVP?s response, the administration had
transitioned GAO Comments
Appendix VI: Comments From the Office of the Former Vice President
Page 49 GAO- 01- 446 EOP E- mail System
and the records were in the process of transfer to the National Archives and
Records Administration.
4. A key feature of an electronic recordkeeping system is that records be
retrievable. To facilitate retrieval of records, the recordkeeping system
must index records and contain fields that can be searched.
5. Best practices for management of records currently exist in the form of
National Archives and Records Administration guidance contained in title 36
of the Code of Federal Regulations (CFR). Even though the majority of this
guidance is prescriptive only for federal records, it does not explicitly
preclude application of these practices in the management of presidential/
vice presidential records. Although the evolution of ARMS continues, the
stability of generally accepted records management best practices
establishes a control foundation for any system, whether manual or
automated.
6. We revised the report to identify ?another EOP component? as EOP?s Office
of Administration.
7. We revised the report to reflect the technical clarification provided. 8.
We revised the report to reflect the technical clarification provided. 9.
The report was revised to reflect that external authorities other than
those of law enforcement scrutinized the e- mail malfunctions. 10. We
revised the report to incorporate the technical clarification
provided. 11. Title 36 CFR, subpart 1234.24, Standards for managing
electronic mail
records, promulgates this standard (36 CFR 1234.24( c)). 12. The report
provides a description of common federal standards for
management and control of federal programs and systems. We did not imply
that the application of such controls alters in any way the determination of
whether a record is presidential/ vice presidential or federal. The
inclusion of this discussion was to emphasize that federal programs, such as
records management programs, and systems, such as ARMS and the e- mail
system, are subject to application of certain management controls. These
controls ensure the effectiveness of programs as well as ensure that
adequate integrity, confidentiality, and availability of electronic
information resources are maintained. Both
Appendix VI: Comments From the Office of the Former Vice President
Page 50 GAO- 01- 446 EOP E- mail System
the Presidential Records Act (PRA) and the Federal Records Act (FRA) require
the implementation of management controls.
13. The new information provided in the Office of the former Vice
President?s comments on the draft has been incorporated into the report.
14. We revised the report to reflect the technical clarification provided.
15. Under the PRA, the Vice President (and thus OVP), was responsible for
management of the Vice President?s records. From the beginning of the
administration, it retained the primary duty to implement records management
controls, and thus, to either modify its records management practice to
accommodate the system transition or to ensure that this responsibility was
assigned to another entity.
16. We attribute this statement to EOP because it was taken from the White
House memo to us on January 19, 2001, which stated that it provided EOP?s
responses to our questions submitted on November 30, 2000.
17. The new information provided by the Office of the former Vice President
has been incorporated into the report.
18. The new information provided by the Office of the former Vice President
has been incorporated into the report.
19. See our response to comment 3 above. 20. In its January 19, 2001,
response to our written questions, the EOP
stated that at some point between issuance of the May 1993 EOP policy and
1998, OVP ceased requiring the retention of paper copies of e- mail records.
We were provided no evidence, or allowed to physically confirm, that some
printed e- mail records were retained. Also, the EOP policy did not
explicitly require that records be captured by ARMS, as indicated by the
comment.
21. We revised the report to reflect the technical clarification provided.
22. As discussed in our response to comment 15 above, the OVP had a
duty to ensure continual preservation of its vice presidential records,
including ensuring that this responsibility effectively transferred to the
Office of Administration.
Appendix VI: Comments From the Office of the Former Vice President
Page 51 GAO- 01- 446 EOP E- mail System
23. We revised the report to reflect the technical clarification provided.
24. The characteristics described in the report are based on National
Archives and Records Administration standards as presented in title 36 CFR.
Creation of records is the manual or automated process during which a
document is produced and is determined to qualify as a record.
Identification is a part of this process in which there is recognition that
a document is a record and a determination is made as to the type of record.
The characteristics described pertain to the monitoring of both
presidential/ vice presidential and federal records.
25. We agree with the Office of the former Vice President?s assessment;
however, the only rationale that was provided to us, either orally or in
writing, was that the letters were ?deserving of respect and deference.? We
revised the report to incorporate the additional rationale provided in its
comments to the draft report.
26. The Committee requested that we assess the adequacy of management
controls over EOP?s records management system, particularly as they relate
to the numerous e- mail system malfunctions experienced by EOP since 1996.
That assessment entailed the examination of management policies, programs,
and practices as they related to effective records management controls and
to the requirements of the PRA and FRA. Our review was not directed at
assessing the intent of individuals responsible for preserving records and
providing responsive records to investigative bodies. As such, we can offer
no conclusions on this matter.
(310307)
The first copy of each GAO report is free. Additional copies of reports are
$2 each. A check or money order should be made out to the Superintendent of
Documents. VISA and MasterCard credit cards are also accepted.
Orders for 100 or more copies to be mailed to a single address are
discounted 25 percent.
Orders by mail:
U. S. General Accounting Office P. O. Box 37050 Washington, DC 20013
Orders by visiting:
Room 1100 700 4 th St., NW (corner of 4 th and G Sts. NW) Washington, DC
20013
Orders by phone:
(202) 512- 6000 fax: (202) 512- 6061 TDD (202) 512- 2537
Each day, GAO issues a list of newly available reports and testimony. To
receive facsimile copies of the daily list or any list from the past 30
days, please call (202) 512- 6000 using a touchtone phone. A recorded menu
will provide information on how to obtain these lists.
Orders by Internet
For information on how to access GAO reports on the Internet, send an email
message with ?info? in the body to:
Info@ www. gao. gov or visit GAO?s World Wide Web home page at: http:// www.
gao. gov
Contact one:
Web site: http:// www. gao. gov/ fraudnet/ fraudnet. htm
E- mail: fraudnet@ gao. gov
1- 800- 424- 5454 (automated answering system) Ordering Information
To Report Fraud, Waste, and Abuse in Federal Programs
*** End of document ***