[Report on the Assessment of Electronic Government Information Products]
[Report ]
[Entire Report (including Appendices)]
[From the U.S. Government Printing Office, www.gpo.gov]

REPORT ON THE

ASSESSMENT OF ELECTRONIC GOVERNMENT INFORMATION PRODUCTS

Prepared under contract 
(#RN 97007001)

by

Westat
Rockville, Maryland

for the

UNITED STATES NATIONAL COMMISSION ON 
LIBRARIES AND INFORMATION SCIENCE

commissioned by the

UNITED STATES GOVERNMENT PRINTING OFFICE
SUPERINTENDENT OF DOCUMENTS

March 30, 1999


Mr. Michael F. DiMario
The Public Printer
The Government Printing Office
North Capitol and H Sts. NW
Washington, D.C. 20401

Dear Mr. DiMario,

It is with great pleasure that I forward herewith a copy of the Final Report
prepared by Westat, Inc., the contractor selected by the Government to undertake
Phase II of the three-part study called "Assessment of Electronic Government
Information Products."  As you requested, the U.S. National Commission on
Libraries and Information Science (NCLIS) planned and implemented this research
survey, pursuant to an interagency agreement between NCLIS and the Government
Printing Office (GPO), approved by the Joint Committee on Printing (JCP).

This report follows on the process begun with the congressional requirement,
contained in the Senate Report on H.R. 1854, the FY 1996 Legislative Branch
Appropriations Act (P.L. 104-53), to identify the measures necessary for a
successful transition to a more electronic Federal Depository Library Program. 
That requirement resulted in a study published by the Government Printing Office
in June 1996.  There was a consensus, however, that additional work was required
(1) to identify the electronic formats and mediums used and/or planned by
Federal publishing entities, and (2) to determine whether public or private
sector standards do, or could, play a stronger role in reducing the
unnecessary proliferation of these formats and mediums.  These questions
precipitated this survey.

I am extremely pleased to note that the survey enjoyed the active support and
participation of all three branches of Government.  Twenty-four different
Federal entities participated, including the Supreme Court, several committees
of the Congress, one regulatory commission and 19 Executive Branch agencies,
including most of the Cabinet Departments.  In addition to this broad and
diverse Federal involvement in the survey, an impressive 74 percent of the
survey forms sent to the agencies were returned completed.  I believe this level
of interest and support is highly unusual, and could, perhaps, be construed as a
reflection of agency desires to help establish a systematic baseline for
measuring and monitoring the rapidly changing and evolving kinds and mix of
preferred mediums, formats, and standards.

Our representatives and your staff have been in close, harmonious contact from
the earliest stages of planning for the survey, right up until the final stages
of review of the final report.  I want to take this opportunity to thank
especially both the former and present Superintendents of Documents, as well as
the staffs of the present and former directors of the Library Programs Service,
and the Office of Electronic Information Dissemination Service, for the superb
support NCLIS and the contractor received throughout the process.

I also want to recognize the key role played by Forest Woody Horton, Jr.  As
consultant to NCLIS, Woody's broad knowledge of how Government works and his
deep understanding of Information Resources Management helped to move the study
along most effectively.

Finally, I would like to recognize the support of Vice-Chair Martha B. Gould,
Commissioners C.E. ("Abe") Abramson who chairs the NCLIS Access to Government
Information Committee, Joan R. Challinor, and Josï¿½-Marie Griffiths, all of whom
have been staunch advocates throughout.  I believe you are also aware of the
strong interest and support NCLIS Executive Director Robert S. Willard
personally accorded this study, beginning very early with his tenure as a
commissioner and extending to the present day.

The long review and analysis process of the contractor's statistical
tabulations, findings, and observations has just begun.  This demanding process
will take some time, in part because the number of interested communities is so
large, and in part because the subject matter is so technical, involving the
full range of information handling formats, mediums, and standards, and quite
diverse agency plans and practices.  Ultimately, actions needed to be taken will
most likely involve new or strengthened policies, rules, and regulations, as
well as the adoption of technical standards, some of which could have
legislative ramifications.

It is now the Commission's intention to begin Phase III.  We will take the now
completed Phase II Westat report, as well as the Phase I report completed in
1997 by the National Academy of Sciences, as points of departure.  They will be
reviewed and we will determine if additional fact gathering is required.  We can
then move forward to draw conclusions and make recommendations to the Congress
and the President from the multitude of facts and expert opinions received thus
far.

NCLIS will continue to consult with GPO, along with various knowledgeable
individuals, interagency and special advisory groups, all of whom have been
assisting us throughout the Phase I and II efforts, as we prepare a plan for the
Phase III initiative.  My hope is that we will keep most of the broader advisory
team we have utilized thus far in place until we have completed Phase III.

Finally, I want to thank you for your personal leadership, without which we
could have never moved ahead with this complex, landmark task.

Sincerely yours,

Jeanne Hurley Simon
Chairperson




Acknowledgments


The study was performed by Westat under contract with the U.S. National
Commission on Libraries and Information Science (NCLIS). The Government Printing
Office commissioned this study as part of the transition to a more electronic
FDLP.  Westat's Task Leader was Denise Glover, and the project staff included
Sarah Bennett-Harper, Debbie Alexander, and Ethel Sanniez.  Libby Farris served
as Westat's Project Director.  Forest Woody Horton, Jr., a consultant to NCLIS,
served as the Project Director.  Francis J. Buckley, Jr., the Superintendent of
Documents, Gil Baldwin, and T.C. Evans served as the key Government Printing
Office (GPO) liaison officials throughout all phases of study design,
implementation, and evaluation.  Wayne P. Kelley, former Superintendent of
Documents, and James D. Young, former Director of GPO's Library Programs
Service, were instrumental in recognizing the need for this study and in shaping
its direction.  Robert S. Willard, NCLIS Executive Director, and Judy Russell,
NCLIS Deputy Director, provided strong overall guidance and supervision.

Westat also wishes to thank the Depository Library Council, NCLIS Chairperson
Jeanne Hurley Simon, and Vice-Chair Martha B. Gould, and the NCLIS Committee on
Access to Public Information, chaired by Commissioner C. E. ("Abe") Abramson,
for their interest and support.  In addition, Westat expresses its deep
appreciation to the 24 participating Federal agencies, especially the Chief
Information Officers, the agency coordinators, the product respondents, and key
officials in the agency library, printing and publishing, information
technology, public affairs, and other functional offices.  Finally, Westat
expresses appreciation to the various experts interviewed and the depository
libraries visited, all of which are listed in appropriate appendices.




Table of Contents

Section											Page

Acknowledgments										i
Executive Summary										xi

1.	Introduction and Background							1
	The Federal Depository Library Program					1
	How the Federal Depository Library Program Works			2
	Background of the Study								2
	Project Phases									3
	Study Goals and Objectives (Phase II)					4
	Scope and Organization of the Report					5

2	Methodology										7
	Product Selection									7
	Coordinator Briefings								8
	Questionnaire Design								9
	Distribution of the Questionnaires						9
	Followup for Nonresponse, Data Retrieval, and Inconsistency		10
	Methodology for Qualitative Data Collection				11
	Site Visits to Depository Libraries						11
	Purpose and Procedures for Agency Meetings				12
	Expert Interviews									14

3	Survey Analysis and Findings							17
	Structure of the Questionnaire						17
	Section A Responses								18
	Section B Responses								19
		Types of Data Contained in Product					19
		Types of Mediums Used							20
		Format Types Used								23
		User Interfaces								28
		Searchability of Product						30
		Product Host								31
		Retrievability of Product						32
	Section C Responses (Planned Product Profile)				33
		Types of Data								33
		Types of Mediums								34
		Types of Formats								36
	Section D Responses (Other Information)					37
		Metadata									37
		Permanent Public Access							38
		Permanent Retention							39
		Ensuring Authenticity							40
		Updating/Refreshing Plans						41
		Changing Supporting Technology					42
		User Fees									43
		Licensing									43
		Public Domain								44
	Section E Responses								45
Study Questions										46
	Preferred Medium and Format Standards					46
	Public Access to Products							50
	Other Issues:  Authenticity and Metadata					53

4	Qualitative Findings								55
	Site Visits to Federal Depository Libraries				55
		Highlights of Site Visits to Three Depository
		Libraries									55
			User Needs and Concerns						55
			Librarians' Concerns:  User Fees, Hardware,
			Training, and Costs						56
	Agency Meetings									56
		Agency Meeting Highlights						57
			Preferred Mediums and Formats					57
			Assessing User Needs						58
			Information Life Cycle Management, Permanent
			PublicAccess, and Permanent Retention			58
			Cost-Effectiveness of Various Mediums and
			Formats								58
	Expert Interviews									58
		Interviews With Webmasters						59
			Preferred Formats							59
			User Needs								59
		Interviews With Preservation Specialists				59
			Goals of Preservation						59
			Barriers to Preservation of Digital Materials		60
			Current Preservation Models and Initiatives		60
			CLIR Initiatives							60
		Interviews With Information Resources Management
		Specialists									61
			Barriers to Successful Implementation of
			Information Resources Management Initiatives		61

5	Discussion of Quantitative and Qualitative Findings			63
	Preferred Mediums and Format Standards					63
	Evaluating Websites								65
	Cost-Effectiveness of Formats and Mediums					65
	Depository Library Needs							66
	Public Access									67
		Permanent Public Access to and Permanent Retention of
		Electronic Government Information					67
		Perspectives on Permanent Public Access and Information
		Life Cycle Management from Information Resources
		Management Experts							68
		Current Initiatives on Permanent Public Access and
		Permanent Retention							69

	Next Steps										71

	Bibliography									73




List of Appendixes

Appendix	

A	Agency Study Coordinator Meetings Agenda					A-1

B	List of Agency Coordinators and Other Key Officials			B-1

C	List of Participating Agencies and Products Surveyed			C-1

D	Coordinator and Respondent Cover Letters					D-1

E	Questionnaire and Glossary of Terms						E-1

F	Site Visits to Three Federal Depository Libraries and
	Interview Questions								F-1

G	Electronic Government Information Products Assessment
	Agency Meetings Held and Discussion Questions				G-1

H	Assessment of Electronic Government Information Products
	List of Expert Interviews and Interview Questions			H-1

I	Sample Agency Meeting Agenda Electronic Government Information
	Products Assessment								I-1

J	Task 16-Assessment of Electronic Government Information
	Products-Statement of Work							J-1




List of Tables

Table												Page

1	Number of surveys returned by each agency surveyed			18
2a	Number and percent of types of data, by the type of data
	contained										20
2b	Number and percent of types of data, by the primary type
	of data 										20
3a	Number and percent of mediums publicly available, by the
	type of medium used and primary medium used				22
3b	Number and percent of mediums publicly available, by the
standard for each medium used							23
4a	Frequency and percent of formats used, by the type of format
	used and primary type of format used					25
4b	Number and percent of formats used, by the standard for each
	format used										26
5	Number and percent of products reported as being in an online
	medium										28
6a	Number and percent of online approaches used, by type of
	online tool used									29
6b	Number and percent of online approaches used, by the standard
	for each online tool used							30
7	Number and percent of responses regarding searchability of
	the product										31
8	Number and percent of responses regarding where the product
	is hosted										31
9	Number and percent of responses concerning the retrievability
	status of the product								32
10	Number and percent of respondents reporting plans to
	discontinue publication of the product					33
11	Number and percent of responses regarding the planned changes
to the type of data contained in the product				34
12	Number and percent of responses regarding the timeframe for
	planned changes to the type of data contained in the product	34
13	Number and percent of responses regarding the planned changes
	to the mediums used for the future						35
14	Number and percent of responses regarding the timeframe for
	planned changes to product medium used					35
15	Number and percent of responses regarding the planned changes
	to the  formats the product will contain					36
16	Number and percent of responses regarding the timeframe for
	planned changes to the product format used				36
17	Number and percent of respondents reporting a metadata record
	for the product									37
18	Number and percent of responses regarding the entity providing
	permanent access to the product						39
19	Number and percent of responses regarding products for which
	access will be provided in the future					39
20	Number and percent of responses regarding permanent retention
	of the product									40
21	Number and percent of respondents who reported the agency
	ensures authenticity for the product					41
22	Number and percent of responses regarding how frequently the
	product is updated or refreshed						42
23	Number and percent of responses regarding the plans for
	supporting technology of the product					42
24	Number and percent of respondents reporting that user fees are
	charged for the product								43
25	Number and percent of respondents reporting about the use of
	licensed commercial search and retrieval software for the
	product										44
26	Number and percent of responses regarding coverage by the
	agency software license								44
27	Number and percent of respondents reporting the public
	domain status of the product							45
28	Number and percent crosstabulations of products in both paper
	and CD-ROM formats								47
29	Number and percent crosstabulations of products in both CD-ROM
	and web formats									47
30	Number and percent crosstabulations of products in both paper
	and web formats									48
31	Number and percent crosstabulations of products in both HTML and
	PDF formats										48
32	Number and percent crosstabulations of products in both HTML and
	GIF formats										49
33	Number and percent of products that use HTML with GIF and ASCII
	formats										49
34	Number and percent of products that use HTML with PDF and ASCII
	formats										49
35	Number and percent crosstabulations of products that are
	permanently public accessible and scheduled for retention with
	the National Archives and Records Administration (NARA)		50
36	Number and percent crosstabulations for products with licensed
	commercial search and retrieval software and user fees charged
	for the product									51
37	Number and percent crosstabulations of those products with
	licensed commercial search and retrieval software and the
	product is scheduled for permanent retention by the National
	Archives and Records Administration (NARA)				52
38	Number and percent crosstabulations of those products for which
	agencies ensure authenticity and permanent public access		53
39	Number and percent crosstabulations of those products for which
	agencies ensure authenticity and another agency provides
	permanent public access								53
40	Number and percent crosstabulations of products that are
	hosted by the agency and have a metadata record				54
41	Number and percent crosstabulations of products that are
	hosted by another agency and have a metadata record			54




REPORT ON THE
ASSESSMENT OF ELECTRONIC GOVERNMENT INFORMATION PRODUCTS

Prepared under contract (#RN 97007001)
by
Westat
Rockville, Maryland

for the

UNITED STATES NATIONAL COMMISSION ON LIBRARIES AND INFORMATION SCIENCE

commissioned by the

UNITED STATES GOVERNMENT PRINTING OFFICE
SUPERINTENDENT OF DOCUMENTS

March 30, 1999

Executive Summary


The Federal Depository Library Program (FDLP) has served and continues to serve
the American public by ensuring localized access to Federal Government
information.  The mission continues to be as important today to the fundamental
success of our democracy as it was when the FDLP was created.  The FDLP's
original mandate, to assist Americans regardless of economic, education, or
geographic considerations, is one that must not be lost as we strategically and
thoughtfully use the tools of the electronic age to enhance that mandate.

Letter to Michael F. DiMario, the Public Printer, from Senators John Warner and
Wendell Ford of the Senate Committee on Rules and Administration, May 24, 1996.



Background


Congress established the antecedents to the Federal Depository Library Program
(FDLP) in the Act of 1813 to ensure that the American public has access to its
Government's information.  The mission of the FDLP, part of the Superintendent
of Documents (SuDocs) in the Government Printing Office (GPO), is to assure
current and permanent public access to the universe of information published by
the U.S. Government.  Depository libraries safeguard the public's right to know
by collecting, organizing, maintaining, preserving, and assisting users with
information from the Federal Government.  GPO provides that information at no
cost to designated depository libraries throughout the country.  These
depository libraries, in turn, provide local, no-fee access to Government
information in all formats in an impartial environment with professional
assistance.  Any member of the public can visit these depository libraries and
use the Federal depository collections.

In order to administer the FDLP, as required by the enabling legislation for the
program, 44 U.S.C. Chapter 19, the SuDocs is responsible for the acquisition,
classification, format conversion, dissemination, and bibliographic control of
tangible and electronic Government information products; the inspection of
depository libraries; and the continuing education and training initiatives that
strengthen the ability of depository library personnel to serve the public.  An
emerging new responsibility is to ensure that electronic Government information
products disseminated through the FDLP, or incorporated in the FDLP Electronic
Collection, remain permanently accessible to the public.  Under 44 U.S.C., 
Sections 1901-1903, and Office of Management and Budget (OMB) Circular A-130, 
Management of Federal Information Resources, Federal agencies should make all
their publications in all formats available to SuDocs for distribution to
depository libraries.


This study to assess electronic medium and format standards for the creation and
dissemination of electronic information products is an essential step toward
ensuring a successful and cost-effective transition to a more electronic FDLP.
The three goals of this assessment were to:

   > Identify medium and format standards that are the most appropriate for
     permanent public access;

   > Assess the cost-effectiveness and usefulness of various alternative medium
     and format standards; and

   > Identify public and private medium and format standards that are, or could
     be used for products throughout their entire information life cycle, not
     just at the dissemination or permanent public access stage.

The Superintendent of Documents will use the results of this work effort to
continue to plan and implement the transition to a more electronic FDLP.  The
five major specific objectives are:

   > First, with respect to electronic publishing practices and plans for
     Federal agencies (including ways in which the FDLP can best accommodate
     them), the objective is to provide an analysis of current practices as well
     as future plans for creating, disseminating, and providing permanent public
     accessibility to electronic information products, and to identify the
     standards for software and electronic mediums and formats that are used
     throughout the product's information life cycle, from creation to archiving
     but especially at the stage of dissemination for permanent public access.

   > Second, with respect to cost-effectiveness of various dissemination mediums
     and formats that are, or could be utilized, the objective is to gather
     information on standards (whether mandated or consensual) that will assist
     the FDLP in making near-term decisions regarding the cost-effectiveness of
     alternative mediums and formats for all FDLP participants.  This
     information should also assist participants in long-term planning for
     permanent public accessibility, and the collection and analysis of overall
     information life cycle costs.

   > Third, with respect to the practical utility of various electronic mediums
     and formats to depository libraries and the public, the objective is to
     identify preferred standards used in various mediums and formats that
     depository libraries will need to support.

   > Fourth, with respect to utilizing standards employed in mediums and formats
     that can be used throughout all stages of the information life cycle
     (including creation, composition, computer terminal display, encryption, 
     secure digital signature with non-repudiation, and secure transmission
     capabilities), for electronic dissemination, but especially permanent
     public accessibility, the objective is to assess standards for basic
     security services in order to provide for secure and reliable transmission
     and document interchange.

   > Fifth, with respect to standards that are being developed and used in the
     private sector, the objective is to identify existing and planned standards
     for the purpose of determining what the FDLP must do to accommodate their
     adoption in terms of hardware/software requirements, staff and user
     education and training, and budgetary impacts.

Methodology


The study utilized both quantitative and qualitative data collection activities: 
a survey of a cross-section of 314 Government information products from 24
agencies and interviews with experts.  The response rate for the survey was 74
percent.  This cross-section of products was not a randomly selected sample due
to cost and time constraints.  Instead, NCLIS and GPO-assisted by various
groups, including the library associations represented by the Inter-Association
Working Group on Government Information Policy (IAWG), the Federal Library and
Information Center Committee (FLICC), the Depository Library Council (DLC), and
the Interagency Council on Printing and Publication Services (ICPPS)- developed
and refined the criteria for product selection.  NCLIS, GPO, and the other
organizations asked knowledgeable members of these groups to identify products
that met one or more of six criteria.

NCLIS distributed the list of preliminary products to agency Chief Information
Officers (CIOs) who were asked to validate and coordinate the final selections
with their appropriate agency personnel.  In addition, NCLIS asked CIOs to
select an agency coordinator.  The coordinator's role was to oversee the
distribution of product questionnaires to the appropriate respondents and to
encourage respondents to complete the questionnaire and return it to Westat.

Product selection was based on six criteria:

   > Increased emphasis on electronic dissemination, rather than continuation of
     paper and microform dissemination;
   > Replacement of older electronic mediums and formats with state-of-the-art
     technologies;
   > Adoption of mandated (Government or private sector) and consensual (common
     agency practice) medium and format standards;
   > Adoption and use of preferred mediums or formats that have widespread
     support from agency, depository library, and user communities;
   > Exemplified cost-effective mediums and standards, especially those that can
     be used throughout the entire information life cycle, rather than the use
     of expensive customized or shelf packages; and
   > Exemplified awareness of the important impact of medium and format
     decisions on permanent accessibility, authentication, and/or security
     encryption protection.

The survey requested information on four main topics:

   > General information about the product and agency that produced it.
   > The product's current profile including the kinds of data the product
     contains, mediums in which it is produced, formats and online approaches
     used (if applicable); and searchability and retrievability of the product.
   > Future plans for the product including changes in its data, mediums, and
     formats.
   > Other issues including metadata, permanent public access, permanent
     retention, authenticity, updating/upgrading plans, user fees, licensing, 
     and public domain. 

The qualitative data collection included site visits to three depository
libraries, meetings with representatives of five Government agencies, and
telephone interviews with six experts. The qualitative data collection included
site visits, agency meetings, and expert interviews. Westat conducted site
visits to three Federal depository libraries:

   > McKeldin Library, University of Maryland College Park, College Park,
     Maryland
   > Washington College of Law Library, American University, Washington, D.C.
   > Montgomery County Rockville Regional Public Library, Rockville, Maryland

The purpose of the visits was to discuss the effects of the transition to a more
electronic Federal Depository Library Program on the end user and on the
services and resources of each library.

Meetings with agency representatives had a twofold purpose:

   > To collect qualitative data about electronic Government information
     products, such as cost-effectiveness of standards, use of locator tools,
     results of user surveys, etc., that were not covered in the survey; and
   > To discuss the procedures for distribution of the questionnaire.

In addition to inviting agency coordinators and respondents, the statement of
work specified that Westat invite representatives of the following offices to
attend the meetings:

   > Public affairs or communications offices
   > Agency printing and publishing units
   > Information technology or electronic information systems offices
   > Agency libraries, and
   > Relevant program offices.

The following six agencies agreed to schedule a meeting: Department of Health
and Human Services, Department of Education, U.S. Supreme Court, Department of
Commerce, Environmental Protection Agency, and the National Archives and Records
Administration.  Only four of the six agencies chose to discuss the qualitative
questions at the meeting.  The other two agencies discussed the questionnaire
only and agreed to respond to the discussion questions in writing, although only
one actually submitted their written questions.

Finally, Westat held four telephone interviews with six content experts.  The
experts included two webmasters (Linda Wallace from the Internal Revenue
Service, and Jerry Malitz from the National Center for Education Statistics);
two preservation specialists (Evelyn Frangakis from the National Agricultural
Library, and Abby Smith from the Council on Library and Information Resources);
and two professors in information resources management (John Bertot and Charles
McClure).  The purpose of expert interviews was to:

>	Solicit opinions of experts on topics not adequately covered on the survey
	or in the agency meetings,
>	Ask questions to provide a broader context in which to view the issues,
	and
>	Explore current initiatives and future directions.


Key Findings


These findings reflect the major results of the survey and qualitative data
collection:

Policy and Planning Issues

1. There is an overall lack of Government information policy guiding electronic
publishing, dissemination, permanent public access, or information life cycle
management, especially as information policy relates to agency missions.
Also, there is a lack of overall coordination of these initiatives at the
Governmental, branch, or even agency level (pp. 68-69).
2. Responsibility for electronic publishing within agencies is decentralized,
diffuse, and unclear.  Some agencies either could not identify or had
difficulty identifying the proper respondent within their own agency, or even
the person who was responsible for the product (pp. 11 and 14).
3. Some Government agencies are monitoring the information needs of their users
to enhance current access to electronic Government information products
(p. 65).
4. There is a lack of specific planning for product development and
technological migration (pp. 34-36; table 23 on p. 42).
5. There is a lack of planning for or consideration of web design approaches 
that comply with the Americans with Disabilities Act (ADA) (table 6a, p. 29)

Permanent Public Access 

6. The concept of permanent public access (PPA) is not well understood.
Respondents also had difficulty distinguishing between PPA for electronic
products and archiving electronic Federal records with the National Archives
and Records Administration (tables 18-20, pp. 39-40).
7. Metadata and their importance to public access are not well understood,
particularly as they may affect PPA.  Only 27 percent of respondents reported
having a metadata record for the products surveyed (table 19, p. 39).
8. For some products, PPA results from the agencies' use of a host disseminator, 
such as GPO Access (p. 11).

Authenticity

9. There is a lack of understanding of what ensuring authenticity entails, and a
lack of planning for or consideration of ensuring authenticity of electronic
Government information products (table 21, p. 41).

Product Characteristics

10. Fifteen percent of the products surveyed are not in the public domain, for
all or part of the product (table 27, p. 45).  In addition, user fees are
charged for 30 percent of the products (table 24, p. 43).
11. The most prevalent types of mediums are the web, paper, CD-ROM, and
bulletin board systems (table 3a, p. 22); the most prevalent formats are
HTML, PDF, GIF, JPEG, TIFF, and ASCII (table 4a, p. 25).
12. The most prevalent types of data contained in the products surveyed are
textual, numerical, bibliographic, and graphical (tables 2a and 2b, p. 20).




Standards

13. There is a lack of standardization for producing Government information
products on CD-ROM (e.g., installation instructions, user documentation) 
(p. 55).
14. The most prevalent medium and format standards identified in the survey
are common agency practice rather than agency-mandated  (tables 3b, 4b, 6b,
pp. 23, 26, and 30).
15. Some Government agencies have established guidelines or best practices for
presenting and organizing Government information products on the web,
although full compliance with the guidelines is a goal that has not yet been
achieved (p. 64).
16. Some Government agencies are exploring a range of innovative formats and
web design approaches for electronic Government information products (p. 57).




Next Steps


As a followup effort, NCLIS indicated that they will use these findings as a
point of departure and analyze them in greater depth.  It is expected that this
followup effort will result in broad conclusions and recommendations to the
President and Congress about how the problems and challenges revealed in this
study can be constructively addressed to improve current and future public
access to electronic Government information.




1
Introduction and Background

Since 1813, the American public has benefited from the ability to gain free
access to Federal Government information.   This unique American right to no-fee
access to Government information is made possible through the Federal Depository
Library Program (FDLP) of the Superintendent of Documents (SuDocs) in the
Government Printing Office (GPO).   The FDLP has significantly contributed to
creating an informed, educated, and culturally enriched U.S. citizenry.

This introduction provides a brief overview of the FDLP and background
information on the purpose and objectives of this study to assess electronic
Government information products.


The Federal Depository Library Program

The mission of the Federal Depository Library Program is to assure current and
permanent public access to the universe of information published by the U.S.
Government.  The FDLP was established by Congress to ensure that the American
public has access to its Government's information.  Depository libraries
safeguard the public's right to know by collecting, organizing, maintaining,
preserving, and assisting users with information from the Federal Government.
The Government Printing Office provides Government information at no cost to
designated depository libraries throughout the country.  These depository
libraries, at their own expense, provide local, no-fee access to Government
information in all formats in an impartial environment with professional
assistance.  Any member of the public can visit these depository libraries and
use the Federal depository collections.

Products distributed by GPO for depository library collections include all
electronic Government information products that are of public interest or
educational value.  By law, the FDLP excludes those products that are solely for
administrative or operational purposes, classified for reasons of national
security, or the use of which is constrained by privacy considerations
(GPO, 1998, p.  4).

In order to administer the FDLP, as required by the enabling legislation for the
program, 44 U.S.C.  Chapters 17, 19, and 41, the SuDocs is responsible for the
acquisition, classification, format conversion, dissemination, and bibliographic
control of tangible and electronic Government information products; the
inspection of depository libraries, and the continuing education and training
initiatives that strengthen the ability of depository library personnel to serve
the public.  An emerging new responsibility is to ensure that electronic
Government information products disseminated through the FDLP, or incorporated
in the FDLP Electronic Collection, remain permanently accessible to the public.
Under 44 U.S.C., Sections 1901-1903, and Office of Management and Budget (OMB)
Circular A-130, Management of Federal Information Resources, Federal agencies
should make all their publications in all produced formats available to SuDocs
for distribution to depository libraries.


How the Federal Depository Library Program Works

GPO provides Government information at no cost to designated depository
libraries throughout the country.  These depository libraries, at their own
expense, provide local, no- fee access with professional assistance to this
information in all formats.   Access to Federal Government information is
available through more than 1,350 depository libraries located throughout United
States and its territories.  Fifty-three of the depositories are regionals, and
the remaining are selective depositories.  The regional libraries receive and
maintain everything that is distributed through the program, unless they are
superseded.  The selective libraries pre-select the types of publications they
wish to receive based on the specific needs and interests of the communities
they serve.  Of the libraries in the FDLP, approximately 50 percent are
academic, 20 percent are public, 11 percent are law, 5 percent are community
college, 4 percent are Federal agency, and 10 percent are special, state, court,
and Federal court libraries.

Before the evolution of electronic publishing media, especially the Internet,
Federal Government agencies published information almost exclusively in a
centralized print environment that facilitated easy distribution to the Federal
depository libraries.  Now, Federal Government agencies are doing their own
electronic publishing and creating and managing their own websites to
disseminate a variety of Government information products.  This study resulted
from Congress's concerns about the short- and long-term effects of electronic
publishing on the ability of all U.S. citizens to continue to gain affordable
and easy access to Government information.




Background of the Study

This study to assess electronic Government information products was authorized
by the Joint Committee on Printing and was sponsored by the Superintendent of
Documents, U.S. Government Printing Office.  The initial need for this project
was identified in GPO's cooperative 1996 Study to Identify Measures Necessary
for a Successful Transition to a More Electronic Federal Depository Library
Program.  This study (see www.access.gpo.gov/su--do cs/dpos/fdlppubs.html#4) was
conducted at the direction of Congress.  In order to conduct the study, the
Public Printer established a working group consisting of representatives from
the following program stakeholders and constituents:
>	GPO,
>	Appropriate congressional committees,
>	Congressional Research Service at the Library of Congress,
>	Office of Management and Budget,
>	National Archives and Records Administration,
>	Federal Publishers Committee,
>	Interagency Council on Printing and Publication Services (ICPPS),
>	Administrative Office of the U.S. Courts, and
>	Depository library community.

One of the committee's major recommendations was to assess electronic medium and
format standards for the creation and dissemination of electronic information
products.  The committee considered this assessment an essential step toward
ensuring a successful and cost-effective transition to a more electronic FDLP.


Project Phases

This project is being undertaken in three phases.  The first phase of the
project consisted of a review by the National Academy of Science's Computer
Science and Telecommunications Board (CSTB) in which CSTB developed a detailed
statement of work that defined the data collection process required to conduct
the assessment (see http://www.nclis.gov/info/g po1.html).

This report is a product of Phase II of the project.  GPO commissioned the
National Commission on Libraries and Information Science (NCLIS) to undertake a
survey and assessment of electronic Government information products.  NCLIS
awarded the contract to Westat, a survey research company, to undertake research
and data collection from Federal agencies in all three Branches, as well as
solicit the opinions of selected knowledgeable experts.   The contract further
called for Westat to complete an analysis of the data and expert opinions for
the purpose of interpreting their general meaning and significance, including
identifying broad emerging trends and patterns, and documenting findings.

In Phase III, NCLIS will identify an appropriate organization to review Phase I
and Phase II findings, as well as to review the data and develop conclusions and
recommendations for GPO, the Congress, and the President.


Study Goals and Objectives (Phase II)

Information gathered from this assessment will be used by the Superintendent of
Documents to facilitate improved public access to Federal Government information
made available to Federal depository libraries and the general public through
the FDLP.   More specifically, for this cross-section of Government information
products, the Phase II goals were to:

>	Identify medium (see glossary in Appendix E for the difference between the
	medium and media) and format standards that are the most appropriate for
	permanent public access,
>	Assess the cost- effectiveness and usefulness of various alternative
	medium and format standards, and
>	Identify public and private medium and format standards that are, or could
	be used for products throughout their entire information life cycle, not
	just at the dissemination or permanent public access stage.

The Superintendent of Documents will use the results of this work effort to 
continue to plan and implement the transition to a more electronic FDLP.  The 
five major specific objectives are:

>	First, with respect to electronic publishing practices and plans of
	Federal agencies (including ways in which the FDLP can best accommodate
	them), the objective is to provide an analysis of current practices as
	well as future plans for creating, disseminating, and providing permanent
	public accessibility to electronic information products, and to identify
	the standards for software and electronic mediums and formats that are
	used throughout the product's information life cycle, from creation to
	archiving, but especially at the stage of dissemination for permanent
	public access.
>	Second, with respect to cost-effectiveness of various dissemination
	mediums and formats that are, or could be utilized, the objective is to
	gather information on standards (whether mandated or consensual) that will
	assist the FDLP in making near-term decisions regarding the cost
	effectiveness of alternative mediums and formats for all FDLP
	participants.   This information should also assist participants in long
	term planning for permanent public accessibility, and the collection and
	analysis of overall information life cycle costs.
>	Third, with respect to the practical utility of various electronic mediums
	and formats to depository libraries and the public, the objective is to
	identify preferred standards used in various mediums and formats that
	depository libraries will need to support.
>	Fourth, with respect to utilizing standards employed in mediums and
	formats that can be used throughout all stages of the information life
	cycle (including creation, composition, computer terminal display,
	encryption, secure digital signature with non-repudiation and secure
	transmission capabilities), but especially for permanent public
	accessibility, the objective is to assess standards for basic security
	services in order to provide for secure and reliable transmission and
	document interchange.
>	Fifth, with respect to standards that are being developed and used in the
	private sector, the objective is to identify existing and planned
	standards for the purpose of determining what the FDLP must do to
	accommodate their adoption in terms of hardware/software requirements,
	staff and user education and training, and budgetary impacts.


Scope and Organization of the Report

The primary data collection activities included a survey and interviews.
Westat, per the requirements established by NCLIS in consultation with GPO,
surveyed a cross-section of electronic information products from Federal
agencies in all three branches of Government and solicited the opinions of
selected knowledgeable experts.  This cross-section of products was not a
randomly selected sample due to cost and time constraints.  Therefore, readers
are cautioned about generalizing the findings to all electronic Government
information products.

Westat surveyed electronic Government information products to determine the
mediums and formats in which products are currently produced and the standards,
if any, that are being used.  The survey also asked respondents questions about
the agency's future plans for adding or changing products, including the mediums
and formats in which they will be disseminated for permanent public access.

This report is limited to presenting and discussing the survey findings and
findings from qualitative site visits, agency meetings, and expert interviews.
Phase III of the project will focus on drawing conclusions and recommendations
based on work conducted during Phases I and II.

The report is organized in five parts: introduction and background, methodology,
survey analysis and findings, qualitative findings, and discussion of
quantitative and qualitative findings.  Please note that Appendix E contains a
glossary of terms and acronyms used on the questionnaire and throughout this
report.




2
Methodology

This second part of the report discusses the following topics:
>	The process of selecting a cross-section of electronic Government
	information products,
>	Agency coordinator briefings,
>	Questionnaire design and development,
>	Nonresponse and data retrieval followup, and
>	The methodology for the qualitative data collection activities, i.e., site 
	visits, agency meetings, and expert interviews.


Product Selection

NCLIS and GPO-assisted by various groups, including the library associations 
represented by the Inter-Association Working Group on Government Information 
Policy (IAWG), the Federal Library and Information Center Committee (FLICC), the 
Depository Library Council (DLC), and the Interagency Council on Printing and 
Publication Services (ICPPS)- developed and refined a set of criteria for 
product selection.  NCLIS, GPO, and the other representatives asked 
knowledgeable members of these groups to identify products that met one or more 
of the following six guidelines:
>	Increased emphasis on electronic dissemination rather than continuation of
	paper and microform dissemination;
>	Replacement of older electronic mediums and formats with state-of- the-art
	technologies;
>	Adoption of mandated (Government or private sector) and consensual (common
	agency practice) medium and format standards;
>	Adoption and use of preferred mediums or formats that have widespread
	support from agency, depository library, and user communities;
>	Exemplified cost- effective mediums and standards, especially those that
	can be used throughout the entire information life cycle, rather than the
	use of expensive customized or shelf packages; and
>	Exemplified awareness of the important impact of medium and format
	decisions on permanent accessibility, authentication, and/or security
	encryption protection.

The products were not randomly selected; therefore, readers are cautioned about
generalizing the findings to all electronic Government information products.

In April 1998, NCLIS distributed the preliminary list of products to agency
Chief Information Officers (CIOs), who were asked to validate and coordinate the
final selections with appropriate agency personnel.  In addition, NCLIS asked
CIOs to select an agency coordinator.  The coordinator's role was to oversee the
distribution of product questionnaires to the appropriate respondents and to
encourage respondents to complete the questionnaire and return it to Westat.
(See Appendix B for a list of coordinators who participated in this study.)

The final product list included 328 products from 24 agencies (Appendix C).
Over the course of the data collection, the number of products decreased from
328 to 314 for the following reasons:
>	Several products were discontinued and no longer exist.
>	Several products were in paper only and agencies had no plans to migrate
	them to an electronic medium; therefore, they fell outside the scope of
	this study.
>	Agency coordinators could not identify respondents for some products, so
	there was no one to complete the questionnaire.
>	Several questionnaires were undeliverable due to unknown or incorrect
	respondent addresses; no alternate respondent could be located in a few
	cases.


Coordinator Briefings

NCLIS and GPO planned and conducted two coordinator briefings in June and July
1998, and asked Westat to attend them (see Appendix A for agenda).  The purpose
of these briefings was to:
>	Provide an overview of the study including background, purpose, goals, and
	schedule,
>	Discuss their specific tasks,
>	Review the draft questionnaire with coordinators and solicit their input
	on changes,
>	Collect their final list of products, and
>	Thank them for their participation and cooperation.

Coordinators were asked to:
>	Assist Westat in pretesting the survey instrument,
>	Identify and brief appropriate internal participating offices,
>	Identify product respondents for survey followup,
>	Schedule and participate in voluntary agency meetings with Westat,
>	Distribute questionnaires to agency respondents,
>	Ensure timely completion and submission of survey instruments, and
>	Cooperate with Westat on followup.

Only a few coordinators brought their final selections to the agency meetings;
most agencies needed much more time to review and finalize their product
selections.  The questionnaire review also served as an informal pretest of the
questionnaire.


Questionnaire Design

NCLIS, with consultation from GPO, developed the initial five-page list of
questions.  This list of questions was included as an appendix to the statement
of work.  Westat worked with GPO and NCLIS from June through July to expand and
refine the list of questions to a 13- page instrument with appropriate
instructions, examples, skip patterns, open-ended questions, please-specify
questions, etc.  Westat pretested the questionnaire informally at the two
coordinator briefings.  The coordinators helped Westat to clarify some
questions, expand the format choices, and add a few more questions.

Westat conducted a more formal pretest with personnel from six Government
agencies.  These pretests led to the following substantive changes in the
questionnaire:
>	Clarification of instructions and wording of several questions,
>	Addition of more format options,
>	Addition of the definition of "product" at the beginning of the
	questionnaire, and
>	Clarification of definitions included in the glossary.

Westat, with final approval by NCLIS and GPO, finalized the questionnaire by
mid-August 1998.  (See Appendix D for the cover letters and Appendix E for the
final questionnaire.)
Distribution of the Questionnaires

During the last week of September and the first week of October, Westat
distributed the questionnaires to 23 agencies through the agency coordinators.
On October 9, 1998, NCLIS requested that Westat add more products to the survey
by including a 24th agency, the Securities and Exchange Commission (SEC).

Westat created a database of products and corresponding coordinators or
respondents and their addresses, phone and fax numbers, and e-mail addresses,
and prepared and mailed packets to the agency coordinators.  The agency
coordinators were responsible for ensuring that each packet was sent to the
appropriate product respondent in a timely fashion.  These packets included the
following materials for each product that was to be surveyed:
>	Cover letter to coordinator,
>	Cover letter to respondent,
>	Questionnaire,
>	Glossary of terms used in the questionnaire, and
>	Postage-paid return envelope.

A few agency coordinators requested that Westat send questionnaires directly to
their product respondents and a copy of the respondents' packets to the
coordinators themselves.  Westat sent questionnaire materials directly to the
respondents at the Department of Commerce, the Department of the Interior, the
Executive Office of the President, and the U.S. Congress.  These respondent
packets included the following materials:
>	Cover letter to respondent,
>	Questionnaire for each product he/she was assigned to survey,
>	Glossary of terms used in the questionnaire, and
>	Postage-paid return envelope.

In addition, the Department of Health and Human Services asked Westat to send an 
e-mail message to the individual product respondents notifying them that they 
could download the final version of the questionnaire and cover letters from the 
PDF file located on the NCLIS website at http://www.nclis.gov/news/nclisqux.pdf 
in order to complete the questionnaire.


Followup for Nonresponse, Data Retrieval, and Inconsistency

Westat made the first calls for nonresponse to agency coordinators.  These calls
began in early November and continued through mid- December.  In addition, NCLIS
sent periodic coordinator bulletins to keep coordinators updated on the progress
of the study and to encourage respondents-through the coordinators-to complete
questionnaires and return them to Westat.

Westat began a second round of nonresponse followup calls to respondents from
mid- December through the end of January 1999.  From mid-November through the
first week in January 1999, Westat made calls directly to respondents for data
retrieval (i.e., missing data) and inconsistencies (i.e., a respondent checked
"yes" to one question, but the next question was answered in a way that
suggested a "no" answer to the first question).

Approximately 40 percent of the questionnaires required some type of data
retrieval followup for one or more questions.  Some questions, such as 16,
18-19, and 21a, concerning metadata, permanent retention,   authenticity, and
the product's supporting technology, presented particular problems.  Westat
added a "don't know" category to these questions as a result of the nonresponse
data retrieval.  In addition, most respondents skipped questions 13d, 14d, and
15d about long-term plans for changing the product.  Data retrieval phone calls
and discussions with agency coordinators suggest respondents skipped these
questions because agencies had not yet developed long-term plans.

  The calls to respondents for data retrieval and data inconsistency revealed
the following reasons for nonresponse:
>	Did not know the answer.
>	Could not identify anyone who knew the answer.
>	Did not understand the question or the concept; using glossary did not
	help.
>	Did not have time to research the answer; had other work priorities.

In a few instances, it was clear that the agency was not in a good position to
respond to the questionnaire, in part because they rely on another agency,
vendor, or contractor to provide electronic access to their products.  Sometimes
these "host disseminators," such as GPO, assisted in preparing the responses
sent in by the publishing entity.

Observations about the data collection process.  Agency coordinators had
difficulty locating a single point of contact from each agency sub-unit who was
knowledgeable about the range and type of electronic information products
created for the agency.  Furthermore, due to the nature of the survey questions,
product respondents had to coordinate responses to some questions with personnel
who often did not work in their program areas.  This process required
respondents to identify personnel with whom they appeared to have little prior
contact, such as records managers, information technology staff, and staff in
planning offices, in order to respond to these questions.  In some cases, this
extra step discouraged respondents from seeking answers to these questions, so
questions were left unanswered.  Also, agencies whose coordinators could not
attend the coordinator briefings and agencies that did not participate in the
agency meetings had more problems with data consistency than did other agencies.


Methodology for Qualitative Data Collection
Site Visits to Depository Libraries

The qualitative data collection included site visits, agency meetings, and
expert interviews.  Westat conducted site visits to Federal depository libraries
from July 30 through September 9, 1998.  The statement of work (Appendix J)
specified that Westat visit three libraries:  one regional academic, one law,
and one public.   Furthermore, GPO suggested that Westat visit the following
specific libraries in the Washington, D.C., metropolitan area:
>	McKeldin Library, University of Maryland College Park, College Park,
Maryland
>	Washington College of Law Library, American University, Washington, D.C.
>	Montgomery County Rockville Regional Public Library, Rockville, Maryland

The purpose of visits was to discuss the effects of the transition to a more
electronic Federal Depository Library Program on the end user and on the
services and resources of each library.  The interview questions, which were
based on readings and discussions with GPO and NCLIS, covered three broad areas:
>	What key issues or concerns do you have about users accessing and using
	electronic Government information products?
>	What are your concerns about providing access to electronic Government
	information products?
>	What specific ideas do you have for improving public access to online and
	electronic Government information products in your library?

The site visits were audiotaped.  In addition, the libraries gave Westat
representatives a tour of the facilities.  (See Appendix F for a list of the
specific interview questions, the names of all interviewees, and detailed site
visit notes.)

Site visit observations.  In addition to the small number of libraries visited,
the problems and concerns of librarians in the D.C.  metropolitan area may not
be representative of those experienced by librarians at most depository
libraries, especially the selective depositories.  Some smaller selective
depository libraries that are located in more remote areas as well as some of
the larger urban selective depositories might have fewer resources (e.g., fewer
computers and trained librarians, training funds, and options for low-cost
Internet providers).


Purpose and Procedures for Agency Meetings

Meetings with agency representatives were held between September 15 and 
September 24, 1998.  The purpose of the meetings was twofold:
>	To collect qualitative data about electronic Government information
	products that were not covered in the survey, such as cost- effectiveness
	of standards, use of locator tools, results of user surveys, etc.; and
>	To discuss the questionnaire and data collection procedures for
	distribution of the questionnaire.

In addition to inviting agency coordinators and respondents, the statement of
work specified that Westat invite representatives of the following offices to
attend the meetings:
>	Public affairs or communications offices,
>	Agency printing and publishing units,
>	Information technology or electronic information systems offices,
>	Agency libraries; and
>	Relevant program offices.

Westat wrote the procedures for scheduling agency meetings and arranging for
logistics, which included developing meeting protocols, agenda, cover letter,
and script for interviewers to schedule meetings.  We then contacted
coordinators and sent them the following materials:
>	Cover letter explaining purpose of meeting and their tasks,
>	Meeting agenda and discussion questions,
>	Press release from NCLIS with background information on the project,
>	Roster of potential agency representatives who will attend meeting (to be
completed by the coordinator), and
>	Respondent product roster (to be completed by coordinator).

Agency meetings held.

Westat contacted 15 of the 24 agencies to hold meetings.  Of the 7 agencies that
were not contacted, 3 had fewer than 10 products.  NCLIS instructed Westat not
to hold meetings with the U.S. Congress and the Executive Office of the
President because NCLIS and GPO worked with them directly.  Ten of the 16
agencies did not respond to Westat's request to schedule a meeting.  The
following six agencies agreed to schedule a meeting:
>	Department of Health and Human Services
>	Department of Education
>	U.S. Supreme Court
>	Department of Commerce
>	Environmental Protection Agency
>	National Archives and Records Administration

Only four of the above six agencies chose to discuss the qualitative questions
at the meeting.  The other two agencies wanted to discuss the  questionnaire
only and agreed to respond to the discussion questions in writing.  However,
only one of them sent in responses.

Westat audiotaped all agency meetings and took notes as agency personnel
discussed the questions.  (Appendix G includes the list of agencies that
participated in meetings, the number of attendees, the discussion questions, and
summary notes from the meetings.)

In addition to the meetings held with Westat, NCLIS and/or GPO representatives
met with approximately 50 agency representatives.  In these meetings, NCLIS and
GPO discussed survey goals and objectives and the process for preselecting
products, in addition to responding to specific questions about the survey.

Agency meeting observations.

Agency participation in the entire project was voluntary but essential.  As with
any voluntary activity, participation is based on availability and timing.  For
example, many agency coordinators were unavailable to schedule meetings during
the summer months, or they were available but product respondents were on
vacation, which may have resulted in fewer agency meetings.

Product respondents needed to attend the agency meetings to review the
questionnaire, although they were not always the most appropriate personnel to
respond to all of the qualitative questions.  The project depended upon the good
faith, interest, and cooperation of agency CIOs and coordinators to participate
in the meetings.   Respondents and participants from the private sector are
often given an honorarium for participating in similar research activities, but
Federal employees are exempt from this process.

Scheduling agency meetings, calling coordinators, and preparing paperwork to
send to coordinators took a considerable amount of planning and coordination and
time, but it did not result in many meetings.   Agencies were cooperative, but
it was difficult for them to identify the "right" personnel to invite to the
meetings, even though coordinators took a significant amount of time to locate
product respondents from other sub- units within their agencies.   Therefore,
answers to the agency meeting discussion questions reflected the perspectives of
only 5 of the 24 agencies surveyed.


Expert Interviews

NCLIS provided a list of experts from which Westat chose six names.  Westat held
four telephone interviews with the six experts between October 27 and November
24, 1998.  The experts included two webmasters, two preservation specialists,
and two professors in information resources management.  The purpose of expert
interviews was to:
>	Solicit opinions of experts on topics not adequately covered on the survey
or in the agency meetings,
>	Ask questions raised during the agency meetings or site visits that
	require further explanation, or to provide a broader context in which to
	view the issues, and
>	Explore current initiatives and future directions.

As with the site visits and agency meetings, Westat audiotaped the interviews.
Appendix H provides a list of experts, interview questions, and a summary of
interview notes.

3
Survey Analysis and Findings

This section of the report presents the survey findings from each of the major
survey questions as they appear in the questionnaire (Appendix E).  Appendix E
also includes a glossary of terms and acronyms used throughout this report.  The
discussion and presentation will then focus on the key study questions explored
on the following topics:
>	Preferred medium and formats used,
>	Planned medium and format changes,
>	Permanent public access issues,
>	Permanent retention issues,
>	Authenticity, and
>	Searchability, proprietary software, and licensing fees.

The final response rate was 74 percent.  Respondents from 24 Government agencies
completed and returned a total of 242 of the 328 questionnaires fielded.  The
word "respondents" refers to the 242 agency personnel who completed the
questionnaire.  Since each agency submitted at least two product questionnaires,
the unit of analysis is the product or product respondent, not the agency
(table1).  The sample was not randomly selected due to cost and time
constraints.  Therefore, readers are cautioned about generalizing the findings
to all electronic Government information products.


Structure of the Questionnaire

The questionnaire is organized into five sections, A through E.  Section A
contains general information about the product and agency that produced it.
Section B contains questions about the product's current profile including the
kinds of data the product contains, mediums in which it is produced, and, if in
an online medium, formats and online approaches used.   This section concludes
with questions on searchability and retrievability of the product.  Section C
relates to the future plans for the product and is designed to solicit
information about changes in the product's data, mediums, and formats.  Section
D addresses the issues of metadata, permanent public access, permanent
retention, authenticity, updating/upgrading plans, user fees, licensing, and
public domain.  The final section, E, includes one open-ended general comments
question.

Section A Responses

Sections A and B of the questionnaire focus on format and medium standards that
address the key objectives of the study.   Section A contains general
information about the product and the agency that produced it, including the
name of the agency and its sub-unit, the product name and description, and the
Uniform Resource Locator (URL) for the site in which the product appears.  A
list of the agencies surveyed and the number of product questionnaires received
from each agency appears in table 1.  (For a description of how products were
selected, refer to the methodology section.)  Appendix C contains the final list
of products surveyed.






Table 1.
Number of surveys returned by each agency surveyed
Agency								Number of surveys
									returned
Administrative Office of the U.S. Courts			5
Department of Agriculture					19
Department of Commerce						14
Department of Defense						8
Department of Education						14
Department of Energy						12
Department of Health and Human Services			19
Department of the Interior					11
Department of Justice						8
Department of Labor						2
Department of State						3
Department of Transportation					9
Department of the Treasury					13
Environmental Protection Agency				16
Executive Office of the President				5
General Services Administration				8
Library of Congress						21
National Aeronautics and Space Administration		6
National Archives and Records Administration		10
Securities and Exchange Commission				11
Smithsonian Institution						11
Social Security Administration				4
Supreme Court of the United States				4
United States Congress						9

SOURCE: National Commission on Libraries and Information Science, Government
Information Product Assessment Questionnaire: 1998.

Section B Responses

Section B covers the current product profile, including:
>	How it is used;
>	What types of data it contains;
>	What mediums the product is available in, what is the primary medium used,
and what are the agency's medium standards;
>	What kinds of formats are used, what is the primary format used, and what
are the agency's format standards;
>	What user interfaces are supported and what web design approaches are
used;
>	If the electronic product can be searched and how;
>	What agency hosts the product on the web; and
>	How the product can be retrieved.

Readers should note that most of the survey questions asked respondents to
"check all that apply"; therefore, the percentages for these questions will
exceed 100 percent.  Also, for the first set of tables in this section (tables 1
through 6), the response categories appear in descending order by number or
percentage.  Therefore, the responses will not match the order in which they
appear on the questionnaire.

Types of Data Contained in Product

Table 2a shows that the frequently mentioned types of data contained in the
products surveyed are textual (188 responses), followed by graphical (142
responses), numerical (141 responses), bibliographic (82 responses), and spatial
(53 responses).  Multimedia, video, and sound are less common, probably because
they reflect the products surveyed and because of the special plug-ins,
hardware, and memory required to open, view, and listen to products that contain
these data types.  The primary data types contained in products surveyed are
textual (57 percent), numerical (21 percent), bibliographic (10 percent), and
graphical (5 percent; table 2b).  These four types of data account for
approximately 93 percent of the products surveyed.

Table 2a.

Number and percent of types of data, by the type of data contained
Type of data						Type of data contained
								Number	Percent
Textual data (books, serials, reports)		188 		77.7
Graphical data
(photos, charts, graphs, tables, drawings)	142		58.7
Numerical data						141		58.3
Bibliographic data					82		33.9
Spatial data (maps, coordinate files)		53		21.9
Multimedia (sound, video, text, graphics)		14		5.8
Video								10		4.1
Sound								9		3.7
Other 							16		6.6
	NOTE: Percents do not add to 100 because respondents could
	choose more than one item.
SOURCE: National Commission on Libraries and Information Science, Government
Information Product Assessment Questionnaire, 1998.



Table 2b.

Number and percent of types of data, by the primary type of data
Type of data						Primary type of data
								Number	Percent
Textual data (books, serials, reports)		138		57.0
Numerical data						50		20.7
Bibliographic data					24		9.9
Graphical data
(photos, charts, graphs, tables, drawings)	13		5.4
Multimedia (sound, video, text, graphics)		3		1.2
Spatial data (maps, coordinate files)		2		0.8
Sound								1		0.4
Video								1		0.4
Other 							10		4.1
	NOTE: Percents do not add to 100 because of rounding.
SOURCE: National Commission on Libraries and Information Science, Government
Information Product Assessment Questionnaire, 1998.

Types of Mediums Used

Respondents were asked to identify all the types of mediums in which the product
is available to the public as well as the primary type of medium used.  The most
common type of medium used among pre-electronic mediums is paper (177
responses), followed by microform (22; table 3a).  The responses in the "other"
category include Fax on Demand, audiotapes, and Braille.  Among electronic
mediums used, it is not surprising that the web is the most common (204
responses), followed by   CD-ROM (70 responses), floppy diskettes (42
responses), hard drive (30 responses), and magnetic tape (18 responses).  These
figures reflect the medium types the public is most likely to easily access, as
well as the availability and growing interest in the web.

Table 3a also displays the frequency and percentage distribution of the primary
types of mediums in which the product is publicly accessible.  The web (42
percent) and paper (41 percent) are the primary types of mediums used, followed
by CD-ROM (8 percent) as a distant third.

Standards for all mediums checked.  For each type of medium checked, respondents
identified one medium standard (see Appendix E glossary) among four types:
>	Agency mandated,
>	Common agency practice,
>	Other, and
>	None.

While most agencies have some type of standards for their pre-electronic and
electronic mediums, they are primarily "common agency practice" rather than
"agency mandated." For pre-electronic mediums, 33 percent of the products in
paper are in an agency- mandated standard (table 3b).  However, 52 percent of
paper products are used as a common agency practice.  Only 13 percent of the
CD-ROM products are in an agency-mandated standard, as compared to 59 percent of
CD-ROMs that are used as a common agency practice.  Eighteen percent of
web-based products were reported to be in an agency-mandated standard, while 70
percent of them are used as a common agency practice.

A considerable number of products in CD-ROM (21 percent) were reported as having
no standards (table 3b).  Compare these numbers to 9 percent (15 products) of
products reported by respondents as having no standards for the use of paper,
and 8 percent (16 products) reported as having no standards for the use of the
web.

		Table 3a.  (Omitted - Will appear at a later date)
		Table 3b.  (Omitted - Will appear at a later date)


Format Types Used

Databases.  Responses to all formats used are shown in table 4a.  Wide Area
Information Server (WAIS) is the most common type of database identified (22
responses), followed by Oracle (17 responses), and dBase (9 responses).  In some
cases, WAIS is reported because the products surveyed are made available through
GPO Access.  The 44 responses in the "other" category reveal few multiple
responses except for Microsoft Access that received 5 "write-in" responses in
this category.

WAIS (24 percent) and Oracle (14 percent) are the primary types of databases
used (table 4a).  Ninety-one percent of the respondents who checked WAIS as one
of the databases used also indicated that the use of WAIS is a common agency
practice, while only one respondent indicated that WAIS is agency mandated
(table 4b).  However, only 44 percent of the respondents identified the use of
Oracle as a common agency practice,   but 39 percent of respondents indicated
their use of Oracle is agency mandated.

Spreadsheets.  For spreadsheet formats used, Excel and Lotus 1-2-3 received 33
and 23 responses, respectively (table 4a).  When respondents were asked to
choose one of the databases as the primary type used, 59 percent chose Excel,
while only 33 percent chose Lotus 1-2-3.  Close to 71 percent of the respondents
also identified the use of Excel as a common agency practice as compared to 38
percent who indicated the use of Lotus 1-2-3 as a common agency practice (table
4b).

Tagged mark-up.   Hypertext Markup Language (HTML) is both the most commonly
used tagged markup language (157 responses) and the primary type of tagged
markup language used (89 percent; table 4a).  The Government agencies surveyed
seldom use Extensive Markup Language (XML) (2 responses), and Standard
Generalized Markup Language (SGML) (14 responses).  This is noteworthy since
SGML is one of the few formats that NARA accepts for electronic records.

Even given the fact that HTML is the primary type of tagged markup format used,
72 percent of the respondents reported that HTML is used as a common agency
practice, while only 13 percent reported that its use is mandated by the agency
(table 4b).  Sixty percent of the respondents who use SGML for their online
products reported it as a common agency practice, while only 13 percent reported
that its use is mandated by the agency.

Image formats.  Portable Document Format (PDF) is the most common image format
(132 responses) and the primary type of format used (49 percent) by the agencies
surveyed in this study (table 4a).  The use of PDF is followed by GIF (99
responses), JPEG (77 responses), then TIFF (36 responses) as image formats used.
Perhaps PDF is the most commonly used format by the agencies surveyed because
the Federal Government disseminates a wide range and large number of forms and
documents that must be printed in the exact format in which they are created.

Almost 64 percent of respondents reported that PDF is a common agency practice,
while 16 percent reported it is mandated by the agency (table 4b).  While a
higher percentage of respondents reported using GIF (69 percent) and JPEG (71
percent) as a common agency practice, PDF is the most used agency-mandated image
format (16 percent).   Audio formats.  The number of responses reported in this
category reflects the small numbers of products surveyed that contain sound (see
table 2a).  WAV (12 responses) is the most commonly used sound format followed
by AU (5 responses), and AIFF with 1 response (table 4a).   WAV is also the
primary type of audio format used (73 percent).

		Table 4a.  (Omitted - Will appear at a later date)

Sixty-two percent or eight of the agency respondents who indicated using WAV
reported it as a common agency practice; only two respondents (15 percent)
reported that WAV is an agency- mandated standard (table 4b).  Perhaps it is not
surprising that WAV is the most commonly used audio format; since it was built
into Windows95, it has become the de facto standard for sound on PCs.  AIFF is
the standard audio format for Macintosh computers (PC Webopaedia; see
www.pcwebopaedia.com).

		Table 4b.  (Omitted - Will appear at a later date)

Video formats.  As with the audio formats used, the even smaller number of
responses reported in this category also reflect the small numbers of products
surveyed that contain moving images.  Table 4a shows that Moving Picture Experts
Group (MPEG) (9 responses) is the most commonly used format, followed by MOV  (7
responses) and Audio Video Interleave (AVI) (4 responses). MPEG may be more
commonly used since it generally produces better quality video than AVI (PC
Webopaedia).  Of all the video formats used, however, the primary type of video
format used is MOV (33 percent), followed by MPEG and AVI (27 percent each).


Of the respondents who reported using MPEG, 50 percent indicated its use is a
common agency practice, while only 1 respondent (10 percent) reported that its
use is agency mandated.   Sixty-three percent of the respondents reported that
MOV is used as a common agency practice, and none indicated that its use is
agency mandated (table 4b).

Text formats.  ASCII is by far the most commonly used text format (122
responses) and the primary type of text format used (81 percent; table 4a).
The second most commonly used text format is ANSI (11 responses) followed by
Rich Text Format (RTF) (9 responses).  Seventy-one percent (87) of the
respondents reported that their use of ASCII is a common agency practice, as
compared to 11 percent (14) who reported its use is agency mandated (table 4b).

Word processing formats.   Between the two most popular word-processing software
packages, Microsoft Word and WordPerfect, the latter (75 responses) is more
commonly used than Microsoft Word (55 responses; table 4a).   These responses
are also consistent with the primary type of word processing used.  Sixty-four
percent of respondents reported WordPerfect as the primary type of format used
while only 22 percent of respondents reported Microsoft Word as the primary type
of format used.  PageMaker received the largest number of responses (5) in the
"other" category.  Nineteen respondents (25 percent) reported that WordPerfect
is an agency-mandated format standard, while only 8 respondents (14 percent)
indicated that Microsoft Word is an agency- mandated format standard (table 4b)

Summary of format types used.  Each of the 242 respondents from the 24 agencies
surveyed was asked to identify the primary type of format used of each of the
categories.  The primary types of formats used in each category are WAIS, Excel,
HTML, PDF, ASCII, and to a lesser degree, WAV and MOV.


User Interfaces

Online approaches.   Question 9 on the survey refers to online approaches used.
Eighty-five percent of the respondents reported that their product is in an
online medium (table 5).   These respondents were then asked to respond to a set
of questions on user interfaces supported and web design approaches.

		Table 5.  (Omitted - Will appear at a later date)

User interface supported.  Table 6a shows that Netscape Navigator (195
responses) is a more commonly supported browser than Internet Explorer (170
responses).   However, close to 70 percent of agency respondents indicated that
both of these browsers are almost equally supported as a common agency practice
rather than an agency- mandated standard (table 6b).

		Table 6a.  (Omitted - Will appear at a later date)

In addition, respondents reported that file transfer protocol (FTP), Telnet, and
nongraphical/dial-up shells are also supported by their agencies (table 6a).
Designs that support LYNX, a text-based browser, account for 12 of the 22
responses in the "other" category.  The number of responses for the category
nongraphical/dial-up shell are low (15 responses), especially given the need for
agencies to comply with the American with Disabilities Act by making their sites
more accessible to the visually and hearing impaired.   Like the browsers, the
other user interfaces supported are primarily supported as a common agency
practice rather than an agency-mandated standard.  Almost 83 percent of the 40
respondents who reported their agency supports FTP also reported it is a common
agency practice, while 79 percent of the 27 respondents who reported supporting
Telnet also indicated it as a common agency practice (table 6b).  No respondents
reported that Telnet is an agency-mandated standard; however, 8 percent reported
that FTP is an agency- mandated standard for their surveyed products.  Seventy-
five percent of the respondents reported the support of a nongraphical/dial-up
shell as a common agency practice while only 13 percent indicated that it is
agency mandated.   Web design approaches.  Various web design approaches used,
in descending order, are HTML (150 responses), tables (111 responses), CGI
Scripts (66 responses), frames (53 responses), Javascript (43 responses), Java
Applets (23 responses), and XML (11 responses; table 6a).   ColdFusion was
reported in three of the responses in the "other" category.   The use of these
web design approaches is overwhelmingly a common agency practice rather than an
agency-mandated standard (table 6b).  Basic HTML-tags that consistently display
content in a similar fashion by the most popular browsers-is the only approach
to which almost one-fifth (18 percent) of the respondents reported that its use
is agency mandated.  Less than 10 percent of the respondents using each of the
other approaches indicated that they are agency-mandated standards.  Since the
use of frames, Javascript, Java Applets, and XML may not be supported or enabled
for many users' browsers, the agencies surveyed appear to be adopting them
slowly, if at all.

		Table 6b.  (Omitted - Will appear at a later date)

Searchability of Product

Searchability of an electronic product is important for users because it allows
them to effectively access the information they need.  Most electronic products
are searchable either by full-text with no fielding (74 responses) and/or by
full-text and field (99 responses; table 7).  The "view only" category contains
a higher number of responses than expected (79 responses).   The "other"
category contains the following common responses:
>	Inapplicable because product is in a paper medium (most common response);
>	In PDF, which is not searchable; and
>	Product is indexed by field only.

		Table 7.  (Omitted - Will appear at a later date)

Product "Host"
Most of the products surveyed (199 responses) were hosted by the agency that
created them, although other agencies or institutions might also host the
products since respondents were asked to "check all that apply" for this
question (table 8).  There are fewer responses for products hosted by another
agency (42 responses), a contractor (17 responses), and an educational
institution (9 responses).

		Table 8.  (Omitted - Will appear at a later date)

Retrievability of Product

In order to ensure broad access to the product, the public should be able to
download and save electronic Government information products without
restrictions (GPO, 1996, p.  7).  Responses to Question 11 indicate that for the
most part, products surveyed for this study can be downloaded and saved without
restrictions (173 responses; table 9).   Responses in the second category
indicate that some products cannot be downloaded or saved (20 responses).  A
small number of products (14) cannot be downloaded or saved because their use
requires proprietary software that is not freely distributed (table 9).  Common
write-in responses in the "other" category include
>	Can be downloaded and saved, but subject to restrictions.
>	Can be printed from browser, but not downloaded.
>	Product available only in paper.

The United States Advisory Council on the National Information Infrastructure,
in its publication "A Nation of Opportunity," identifies as one of the basic
principles of Government information and services that "the Federal Government
should not charge for making its information available...nor charge for access
to that information" (GPO, 1996, p.  28)

		Table 9.  (Omitted - Will appear at a later date)


Section C Responses (Planned Product Profile)

Section C contains a series of questions related to the future product profile.
Respondents were asked questions about changes in the types of data, mediums,
and formats used and reported on in Section B of the questionnaire.
Respondents also were asked to identify the time span in which the changes would
occur and to describe the planned changes.

Types of Data

The first question in this section of the questionnaire asked respondents about
plans to discontinue publication of the product.   Only 5 percent (12) of the
respondents planned to discontinue the product (table 10).  Several of the most
commonly listed responses provided for discontinuation of a product was that the
product was a one-time "prototype" or that the paper version of the product
would be discontinued.

		Table 10.  (Omitted - Will appear at a later date)

Table 11 shows responses to question 13 about the kinds of data (i.e.,
bibliographic, textual, graphical) the product will contain in the future.  The
majority of respondents (76 percent) reported that the agency plans no changes
to the product.  Twenty-one percent reported that the agency would add one or
more new types of data.  A total of 3 percent reported either the
discontinuation of one type of data (0.4 percent), or a complete change to new
data types (2.6 percent).  Several respondents reported that the changes in data
types would include adding audio or video and multimedia.

		Table 11.  (Omitted - Will appear at a later date)

Most agency respondents reported that these changes in data types would mainly
occur in the short term (40 responses) and, to a lesser degree, in the medium
term (24 responses; table 12).   Most respondents skipped the question about
long- term plans for changing data types.  Respondents noted in the "please
specify" categories in questions 13c and 13e indicate that respondents' plans
for product changes have not yet been solidified.

		Table 12.  (Omitted - Will appear at a later date)

Types of Mediums Responses to changes in types of mediums parallel those for
changes in data types.  Seventy-six percent of the respondents reported no plans
to change mediums (table 13).  Eighteen percent of respondents reported that
they are planning to add one or more mediums, 2 percent indicated they will
discontinue one or more mediums, and 3 percent reported they will change to a
new type of medium.

		Table 13.  (Omitted - Will appear at a later date)

The two most frequently mentioned additions to medium types are to provide web
access to the product, and to make the product available on CD- ROM.  Most of
the respondents (35) who reported changes in medium types indicated that the
changes will occur in the medium term; 21 respondents indicated that these
changes will occur in the short term (table 14).  Again, most respondents
skipped the question about long-term plans for changing product mediums.  The
few respondents who provided descriptions of their long- term plans mentioned
that they will produce the product in multiple mediums (paper and web), or that
paper items will be migrated to the web.  Other respondents indicated that their
long-term plans are undetermined or undefined.

		Table 14.  (Omitted - Will appear at a later date)

Types of Formats

One might expect to see more dramatic changes in types of formats since the
range of formats is varied and broad (i.e., database, spreadsheet, tagged
markup, image, etc.).  The pattern of responses to question 15 mirrors the
responses to changes in types of data and mediums, except for the change to new
types.   Seventy-two percent of the respondents reported no changes in format
types.   Eighteen percent indicated that they are planning to add one or more
formats, while 9 percent reported they will change to new format types (table
15).  This change to new format types is the largest percentage change in this
category as compared to changes to new types of  data (3 percent; table 11) and
new types of mediums (also 3 percent; table 13).  Respondents who provided
specifics about the changes to new format types indicated these new types would
be PDF and XML.

		Table 15.  (Omitted - Will appear at a later date)

The majority of respondents who reported changes indicated that they will occur
in the short term (36 responses), and/or the medium term (32 responses; table
16).   The majority of respondents did not answer the question about long-term
plans for changing formats.

		Table 16.  (Omitted - Will appear at a later date)


Section D Responses (Other Information)

Section D of the questionnaire contains a variety of questions in an effort to
answer some of the critical issues of public access to electronic information
products:
>	Metadata,
>	Permanent public access (i.e., provided by what agency and how),
>	Permanent retention,
>	Ensuring authenticity,
>	Updating/upgrading plans,
>	User fees,
>	Licensing, and
>	Public domain.


Metadata

Metadata, data about data, are important for public access.  Metadata refers to
describing the content of a document or record allowing users to find Government
information more effectively.  Examples of metadata include Government
Information Locator Service (GILS) and machine-readable cataloging (MARC)
records.  To that end, one of the requirements of the Government Printing Office
Electronic Information Access Enhancement Act of 1993 (Public Law 103-40) was
that the Superintendent of Documents maintain an electronic directory of Federal
electronic information (44 U.S.C., Section 4101).

Only 27 percent of agency respondents reported that their products have a
metadata record, while 69 percent reported no metadata record exists for their
products (table 17).  In the followup question, most respondents identified
their metadata records as either MARC or GILS.  Another 5 percent indicated they
do not know if a metadata record exists.

		Table 17.  (Omitted - Will appear at a later date)

Permanent Public Access

In an electronic age, permanent public access to Government information, a
critical concept in information resources management, presents far-reaching
challenges to the Federal Depository Library Program, Congress, Federal
agencies, and ultimately the American public.  GPO indicates that permanent
public access "means that electronic Government information products within the
scope of the FDLP remain available for continuous, no-fee public access through
the program" (GPO, 1998, p.  19).  GPO recognizes and acknowledges its
responsibility to provide ongoing public access to the electronic Government
information available through the FDLP.   However, in a decentralized networked
environment, agencies are asked to share the responsibility for building,
storing, disseminating, and preserving a broad range of electronic information
products in order to ensure continued public access.

Agency respondents reported that permanent public access is primarily provided
by their agency (177 responses), by another agency (51 responses), and/or by
some other entity (20 responses; table 18).   Respondents reported that
permanent public access is not provided for 28 products (table 18).   However,
on closer examination, the responses to the "please specify" questions indicate
that either respondents may have misunderstood the concept of permanent public
access (as opposed to current access), or they assumed other entities have this
responsibility.  Some of the common responses to "other" agencies include the
Government Printing Office, National Archives and Records Administration (NARA),
and contractors and vendors.  These responses illustrate respondents' lack of
understanding about the difference between permanent public access to electronic
information products through their own agencies or through partnerships with
GPO, and permanent retention of official Government records through NARA.
Furthermore, only 4 of the 28 products for which no permanent public access
currently is provided have future plans for providing permanent public access
(table 19).

		Table 18.  (Omitted - Will appear at a later date)

		Table 19.  (Omitted - Will appear at a later date)

Permanent Retention

The mission of the National Archives and Records Administration is distinct from
that of GPO.  NARA's mission is to preserve and provide public access to
permanently valuable records of the Federal Government.  Federal agencies are
responsible for transferring products to NARA that are scheduled as permanent
records (i.e., official records of the Federal Government as defined by the
Federal Records Act).  Under 36 CFR 1228.188, mediums approved for transfer
include open reel magnetic tape, magnetic tape cartridge, and CD-ROM.   Agencies
currently may not transfer to NARA electronic records that are in a format
\dependent on specific hardware and software.   However, SGML tags are permitted
on electronic textual documents as are records written in ASCII or Extended
Binary Coded Decimal Interchange Code (EBCDIC) with all control   characters and
other non- data characters removed (Lewis Bellardo, Deputy Archivist of the U.S.
in a written response to agency questions, October 14, 1998).

The responses to the questions on permanent retention may reflect the current
status of transferring permanent electronic records to NARA (see questions
findings).  Only 34 percent of agency respondents reported that their products
are scheduled for permanent retention by NARA (table 20).  Sixty- four percent
reported their products are not scheduled for retention, while another 3 percent
reported they do not know if the product is scheduled for retention.   However,
it should be pointed out that at the time of the survey, the schedule that would
have covered electronic records of permanent value was unenforceable under a
court case declaring it null and void; therefore, these figures may be
unreliable.

		Table 20.  (Omitted - Will appear at a later date)

Ensuring Authenticity

Although 64 percent of respondents reported that their agency ensures
authenticity for the products surveyed (table 21), responses to the open-ended
question about how the agency attests to authenticity indicate that respondents
may not fully understand the concept.  Authentication refers to the process
agencies use to ensure the public that the product is an official legitimate
product created and produced by the Federal Government agency and no other
source (see glossary, p.  E-17).  Ensuring authentication includes technical as
well as policy considerations.  Some technical examples of authentication
include digital signature technology, special watermarks, disclaimers, or
statements on the products.  Respondents provided answers that address how the
agency ensures that information or data in the product are valid or reliable- an 
important process, but not the same concept as authenticity.  Common responses
include the following:
>	Program office verifies data.
>	Review CD-ROM contents before public release.
>	Regulations and source/reliability statement regarding data sources.
>	Review and approval within agency.
>	Source of content is the same as the hardcopy version.
>	Test reliability of data every 5 years, or more often.
>	Publications are subjected to review by subject matter expert and peer
	review.

		Table 21.  (Omitted - Will appear at a later date)

Updating/Refreshing Plans

Twenty percent of respondents reported that their products are updated annually,
followed by daily (16 percent), monthly (12 percent), and weekly (5 percent;
table 22).   However, the majority (47 percent) of respondents checked the
"other" response category.  The write-in responses covered a broad range of time
periods in which products are updated.  Below is a sampling of multiple
responses:
>	Quarterly,
>	As needed,
>	Irregularly,
>	Not updated,
>	Semi-annually,
>	Every 2 years,
>	Periodically, and
>	Twice a month with old version staying on line.

		Table 22.  (Omitted - Will appear at a later date)

Changing Supporting Technology

The majority of the respondents (71 percent) reported that there are no plans to
change the product's supporting technology (table 23).   Twenty-eight percent of
respondents reported plans to change the product's supporting technology.

		Table 23.  (Omitted - Will appear at a later date)




User Fees

Public access to no-fee Government information products is one of the core
principles upon which the FDLP is based.  However, users might be charged a fee
if they order certain types of electronic Government information products
directly from GPO or the agency that created the product.  Nine percent of
respondents reported that all users are charged fees, while 20 percent reported
some users are charged fees.  The majority (72 percent) of agency respondents
reported that there are no fees charged to access or use the product surveyed
(table 24).  The followup question asks about specific fee amounts and the
reasons for the charge.  The responses to this question vary greatly.  A few
common responses include the following:
>	No charge for web access.
>	Single paper copy free; charge for additional copies.
>	No subscription fee to libraries and some constituencies.
>	Files can be downloaded from the Internet for free.  There is a charge for
	published books.
>	Fees are for paper products only.

		Table 24.  (Omitted - Will appear at a later date)

Licensing

Many Government agencies purchase licenses from vendors for search and retrieval
software to be used with the product to make the data or information more
accessible to users.   Agencies negotiate various agreements with vendors about
who can use the software free of charge.   The majority of respondents (69
percent) reported that they do not license commercial search and retrieval
software (table 25).  For the remaining 31 percent of respondents who have
licensed commercial software, the license covers use by all the key
constituencies including agency personnel (73 responses), public users (69
responses), agency's primary target constituencies (65 responses), Federal
depository libraries (59 responses), and/or all libraries (59 responses; table
26).

		Table 25.  (Omitted - Will appear at a later date)

		Table 26.  (Omitted - Will appear at a later date)

Public Domain

Public domain, a critical component of public access, means that the
information, product, or publication is not copyrighted and therefore can be
reproduced by anyone without obtaining copyright permission.  One of the goals
of an electronic FDLP is to provide public access to any Government information
product free of copyright or copyright-like restrictions (GPO, 1996, p.  2). 
The majority of respondents, 86 percent, indicated that all parts of their
surveyed products are in the public domain (table 27).  Another 10 percent
indicated that part of the product is in the public domain, while 5 percent
reported that the product is not in the  public domain.  The followup question
that requests an explanation of the second response (i.e., part of product is in
the public domain) uncovered these typical responses:

>	Copyrighted tables are not in the public domain.
>	There are some copyright-protected logos and trademarks.
>	Includes copyrighted material that would require approval for
reproduction.

Respondents offered a wide variety of explanations for products that are not in 
the public domain:
>	Retrieval software is proprietary and use is licensed.
>	Commercial vendors lease the database for distribution.
>	Songs and performances are protected by copyright.
>	Books are available only to eligible blind patrons of our program, by law.

		Table 27.  (Omitted - Will appear at a later date)


Section E Responses

The final section E of the questionnaire contains one open-ended "comments" 
question.  These responses are too broad and disparate to provide a detailed 
itemization.  Most of the comments are explanations of issues covered in the 
survey.  However, below are a few comments that cover issues not directly 
addressed in the survey.
>	Our mission, mandated by the Americans with Disabilities Act (ADA), is to 
	satisfy all browser requirements (e.g., ASCII browsers like LYNX through 
	the latest versions of Netscape and Internet Explorer).
>	We produce printed documents and link to electronic documents maintained 
	on the GPO's server.
>	The product is not published in any electronic form.  It is a collection 
	of individual products that are individually published.
>	I am very new to this area (2 weeks) and received significant contractor 
	assistance in completing this form.
>	In addition to four other web sites, we will soon web-enable our database 
	with some encrypted modules.
>	The database is intended to be accessible to the largest audience possible 
	via free or public domain software whenever possible.
>	This information is available in PDF format on our website to ensure the 
	integrity of the data.  Coding in HTML (particularly tables) could lead to 
	mistakes with such a large amount of numeric data.

No respondents commented on the survey questionnaire, the project in general, or 
the process of filling out the survey.


Study Questions

This section will use findings from two or more survey questions to provide 
additional information on some of the key issues explored in the study.  The 
responses to these questions relate specifically to the products surveyed.

The following questions were chosen because they address one or more of the 
critical study areas: preferred medium and format standards, permanent public 
accessibility, permanent retention, user fees, commercial licensing of search 
and retrieval software, and authenticity.


Preferred Medium and Format Standards

Study Question 1: What combinations of preferred medium standards are currently 
used by the respondents?

The agencies surveyed are creating and using (in descending order):
>	Products both in paper format and on the web.
>	Products both in CD- ROM and the web.
>	Products both in paper and CD-ROM.

Since most of the agencies surveyed create products in more than one medium, 
what combinations of  preferred mediums are they using?  Of the respondents who 
indicated that paper was a medium used and the respondents who reported that CD-
ROM was a medium used, only 19 percent reported that they are using both paper 
and CD-ROM products   (table 28).  Table 29 shows that of the respondents who 
reported that they use CD-ROM, and those who reported that they use the web as a 
medium, 21 percent use both CD-ROM and the web as mediums.   However, of the 
respondents who reported using paper and the respondents who reported using the 
web as a medium, 64 percent use both paper and the web (table 30).  Therefore, 
the respondents surveyed are creating and using products both in paper format 
and on the web much more often than they are creating and using products in CD-
ROM and the web.  An even smaller percentage of products is being created in 
paper and in CD-ROM.  This confirms the earlier finding that paper and the web 
are the preferred mediums used by the agencies surveyed, but provides additional 
information about the combinations of mediums used.

		Table 28.  (Omitted - Will appear at a later date)

		Table 29.  (Omitted - Will appear at a later date)

		Table 30.  (Omitted - Will appear at a later date)

Study Question 2: What combinations of preferred format standards are used by 
the respondents?

The respondents are slightly more likely to use HTML in combination with PDF 
than they are to use HTML together with GIF.  However, they are almost as likely 
to use HTML, GIF, and ASCII together as they are to use HTML, PDF, and ASCII 
together.  Of the respondents who reported using HTML as a tagged markup format, 
and those who reporting using PDF as an image format, 39 percent reported the 
use of both HTML and PDF (table 31).  Of the respondents who checked HTML, and 
those who checked GIF as an image format, 36 percent checked that they used HTML 
in combination with GIF (table 32), slightly less than those who used HTML and 
PDF in combination.  Since PDF is the preferred image format used by agencies 
(table 4), this is not an unexpected finding.

		Table 31.  (Omitted - Will appear at a later date)


		Table 32.  (Omitted - Will appear at a later date)

However, when the formats are used in combinations of three, it appears that 
respondents are almost as likely to use HTML, GIF, and ASCII (21 percent) 
together as they are to use HTML, PDF, and ASCII (22 percent) together (tables 
33 and 34).

		Table 33.  (Omitted - Will appear at a later date)


		Table 34.  (Omitted - Will appear at a later date)

Public Access to Products

Study Question 3: If a product is permanently accessible, is it also likely to 
be scheduled for retention with the National Archives and Records Administration 
(NARA)?

No, the majority of products surveyed that are permanently accessible are not 
likely to also be scheduled for permanent retention with NARA.

Permanent public accessibility and permanent record retention are two distinct 
concepts.  GPO, through the FDLP, has a historical commitment to permanent 
accessibility of paper products, and now to electronic products.  To that end, 
GPO requests that agencies provide information products in all mediums to GPO 
and work with GPO and Federal depository libraries to provide permanent public 
accessibility to electronic products.  Agencies are responsible for transferring 
those products that are scheduled as permanent records to NARA.  However, not 
all records that are scheduled for permanent retention by NARA are products 
within the scope of the FDLP.  For such records, permanent public accessibility 
through the FDLP is not an issue.

Of the respondents who said yes, the product is permanently accessible, and the 
respondents who reported their product is scheduled for retention with NARA, 
only 25 percent reported that the product is both permanently accessible and 
also scheduled for retention with NARA (table 35).  The majority of products 
that are publicly accessible are not likely to also be scheduled for retention 
with NARA.  While there is not information from the survey data to identify 
reasons for this situation, some possibilities are that:
>	the product is not a permanent or official record of the U.S. Government 
	as defined by Federal Records legislation.
>	the product is in a format that is accepted by GPO but in a format that 
	NARA does not currently accept, and therefore could not be transferred to 
	NARA.
>	agencies are overlooking this important part of the information life cycle 
	of electronic products.

		Table 35.  (Omitted - Will appear at a later date)


Study Question 4: Is the licensing of search and retrieval software likely to be 
a barrier to unrestricted public access?

No, for the products surveyed, the licensing of commercial search and retrieval 
software by the agency does not appear to be a barrier to unrestricted (no fee) 
use.  Of the respondents who reported that they license commercial search and 
retrieval software for their products, and those who reported that all users are 
charged a fee for the products, only 2 percent who license commercial search and 
retrieval software also charge a fee for all users (table 36).  A slightly 
larger number of respondents (4 percent) who use commercial search retrieval 
software for their products also charge a fee for some users.  Twenty- five 
percent of respondents who license search and retrieval software for their 
products charge no user fees.

		Table 36.  (Omitted - Will appear at a later date)

Study Question 5: Are respondents who have purchased commercial search and 
retrieval software for their products also transferring the products to NARA?

No, based on the products surveyed here, respondents are not transferring 
permanent records to NARA for products in which they have purchased commercial 
search and retrieval software.  Of the respondents who reported issuing 
commercial search and retrieval software, and those who reported scheduling 
products for permanent retention with NARA, only about 10 percent who have 
purchased commercial software for products have also scheduled their products 
for permanent retention with NARA (table 37).

		Table 37.  (Omitted - Will appear at a later date)


Other Issues:
Authenticity and Metadata

Study Question 6: If an agency ensures authenticity, is it also likely to 
provide permanent public access to the product or do agencies rely on another 
agency to provide permanent public access?

Yes, based on the products surveyed, agency respondents who ensure authenticity 
for their products are also more likely to provide permanent access to them 
directly, rather than through another agency.

Of those respondents who reported they ensure authenticity and those who 
reported they provide direct permanent public access to their products, 47 
percent both ensure authenticity for their products and provide direct permanent 
access to them (table 38).  However, only close to 14 percent of the respondents 
who reported they ensure authenticity for their products also reported that 
another agency provides permanent public access to the product (table 39).

		Table 38.  (Omitted - Will appear at a later date)

		Table 39.  (Omitted - Will appear at a later date)

     Study Question 7: Are online products hosted by the agency that created it 
more likely to have a metadata record than products hosted by another agency?

Yes, based on the products surveyed, those that are hosted by the agency that 
created it are more likely to have a metadata record than those hosted by 
another agency.

Tables 40 and 41 show that almost 20 percent of the products that are hosted by 
an agency also have a metadata record, while only 7 percent of the products that 
are hosted by another agency also have a metadata record.

		Table 40.  (Omitted - Will appear at a later date)

		Table 41.  (Omitted - Will appear at a later date)


    4 Qualitative Findings

This section of the report highlights the qualitative findings from the three 
site visits with Federal depository libraries, five agency meetings, and six 
expert interviews.   Appendices F through H include interview questions and 
detailed responses from the site visits to depository libraries (F), agency 
meetings (G), and expert interviews (H).



Site Visits to Federal Depository Libraries

The purpose of the site visits to the three depository libraries was to identify 
the key issues and concerns librarians have about providing public access to 
electronic Government information products through the Federal Depository 
Library Program.  (See Appendix F for a complete list of questions posed to 
librarians.)

The site visits were held with one regional depository library and two selective 
depository libraries in the Washington, D.C., metropolitan area.  It is 
important to note that the three libraries visited may not be representative of 
all depository libraries in terms of the geographical location and library user 
characteristics (e.g., education level, socioeconomic status, etc.).   
Therefore, readers are cautioned about generalizing these observations to all 
depository libraries.   Highlights of the three librarians' responses are 
provided below.  Appendix F contains a detailed description of the librarians' 
responses to the interview questions.


Highlights of Site Visits to Three Depository Libraries

User Needs and Concerns

>	Librarians interviewed noted that the general public is still more 
	comfortable using Government information products in paper and microfiche 
	than they are using the Internet.  Patrons (and librarians) are least 
	comfortable using products on CD-ROM.
>	Librarians expressed concern about the difficulty patrons experience in 
	accessing Government-produced CD-ROMs that are not standardized.  They 
	reported that the search and retrieval software is different for each CD, 
	CD-ROMs often have no installation instructions or user documentation, and 
	they are not user-friendly.
>	Librarians indicated that some users are still intimidated by electronic 
	mediums and computers.  Most users ask librarians to help them search for 
	materials on the web and frequently need help downloading large files.
>	Librarians noted that since most Government websites only contain the most 
	recent information, they are concerned about users having permanent public 
	access to retrospective Government information on the web in the future.

Librarians' Concerns: User Fees, Hardware, Training, and Costs

>	Although none of the libraries visited currently charge fees for printing 
	materials from the Internet or CD-ROMs, all three librarians are either 
	considering charging fees or are planning to charge fees and expressed 
	concerns about how this will affect their patrons.
>	Users do not have access to enough workstations, so the libraries must 
	limit use.   Also, if libraries had additional money for hardware, they 
	would order hardware in support of CD-ROMs (e.g., a new CD-ROM server and 
	an 18-disk CD changer).  (Even though CD-ROM is the least preferred medium 
	and declining in number in the FDLP.)
>	All librarians interviewed expressed concerns about finding time and money 
	to train librarians and staff, especially on using CD- ROM products, but 
	also on downloading files, effectively searching the Internet for 
	Government information, and creating and maintaining web pages.   They 
	welcome any additional training on using GPO Access, Geographic 
	Information Systems, etc.
>	Time and money permitting, librarians expressed interest in establishing 
	partnerships with GPO and other Government agencies to put some 
	retrospective online Government information on their servers so users can 
	have reliable access to it in the future.  In addition, librarians would 
	like to provide outreach to public schools, community centers, etc., to 
	educate students and adults about the wide variety of valuable information 
	available from the Federal Government.
>	One librarian expressed strong feelings about the need for Congress to 
	provide long-term financial support to Federal depository libraries so 
	they can continue to provide permanent public access to digital materials. 
	This librarian's perspective was that the cost to provide access to 
	electronic Government information is steadily increasing.


Agency Meetings

Meetings were held with four agencies between September 14 through September 24, 
1998:
>	Department of Health and Human Services
>	Environmental Protection Agency
>	U.S. Department of Education
>	U.S. Department of Commerce

Although meetings also were held with the U.S. Supreme Court and the National 
Archives and Records Administration (NARA), these two agencies did not respond 
to the agency discussion questions in the agency meetings; they chose to discuss 
the survey questionnaire only.  However, Lewis Bellardo, the Deputy Archivist of 
the United States, sent in written responses to the discussion questions 
(Appendix G).

The National Commission on Libraries and Information Science (NCLIS) provided 
discussion questions for the agency meetings; Westat modified some of the 
questions with NCLIS's approval.  The purpose of the agency meetings was to 
supplement survey data by collecting more general information on electronic 
Government information products that are not product-specific.  For example, one 
of the survey objectives is to assess the cost-effectiveness and usefulness of 
preferred medium and format standards, an issue that was not directly addressed 
on the survey.  In addition, the agency meetings afforded NCLIS, GPO, and Westat 
an opportunity to review the survey questionnaire with agency respondents and to 
address any questions they might have.

Highlights of the agency meetings are provided below.  For a more detailed 
summary of the responses to the 12 questions posed to agencies, see Appendix G.

Agency Meeting Highlights

Preferred Mediums and Formats
>	Agencies interviewed reported using the same preferred medium and format 
	standards as those reported by survey respondents: web, CD-ROM, bulletin
	board; HTML, PDF, and ASCII.  Additional preferred formats mentioned by
	agency representatives include TIFF, JPEG, and Lotus/Domino.
>	All agencies are exploring a wide range of innovative and creative web
	design approaches including the use of SQL, Oracle, ColdFusion, and
	animated GIFs.  Some examples of ways in which agencies are utilizing web
	technologies include data warehousing, interactive GIS, multimedia CD-ROM,
	live "real-time" web casting of selected speeches, and real-time
	forecasting of air pollution levels for 22 states.
>	Four of the five agencies have guidelines or "best practices" for 
	presentation and organization of products or publications on the web. 
	Most of the guidelines discuss preferred formats for some types of 
	products.   The most common problem experienced by the agencies in this 
	regard is compliance issues (i.e., encouraging personnel to adhere to 
	them).
>	There are some trends for migrating certain families of products to the 
	web for newsletters, training manuals, annual reports, and conference 
	proceedings and presentations.
>	Agencies consider many factors when making decisions to create/retain 
	products in more than one medium: budget, cost, accessibility to users, 
	and size of audience the product reaches.  The decision-making process 
	varies from agency to agency and sub-unit to sub-unit.

Assessing User Needs
>	All agencies reported involving users in testing and evaluating the 
	usefulness of the web and CD-ROM products.  The most frequently used 
	assessment methods are focus groups, videotaping of users, and online user 
	surveys.   Agencies are using the results of these evaluation methods to 
	add and change some formats and mediums as well as content.
>	Four of the five agencies interviewed reported that they maintain 
	some type of GILS records to help the public locate their information 
	resources.

Information Life Cycle Management, Permanent Public Access, and Permanent 
Retention
>	No agencies are addressing the following key information resources 
	management issues: permanent public access, information life cycle 
	management, and permanent retention.  (The expert interviews provide some 
	insight into the reasons that agencies are not addressing these issues. 
	See the summary section of this report.)

Cost-Effectiveness of Various Mediums and Formats
>	No agencies have conducted a formal cost-benefit analysis for creating 
	products in formats and mediums for distribution to the Federal Depository 
	Library Program.  Generally, agency representatives reported it costs less 
	to create products for the web because they can avoid production, 
	printing, and distribution costs for paper and CD-ROM products.

Expert Interviews

The interviews with six experts also enriched and supplemented the survey 
findings.  Since the interviews were conducted after the site visits and agency 
meetings, they were helpful in providing a broad context within which the survey 
findings could be viewed.

The expert interviews were conducted between October 27 and November 24, 1998.   
Telephone interviews were held with two webmasters, two preservation 
specialists, and two professors of information resources management.  These 
experts were selected from a list provided by the NCLIS.   Highlights from each 
set of interviews are provided here.  (See Appendix H for a detailed summary of 
each telephone interview.)

Interviews With Webmasters

Highlights from interview with webmasters Jerry Malitz, National Center for 
Education Statistics (NCES), and Linda Wallace, the Internal Revenue Service 
(IRS), on October 27, 1998.

Preferred Formats
>	The IRS, unlike the other agencies surveyed, primarily uses SGML, followed 
	by PDF, HTML, and Postscript.  They train their authors to use SGML 
	because they consider it "intelligent data" that can automatically 
	generate other formats (e.g., web, BBS, Fax on Demand) through templates 
	and filters.   All NCES publications are in PDF, then HTML (optional); 
	they rarely put an entire publication in HTML format only.
>	The IRS has conducted a cost-benefit analysis of the costs of delivering 
	requests through different formats.  They have found that it costs $3 per 
	phone call to fill a request, 1 cent to access their Internet site for 
	forms, etc., and $2.50 to make a CD-ROM containing 5 years of IRS 
	publications.
>	IRS indicates that it provides permanent public access to tax information 
	online for 5 years, and from their "core repository library" for about 14 
	years, but not for every application.  However, this 5 to 14 years means 
	that IRS provides current but not permanent public access to their 
	Government electronic products.
>	All IRS documents are ADA-compliant, online searchable, and downloadable.

User Needs
>	Both IRS and NCES assess and evaluate the effectiveness of their web sites 
	with advisory groups (IRS), or for NCES, through an Internet Working Group 
	made up of representatives from each program area.
>	Both agencies have GILS records.


Interviews With Preservation Specialists

Highlights from interview with preservation specialists Evelyn Frangakis from 
the National Agricultural Library (NAL) and Abby Smith from the Council on 
Libraries and Information Resources (CLIR), November 10, 1998.

Goals of Preservation
>	It is useful to think about preservation goals such as enhancing the long-
	term preservation of and access to information of enduring value for as 
	long into the future as possible.
>	There is no standard accepted method of ensuring long-term access to 
	digital information.  It may be more accurate to say that one of the 
	primary goals of preservation is to set up systems that "sustain 
	predictable levels of loss."

Barriers to Preservation of Digital Materials
>	The concept of preservation in the traditional preservation world examines 
	the concept of permanence, but in the print world the concept of 
	permanence relates to chemical inertness and mechanical durability.  These 
	concepts do not translate easily into a digital world.
>	There are two problems with digital preservation: (1) media in which 
	information resides may be unstable; and (2) software/hardware 
	configurations on which information is stored becomes obsolete so quickly 
	that even when one migrates information from one system to another, much 
	of the data and functionality are lost.
>	Other barriers to digital preservation include that it is difficult to 
	understand what we can and cannot do under current copyright law, and any 
	transmission link is as strong as the weakest link.  The weak link in the 
	transmission of electronic information is human beings, not technology 	 
	(e.g., no one agency or organization has stepped forward to address issues 
	like information life cycle management).   Preservation of information 
	must be thought about at the creation stage, not after the information has 
	been collected and disseminated.
>	One of the core infrastructure problems is the need to create a failsafe
	archives mechanism for materials that disappear from the web.

Current Preservation Models and Initiatives
>	NAL and partner institutions are implementing a model for permanent public
	access and preservation of agricultural literature that addresses all the
	key issues in information resources management: inventory and life cycle
	of information, permanent public access, technical requirements, and user
	access and retrieval.  (NAL is one of the few examples for ensuring a
	failsafe archives for preservation of agricultural literature.)

CLIR Initiatives
>	CLIR commissioned a report by Jeff Rothenberg from RAND Corporation on 
	emulation.  (Emulation is the process of imitating one system with another 
	so both accept the same data, execute the same programs, and achieve the 
	same results.)
>	CLIR commissioned an analysis of migrating file formats to do a risk 
	assessment associated with those file formats during migration.
>	CLIR identified a computer scientist at Carnegie Mellon University, John 
	Ockerbloom, who has developed a system of file conversion called TOM 	 
	(Typed Object Model), a type of migration that converts web-based 
	materials to different file formats.

Interviews With Information Resources Management Specialists

Highlights from interviews with John Bertot, November 18, and Charles McClure, 
November 28, 1998.  (These two telephone interviews were held separately.)

Barriers to Successful Implementation of Information Resources Management 
Initiatives
>	Agencies are struggling with issues such as permanent public access, 
	information life cycle management, and permanent retention due to a 
	general lack of information resources management (IRM), as well as 
	organizational policy integration for Federal Government legislation and 
	initiatives.
>	Agencies do not view information as a strategic resource that is directly 
	related to agency missions.  Most Government IRM initiatives focus on the 
	technology side of IRM because it is tangible.
>	Sometimes smaller agencies are more successful in implementing IRM 
	initiatives due to fewer organizational and communication barriers to 
	working collaboratively.
>	The Information Technology Management and Reform Act of 1996 did little to 
	clarify the role of the CIO and IRM staff, so agencies are now struggling 
	with what to do with these functions.
>	Agency resources are now almost exclusively devoted to Y2K efforts with 
	little time and resources left to devote to IRM, standards, and 
	operability.
>	Staffing and training are critical for both IRM and CIO staff.
>	Challenges for agencies in the next few years include how to coordinate 
	information technology and information technology management, 
	interoperability and standards that cut across agencies, and education and 
	training of staff.




5
Discussion of Quantitative and Qualitative Findings

This section synthesizes, integrates, and discusses issues and the major themes 
that emerge from the survey and the qualitative data collection activities, 
including the interviews with Federal depository librarians, agency personnel, 
and other experts, and the literature review.  The section is arranged by the 
following key study issues:
>	Preferred mediums and formats,
>	Evaluating websites,
>	Cost-effectiveness of formats and mediums,
>	Depository library needs,
>	Public access (public domain and user fees),
>	Permanent public access and preservation.


Preferred Mediums and Format Standards

Survey respondents and agency representatives reported they most often use the 
following mediums:
>	Paper
>	Web
>	CD-ROM
>	Bulletin board systems (to a lesser degree)

Both respondents and representatives also reported use of the following formats:
>	HTML
>	PDF
>	GIF
>	ASCII
>	TIFF

However, most agencies whose products were surveyed use these mediums and 
formats as a common agency practice, rather than as an agency mandate.  In 
addition, agency representatives and webmasters reported they use SGML, Oracle 
(with ColdFusion or SQL), JPEG, and TIFF because these formats meet the 
information needs of their individual constituents or are used in some of their 
creative web approaches.   The IRS is one of the few agencies interviewed that 
uses SGML.  IRS' Linda Wallace, one of the webmasters who served as an expert 
consultant for this project, indicates that most agencies do not use SGML 
because it is difficult to use.   But Wallace noted that IRS uses SGML because 
it is   much more robust, and it is easy to change a document format to match 
customer needs (e.g., tax law information for consumers and for lawyers).  (See 
Appendix H for detailed notes on the telephone interview with Linda Wallace.)

A few survey respondents indicated they are planning to change to or add XML or 
other object-oriented formats.  XML may be appealing to some agencies because 
data can be stored in a format provided by XML that is transferable to a wide 
range of hardware and software environments (Bryan, 1998, p.  14).  In addition, 
according to Stuart Culshaw, XML makes it easier for authors to produce 
documents for many different output mediums (i.e., paper, online help, web) from 
a single source (Culshaw, 1998, p.  7).

Most of the agency representatives who participated in the meetings also 
reported that their agencies have established written guidelines or "best 
practices" that specify preferred formats for the presentation of information on 
the web.  Even though these guidelines are not agency-mandated, they seem to be 
a common agency practice.  Several of the agencies interviewed indicated they 
have modified or adopted their  agency guidelines from the guidelines 
established by the Federal Web Consortium in 1996.

The Consortium, founded in 1994 by the National Science Foundation and the 
U.S. Nuclear Regulatory Commission, established guidelines with other Government 
agencies (see http://www.dtic.mil/staff/ct homps/ guidelines/).  The guidelines 
provide suggestions to help the Federal community accomplish agency missions to 
improve services to customers.  Consortium guidelines cover a wide range of 
topics including:
>	Home page checklist (content, navigation/organization, style/markup);
>	File formats (i.e., agencies should not be restricted to proprietary 
	formats such as WordPerfect, Microsoft Word, SAS, PDF);
>	Rationale for using certain kinds of formats such as HTML, GIF, and JPEG;
>	Guidelines for formats to be used for downloading or display (e.g., HTML, 
	GIF, JPEG, PDF, Postscript); and
>	Emerging standards.

Agency representatives indicated that one of their biggest challenges is to 
convince personnel from all program areas to follow the agency's internal 
guidelines when creating products for the web.  Another challenge for agencies 
is to consolidate web guidelines from different agency sub-units so they are 
complementary rather than contradictory.

Evaluating Websites

Agency representatives, per OMB Circular A-130 and the Government Performance 
and Results Act of 1993, are assessing the usefulness of their websites and CD-
ROMs as part of a larger effort to measure program effectiveness.  Focus groups, 
online customer surveys, and videotaping of customers online are the most common 
ways in which agencies evaluate and test products on their websites.

One objective of the evaluation is to test both formats and web approaches.  
Based on the evaluation results, agencies may change or add formats.  For 
example, one agency, after testing their site with children, eliminated PDF 
files on the site and made it more interactive.  Another agency made the 
decision to keep their BBS because many of their international users do not have 
ready access to the web.  One agency webmaster indicated that the needs of their 
business clients, who participate on their advisory board, help drive their 
format needs.  A fourth agency stores its documents in TIFF format for image and 
textual data.  As customers request documents, the agency converts them to PDF 
so customers can download the material.  A fifth agency created a simple set of 
rules for producing CD-ROMs based upon user input: keep it simple to use, 
intuitive, and self-tutorial.

User needs for easy access to electronic information products will continue to 
affect how agencies make decisions about formats and mediums.  Bertot and 
McClure suggest that more agencies should continue to monitor the information 
needs of the public as well as targeted constituencies to enhance current access 
to electronic Government information products (Bertot and McClure, 1997, p.  
288).




Cost-Effectiveness of Formats and Mediums

None of the agency representatives who attended the agency meetings has 
conducted a formal cost-benefit analyses for producing or creating products in 
preferred or emerging formats, mediums, or online approaches for distribution to 
the FDLP.  Most agencies reported that migrating products to the web 
substantially reduces printing and distribution costs associated with paper 
mediums.  However, the crosstabs in tables 28-29 reveal that many Government 
products are still produced in more than one medium and often in more than one 
format.   Providing permanent public access to electronic mediums ultimately may 
exceed the one-time costs associated with producing and distributing the same 
information in print or microform (GPO, 1996, p.  24 and A71-A74).

In her role as Chief, Electronic Information Services, at the Internal Revenue 
Service, Linda Wallace has analyzed the costs of delivering documents to 
customers (Appendix H).  She found that:
>	It costs IRS $3 per call for the public to call into their toll-free
	number and for IRS to fill the request.
>	The cost to IRS for the public to use the Internet to access and use the
	forms is 1 cent, a difference of 300 to 1.  (However, this shifts the cost
	to the public, who must have access to the Internet.)
>	It costs IRS $2.50 to make and distribute to all public libraries 	
	(including the Federal depository libraries) each CD-ROM containing 5
	years of tax forms, instructions, and publications.

Based on these numbers, the IRS has made some internal decisions about where 
they will focus their resources and time in order to reach the maximum number of 
customers in the most cost-effective manner.


Depository Library Needs

Since depository librarians serve as the intermediary between the users and 
electronic information products, their observations and experiences about user 
and library needs are critical.  In general, the five agencies interviewed 
focused on public users or their target audiences rather than depository library 
users when discussing usage of their electronic Government information products.

First, the librarians interviewed emphasized that many patrons still prefer 
Government information in paper mediums, followed by the web and then CD-ROM.   
The respondents surveyed indicated that many of their products are produced both 
in paper and on the web.

Second, librarians expressed concerns about lack of standardization for 
producing Government CD- ROMs.  One agency representative indicated that they 
are undertaking several initiatives to make their CD-ROMs more user- friendly by 
making them as intuitive as possible and incorporating a user testing component 
into the production schedule.

A third important concern for the librarians interviewed is the rising cost of 
computer hardware and the simultaneous rise in user expectations for state-of-
the-art computer workstations.  Although the three libraries recently received 
updated computer workstations that met or exceeded the recommended minimum 
guidelines for depository libraries, they are beginning to change their policies 
on access to workstations by placing a time limit on their use.

A fourth issue concerns the rising costs to purchase and maintain new equipment, 
which have caused depository librarians to reconsider their policies on charging 
printing fees.  One librarian indicated that their library already charges 
patrons for photocopying materials; this change is not dramatic, but it does 
affect the concept of no-fee access when an overwhelming number of products are 
offered on the Internet.

Fifth, time and resources to train library staff (and patrons) on how to use the 
new technology (i.e., how to download files), conduct Internet searches, design 
and develop their own websites, and load, search, and use CD-ROMs are major 
concerns expressed by the depository librarians interviewed.  The fact that 
Government information exists in a variety of mediums and formats only increases 
rather than diminishes the need for training.

Finally, all librarians are troubled by how GPO, the FDLP, and Government 
agencies will address the problems of permanent public access to electronic 
information products that are constantly being replaced and updated by new ones.  
In addition, the preservation of retrospective electronic Government information 
is an issue of concern.




Public Access

The survey data revealed that 15 percent of the products surveyed are not in the 
public domain, for all or part of the product.  In addition, user fees are 
charged for 30 percent of the products.  These data suggest that these two 
critical public access goals have not yet been achieved.




Permanent Public Access to and Permanent Retention of Electronic Government 
Information 

Perhaps more than any other issues, permanent public access and preservation 
pose two of the greatest challenges to the FDLP, and ultimately to the public.  
Each of the experts raised different issues and shared various perspectives 
about these issues.  It might be helpful here to summarize their perspectives 
and describe initiatives underway to address the problems associated with the 
provision of permanent public access and preservation. 

Most of the survey respondents indicated that permanent access is currently 
provided for the products surveyed, although most of the responses indicated 
that this concept is not fully understood and that access is not provided by the 
agency responsible for the product.  Instead, they are relying on GPO, Federal 
depository libraries, the National Technical Information Service, or other 
agencies to provide this permanent public access. In its policy and planning 
document, Managing the FDLP Electronic Collection (see ttp://www.access.gpo.gov/
su--docs/dpos/ecplan.html), GPO states that "the `first- level' collection 
management activity depends upon knowledge that the products exist.  In order to 
ensure current and permanent access, GPO will .rely on notification from and 
outreach to other agencies and notification from the depository library 
community."

The responses of agency representatives on the issue of permanent public access 
may provide additional information about the problem. Most agency epresentatives 
said their agencies had not discussed the issue or were exploring the issue to 
see how it should be addressed, and they indicated that they did not understand 
the concept of permanent public access in relation to permanent retention.  The 
one exception was the representative from National Archives and Records 
Administration, who is clear about the agency's role to provide permanent public 
access to its own products.





It might be helpful here to clarify the distinctions between the two concepts.   
GPO's definition of permanent public access "means that electronic Government 
information products within the scope of the FDLP remain available for 
continuous, no-fee public access through the program" (GPO, 1998, p. 19).


Lewis Ballardo, deputy archivist of the United States, in a recent article in 
the Washington Post (March 12, 1999, p. A01) stated that the problem of digital 
preservation must be addressed "or memory will be lost for the latter half of 
the 20th century." In addition, Bellardo, in a written response to agency 
questions, articulated agency responsibilities to GPO for permanent public 
access and to NARA for permanent retention.   GPO will accept products in all 
mediums to provide continuous, no-fee public access, if notified by agencies 
that access is being discontinued.  Agencies are responsible for transferring 
those products that are scheduled as permanent records (official records as 
defined by Federal Records legislation) to NARA.  

Linda Wallace described the IRS' methods for providing current public access to 
their materials. Using SGML format, the IRS has built and maintains a core 
knowledge repository to generate media output in any application to respond to 
customer needs.  The repository maintains materials for 14 years, but not for 
every application. In addition, all tax forms, publications, instructional 
materials, etc., are available online for 5 years.  Since none of the agencies 
interviewed is providing permanent public access to its products, it was useful 
to ask two information resources management experts, John Bertot and Charles 
McClure, to provide some larger context within which the problem can be viewed. 

Perspectives on Permanent Public Access and Information Life Cycle Management 
from Information Resources Management Experts  

Both Bertot and McClure have extensively studied and taught information esources 
management (IRM). They attribute the lack of successful implementation of IRM 
initiatives in the Federal Government to the following factors:
>	There is no comprehensive integrated Federal IRM policy; current policies 
	do not adequately address permanent public access, information life cycle,
	and electronic records management.
>	There is no strategic vision of IRM by agencies; information is not viewed 
	as a resource that should be used to accomplish agency missions.
>	Most agency initiatives focus on the technology side of IRM because it is 
	tangible.
>	Most agencies are targeting their information technology resources 
	toward Y2K efforts.
>	 There is no clear distinction between the role of information resources 
	managers and CIOs.
>	There is no ongoing training for IRM and CIO staff. 

(See Appendix H for detailed notes on telephone interviews with Bertot and 
McClure, and Bertot and McClure, 1997, pp.  280- 282.) 

There are many IRM policy instruments from the Paperwork Reduction Act of 1980 
and 1986, OMB Circular A-130 (1985; and 1993 and 1994 revisions) through the 
Information Technology Management Reform Act (ITMRA) of 1996 and Executive Order 
13011 (July 1996).  But Bertot and McClure (1998) emphasize that there is still 
a lack of an integrated policy.  For example, in their focus group with IRM 
managers, Bertot and McClure noted that managers felt that the Paperwork 
Reduction Act assumed that the managers understood and knew how to manage the 
information life cycle, but they agreed that agency management at all levels 
never grasped the concept either in theory or in practice.  In addition, the 
ITMRA that created a position for an agency- based CIO to oversee agency IRM 
activities and to provide education for agency IRM personnel and agency managers 
(among other things) does not clarify the relationships between and among CIOs 
and IRM managers. Consequently, it is ambiguous about whether the agency CIO's 
organization replaces, incorporates, or is separate from current agency IRM 
functions. 

Given this larger context, it is not surprising that IRM issues such as 
information life cycle management, preservation, and permanent public access 
have not been adequately addressed.   Conventional organizational barriers such 
as size, culture, poor communication and interaction across and within agencies, 
and lack of ongoing, strategic training for IRM and CIO staff may exacerbate 
these challenges faced by agencies (telephone interview with Bertot, Appendix 
H). (As an example, McClure states that IRM graduate students' degrees are 
useful for about 1-2 years after they graduate.  After that, their skills are 50 
percent out of date; telephone interview with McClure, Appendix H.)  Several 
experts are involved in initiatives that address some of these important IRM 
issues.




Current Initiatives on Permanent Public Access and Permanent Retention

Several agencies, organizations, and Federal depository libraries with partner 
institutions are exploring ways to address the problems of permanent public 
access, preservation, and electronic records management.  Appendix H contains 
more detailed information about each of these initiatives that will be 
summarized here.

Abby Smith from the Council on Libraries and Information Resources (CLIR), and 
Evelyn Frangakis from the National Agricultural Library (NAL) are supporting 
research and testing models for permanent public access and preservation.  The 
three CLIR initiatives are described below:
>	A commissioned report by Jeff Rothenberg from RAND Corporation on 
	emulation.  The report has been completed and was published in January
	1999.  The report describes the weaknesses of migration and the strengths
	of emulation and sets up a research agenda to develop emulation.  (Log
	onto publications on CLIR site for a summary of Rothenberg's report:
	http://www.clir.org/pubs/reports/rothenberg/contents.htm l.)
>	A commissioned analysis of migrating file formats to support a risk 
	assessment associated with those file formats during migration.  The study
	by Cornell University, using data from the Mann (agricultural) Library,
	will use numeric file formats and databases and text formats.  The report,
	to be finished by September 1999, will include analysis and a template
	that others can use for doing a risk assessment of migration of those file
	formats.
>	CLIR is working with John Ockerbloom, a computer scientist at Carnegie 
	Mellon University (CMU) who has developed a system of file conversion
	called TOM (Typed Object Model).  (See www.cs.cmu.edu/afs/cs.cmu.edu
	user/spok/www/defense/index.html).  CLIR would like to see if they can
	bring his concepts into fuller application.

Smith describes NAL and its efforts to provide permanent public access and to 
preserve agricultural literature as one of few examples where a failsafe 
archives might work, partly because NAL is a national library dedicated to one 
type of literature.  Evelyn Frangakis is involved in NAL's efforts to develop 
its own preservation program that includes a traditional preservation program 
and digital efforts.  Their digital efforts are two-pronged:  
>	Conversion of brittle paper materials into digital products by working 
	with the best available guidelines to implement good preservation
	practices.  They will make this digital material available on the web.
>	Development of a program to preserve USDA digital materials (i.e., 
	materials that are born digitally).

In addition, Frangakis is also involved in a national effort to preserve 
agricultural literature.  The U.S. Department of Agriculture's Digital 
Publications Preservation Steering Committee was established in 1998 to oversee 
the implementation of the plan, A Framework for the Preservation of and 
Permanent Public Access to USDA Digital Publications. This group met for the 
first time in October 1998.  The plan may serve as a model that other agencies 
or institutions can adapt.  USDA is incorporating the following needs and 
considerations into its framework: 
>	Inventory and life cycle management, 
>	Technical requirements, and 
>	User access and retrieval. 

USDA is moving ahead to implement the plan.  The USDA CIO accepted the report, 
and under Frangakis' guidance, NAL established a national steering committee 
made up of representatives from USDA and from agribusiness, the research library 
community, the U.S. Agricultural Information Network (USAIN), Federal partners, 
etc.  The group will meet on a quarterly basis for the first 2 years.  They will 
establish test groups to explore the technical and funding issues.  They are 
hoping to secure funding for a pilot project to test the framework on an agency 
within USDA to see how manageable it will be for full-scale implementation (see 
Appendix H for a detailed description of the Framework).  

Finally, GPO has established partnerships with several depository libraries and 
Federal agencies to provide permanent public access to remotely accessible 
electronic Government information products.  Three such partnerships include: 
>	Partnership with the University of Illinois at Chicago's Richard J.  Daley 
	Library and the U.S. Department of State (DOS) to provide permanent access
	to remotely accessible electronic DOS information products.  
>	An Online Computer Library Center/GPO pilot project with the U.S. 
	Department of Education/National Library of Education (NLE) provided free
	public access through the FDLP to remotely accessible electronic
	Educational Resources Information Center (ERIC) documents.
>	A project with the Department of Energy (DOE) Office of Scientific and 
	Technical Information (OSTI) to provide public and depository library
	access to DOE technical reports in image format via the web service called
	"DOE Information Bridge" (Aldrich, 1998). 

Preservation specialists Smith and Frangakis noted that technology is not the 
biggest barrier to permanent access and preservation; the human infrastructure 
is not in place yet that would ensure permanent access and preservation 
telephone interview with Smith and Frangakis, Appendix H).  The plans and 
initiatives described here, coupled with the recommendations for training, 
policy integration, and support for best practices to implement policies are a 
few of the strategic actions that appropriate agencies, libraries, and 
institutions should undertake to ensure that future generations will have 
unrestricted, no-fee access to Government information in all formats.




Next Steps 

As a followup effort, NCLIS indicated that they will use these findings as a 
point of departure and analyze them in greater depth.  It is expected that
this followup effort will result in broad conclusions and recommendations to the 
President and Congress about how the problems and challenges revealed in this 
study can be constructively addressed.





                             Bibliography

Books and Articles

Achenrach, Joel.  1999.   The too-much information age:  Today's glut 
jams libraries and lives, but is anyone wiser


?  Washington Post, March 
15, p.  A01.

Adler, Prudence S.  1996.   Federal information dissemination policies 
and practices: One perspective on managing the transition.  Journal of 
Government Information 23, 4:435-441.  

Adler, Prudence S.  1998.   The times they are a changin' for our 
depository libraries.   Managing Technology, The Journal of Academic 
Librarianship, September.  

Aldrich, Duncan.  1998.   Partners on the Net: FDLP partnering to 
coordinate remote access to Internet- based government information.   
Government Information Quarterly 15, 1:27- 38.  [This issue of 
Government Information Quarterly is devoted entirely to a symposium on 
Federal depository libraries.  The issue was edited by John A.  Shuler 
and Gary Cornwell.] 

Beachboard, John C.  1997.   Assessing the Information Technology 
Management Reform Act from a bureau's perspective.   Government 
Information Quarterly 14, 3:  291-311.  

Bertot, John Carlo.  1997.   The impact of federal IRM on agency 
missions: Findings, issues, and recommendations."  Government 
Information Quarterly 14, 3:235- 253.  

Bertot, John Carlo, and Charles R.  McClure.   1997.  Key issues 
affecting the development of federal IRM: A view from the trenches.   
Government Information Quarterly 14, 3:271- 290.  

Bertot, John Carlo, Charles R.  McClure, William E.  Moen, and Jeffrey 
Rubin.  1997.  Web usage statistics: Measurement issues and analytical 
techniques.   Government Information Quarterly 14, 4:373- 395.  

Bryan, Martin.  1998.  An introduction to the extensive markup language 
(XML).   Bulletin of the American Society for Information Science 25, 
1:11-14.  

Culshaw, Stuart.  1998.   SGML, HTML, and XML: Sorting out the puzzle.   
Information Standards Quarterly 10, 2:6-8.  

Depository Administration Branch, Library Division.  Library Programs 
Service.   1997.  List of classes of United States Government 
publications available for selection by depository libraries.   
Washington, DC: U.S. Government Printing Office.    

Dugan, Robert E., and Ellen M.  

Dodsworth.   1994.  Costing out a depository library: What free 
government information?  Government Information Quarterly 11, 3:  285-
300.  

General Accounting Office.   1988.  Federal information: Agency needs 
and practices.   GAO/GGD- 88_115FS.   Washington, DC:  Government 
Accounting Office.  

Gorman, G.  E., and R.  H.  Miller.  1997.   Collection management for 
the 21st century: A handbook for librarians.   Westport, CT: Greenwood 
Press.  

Hernon, Peter.  1994.   Information life cycle: Its place in the 
management of U.S. Government information resources.   Government 
Information Quarterly 11, 2:143- 170.  

Hernon, Peter.  1994.   Discussion forum: A time of change.   
Government Information Quarterly 11, 2:137- 142.  

Hernon, Peter.  1994.   Information life cycle: Its place in the 
management of government information resources.   Government 
Information Quarterly 2:143-170.  

Hernon, Peter, and Charles R.  McClure.  1993.   Electronic U.S. 
government information: Policy issues and directions.  In Annual Review 
of Information Science and Technology, vol.  28, edited by Martha E.  
Williams.   Medford, NJ:  American Society for Information Science.  

Marcum, Deanna B.  1996.   The preservation of digital information.   
The Journal of Academic Librarianship 22, 6:451-454.  

McClure, David L.  1997.   Improving Federal performance in the 
information era: The Information Technology Management Reform Act of 
1996.   Government Information Quarterly 14, 3:255- 269.  

Okay, John, and Roxanne Williams.  1993.   Interagency workshop on 
public access: A summary for historical purposes.   Government 
Information Quarterly 10, 2:237- 253.  

Radack, Shirley M.  1994.   The Federal Government and information 
technology standards: Building the national information 
infrastructure."  Government Information Quarterly 11, 4:373- 386.  

Ryan, J., Charles R.  McClure, and R.  T.  Wigand.  1994.   Federal 
information resources management: New challenges for the nineties.   
Government Information Quarterly 11, 3:301- 314.  

Ryan, Susan M.  1996.   Downloading democracy: Government Information 
in an electronic age.   Cresskill, NJ: Hampton Press, Inc.    

Schwartz, Candy, and Mark Rorvig, eds.  1997.  Digital Collections: 
Implications for users, funders, developers and maintainers.   
Proceedings for the 60th American Society for Information Science 
Annual Meeting, 1997, vol.  34.   Washington, DC,  November 1-6.  
Medford, NJ: Information Today, Inc.  

Tennant, Roy.  1999.   Beyond GIF and JPEG: New digital image 
technologies.   Library Journal (February): 111-112.  

Turock, Betty J., and Carol C.  Henderson.  1996.   A model for a new 
approach to federal government information access and dissemination.   
Journal of Government Information 23, 3:227-240.  

Uhlir, Paul.  1997.   Framework for the preservation of and permanent 
public access to USDA digital publications.   Washington, DC:  National 
Academy of Sciences, National Research Council.  

U.S. Code 44: 1901-1916 Title 44-Public Printing and Documents.  
Chapter 19, Depository Library Program.  

U.S. Congress.  1988.   Informing the nation: Federal information 
dissemination in an electronic age.  LCCN 88-600567.   Washington, DC: 
Office of Technology Assessment.  

U.S. Congress.   House of Representatives.   Congressional Record, 
House.   H11245.  October 19, 1998.  

U.S. Department of Commerce.   National Telecommunications and 
Information Administration.   United States Advisory Council on the 
National Information Infrastructure.  1996.   A nation of opportunity: 
Realizing the promise of the information superhighway.   Washington, 
DC: U.S. Government Printing Office.  

U.S. Government Printing Office.  1995.   Biennial report to Congress 
on the status of GPO access: A service of the U.S. Government Printing 
Office.  Washington, DC:  U.S. Government Printing Office.  

U.S. Government Printing Office.  1996.   Report to the Congress.  
Study to identify measures necessary for a successful transition to a 
more electronic Federal Depository Library Program.   As Required by 
Legislative Branch Appropriations Act, 1996 Public Law 104-53.  
Washington, DC: U.S. Government Printing Office.  

U.S. Government Printing Office.   Superintendent of Documents.  
Library Programs Services.   1998.  Managing the FDLP electronic 
collection.   Washington, DC:  U.S. Government Printing Office.    





                              Web Sites 


Acrobat in Action.  The Acrobat info source on the web.  
http://www.purepdf.com/action/gov.html.   Accessed February 24, 1999.

Guidelines for Accessible Web Pages.   Microsoft, accessibility and 
disabilities.  For developers, writers and designers:  For web page 
designers.  http://www.microsof t.com/enable/dev/we b_guidelines.htm.   
Accessed February 24, 1999.  

Introduction to Accessible Web Pages.   Microsoft, accessibility and 
disabilities.  For developers and authors: For web page designers.  
http://www.microsof t.com/enable/dev/we b_intro.htm#tools.   Accessed 
February 24, 1999.  

McClure, Charles.  Guidelines for electronic records management on 
state and federal agency websites.  In Analysis and development of 
model quality guidelines for electronic records management on state and 
federal websites.  Chapter 6.   http://istweb.syr.edu/
~mcclure/nhprc/nhpr c_chpt_6.html.   Accessed February 24, 1999.   

National Academy of Science's Computer Science and Telecommunications 
Board (CSTB).  Review in which CSTB developed a detailed statement of 
work that defined the data collection process required to conduct the 
assessment.  http://www.nclis.gov/info/gpo1.html.   Accessed February 
24, 1999.  

National Archives and Records Administration Guidelines for Digitizing 
Archival Materials for Electronic Access.   http://www.nara.gov/ nara/vision/eap/eaps pec.html.  Accessed February 24, 1999.  

National Commission on Libraries and Information Science.   1998.  
National Commission on Libraries and Information Science, government 
information product assessment questionnaire.   http://www.nclis.gov
/news/nclisqux.pdf.   Accessed March 15, 1999.  

Ockerbloom, John.  1998.   Mediating among diverse data formats.   
Thesis defense.   http://www.cs.cmu.e du/afs/cs.cmu.edu/us 
er/spok/www/defens e/index.html.   Accessed March 15, 1999.    

Rothenberg, Jeff.  1998.   Avoiding technological quicksand: Finding a 
viable technical foundation for digital preservation.   
http://www.clir.org/p ubs/reports/rothenbe rg/contents.html.   Accessed 
March 15, 1999.  

The Federal Web Locator.   This site is a service provided by the 
Villanova Center for Information Laws and Policy and is intended to be 
the one site to locate federal government information on the World Wide 
Web.   http://www.law.vill.e du/fed- agency/fedwebloc.ht ml.  Accessed 
February 24, 1999.  

U.S. Department of Education Guidelines.  (The Federal Consortium 
Guidelines are based on the guidelines set by the Department of 
Education.)  http:/www.ed.gov/int ernal/wwwstds.html.   Accessed 
February 24, 1999.  

U.S. Department of Health and Human Services.   Information resources 
management policy.   http://www.hhs.gov/policy/irm-pol.html.   
Accessed February 24, 1999.  

U.S. Department of Health and Human Services.   World Wide Web 
applications and the Internet.  Best practices and guidelines.   
http://www.hhs.gov/ progorg/oirm/bestgui d.html.  Version as of May 26, 
1998.  U.S. Environmental Protection Agency.   Information resources 
management (IRM) policy, standards, guidelines and planning documents.   
http://www.epa.gov/i rmpoli8/.  Accessed February 24, 1999.   

U.S. Government Printing Office, Library Programs Service.   Managing 
the FDLP electronic collection:  A policy and planning document.   
October 1, 1998.   http://www.access.g po.gov/su_docs/dpos / 
ecplan.html.   Accessed March 25, 1999.  

U.S. Government Printing Office.  1996.  Final report to Congress:  
Study to identify measures for a successful transition to a more 
electronic Federal Depository Library Program.   http://www.access.g 
po.gov/su_docs/dpos /fdlppubs.html#4.   Accessed March 15, 1999.  

Uncle Sam Migrating Government Publications.   Government Publication 
Department.   Regional Depository Library, University of Memphis.  Last 
updated September 2, 1998.  http://www.lib.mem phis.edu/gpo/mig.ht m.  
Accessed February 24, 1999.    WWW Federal Consortium Guidelines.  1996 
revised guidelines.   http://www.dtic.mil/ staff/cthomps/guideli nes/.  
Accessed February 24, 1999.    


----

Appendix A

Agency Study Coordinator Meetings Agenda


Assessment of Electronic Government Information Products 
Agency Study Coordinator Meetings
July 23, 1998, and August 4, 1998

The Benton Foundation
1634 I Street, N.W.
11th Floor
Washington, D.C.


Agenda

9:30 - 9:45	  Background and Welcome
	          Robert S. Willard, Executive Director, National Commission on 
                  Libraries and Information Science (NCLIS) Francis J. Buckley, 
Jr., 
                  Superintendent of Documents, Government Printing Office (GPO)

9:45 - 10:15      Objectives, Timetable, and Required Actions
	          Forest Woody Horton, NCLIS Study Manager

10:15 - 10:30     Coffee Break

10:30 - 12:15     Specific Data Requirements (question-by-question review of 
survey 
                  instrument using GPO Access as a specific product example)

	          Gil Baldwin, GPO
	          T.C. Evans, GPO
	          Ric Davis, GPO
	          Denise Glover, Westat
	          Elizabeth Farris, Westat
	          Steve Fischer, Westat

12:15	          Wrap-up and Closing
	          Forest Woody Horton

12:30	          Adjourn

-----

Appendix B

List of Agency Coordinators and Other Key Officials


                   Agency Coordinators and Other Key Officials

Agency                                             Agency Coordinator(s)

                             Legislative Branch

Library of Congress                                Nancy Davenport 
                                                   Maggie Smith

United States Congress                             Eric Peterson


                             Judicial Branch

Administrative Office of the U.S. Courts           Gloria Malkin

Supreme Court of the United States                 Shelley L. Dowling 
                                                   Wilma M. Grant
                             
                             Executive Branch

Department of Agriculture                          Betty Behal

Department of Commerce                             Vera Whisenton 
                                                   Cynthia Banicki

Department of Defense                              William Beyer 
                                                   Rick Silva

Department of Education                            Chiquitta Thomas 
                                                   Linda Tague

Department of Energy                               Karen Spence

Department of Health and Human Services            Deborah Burris 
                                                   Fred Wood

Department of the Interior                         Claude Christensen 
                                                   Bob Mehnert

Department of Justice                              Regina Byrd 
                                                   Chris Rudy 
                                                   Dennis Feldt

Department of Labor                                Maureen Hill
                                                   Deborah Klein

Department of State                                Colleen Hope 
                                                   Dan Clemmer

Department of Transportation                       Barbara Post 
                                                   Robert Zarnetske

Department of the Treasury                         Gladys Myatt 
                                                   Mike Conklin

Environmental Protection Agency                    Richard Huffine

Executive Office of the President                  Peter Weiss

General Services Administration                    Odessa Brown

National Aeronautics and Space Administration      Roland Ridgeway 

National Archives and Records Administration       Debra Leahy

Securities and Exchange Commission                 Bert Lee

Smithsonian Institution                            Robert Schelin

Social Security Administration                     Terry Hynes

----

Appendix C

List of Participating Agencies and Products Surveyed



                  Participating Agencies and Products Surveyed


                         Legislative Branch                               Survey 
                                                                          status 

Library of Congress

American Memory: Historical Collections                                       CM
Braille Books on Disk                                                         CM
Cataloger's Desktop                                                           CM
CD BLND                                                                       CM
Classification Plus                                                           CM
Country Studies                                                               CM
French Poster Art                                                             CM
THOMAS                                                                        CM
Thomas: A Century of Lawmaking for a New Nation                               CM
Thomas: Bill Summary and Status                                               CM
Thomas: Committee Reports                                                     CM
Thomas: Congressional Record                                                  CM
Thomas: Early Congressional Documents                                         CM
Thomas: Enactment of a Law                                                    CM
Thomas: Federalist Papers                                                     CM
Thomas: Floor Activities for the House & Senate                               CM
Thomas: House Committee Schedules                                             OS
Thomas: House Roll Call Votes                                                 CM 
Thomas: How Our Laws are Made                                                 CM
Thomas: Major Legislation                                                     CM
Thomas: Senate Roll Call Votes                                                CM
Thomas: Text of Bills                                                         CM


United States Congress

Congressional Bills                                                           CM
Congressional Directory                                                       CM
Congressional Record (bound permanent)                                        CM      
Congressional Record (daily)                                                  CM
Hearings, Reports, & Prints:  House Agriculture Committee                      *
Hearings, Reports, & Prints:  House Appropriations Committee                   *
Hearings, Reports, & Prints:  House Education and Workforce Committee          *
Hearings, Reports, & Prints:  House Government Reform and Oversight Committee  *
Hearings, Reports, & Prints:  House Judiciary Committee                        *
Hearings, Reports, & Prints:  Joint Committee on Taxation                     CM
Hearings, Reports, & Prints:  Senate Appropriations Committee                 CM
Hearings, Reports, & Prints:  Senate Armed Services Committee                 CM
Hearings, Reports, & Prints:  Senate Commerce Committee                        *
Hearings, Reports, & Prints:  Senate Foreign Relations Committee               *
Hearings, Reports, & Prints:  Senate Judiciary Committee                      CM
Hearings, Reports, & Prints:  Rules and Administration Committee               *
Serial Set                                                                     *
United States Code                                                            CM


                         Judicial Branch


Administrative Office of the U.S. Courts     
     
Federal Rules of Civil Procedures                                             CM
Judicial Business of the United States Courts                                 CM
The Official Bankruptcy Forms Collection                                      CM
The Third Branch                                                              CM
Understanding the Federal Courts                                              CM


Supreme Court of the United States

Bench Opinions of the Supreme Court                                           CM
Rules of the Supreme Court                                                    CM
Slip Opinions of the Supreme Court                                            CM
U.S. Reports                                                                  CM


                        Executive Branch


Department of Agriculture

Agent Orange                                                                  CM
AgExporter                                                                    CM
AGRICOLA Database                                                              *
Agricultural Prices, Monthly                                                  CM
Agricultural Statistics                                                       CM
Aquaculture                                                                   CM
Continuing Survey of Food Intake By Individuals                               CM
Crops County Data                                                             CM 
Fact Sheets                                                                   CM
FAS Hot Country Pages                                                         CM
Fire Effects Information System                                               CM
Forest Land Distribution Data for the U. S.                                   CM
Leaflets                                                                      CM
Market News                                                                   CF
NAFTA Agricultural Fact Sheets                                                CM
Ornamental Horticultural Multimedia Project                                   CM
Outlook for U.S. Agricultural Exports                                         CM
Statistical Bulletins                                                         CM
The Plant Genome Database Collaboration                                       CM
Wildland Fire Assessment System                                               CM
World Agricultural Supply and Demand Estimates                                 *


Department of Commerce

AgroBase Database                                                              *
Cen-Data                                                                      OS
CenStats                                                                       *
Census of Population and Housing                                               *
Commerce Business Daily/CBDNet                                                CM
County and City Data Book                                                     CM
Current Population Reports                                                     *
Economic Census Reports                                                        *
Federal Research in Progress Database                                          *
Geophysics of North America                                                    *
GOV.Research-Center                                                            *
Imports/Exports CD                                                            CM
Local Climatalogical Data (for states)                                        CM
National Trade Data Bank                                                      CM
NTIS Database                                                                  *
Regional Economic Information System (REIS)                                   CM
Solar Radio Bursts                                                             *
Statistical Abstract of the United States                                     CM
STAT-USA/Internet/State of the Nation                                         CM
STAT-USA Newsletter                                                           CM
Survey of Current Business                                                    CM
TIGER/Line                                                                    CM
U.S. Industry and Trade Outlook                                               CM
USA Counties                                                                  CM
Zip Code Area CD-ROM                                                          CM


Department of Defense

Airman Magazine                                                               CM
Airpower Journal                                                              CM
All Hands                                                                      *
Defense Logistics Agency Publishing System                                    CM
Joint Electronic Library                                                      CM
Marines Magazine                                                              CM
Notice to Mariners                                                            CM
Soldiers Magazine                                                             CM
Technical Manuals                                                             CM


Department of Education

Condition of Education                                                        CM
Digest of Education Statistics                                                CM
Disability Statistics Abstract                                                CM
Disability Statistics Report                                                  CM
EDsearch, Education Statistics on Disk                                        CM
ERIC Database                                                                 CM
Guide to Education Programs                                                   CM
Helping Your Child (series)                                                   CM
NAEP (Nat'l Assessment of Educational Progress) Mathematics                   CM
National Education Goals Report                                               CM
Projections of Education Statistics                                           CM
Resources in Education                                                        CM
Student Guide                                                                 CM
Think College? Me? Now?                                                       CM


Department of Energy

Country Analysis Briefs (CABs)                                                 *
DOE Directives                                                                 *
DOE Information Bridge                                                        CM
DOE R&D Project                                                               CM
Electric Power Annual                                                         CM
Electric Power Monthly                                                        CM 
Emissions of Greenhouse Gases in the U.S.                                      *
Energy Consumption & Carbon Emissions by Region                                *
Energy Files                                                                  CM
Energy InfoDisc                                                               CM
Energy Science and Technology Database                                        CM
International Energy Outlook                                                  CM
Monthly Energy Outlook                                                         *
National Education Goals Report                                                *
Petroleum Supply Annual                                                       CM
State Energy Data Reports                                                     CM
State Energy Data System                                                      CM
Weekly Petroleum Status Report                                                CM
World Energy Consumption                                                       *


Department of Health and Human Services

Annual Pesticide Residue Monitoring Report                                    CM
Biosafety in Microbiological and Biomedical Laboratories                      CM
CDC Wonder                                                                    CM
Clinical Preventive Services                                                  CM
EHP (Environmental Health Perspectives)                                       CM
FDA Almanac                                                                   CM
FDA Compliance Policy Guide                                                   CM
HCFA Health Watch                                                             CM
HCFA's Laws, Regulations, Manuals                                              *
Healthcare Cost & Utilization Project (HCUP-3)                                CM
Medicare and You                                                              CM
Medicare Compare                                                              CM
Morbitity & Mortality Weekly report                                           CM
National Health Interview Survey, State Data Files                            CM
NIOSHTIC Database                                                             CM
PubMed                                                                        CM
Research Activities (Monthly Newsletter)                                      CM
RTECS Database                                                                CM
SETS: Statistical Export & Tabulation System                                  CM
Vital and Health Statistics (Rainbow Series)                                  CM


Department of the Interior

Contaminant Information Mgt & Analysis System                                 CM
Endangered Species Bulletin                                                   CM
Geographic Names Information System                                           CM
Metal Industry Indicators (MII)                                               CM    
Mineral Industry Surveys (monthly & quarterly)                                CM
Minerals Yearbook                                                             CM
National Survey of Fishing, Hunting & Wildlife &                               *
National Wildlife Refuge System Profiles Database                             CM
New Publications of the Geological Survey                                     CM
NPS Statistical Abstract                                                      CM
Preservation Briefs (numbered series)                                          *
The National Register Information System (NRIS)                               CM
Water Resources Abstracts                                                     CM

                       
Department of Justice

Correctional Populations of the United States                                 CF
FBI Law Enforcement Bulletin                                                  CM
FOIA Annual Report                                                            CM
Justice Information Center Publications                                       CM
NCJRS Abstracts Database                                                      CM
Prison and Jail Inmates                                                       CF     
Sourcebook of Criminal Justice Statistics                                     CF
Uniform Crime Reports: Crime in the United States                             CM


Department of Labor

General Wage Determinations Issued Under Davis-Bacon                           *
Monthly Labor Review                                                          CM
Occupational Outlook Handbook                                                 CM
OSHA Documents and Files                                                       *


Department of State

Background Notes                                                              CM
Diplomatic List                                                                *
Dispatch                                                                      CM
Key Officers of Foreign Service Posts                                          *
Maximum Travel Per Diem Allowance for Foreign Area                             *
Medical Information for Americans Traveling Abroad                            CF
Treaties and Other International Acts Series                                   *


Department of Transportation

FAA Statistical Handbook                                                      CM
Highway Statistics                                                            CM
National Transit Database                                                     CM
National Transportation Statistics                                            CM
Nationwide Personal Transportation Survey                                     CM
Rail Waybill Data                                                             CM
Transportation Expressions                                                    CM
Transportation Statistics Annual Report                                       CM
Worldwide Transportation Directory                                            CM


Department of Treasury

A Visitors Guide                                                              OS
Arson Investigations Guide, ATF P 2220.1                                       *
Bonded Warehouse-Manual for Proprietors, Importers & Customs Officers         CM
Business Taxpayers Information Publications, Pub 1194.B                       CM
Buying Treasury Securities, PD P 0009                                          *
Counterfeiting & Forgery                                                       *
Customs Valuation Encyclopedia, 1980-1997                                      *
Daily Treasury Statements                                                     CM
Federal Tax Forms                                                             CM
Firearms State Laws & Published Ordinances, ATF P 5300.5                      CM
Importing a Car (Pub 520)                                                     CM
Know Your Money                                                                *
Payment of Tax by EFT, ATF P 5000.10                                          CM
Quarterly Journal (Online)                                                    CF
Tables for Redemption Values for US Series E Savings Bonds & Savings Notes    CM
Tables of Redemption Values for US Series EE Savings Bonds                    CM
Tax Guide for Small Businesses, Pub 334                                        *
The History of Money                                                          OS
The Money Story                                                               CF
Treasury Bulletin                                                              *
U.S. Mint Gift Collection                                                     CF
Your Federal Income Tax, Pub 17                                               CM 
Your Guide to Federal Firearms Regulations, ATF P 5300.4                       *


Environmental Protection Agency

Center for Environmental Info. & Statistics                                   CM
Consumer Handbook for Reducing Solid Waste                                    CM
Envirofacts Warehouse                                                         CM
EPA Online Library System (OLS)                                               CP
Federal Register Environmental Subset                                         CM
Guide to Environmental Issues                                                 CM
Integrated Risk Information System Database (IRIS)                            CP
National Air Quality and Emission Trends Report                               CM
National Environmental Publications Information                               CM
National Water Quality Inventory: Biennial Report to Congress (305b report)   CM
Recycle City                                                                  CM
Sector Facility Indexing Project                                              CM
Setting the Record Straight: Secondhand Smoke is a Preventable Health Risk    CM
Superfund Hazardous Waste Site Query (CERCLIS Data)                           CM
Surf Your Watershed                                                           CM            
Test Methods for Evaluating Solid Waste: Physical/Chemical Methods (SW-846)   CM


Executive Office of the President

Art in the White House -- A Nation's Pride                                     *
Best Kept Secrets in Government                                               CM
Budget of the United States                                                    *
FBIS Publications Reports                                                      *
From Red Tape to Results: Creating a Government &                             CM
Interactive Citizens' Handbook                                                 *
NAICS Manual                                                                   *
OMB Circulars                                                                  *
Putting Customers 1st, Serving the American Public                            CM
Reinvention Express                                                           CM
The White House for Kids                                                       *
Virtual Library                                                                *
White House Briefing Room                                                      *
White House History  (web)                                                     *
White House Tour (CD-ROM)                                                      *
World Factbook                                                                CM
World News Connection                                                          *


General Services Administration

Catalog of Federal Domestic Assistance                                        CM
Consumer Information Catalog                                                  CF
Consumer Information Series                                                   OS
Consumer Resource Handbook                                                    CM
Federal Acquisition Regulation                                                 *
Government Registration Service                                               CM
MarkeTips                                                                      *
U.S. Gold                                                                     CM
U.S. Government TDD/TTY Directory                                             CM
United States Online Directories                                               *
US Government Blue Pages Online Directory                                     CM
US Real Property Sales List (online title varies)                             CM


National Aeronautics and Space Administration

Aerospace Medicine and Biology                                                CM
CASI Technical Report Server                                                  CM
NASA Thesaurus                                                                CM
NASA Video Catalog                                                            CM
Patent Abstracts                                                              CM
Scientific, Technical Aerospace Reports                                       CM


National Archives and Records Administration

Code of Federal Regulations                                                   CM
Digital Classroom                                                             CM
Emerging Nation                                                               CM
Federal Register                                                              CM
National Archives Information Locator (NAIL)                                  CM
Online Exhibit Hall                                                           CM
Privacy Act Issuances                                                         CM
Public Laws (slip laws)                                                       CM
United States Government Manual                                               CM
Weekly Compilation of Presidential Documents                                  CM


Securities and Exchange Commission

Administrative Proceedings                                                    CM
Cold Calling                                                                   *
Commission Legal Briefs                                                        *
Completed Initiatives of Interest to Small Business                            *
Directory of Companies Filing Annual Reports                                   *
EDGAR Database of Corporate Information                                        *
Financial Facts Tool Kit                                                       *
Investment Fraud & Abuse Travel to Cyberspace                                  *
Litigation Actions and Proceedings Bulletin                                   OS
Litigation Releases                                                           CM
Official List of Section 13 (f) Securities                                     *
Official Summary of Security Transactions & Holdings                           *
Pending Initiatives of Interest to Small Business                              *
Plain English Handbook: How to Create Clear SEC                                *
SEC Annual Report                                                              *
SEC Concept and Interpretative Releases                                       CM
SEC Decisions                                                                 CM
SEC Docket                                                                    OS
SEC Final Rules                                                               CM
SEC News Digest (daily)                                                       CM
SEC Opinions                                                                  CM
SEC Proposed Rules                                                            CM
SEC Special Studies                                                           CM
Staff Accounting Bulletins                                                    CM
Trading Suspensions                                                           CM


Smithsonian Institution

Annals of the Smithsonian Institution                                         CM
Anthology of American Folk Music                                              CM
Gallery Exhibition Catalogues                                                  *
SIRIS: SI Research Information System                                         CM
Smithsonian (magazine)                                                         *
Smithsonian Contributions to Anthropology                                     CM
Smithsonian Contributions to Botany                                           CM  
Smithsonian Contributions to Earth Sciences                                   CM
Smithsonian Contributions to Marine Sciences                                  CM
Smithsonian Contributions to Paleobiology                                     CM
Smithsonian Contributions to Zoology                                          CM
Smithsonian Folklife Festival: Culture of, by & . . .                          *
Smithsonian Folklife Studies Series                                           CM
Smithsonian Institute Research Reports                                         *
Smithsonian Photographs Online                                                 *
Smithsonian Studies in Air and Space                                          CM
Smithsonian Year                                                               *


Social Security Administration

Documents Published by the SSA Historian                                      CM
Publications Information Pamphlets and Fax Sheets                             CM
Request a Personal Earnings & Benefits System                                 CM
SSA Publications on CD-ROM                                                    CM

----

Appendix D

Coordinator and Respondent Cover Letters

October 1998


Dear Agency Coordinator:
On behalf of the National Commission on Libraries and Information Science 
(NCLIS) and the Government Printing Office, we would like to thank you for 
agreeing to distribute this survey of Government Electronic  Information 
Products to agency respondents.

The purpose of this assessment is to:  (1) identify medium and format standards 
that are the most appropriate for permanent public access, (2) assess the cost-
effectiveness and usefulness of alternative medium and format standards, and (3) 
identify public and private medium and format standards that are or could be 
used for products throughout their entire information life cycle.  The broader
objective of the survey is to study the long-term impacts of shifting
government information products from paper and microform mediums to federal
agency web sites and other electronic mediums.  We want to ensure that as more
government information is available in a variety of electronic mediums and 
formats, the American public continues to have free and easy local access to
this information through the Federal Depository Library Program.  

We are requesting that you, as the agency coordinator, promptly distribute the 
questionnaires to the appropriate personnel in your agency.  If you have any 
questions about this survey, please call Denise Glover at Westat: 301-251-2269 
or toll-free at 800-937-8281, ext. 2269.  We will send you a copy of the final 
report once it is completed. 

The following suggested procedures and instructions will assist you to 
successfully distribute the product questionnaires.  However, some agency 
coordinators have decided to use different data collection procedures.  If you 
are one of those coordinators, it is essential that you explain your procedures 
to your respondents.

1.  All agency coordinators will receive a full packet from Westat on September 
29 or shortly thereafter that contains a cover letter and an information copy of 
the questionnaire.  The coordinator's packet will also include a sealed packet 
for each product.  Please make sure you have a packet for each product on your 
final list of product selections.  If you are missing information or have 
incorrect information, please call Debbie Alexander at Westat: 301-294-2088, or 
toll-free at 800-937-8281, ext. 2088.

2.  The product packet includes a cover letter to the respondent, a product 
questionnaire, a glossary of terms, and a postage-paid return envelope.  

3.  Each product questionnaire contains the following preprinted information:  a 
five-digit ID, the agency's name and submit, and the name of the product.

4.  Upon receipt of the packet, please distribute the product packets to the 
appropriate product respondents in your agency.

5.  Due to the specialized nature of many of the questions asked on the survey, 
it may be necessary for product respondents to consult with other agency 
personnel such as records managers, information technology staff, planning 
offices, and others to complete the survey in its entirety.  We recommend that 
you ask product respondents to leave blank any questions they feel uncomfortable 
answering and to make arrangements for someone in their office or another office 
with more appropriate knowledge and expertise to answer those questions.  If 
product respondents decide to take this action, we strongly suggest that, if 
possible, you standardize the arrangements in advance and ensure that all agency 
respondents are aware of your procedures.  One purpose of standardizing 
arrangements is to ensure that product respondents do not lose control of a 
questionnaire because multiple individuals and offices are handling it.  

6.  Product respondents are responsible for ensuring that all questions are 
answered on their respective questionnaires, even if they must consult with 
other personnel.

7.  To ensure consistency and completeness, you might want to ask your assigned 
product respondents to allow you to review the surveys before they send them to 
Westat.  You might also request that respondents send you a copy of the 
completed questionnaire for your files.

8.  Product respondents should return completed questionnaires directly to 
Westat (not to you) in the enclosed postage-paid envelope.  If product 
questionnaires are not completed and returned to Westat by the October 30, 1998 
deadline, Westat will attempt to follow up directly with the designated product 
respondent.  However, if respondent information is unavailable or unknown, 
Westat will contact you for followup.  

Respondents are requested to return the survey by October 30, 1998 by mailing it 
in the postage-paid envelope to:

Denise Glover
TA 2064
Westat, Inc. 
1650 Research Boulevard 
Rockville, MD 20850

Thank you very much for your assistance and cooperation.  We appreciate the time 
and hard work you have invested in the coordination activities.

Sincerely,




Robert S. Willard                                Francis J. Buckley, Jr.
Executive Director                               Superintendent of Documents
National Commission on Libraries                 Government Printing Office
and Information Science 










                                                                   January, 1999


Dear Respondent:

On behalf of the National Commission on Libraries and Information Science 
(NCLIS) and the Government Printing Office, we are requesting that you complete 
this survey of Government Electronic Information Products.

The purpose of this assessment is to:  (1) identify medium and format standards 
that are the most appropriate for permanent public access, (2) assess the cost-
effectiveness and usefulness of alternative medium and format standards, and (3) 
identify public and private medium and format standards that are or could be 
used for products throughout their entire information life cycle.  The broader 
objective of the survey is to study the long-term impacts of shifting government 
information products from paper and microform mediums to federal agency web 
sites and other electronic mediums.  We want to ensure that as more government 
information is available in a variety of electronic mediums and formats, the 
American public continues to have free and easy local access to this information 
through the Federal Depository Library Program.

We recognize that many respondents will complete surveys for more than one 
agency product.  We also understand the burden this imposes upon you and 
appreciate the time and effort you will devote to completing the survey.  
However, we want to underscore the importance of your participation.  We will 
send you a copy of the report summarizing the data from these surveys once it is 
completed.  

Please carefully read all of the following suggested procedures and instructions 
that will assist you to successfully complete and return the product 
questionnaires.  However, some agency coordinators have decided to use different 
data collection procedures.  You should contact your coordinator directly to 
find out if you are to follow the procedures outlined here.  If your coordinator 
has made other arrangements, please contact him/her to determine what they are.

1.  Once you receive your packet(s) from your agency coordinator, please check 
to make sure each packet contains the following information for each product you 
are to survey: a cover letter, questionnaire, glossary of terms, and postage-
paid return envelope.  If you feel you are missing information or have incorrect 
information, please first contact your agency coordinator. (Your coordinator's 
contact information is available on the NCLIS web site at www.nclis.gov.)  If 
your coordinator is unable to provide this information, call Debbie Alexander at 
Westat: 301-294-1088, or toll-free at 800-937-8281, ext. 2088. 

2.  Each product questionnaire contains the following preprinted information: a 
five-digit ID, the agency's name and subunit, and the name of the product.

3.  Due to the specialized nature of many of the questions asked on the survey, 
it may be necessary for you to consult with other agency personnel such as 
records managers, information technology staff, planning officers, and others to 
complete the survey in its entirety.  We suggest you leave blank the responses 
to questions you feel uncomfortable answering and then arrange for someone in 
your office or another office with more appropriate knowledge and expertise to 
answer those questions.  Your agency coordinator may have standardized those 
arrangements in advance and made you are aware of their procedures.  The purpose 
of standardizing arrangements is to ensure that you do not lose control of a 
questionnaire because multiple individuals and offices are handling it. 

4.  Please note that you are responsible for ensuring that all questions are 
answered on your respective questionnaires, even if you must consult with other 
personnel.

5.  To ensure consistency and accuracy, coordinators might request that they be 
allowed to review the completed surveys before you return them to Westat. 

6.  Return completed questionnaires directly to Westat in the enclosed postage-
paid envelope by January 15, 1999.  Your coordinator may request that you also 
send him/her a copy of the completed questionnaire.  

7.  If you have any questions about this survey, please call Denise Glover at 
Westat: 301-251-2269, or toll-free at 800-937-8281, ext. 2269.

Please return the questionnaire by January 15, 1999 by mailing it to:

Denise Glover
TA 2064
Westat, Inc.
1650 Research Blvd. 
Rockville, MD 20850

Thank you very much for your assistance and cooperation. 

Sincerely,




Robert S. Willard                            Francis J. Buckley, Jr.
Executive Director                           Superintendent of Documents
National Commission on Libraries             Government Printing Office
and Information Science 

----

Appendix E

Questionnaire and Glossary of Terms




             NATIONAL COMMISSION ON LIBRARIES AND INFORMATION SCIENCE
                              GOVERNMENT INFORMATION 
                         PRODUCT ASSESSMENT QUESTIONNAIRE

Directions:  This form is to be completed only for products that are either 
already in electronic mediums or products that are to be migrated to electronic 
mediums.  Do not complete this form for products that will remain in paper or 
microform mediums only.  A product is defined as "a Government publication or 
other work of the United States Government conveyed in a tangible physical 
medium such as a book, CD-ROM, etc., or disseminated through an electronic 
Government information service and intended for public dissemination."  (See 
enclosed glossary for definitions of terms used throughout this questionnaire.)  
Complete one questionnaire for each product.  (Please note that a Website is not 
considered a product, although products might be on a Website.)


A.  GENERAL INFORMATION
1.  Agency Name: ________________________________________________
    Sub-Unit: ___________________________________________________
2.  Name of Product: ____________________________________________
3.  Brief Description of Product: _______________________________

4.  URL for Product Website: ____________________________________
    __ Check if no Website

B.  CURRENT PRODUCT PROFILE  
5.  How is this product used by the end user?  (Check all that apply.)
a.  Information access and retrieval                             __
b.  Data analysis (e.g., to support analysis by end user)        __
c.  Other (specify)                                              __



6.  What types of data are contained within this product?  

In Column A, indicate which type of data this product contains. Check all that 
apply.)

In Column B, indicate the primary type of data contained in this product. for 
example, if you checked boxes for items b and f in column A, indicate which of 
the two is the primary type of data by placing a check in the appropriate  box 
in column B.

Type of data              A. Type of data                B. Primary type of data
                             contained                      (Check only one)
                       (Check all that apply)

a.  Bibliographic data
b.  Graphical data (photos, charts, graphs, tables, drawings)
c.  Numerical data
d.  Sound
e.  Spatial data (maps, coordinate files)
f.  Textual data (books, serials, reports)
g.  Video
h.  Multimedia (sound, video, text, graphics)
i.  Other (specify)


7a.  In what mediums is this product publicly available? 

In Column A, indicate which type of medium is used.  (Check all that apply.) 

In Column B, for each medium used, indicate whether there is a medium standard 
that is mandated by the agency, a common practice in the agency (although not 
mandated), other (i.e., not agency-mandated standard or common agency practice, 
but new and promising and beginning to be used, etc.), OR none of these.

In Column C, for each type of medium used, check one box to indicate the primary 
type used.



7b.	If you checked "Other" category in Question 7a, Column B, please explain.

___________________________________________________________________

8a.  Which of the following types of formats does this product use? 

In Column A, indicate which type of format is used.  (Check all that apply.)
In Column B, for each format used, indicate whether there is a format standard 
that is mandated by the agency, a common practice in the agency although not 
mandated, other (i.e., not agency-mandated standard or common agency practice, 
but new and promising and beginning to be used, etc.), OR none of these.
In Column C, for each type of format used, check one box to indicate the primary 
type that is used within each major category (e.g., database, spreadsheet, word 
processing, etc.).



8b.  If you checked "Other" category in Question 8a, Column B, please explain.

___________________________________________________________________

9a.  Is this product in an online medium?			
     Yes	 __ (Continue with question 9b)	No	 __ (Skip to question 10a)

9b.  Which of the following online approaches are used? 

In Column A, indicate which type of approach is used.  (Check all that apply.)
In Column B, if the approach is used, indicate whether it is mandated by the 
agency, a common practice in the agency although not mandated, other (i.e., not 
agency-mandated standard or common agency practice, but new and promising and 
beginning to be used, etc.), OR none of these.



9c.	If you checked "Other" category in Question 9b, Column B, please explain.

___________________________________________________________________

Searchability of Product  
10a.	Please indicate whether this product is...  (Check all that apply.)

a.	Included as part of a full-text searchable database with no fielding __ 
b.	Indexed by full-text and field __ 
c.	Available as "view only" - non-searchable __ 
d.	Other (specify) __ 

10b.	This product is officially hosted by...  (Check all that apply.)  (Host 
refers to the primary site where the public can find the product.)
		
a.	Your agency __ 
b.	Another agency  (specify) __ 
c.	Contractor __ 
d.	Educational institution __ 
e.	Other (specify) __ 


Retrievability  
11. This product and any associated software... (Check all that apply.)

a.	Can be downloaded, saved, and is not subject to any restrictions on use or

re-use by the end user __ 
b.	Cannot be downloaded, saved, and/or re-used because it is part of a 
database and does not exist as a distinct product __ 
c.	Cannot be downloaded, saved, and/or re-used because it requires 
proprietary software that is not freely distributable __ 
d.	Other (specify) __ 


C.	PLANNED PRODUCT PROFILE  (This next section refers to future plans for the 
product.)

Type(s) of Data - Future Plans

12a.	Are there any plans to discontinue publication of this product? 
Yes __ (Continue with question 12b)
No __ (Skip to question 13a)

12b.	If yes, please explain.	 (Skip to Section D.)

13a.	What kind of data will this product contain?  If product contains more 
than one type of data, respond for all data types. (Check one.)

Retain existing type(s) of data, no changes planned __ (Skip to question 14a)
Retain existing type(s) of data and add items of one or more new types of 
data(specify) __ (Continue with question 13b)
Discontinue one or more types (specify) __________________________________ __ 
(Continue with question 13b)
Change to new type(s) of data (specify) __ (Continue with question 13b)

13b.	Change(s) will occur in the: (Check all that apply.)

	Short term:  within 1 year or less __ 
	Medium term:  within 2 to 5 years __ 
	No changes indicated __ (Skip to question 14a)

If you checked both "short term" and "medium term" in question 13b, continue 
with question 13c.  Otherwise, skip to question 13d.

13c.	If you have short-term and medium-term plans, how are your short-term 
plans different from your medium-term plans? For example... (Check all that 
apply.)

Short-term plans call for one type of data, but medium-term plans call for a 
different type of data.
(specify) ____________________________ 

In the medium term there are plans to combine resources to create a new version 
of a product using a different type of data that may be different from short-
term plans. 
(specify) ____________________________ 

Other (specify) (specify) ____________________________ 


13d.	Do you have any long-term plans (6 or more years) for changing this 
product to new types of data?  

Yes __ (Continue with question 13e)
No __ (Skip to question 14a)

13e.	If yes, please describe them here.
___________________________________________________________________
___________________________________________________________________
___________________________________________________________________


Medium - Future Plans
14a.	What kind of medium(s) will this product use?  If product is delivered in 
more than one medium, respond for all mediums. (Check one.)
	
Retain existing medium(s), no changes planned __ (Skip to question 15a)
Retain existing medium(s) and add items of one or more new types of mediums 
(specify) __ (Continue with question 14b)
Discontinue one or more types (specify) ____________________________  __  
(Continue with question 14b)
Change to new type(s) of medium(specify) __ (Continue with question 14b)
		
		
	No agency-mandated medium applies __ (Continue with question 14b)

14b.	Change(s) will occur in the: (Check all that apply.)

	Short term:  within 1 year or less __
	Medium term:  within 2 to 5 years __ 
	No changes indicated __ (Skip to question 15a)

If you checked both "short term" and "medium term" in question 14b, continue 
with question 14c.  Otherwise, skip to question 14d.


14c.	If you have short-term and medium-term plans, how are your short-term 
plans different than your medium-term plans?  For example... (Check all that 
apply.)
		
Short-term plans call for one type of medium, but the medium-term plans call for 
a different type of medium. (specify) ______________________________

In the medium term there are plans to combine resources to create a new version 
of a product in a medium that may be different from short-term plans. (specify) 
______________________________

	Other (specify) ______________________________

14d.	Do you have any long-term plans (6 or more years) for changing mediums for 
this product?  

Yes __ (Continue with question 14e)
No __ (Skip to question 15a)

14e.	If yes, please describe them here.
___________________________________________________________________
___________________________________________________________________
___________________________________________________________________

Format - Future Plans
15a.	What kind of format(s) will this product contain?  If the product uses 
more than one format, respond for all formats. (Check one.)

	Same as existing format(s), no changes planned __ (Skip to question 16a)
Retain existing format(s) and add one or more new format types(specify) __ 
(Continue with question )
___________________________________________________________________

	Change to new format type(s)  (specify) __ (Continue with question 15b)
___________________________________________________________________

	Discontinue one or more types (specify) __________________________________
	 __ (Continue with question 15b)
	No agency-mandated format applies __ (Skip to with question 16a)

15b.	Change(s) will occur in the: (Check all that apply.)
	Short term:  within 1 year or less __ 
	Medium term:  within 2 to 5 years __ 
	No changes indicated __ (Skip to question 16a)

If you checked both "short term" and "medium term" in question 15b, continue 
with question 15c.  Otherwise, skip to question 15d.


15c.	If you have short-term and medium-term plans, how are your short-term 
plans different from your medium-term plans?  For example... (Check all that 
apply.)
	
Short-term plans call for one kind of format, but the medium-term plans call for 
a different type of format.  (specify) __ 
___________________________________________________________________
In the medium term there are plans to combine resources to create a new version 
of a product using a different format that may be different from short-term 
plans. (specify) __ 
___________________________________________________________________
	Other (specify) __ 
___________________________________________________________________



15d.	Do you have any long-term plans (6 or more years) for changing formats for 
this product?  

Yes __ (Continue with question 15e)
No __ (Skip to question 16a)

15e.	If yes, please describe them here.
___________________________________________________________________
___________________________________________________________________
___________________________________________________________________


D. OTHER INFORMATION

Metadata

16a.	Is there a metadata record for this product (e.g., GILS, MARC)?

Yes __ (Continue with question 16b)
No __ (Skip to question 17a)

16b.	If yes, please specify	
___________________________________________________________________


Permanent Access
17a.	Permanent public access to this product is currently provided by:  (Check 
all that apply)

	Your agency __ (Continue to question 17b)
	Another agency (specify) __ (Continue to question 17b)
	Other (specify) __ (Continue to question 17b)
	No permanent public access provided __ (Skip to question 17c)

17b.	How is permanent public access provided?  (specify)	
___________________________________________________________________
(Go to question 18a)

17c.	Are there plans to provide permanent public access in the future for this 
product?

Yes __ 
No __ 


Permanent Retention

18a.	Is this product scheduled for permanent retention by the National Archives 
and Records Administration?

Yes __ (Continue with question 18b)
No __ (Skip to question 19a)

18b.	What is the planned retention period for this product?  (specify)	
___________________________________________________________________


Ensuring Authenticity

19a.	Does the agency ensure authenticity (official status determination) for 
this product?

Yes __ (Continue with question 19b)
No __ (Skip to question 20)

19b.	How does the agency attest to authenticity?  (specify)	
___________________________________________________________________

Updating/Upgrading Plans

20. How frequently is this product updated or refreshed?  (Check one.)

Daily __ 
	Weekly __ 
	Monthly __ 
	Annually __ 
	Other (specify) __ 

21a.	Are there plans for changing the product's supporting technology?

Yes __ (Continue with question 21b)
No __ (Skip to question 22a)

21b.	If yes, specify	
___________________________________________________________________

User Fees
22a.	Is a user fee charged for this product?  (Check one.)

Yes, for all users __ (Continue with question 22b)
Yes, for some users __ (Continue with question 22b)	
No		03  (Skip to question 23a)

22b.	If yes, explain and specify amount of fees.	
___________________________________________________________________

Licensing 

23a.	Has the agency licensed commercial search and retrieval software for use 
with this product?

Yes __ (Continue with question 23b)
No __ (Skip to question 24a)

23b.	Specify the software vendor and product name. 	
___________________________________________________________________

23c.	Does the agency's license cover use by...  (Check all that apply.)

a.	Agency personnel __ 
b.	Agency's primary target constituencies __ 
c.	Federal Depository Libraries __ 
d.	All libraries __ 
e.	Public users __ 
f.	Other __ 


Public Domain

24a.	Is this product in the public domain?  (Check one.)

Yes, for the entire product __ (Continue with question 25)
Yes, for part of product __ (Continue with question 24b)	
No __ (Skip to question 24c)

24b.	If yes for part of product, please explain.	
___________________________________________________________________


24c.	If no, has the agency entered into an arrangement with the private sector 
that would limit use of this information?  
(Please briefly explain the arrangement.)
___________________________________________________________________
___________________________________________________________________
___________________________________________________________________


E.  COMMENTS

25.  If you wish to comment on matters that you believe are not otherwise 
adequately covered in this survey, do so here.  
___________________________________________________________________
___________________________________________________________________
___________________________________________________________________
___________________________________________________________________
___________________________________________________________________
___________________________________________________________________
___________________________________________________________________

26.  Key person completing this form.
	Contact Name: ___________________________________________
	Title: __________________________________________________
	Telephone: ____________________ Fax: ____________________
	E-Mail: ____________________ 

Other person(s) providing responses to questions or assistance in completing 
this form.

27.	Contact Name: ____________________________________________

Title: ________________________________________________________
	Telephone: ____________________ Fax: ____________________
	E-Mail: ____________________ 

28.  Contact Name: ____________________________________________


Title: ________________________________________________________
	Telephone: ____________________ Fax: ____________________
	E-Mail: ____________________ 

THANK YOU.  PLEASE RETURN THIS SURVEY TO:

DENISE GLOVER
WESTAT, ROOM TA2064
1650 RESEARCH BOULEVARD
ROCKVILLE, MD  20850
FAX:  301-517-4134
PHONE: 301-251-2269
[email protected]

FOR QUESTIONS ABOUT COMPLETING THE SURVEY, CONTACT DENISE GLOVER.

PLEASE KEEP A COPY OF THIS SURVEY FOR YOUR RECORDS.




              Glossary of Terms, Abbreviations, and Acronyms for the 
              Government Information Product Assessment Questionnaire


ASCII-An acronym for American Standard Code for Information Exchange, ASCII is 
an international standard in which numbers, letters, punctuation marks, symbols, 
and control codes are assigned numbers from 0 to 127. 

AIFF-Short for Audio Interchange File Format, a common format for storing and 
transmitting sampled sound. 

ANSI-Acronym for the American National Standards Institute, a voluntary 
organization that creates standards for the computer industry.  In addition to 
programming languages, ANSI sets standards for a wide range of technical areas, 
from electrical specifications to communications protocols.

AU-Short for audio, a common format for sound files on UNIX machines. 

AVI-A format developed by Microsoft Corporation for storing video and audio 
information.  

Accessibility-The degree to which the public is able to retrieve or obtain 
Government information products, either through the Federal Depository Library 
Program (FDLP) or directly through an electronic information service established 
and maintained by a Government agency or its authorized agent.  The other 
aspects of accessibility include the degree to which these Government 
information products are available to the public in a useful format or medium 
and in a time frame in which the information has utility.  In the context of the 
FDLP, accessibility includes the degree to which Government information is 
accurately identified and described bibliographically, the information's 
availability is made known to the public, and technological, social, economic, 
political and physical barriers to gaining access are minimized.

Authentication-The certification of a Government information product attesting 
to its legitimate official status.  Ensuring the authenticity of a product 
involves product design, planning, and policy development, as well as technical 
considerations.

Availability-The degree to which information is physically or electronically 
obtainable through the intentional or unintentional provision of Government 
information products to the public. In the context of the FDLP, availability 
includes the measures taken by Government agencies and the FDLP to include 
Government information products in the program. Accessibility is meaningless if 
information is unobtainable from its source.

BBS-Bulletin Board System, an electronic message center.  Most bulletin boards 
serve specific interest groups.  They allow users to dial in with a modem, 
review messages left by others, and leave their own message. 

CGI Scripts-Abbreviation for Common Gateway Interface, a specification for 
transferring information between a World Wide Web server and a CGI program. CGI 
programs are the most common way for Web servers to interact dynamically with 
users. An increasingly common way to provide dynamic feedback for Web users is 
to include scripts or programs that run on the user's machine rather than the 
Web server.

CD-ROM-Compact Disk-Read Only Memory; an optical disk from which information may 
be read but not written.

DVD-Short for digital versatile disk or digital video disk, a new type of CD-ROM 
that holds a minimum of 4.7GB (gigabytes), enough for a full-length movie. 

Digital Image-An electronic version of a bit-mapped image of a document or other 
information format that allows text to be searched at the character level; 
"digitalize" means the process and accompanying technologies required to effect 
the conversion from bit-mapped (e.g., a fax) to searchable format.

Dissemination-The act of making Government information products accessible to 
the public through distribution to depository or program libraries or by using a 
Government electronic information service.

Electronic Government Information-Information that is organized, stored, and 
disseminated using electronic or optical mediums as opposed to paper-based or 
microfiche-based mediums.

FTP-An acronym for of File Transfer Protocol, the protocol (agreed-upon format) 
used on the Internet for sending files.

Format-The manner in which data, documents, or literature are organized, 
structured, named and described, classified, and arranged.  For example, full 
narrative texts in English language could be in the following forms: books or 
articles, abstracts of text used in reviews and summaries, indexes and catalogs, 
maps, photographs, drawings, sound recordings, video tapes, bibliographies, and 
statistical and other numeric kinds of tabulations.  A screen format is the 
layout of fields on the screen.  A report format is the layout of the printed 
page including print columns and page headers and footers.  A record format is 
the layout of fields within a record.  A file or database format is the layout 
of fields and records within a data file, layout codes within a word processing 
document, or display lists (vector) or bit maps (raster) within a graphics file.  
The term is sometimes used to refer to the way digital data is encoded or 
inscribed.  Archivists used the terms "genre" or "form of material" to cover 
what is meant by format in this context.

Frames-A feature supported by most modern Web browsers that enables the Web 
author to divide the browser display area into two or more sections (frames). 
The contents of each frame are taken from a different Web page. 

GIF-Pronounced jiff or giff (hard g), it is short for graphics interchange 
format, a bit-mapped graphics.  GIF supports color and various resolutions. It 
also includes data compression, making it especially effective for scanned 
photos.

GILS-Short for Government Information Locator Service.  A metadata tool for 
identifying, locating, and describing publicly available Federal information 
resources, including electronic information resources. 

Gopher-A method of making menus of material available over the Internet.  Gopher 
pre-dates the World Wide Web for organizing and displaying files on Internet 
servers. 

Government Electronic Information Service-The system or method by which a 
component of the Government, or its authorized agent, disseminates Government 
information products to the public via such means as a network or use of CD-ROMs 
at a kiosk.

Government Information-Refers to information, regardless of form, medium, or 
format, that is created or compiled by employees of a component of the 
Government, or at Government expense, or as required by law.  Government 
information as used here does not include information for official use only, 
information classified for reasons of national security, or information used 
strictly for administrative or operational purposes (e.g., not of public 
interest or educational value).

Government Information Product-A Government publication or other work of the 
United States Government conveyed in a tangible physical medium such as a book 
or CD-ROM, or disseminated through an electronic Government information service 
established and maintained by a Government agency or its authorized agent.

Graphical User Interface-A program interface that takes advantage of the 
computer's graphics capabilities to make the program easier to use. 

HTML (HyperText Markup Language)-The authoring language used to create documents 
on the World Wide Web. 


HTTP-Short for HyperText Transfer Protocol, the underlying protocol used by the 
World Wide Web.  HTTP defines how messages are formatted and transmitted and 
what actions Web servers and browsers should take in response to various 
commands. 

Information Intermediary-Refers to any person, institution, or mechanism that 
adds value to information products so that they are more useful to information 
users.  Intermediaries perform their work at the middle information life cycle 
stages-that is, between information creation and information disposition or 
destruction.  Federal libraries and information centers are examples of 
intermediaries.

Information Life Cycle-The various stages through which information passes, 
including creation, production or collection, review and editing, organization 
and reorganization, packaging, storage, search and retrieval, communication and 
re-communication, dissemination, disposition, archiving, and destruction.

JPEG-Joint Photographic Experts Group (JPEG) is an image compression format used 
to transfer color photographs and images over computer networks. 

Java Applets-The use of small Java programs.  Java Applets allows Web pages to 
include functions such as animations, calculators, and other fancy tricks. 

Javascript-A scripting language developed by Netscape to enable Web authors to 
design interactive sites.

MARC-Short for machine-reading cataloguing.  The USMARC formats are standards 
for the representation and communication of bibliographic and related 
information in machine-readable form.  The Library of Congress, in consultation 
with various user communities, maintains USMARC formats.

MPEG-Short for Moving Picture Experts Group, a working group of ISO.  MPEG 
generally produces high-quality video. 

Medium-The physical, chemical, or biological substrate used to create, organize, 
store, search for, retrieve, disseminate, or permanently archive data, 
documents, or literature including, for example, paper, microforms, fiber optic 
cables, photographic film, CD-ROM, floppy diskettes, magnetic storage devices,
 sound recordings, and videotape.

Metadata-Metadata is data about data.  Metadata describes how and when and by 
whom a particular set of data was collected and how the data are formatted.  
These data can be collections or individual instances of objects or documents, 
Internet resources, etc.

Migration-The transfer of an information product from one hardware type, 
software package, system, network, format, or medium to another.  The transfer 
of an information product from a pre-electronic medium such as paper or 
microform to an electronic medium such as CD-ROM is an example of migration.

PDF-Short for Portable Document Format.  A file format developed by Adobe 
Systems, PDF captures formatting information from a variety of desktop 
publishing applications, making it possible to send formatted documents and have 
them appear on the recipient's monitor or printer as they were intended.

Permanent Public Access-The indefinite, continuing accessibility of Government 
information products by the public including the policies, programs, formats, 
mediums, and standards used.  Simply because data such as weather and tide 
information, lists of Government contractors, editions of annual reports, and 
statistical series are continuously updated with more current information does 
not necessarily mean that provision should not be made for retaining (making 
permanently accessible) the replaced information.  Care should be exercised in 
this regard, however, to ensure no harm is done to the public by making 
available out-of-date information.

Preferred Medium or Format Standard-A medium or format standard that may not be 
agency mandated but is either common agency practice or applies to a new or 
promising product format or medium.

Public Domain-A term of American copyright law referring to works that are not 
copyright protected; free for all to use without permission. 

Refreshing-A technical term meaning the manner in which information mediums and 
technologies are periodically reassessed and upgraded to ensure that they are 
not becoming obsolete, thereby risking the loss of information and the 
compromising of Permanent Public Access policies. The term is also sometimes 
applied to formats and standards (not just mediums) that are in danger of 
becoming obsolete and need to be replaced or upgraded.

Rich Text Format (RTF)-A standard for specifying formatting of documents.  RTF 
files are actually ASCII files with special commands to indicate formatting 
information, such as fonts and margins.

SGML-Short for Standard Generalized Markup Language, SGML is a system for 
organizing and tagging elements of a document.  SGML was developed and 
standardized by the International Organization for Standards (ISO) in 1986. 

Standard-An agreed-upon authoritative convention, whether formal or informal, 
whether official or unofficial, whether de facto or de jure, by which 
information products are created, produced, formatted, published, stored, 
communicated, and moved through the remainder of the stages of the information 
life cycle.  Standards (and the more informal concept and term "guidelines") 
minimize incompatibility and interoperability problems when an information 
sender tries to move data, documents, or literature into and out of several 
different formats and mediums to a receiver.  Standards may be set by formal 
national or international standard-setting bodies, or by agencies, or by groups 
of users informally by common consent.

TIFF-The acronym for Tagged Image File Format, a graphic file format developed 
by Aldus and Microsoft.  

Telnet-A terminal emulation program for TCP/IP networks such as the Internet.  
The Telnet program runs on a user's computer and connects his/her PC to a server 
on the network.  This enables users to control the server and communicate with 
other servers on the network.

Type of Data-The general nature of the information content of a Government 
information product and how it is arranged, structured, and presented for ease 
of handling in a medium.  Some examples include textual data, graphical data, 
spatial data, numerical data, etc.

WAIS-Short for Wide Area Information Server. A distributed information service 
that offers simple natural language input, indexed searching for information 
retrieval, and a relevance feedback mechanism. It has an easy-to-use interface 
that searches all documents relative to your query, ranks them, and makes them 
available to retrieve.

WAV-The format for storing sound in files developed jointly for Microsoft and 
IBM. 

WORM-Short for write once, read many, an optical disk technology that allows one 
to write data onto a disk just once.

World Wide Web (WWW)-WWW is a system of Internet servers that support specially 
formatted documents.  The documents are formatted in a language called HTML 
(HyperText Markup Language) that supports links to other documents, as well as 
graphics, audio, and video files.

XML-Short for eXtensible Markup Language.  XML is a pared-down version of SGML, 
designed especially for Web documents.  It enables designers to create their own 
customized tags to provide functionality not available with HTML.

----

Appendix F

Site Visits to Three Federal Depository Libraries and Interview Questions


                Site Visits to Three Federal Depository Libraries


Libraries Visited                                Interviewees

McKeldin Library                                 Linda Spitzer, Acting Head
University of Maryland                           Government Documents and Maps
College Park, Maryland
(Regional Federal Depository Library)
Date visited: July 30, 1998



Washington College of Law Library                Joanne Zich, Chief
American University                              Government Documents and Media
Washington, D.C.                                 Services
(Selective Federal Depository Library)           Patrick Kehoe, Library Director
Date visited: August 11, 1998



Rockville Regional Library                       Judith Horowitz, Documents 
Montgomery County Public Libraries               Coordinator
Rockville, Maryland
(Selective Federal Depository Library)
Date visited: September 9, 1998




Interview Questions for Site Visits to Federal Depository Libraries

Date of Site Visit ____________________________

Contact Name __________________________________________________________________

Title _________________________________________________________________________

Library Name __________________________________________________________________

Address _______________________________________________________________________

_______________________________________________________________________________

Phone____________________________________     Fax _____________________________

Email ____________________________________


Background Information 

1. How long have you been in your current position?

2. Describe range of duties related to Government documents? 

3. How long has this institution been a Federal depository library?

4. Who are your primary and secondary users of Government information products?

5. Do you know what mediums patrons tend to be more comfortable using? (e.g., 
paper, CD-ROMs, Internet, diskettes or does it matter?)  What about what formats 
they prefer to use? (text and graphics)


Accessibility for Users

6. What key issues or concerns do users have about accessing Government 
electronic information products?  Have you conducted any user surveys or focus 
groups to determine the needs of patrons who use Government information products 
and services?

If they do not mention these issues, probe for them:

* Bibliographic and findings tools to identify and describe online and 
electronic Government products

* Problems with changing URLs

* Charging user fees (specify types of fees: printing costs, fees to access 
products)

* Copyright restrictions (can you give an example?)

* Downloading large files from the Internet

* User guides and documentation

* Providing reference and other public services to people with disabilities

* Other (specify)


Concerns about Services and Resources

7. What are your concerns about providing access to electronic Government 
information products?


If they do not mention, probe for:

* Are you notified that a product is available through electronic and online 
storage?  If yes, how?

* Are you currently notified that a product has been removed from the Internet? 
If so, how?

* Is there a better way to inform the depository libraries of additions, 
discontinued products, or modified products?  If so, what are your 
recommendations?

* What about retention and preservation of Government electronic information 
products in the future?



(Hardware and software issues to probe for:)

* What do you think about the latest minimum recommended specifications for 
public access workstations?


* Is your library able to keep pace with these minimum specifications?

* Do the current recommended minimum specifications accommodate current online 
and electronic Government information products? 

* Other hardware/software issues



Recommendations for Improving Access



8. Within the last three years, have you or your predecessor conducted any 
formal or informal studies (i.e., surveys, observations) of your users to 
identify specific needs or concerns they have in using Government electronic 
information products?


9. If yes, could you please briefly describe what you found and what, if 
anything, you did to make adjustments in your services or resources?


10. If you were given a pot of money (say $25,000) to improve access to 
Government electronic information products, how would you spend the monies, in 
priority order?


11. If you had unlimited resources, what would you do to improve public access 
to online and electronic Government information products? 


12. Are there other issues or concerns you would like to share with us about the 
transition to a more electronic FDLP that we haven't already covered?




NCLIS Assessment of Electronic Government Information Products

Summary Notes for Site Visits to Three Depository Libraries


1. What mediums do patrons tend to be most comfortable using? (e.g., paper, CD-
ROMs, Internet, diskettes)?

* Paper and microfiche; Internet a close second.
* CD-ROM is the least preferred medium (by librarians and patrons).


2. What key issues or concerns do users have about accessing Government 
electronic information products? 


Accessing Electronic Resources

All three librarians expressed major concerns about the problems in using 
Government-produced CD-ROMs that are not standardized:

* The search and retrieval software is different for each CD, often they come 
with no installation instructions or user documentation, and they are not user-
friendly. 
* Librarians must call in a computer technician to load the CDs and show 
librarians or users how to access the information. 
* Two of the three libraries have significantly reduced the number of CD-ROM 
titles they select because of the above-mentioned problem.  The third librarian 
indicated that they have a ton of CD-ROMs that are not used due to the problems 
with loading, accessing, and using them.

Users are still intimidated by electronic mediums and computers.  Most users ask 
librarians to help them search for materials on the web. 

Since most Government websites only contain the most recent information, all 
librarians expressed concerns about users accessing retrospective Government 
information on the web.  Two selective depository librarians often send their 
patrons to the regional depository to use their paper files for some historical 
Government information. 


3. What key library issues or concerns do you have about providing access to 
electronic Government information products? 


Charging Fees

Two of the three libraries do not currently charge fees for printing materials 
from the Internet or CD-ROMs.  One library was planning to change its policy 
beginning in September 1998 because it cannot continue to fund this expense.  
The second library is also contemplating changing its policy after installing 
the next generation computer system.  


Hardware issues

* Patrons expect the library to have state-of-the-art equipment and expect 
librarians to know how to use it.  Although all librarians felt their libraries 
had state-of-the-art equipment, they were concerned about how they could meet 
rising user expectations for the "best" computer hardware.
* Users do not have access to enough workstations, so the libraries must impose 
a time limit on computer use.
* Costs of hardware, telecommunications, and access to electronic information 
have increased dramatically.  For example, one library reported spending $60,000 
on these services and equipment 10 years ago; now they spend about $270,000 on 
hardware including maintenance agreements. 


Training Issues

All librarians expressed concerns about finding time and money to train 
librarians and staff, especially on the use of CD-ROM products, but also on 
downloading files, effectively searching the Internet for Government 
information, and creating and maintaining web pages.


4. If you were given a pot of money (e.g., $25,000) to improve access to 
Government electronic information products, how would you spend the monies, in 
priority order?

* New CD-ROM server; an 18-disk CD changer (3 responses).
* More staff training (2 responses).


5. If you had unlimited resources, what would you do to improve public access to 
online and electronic Government information products? 

* Form partnerships with GPO and a Government agency to put some retrospective 
Government information on their server so it will be accessible to users in the 
future (2 responses).
* Provide outreach to public schools, community centers, etc., to educate 
students and adults about the wide variety of valuable information available 
from the Federal Government (2 responses). 


Other Comments:

One librarian expressed strong feelings about the need for Congress to provide 
long-term financial support to Federal depository libraries so they can provide 
permanent public access to digital media.  

----

Appendix G

Electronic Government Information Products Assessment Agency Meetings
Held and Discussion Questions


   Electronic Government Information Products Assessment Agency Meetings Held

Agency                                                       Number of Attendees


Department of Health and Human Services                                       10
Meeting Date:  September 14, 1998


Supreme Court of the United States                                            11
Meeting Date: September 15, 1998


Environmental Protection Agency                                               12
Meeting Date: September 17, 1998


Department of Education                                                        5
Meeting Date: September 22, 1998


Department of Commerce                                                         8
Meeting Date: September 23, 1998


National Archives and Records Administration                                   7
Meeting Date: September 24, 1998






              Electronic Government Information Products Assessment
                         Agency Meeting Discussion Questions


1. Does your agency have preferred medium and format standards for Government 
electronic information products at the permanent public accessibility stage?  If 
so, what are the top three?  What factors does your agency consider in 
determining preferred standards (e.g., user needs, agency's dissemination 
requirements or policies, cost, security etc.)?  What about specific standards 
for CD-ROMs as they relate to user documentation, installation, search 
software, etc.?


2. Can you give us any examples of particularly innovative and creative product 
formats, mediums and/or online approaches?  We have in mind formats, mediums, or 
online approaches that may well point to the wave of the future, not only for a 
particular product but also for other kinds of products, yet is neither an 
agency-mandated standard nor even a common agency practice.


3. Is there any difference between your agency's preferences for mediums and 
formats as opposed to the preferences of intermediary distributors?  If so, what 
are those differences, and why are the two preferences different?


4. Has your agency involved external user groups in assessing the value and 
effectiveness of the dissemination of Government electronic information 
products?  If so, are there formats and mediums that seem particularly 
appropriate for public dissemination to users who may be economically, 
technically, or physically disadvantaged?


5. Does your agency follow any internally or externally prescribed guidelines 
for the presentation and organization of products in online formats?  If so, 
what are they (e.g., WWW Federal Consortium, FIPS Guidelines, agency or 
departmental publication specifications or guidelines)?


6. Has your agency undertaken any kind of cost benefit analyses for producing or 
creating products in preferred or emerging formats, mediums, or online 
approaches for distribution to the FDLP?  If so, which ones appear to be the 
most cost-effective? 


7. What factors does your agency consider in deciding to create or retain 
products in more than one medium?  Is this a common agency practice?


8. Are there trends with respect to migrating specific families of products from 
pre-electronic mediums to electronic mediums or formats?  For example, are 
loose-leaf publications, training manuals, annual reports, conference 
proceedings, newsletters, rules and regulations, scientific journals, etc., 
targeted for migration to a particular medium?  If so, which mediums and formats 
are used for specific families of products?


9. Has your agency identified any medium and format standards that seem 
particularly appropriate for use throughout a product's entire information life 
cycle, not just at one stage (i.e., creation, storage and retrieval, 
communication and dissemination, archiving and disposition) for electronic 
Government information products?  If so, which ones?
 
 
10. How do you determine whether a product should be made permanently publicly 
accessible when you create or produce it?  If so, what criteria do you use to 
determine which products will be permanently publicly accessible?  Can you give 
us any examples of how you ensure permanent public accessibility for a given 
product?
 

11. Does your agency routinely provide locator tools (e.g., GILS or specific 
agency locators) to enhance access to information sources and services available 
to external users and customers?  If so, is this an official policy, common 
agency practice, or both?


12. Are there trends for facilitating public access to your agency products by 
including them in broad electronic Federal Government information services such 
as GPO Access, LOC Thomas, and NTIS FedWorld?  Are you using any particular 
guidelines to facilitate that decision, and if so, what are they?


                 Summary of Responses to Agency Meeting Questions


1. Does your agency have preferred medium and format standards for Government 
electronic information products at the permanent public accessibility stage?  If 
so, what are the top three?  What factors does your agency consider in 
determining preferred standards.  

Agencies reported using the following electronic mediums most often: CD-ROM, 
Internet, and Bulletin Board System (BBS).  The most frequently used formats 
include:  

* HTML, PDF, ASCII 
* TIFF, GIF, JPEG
* Lotus/Domino

In determining medium and format standards, agencies consider the amount of 
information or files used, the timeliness of the information (e.g., more recent 
products or publications often placed on the web), and user needs for easy and 
quick access to information.  For example, some agencies are looking to Windows 
as an interface for CD-ROM products since the public is used to seeing and using 
Windows.


2. Do you have any examples of particularly innovative and creative product 
formats, mediums, and/or online approaches?  We have in mind formats, mediums or 
online approaches that may well point to the wave of the future for not only a 
particular product but also for other kinds of products, yet is neither an 
agency-mandated standard nor even a common agency practice.

Almost all the agencies interviewed are exploring a wide range of innovative and 
creative format, medium, and web applications.  Below is a sampling of some of 
the interesting online approaches and formats used by the agencies interviewed. 

* Data warehousing: Taking information not previously publicly accessible and 
integrating  it into an online format.  The format used is an Oracle database 
using SQL to query.

* Interactive Geographic Information System (GIS).  Provide mapping capability 
through GIS combined with regulatory information to create dynamic maps.  

* Online catalog of all products on the agency's website, using Oracle with a 
ColdFusion interface with their search engine (Verity).

* Creating user guides for CD-ROMs as pop-up HELP or short Read-Me files so 
users will be more inclined to use HELP.

* Live "real-time" web casting of selected speeches.  Format: RealPlayer 
software, available free from the web. 

* Radio news broadcast news service.  Provides daily radio sound-bits for news 
reports. Format: RealAudio RealPlayer software and WAV file format for 
downloading.

* Real-time forecasting of air pollution levels for 22 states from one site. 
Format: animated GIFs created by nonproprietary software designed by computer 
center in North Carolina.  

* Multimedia CD-ROM (i.e., art, music, animation, film, and video).

* Searchable electronic inventory of all proposals funded over the last 30 years 
so the agency can analyze its own information and make it available to others.  
Format: legacy database put into WAV database using HTML on the fly. 

* Documents stored in TIFF format for image and textual data.  As customers 
request documents, the agency converts them to PDF so customer can download.


3. Is there any difference between your agency's preferences for mediums and 
formats as opposed to the preferences of intermediary distributors? 


Generally, agencies indicated that intermediary distributors do not find agency 
formats to be restrictive.  However, the distributors often modify formats 
(e.g., from HTML to ASCII, or reformat data using compression technology).


4. Has your agency involved external user groups in assessing the value and 
effectiveness of the dissemination of electronic Government information 
products?  If so, are there formats and mediums that seem particularly 
appropriate for public dissemination to users who may be economically, 
technically, or physically disadvantaged?

Yes, all agencies reported that they involve users in assessing some aspects of 
their products, per OMB Circular A-130.  Examples include the following:

Focus groups are used to: 

* Determine the capability of new products (e.g., Can you use the same 
technology for the newer version of a product? Do users lose anything (e.g., 
Macros) when they update a product?).

* Determine ways to create more user-friendly CD-ROMs that resulted in the 
agency establishing three principles for producing CD-ROMs: make them simple to 
use, intuitive, and self-tutorial.

* Determine how information is presented on the web (Alpha and Beta testing).

* Solicit feedback on usability and accessibility; focus groups conducted with 
tribal Governments, teachers, librarians, children, etc.

* Learn about expectations, behavior, and problems in accessing products on the 
web experienced by the elderly.

* Interview and videotape users to assist agency redesign of website. 

* Solicit ideas from educators at professional conferences about topics that 
they would like to see presented in online products. 

For technologically or economically disadvantaged customers, one agency sub-unit 
faxes free copies of information printed from the CD-ROM or the Internet.  
Another agency sub-unit reported they try to reach economically disadvantaged 
customers by training rural community leaders in isolated areas on ways to 
access health-related information on the web .


5. Does your agency follow any internally or externally prescribed guidelines 
for the presentation and organization of products in online formats?  If so, 
what are they (e.g., WWW Federal Consortium, FIPS Guidelines, agency or 
departmental publication specifications or guidelines)?

Most agencies have developed guidelines or "best practices" for presentation of 
products in online formats.  However, several agency representatives indicated 
that the real challenge is in convincing agency staff to comply with the 
guidelines.  The following agencies provided information on their guidelines:

1. NARA: NARA Guidelines for Digitizing Archival Materials for Electronic Access 
(not to be considered a standard for digital imaging).   

(http://www.nara.gov/nara/vision/eap/eapspec.html)

2.  Federal Web Consortium's guidelines are based on Dept. of Education's 
guidelines (http://www.ed.gov/internal/wwwstds.html).  One sub-unit, NCES, also 
has developed guidelines.

3. EPA used WWW Federal Consortium guidelines 
(http://www.dtic.mil/staff/cthomps/guidelines/) to develop their own guidelines 
for presentation. 

4. Census uses a process and structure for submitting items for the web, but it 
is not yet formalized.

5. No departmental guidelines exist for the fifth agency, but most sub-units 
have some kind of guidelines for presentation and organization, although they 
might vary among sub-units.



6. Has your agency undertaken any kind of cost-benefit analyses for producing or 
creating products in preferred or emerging formats, mediums, or online 
approaches for distribution to the FDLP?  If so, which ones appear to be the 
most cost-effective? 


Generally, agencies have not conducted a formal cost-benefit analysis.  Agency 
representatives made the following observations, however:  

* One agency sub-unit tracks the number of customers who purchase a product and 
compares this amount against the cost of producing it.  They discontinued a 
product on CD-ROM because so few people could afford to buy it.

* The web reduces administrative costs for printing and mailing hard copies of 
publications.

* One agency reported a dramatic decrease (by 25,000) in the number of 
publications 
requested under the Freedom of Information Act due to the web.

* One agency tracks the number of people that a product can potentially reach 
(e.g., they sell 2,000 paper copies through GPO, but have 9,000 hits on the 
website).

* One agency reported that they order fewer publications to fill customer 
requests as a result of the web.


7. What factors does your agency consider in deciding to create or retain 
products in more than one medium?  Is this a common agency practice?


Agency representatives reported that they consider a variety of factors in 
creating and retaining products in more than one medium, although they did not 
characterize these factors as a common agency practice.  Several agencies 
reported that these issues are considered on a case-by-case basis or by the 
individual program unit.  The key factors considered are:

1. Budget (e.g., some products in CD-ROM are too expensive to make available to 
a small audience).

2. Cost (e.g., cost to print and mail product as opposed to make it available on 
the web).

3. Needs of technologically disadvantaged users (e.g., one agency maintains its 
Fax on Demand service, even though it is not cost-effective).

4. Accessibility (e.g., one sub-unit stores products in TIFF image format so 
they can produce them in whatever medium of output customers want).

5. Size of audience (e.g., agencies survey users and use web software to track 
use).

6. Number of queries or type of customer requests.  Customers will often request 
a publication or product in more than one medium.

7. Regulatory requirements.  Some products must be produced in paper regardless 
of whatever other medium it is produced in.

8. Type of product/publication  (e.g., searchable databases are only suitable 
for electronic mediums).

8. Are there trends with respect to migrating specific families of products 
(e.g., loose-leaf publications, training manuals, annual reports, conference 
proceedings) from pre-electronic mediums to electronic mediums or formats?  If 
so, which mediums and formats are used for specific families of products?


The general trend reported by agency representatives is to migrate more products 
to the web, especially recent ones.  Some examples include:

* Conference proceedings and presentations online in PowerPoint or PDF.

* Newsletters in HTML. 

* Training manuals and annual reports in HTML and PDF.

* Information for records managers are posted to Gopher, but will move to the 
agency's website in 1999.


9. Has your agency identified any medium and format standards that seem 
particularly appropriate for use throughout a product's entire information life 
cycle, not just at one stage (i.e., creation, storage and retrieval, 
communication and dissemination, archiving and disposition) for electronic 
Government information products?  If so, which ones?
 
 
Most agencies have either not addressed this issue of information life cycle or 
are struggling with it. 

* NARA has established medium and format standards for transferring permanent 
records to the National Archives in 36CFR 1228.188. 

* One sub-unit reporting putting documents in ASCII, but using Oracle for 
database management.

* One sub-unit is beginning to think about standardization for some documents.  
They draft documents in Lotus Notes (GroupWare) and publish final document in 
another database that goes onto the web.  They use Rich Text Format (RTF) to 
accommodate images and text.

* Several sub-units indicated that the technology is changing so rapidly they 
cannot establish standards.

 
10.  How do you determine whether a product should be made permanently publicly 
accessible when you create or produce it?  If so, what criteria do you use to 
determine which products will be permanently publicly accessible?  Can you give 
us any examples of how you ensure permanent public accessibility for a given 
product?
 
No agencies could provide responses to this question or indicated that this 
issue has not been resolved.  Some observations:

* One sub-unit is committed to making paper and CD-ROM-based products available 
for permanent access, but they are less clear about their commitment to products 
on the Internet.

* Some agency representatives did not understand the differences between 
permanent public access and permanent records. 

(The experts interviewed for this study provide some insight into the reasons 
that agencies are not addressing this issue.)


11. Does your agency routinely provide locator tools (e.g., GILS or specific 
agency locators) to enhance access to information sources and services available 
to external users and customers?  If so, is this an official policy, common 
agency practice, or both?

Most agencies indicated that the web format supercedes the original GILS 
concept.  However, most agencies have their own locators:

* NTIS has a catalog and maintains some GILS records.

* EPA's website has a GILS record and they put all Internet products on one 
server so there is one access point for all their products.

* Development and maintenance of GILS records is official agency policy for 
NARA.


12. Are there trends for facilitating public access to your agency products by 
including them in broad electronic Federal Government information services such 
as GPO Access, LOC Thomas, and NTIS FedWorld?  Are you using any particular 
guidelines to facilitate that decision, and, if so, what are they?


About half of the agencies use GPO Access or NTIS FedWorld.  The other half 
relies more heavily on individual agency websites with good links.  

----

Appendix H

Assessment of Electronic Government Information Products
List of Expert Interviews and Interview Questions



             Assessment of Electronic Government Information Products
                List of Expert Interviews and Discussion Questions

Interviewees                                         Date of Telephone Interview


Jerry Malitz, Webmaster                                         October 27, 1998
National Center for Education Statistics
U.S. Department of Education
Washington, D.C.

Linda Wallace, Chief                                            October 27, 1998
Electronic Information Services
Internal Revenue Service
Washington, D.C.

Evelyn Frangakis, Preservation Officer                         November 10, 1998
National Agricultural Library
U.S. Department of Agriculture
Beltsville, Maryland

Abby Smith, Director of Programs                               November 10, 1998
Council on Library and Information Resources
Washington, D.C.

John Bertot, Associate Professor                               November 18, 1998
State University of New York at Albany
Albany, New York	

Charles McClure, Distinguished Professor,                      November 24, 1998 
School of Information Studies
Syracuse University
Syracuse, New York



                        Interview Questions for Webmasters: 
                          Jerry Malitz and Linda Wallace
                                (October 27, 1998)


Role of  Webmaster

1. How long have you been in your current position as  webmaster? When and how 
was the position created?  Were you the first  webmaster in your agency?  How 
does the position reside administratively in the structure of your agency? What 
office or unit do you report to?


2. Please describe your current job responsibilities and duties.  What portion 
of the following skills, experience, and knowledge do you use to perform your 
job: technical, administrative, analytical, program, other?  


3. Is there a formal or informal structure for working with staff and 
administrators in other departments or units (e.g., program staff, IT, 
publications, public relations, records managers, librarians, etc.)? If yes, 
please describe how you interact with them?

4. What do you envision as the future role of the  webmaster in Federal 
Government agencies?  Do you see your role as being very different in 5 years 
than it is now? How?


Format Standards and Public Accessibility

5. Please describe the  website development process in your agency from the time 
you receive or generate requests through design, development, evaluation, 
testing, and implementation, etc.


6. Has your agency developed policies or guidelines including format standards 
to ensure technical consistency in the development of web products that are 
intended for public dissemination?  What are the most frequently used file 
formats and why?  Can you identify any formats you plan to use in the future?


7. Are there limitations or specific designations of software tools that may be 
used to develop and implement web pages or sites?  What standards are applied to 
configuration control and arrangement of web-based applications?  Do you have a 
direct role in determining these standards, or are they developed at an agency 
or departmental level?


8. Are there general security standards applied to the availability or 
distribution of web-based information? What is your role in determining or 
implementing these standards? Which software products do you use to implement 
these standards?


9. How do you evaluate the effectiveness of your  websites?  Who is involved in 
the process?  What methodology and criteria have you or others used to evaluate  
websites?


10. Has your agency discussed the concept of permanent public accessibility as 
it relates to Government electronic information products intended for public 
dissemination?  How is your agency addressing the concerns of librarians, GPO, 
and others for ensuring permanent public accessibility for electronic Government 
information products?


11. What consideration are you giving to creating a metadata record for your 
information resources or services on the web (e.g., GILS, MARC, or specific 
agency locators)?  


Cost Analysis

12. Linda, we know you have collected data on the comparative costs of 
delivering services to customers via different delivery mechanisms such as mail, 
e-mail, Fax on Demand, kiosks, Internet, telephone, walk-in, CD-ROM, etc.  What 
have you learned about the costs of delivering services to customers using these 
different systems? Which delivery mechanisms are the most cost-effective for 
what types of services?  


         NCLIS Assessment of Electronic Government Information Products

Summary Notes from Interview with Linda Wallace (IRS) and Jerry Malitz (NCES)


1. Please describe your current job responsibilities and duties.

WALLACE (IRS):

(Wallace was a telecommunications expert and technical advisor to CIO when IRS 
asked her to be webmaster.  She also holds the title of Chief, Electronic 
Information Services.  She is responsible for all electronic information 
products including Fax on Demand, Internet, e-mail, etc.)

Wallace's three major areas are content, applications, and development.  Her 
office: 

* Generates new services and authorware, including creating automated filters 
and templates through core knowledge repository.

* Participated in the development of SGML format (standard format for IRS since 
1970s).

* Interacts with customers to automate a standard way to build a core knowledge 
repository.  Repository contains automated templates and filters to generate 
media output to serve customers via Internet, Fax on Demand, CD-ROM, bulletin 
board system, telephone, or mail requests.

a. Core repository can satisfy 95 percent of requests using 86 different 
variables or attributes that are indexed so everything is searchable. All 
documents include individual catalog and document numbers.

b. Documents are always authored in SGML and can automatically be converted into 
a different format or posted on the web, BBS, etc., in 10-12 hours to fill 
customer requests.

c. Filters and templates are solely by this group.  They also track history of a 
document.

d. A knowledge base is being built by developing a database of frequently asked 
questions.

e. They use ICON tagging to provide accessibility to the visually impaired. All 
IRS documents are ADA-compliant, online searchable, and downloadable.

MALITZ (NCES):

As Technology Outreach Officer for NCES, Malitz services state education 
agencies, school districts, etc.

a. Each one of NCES's 30 programs has a web publisher and a web liaison. 

b. Contractors actually prepare materials for the web once program officer has 
approved content.

c. Malitz sets standards, guidelines, and procedures for web publisher to 
follow.

d. Web publisher in program area develops website on a separate development 
server; Malitz reviews and makes technical changes to ensure that the site meets 
minimum standards and guidelines. 

e. Sometimes, he develops a new application for others to use (e.g., NEWS FLASH 
subscription service featuring daily breaking news from the Department of 
Education).



2. Do you have a formal or informal structure for working with staff and 
administrators in other departments or units (e.g., program staff, IT, 
publications, public relations, records managers, librarians, etc.)?  If yes, 
please describe how you interact with them.

WALLACE:

Wallace deals with high-level senior executives, reviewing their business plans, 
problems, and goals, then recommending solutions that include productivity 
measures, production rates, and cost per person.  In one case, she recommended a 
business CD-ROM.

* Establishes strong liaisons with industry (has marketing person on staff). 
They receive one-half of funding from industry to support business projects that 
benefit industry and IRS customers. 

* Established various delivery service programs: Internet in '96, Fax on Demand 
in '96, CD-ROM in '95, BBS a while ago.

MALITZ

* Each one of NCES's 30 programs has a web publisher and a web liaison.

* Contractors prepare materials for the web once content has been approved by 
the program officer.

* Web publisher in program area develops web site on a separate development 
server; when finished, Malitz reviews and makes technical changes to ensure that 
site meets minimum standards and guidelines.

* Sometimes, Malitz develops a new application for others to use such as the 
NEWS FLASH subscription service that features daily breaking news from the 
Department of Education.

* Malitz's work is divided fairly evenly between technical, administrative, 
analytical, and program areas.




3. What do you envision as the future role of the webmaster in Federal 
Government agencies?  How you see your role as in 5 years as compared to now? 

MALITZ:

The role of webmaster will be completely different.  In the future, he/she will 
have more of a coordinating function and will set policies and procedures. The 
program staff will be forced to do their own work on the web, just as they now 
do their own word processing and e-mail.

WALLACE:

The role of the webmaster will be that of an enabler for business units with 
everyone involved.  There will be more of a focus on multimedia (e.g., BBS, CD-
ROM, Fax on Demand) and not just the Internet. 


4. Please describe the website development process in your agency from the time 
you receive or generate requests through design, development, evaluation, 
testing, and implementation, etc.

WALLACE:

Her unit receives and generates requests. Requests from the core repository can 
fit into an existing template filter application. 

a. Staff and contractors conduct testing and implementation.

b. Second year after web in operation, requests for paper copies of forms 
dropped by 50 percent.

c. Provide hidden codes to track where returns come from: fax, Internet, 
libraries, phone requests, etc.

d. Evaluation includes a simple three-question customer service survey on 
content: did you get what you needed, where would you have gone if not here?  
They build evaluation into every step of the process.

e. A panel of experts measures the effectiveness of their websites.  Also, they 
have partnered with schools to recruit instructors and students to review site 
before they go "live."

f. One person reviews all e-mail messages that contain feedback on website; use 
automated sorters by key words to batch the type of feedback received.


MALITZ:

Before website, customers were very specialized. Most were data file users.  
After they created their website, their customer based increased tremendously.  
Now the culture is different and NCES is dealing with questions from the general 
public. 

a. NCES uses a developmental server but is planning to implement a Point-to-
Point Tunnel Protocol (PPTP) so the developmental server is behind the firewall 
and no longer open to everyone.  Only web publishers and contractors will have 
access to server. 

b. Malitz never reviews content; that is done by individual program staff.

c. NCES conducts customer surveys of users to develop and refine sites.

d. Malitz does database development and tests multiple browsers.  NCES is UNIX-
based; rest of ED is Windows-based.


5. Has your agency developed policies or guidelines including format standards 
to ensure technical consistency in the development of web products that are 
intended for public dissemination?  What are the most frequently used file 
formats and why?  Can you identify any formats you plan to use in the future?

WALLACE:

Formats most frequently used are SGML, PDF, HTML, and Postscript, respectively.  
They will add XML soon.  They train authors to use SGML. SGML is "intelligent" 
data that can automatically generate other formats.  Most agencies do not use 
SGML because it is harder to author in.  Wallace's agency uses it because it is 
much more robust, and it is easy to change a document format to match customer 
needs (e.g., tax law information for consumer and for lawyers).

MALITZ:

NCES uses PDF, then HTML (optional).  They rarely put entire publication in 
HTML.



6. Are there limitations or specific designations of software tools that may be 
used to develop and implement web pages or sites?  What standards are applied to 
configuration control and arrangement of web-based applications?  Do you have a 
direct role in determining these standards, or are they developed at an agency 
or departmental level?

WALLACE:

Their focus is knowledge-based, not web application. Her department sets the 
standards.  They use C++ and Perl.


MALITZ:

All publications are in PDF; all else in HTML. They use SQL databases to support 
the web. Malitz has a direct role in determining standards. 


7. Are there general security standards applied to the availability or 
distribution of web-based information? What is your role in determining or 
implementing these standards? Which software products do you use to implement 
these standards?


WALLACE:

The IRS uses an automated redacting scheme; with one keystroke, they can create 
a public and specialized version of the same document.  They apply all security 
standards from the Government, including SSA, Treasury, etc.  They cannot reveal 
security software.

MALITZ:

Only a few people who use developmental server have access; all must be 
registered users.  NCES uses PPTP encryption for the developmental server.


8. Has your agency discussed the concept of permanent public accessibility to 
electronic Government information products intended for public dissemination?  
How is your agency addressing the concerns of librarians, GPO, and others for 
ensuring permanent public accessibility for electronic Government information 
products?

WALLACE:

All tax forms, instructions, publications etc., are available for 5-6 years 
online.  The core knowledge repository maintains material for 14 years, but they 
do not keep every application back that far.  IRS can fill e-mail requests for 
information or forms from earlier years.  In addition, they provide GOLD CARD 
SERVICES for librarians. Librarians have their own page, track orders, and talk 
"live" with one another. IRS gives their orders priority.

MALITZ:

The issue of permanent public accessibility is currently under discussion.


9. What consideration are you giving to creating a metadata record for your 
information resources or services on the web? (e.g., GILS, MARC or specific 
agency locators)?  


WALLACE:

GILS records are a subset of the 86 variables that go into the core knowledge 
repository.


MALITZ:

The Dept. of Education has an agency locator with total search capability.  They 
also participate in FedStats, White House Briefing Room, etc.


10. Linda, we know you have collected data on the comparative costs of 
delivering services to customers via different delivery mechanisms such as mail, 
e-mail, Fax on Demand, kiosks, Internet, telephone, walk-in, CD-ROM, etc.  What 
have you learned about the costs of delivering services to customers using these 
different systems? 

WALLACE:

Breakdown of comparative costs follows: 

a. It costs IRS $3 per call for the public to call into their toll-free number 
and for IRS to fill the request.   The cost to IRS for the public to use the 
Internet to access and use forms is 1 cent, a difference of 300 to 1.

b. The costs to create forms on Internet have gone down, but the cost to fill 
phone requests remains the same.

c. It costs IRS $2.50 to make and distribute to public libraries each CD-ROM 
containing 5 years of tax forms, instructions, and publications.  This is less 
than it takes for IRS to respond to one telephone call.  The IRS also sends tax 
CD-ROMs to the depository libraries.  They can mount them on their PCs or allow 
customers to check them out.

d. They found that kiosks are very expensive; ATMs are cost-effective 

Final Comments from Wallace

The answer to public accessibility is not the Internet; it is multimedia.  
Delivery mechanisms must meet the individual needs of the customers; no one size 
fits all.




                  Interview Questions for Preservation Specialists: 
                            Evelyn Frangakis and Abby Smith 
                               (November 10, 1998)


1. How long have you been in your current positions?  Please describe your 
current job responsibilities and duties.  What portion of the following skills, 
experience, and knowledge do you use to perform your job: technical, 
administrative, analytical, and other?


2. What are the key problems associated with digital preservation?


3. What key policy, organizational, economic and other non-technical issues need 
to be addressed or solved to facilitate digital preservation?


4. What technological strategies or models have various organizations such as 
the Association of Research Libraries, the Digital Library Federation, National 
Archives, etc., identified to address these problems?  Evelyn, one of the NCLIS 
staff mentioned that NAL has established a structure or framework that addresses 
this problem.  Could you please talk more about that? If you have any handouts 
you can fax to us, that would also be helpful.   Abby, can you describe some of 
CILR's recent efforts to address the issue of digital preservation, including 
the survey by Jeff Rothenberg of the RAND Corporation?


5. What do we know about specific file formats or mediums that might facilitate 
digital preservation such as SGML, CD-ROM, etc.?


6. Are there any important preservation issues that we have not addressed in the 
above-listed questions?  If so, please discuss them.


7. Could you please refer us to any important articles on this topic that have 
been published in the last year?


Summary of Notes from Conference Call with Two Preservation Specialists:
Evelyn Frangakis (NAL) and Abby Smith (Council on Library and Information 
Resources)


1. How long have you been in your current position?  Please describe your 
current job responsibilities and duties.  What portion of the following skills, 
experience, and knowledge do you use to perform your job: technical, 
administrative, analytical, and other?

ABBY SMITH (CLIR)

* Been with CLIR since Sept. 1997 as director of programs.

* Provide program coordination among the four areas: economics of information, 
leadership in libraries and archives, digital libraries, preservation and 
access.

* Her primary program responsibility is in preservation and access in libraries, 
traditional and digital.

* 10 staff members; 6 professionals, 4 admin. support staff.

* Spends 75 percent of time on policy-related issues and the remaining 25 
percent of time spent on administrative functions ( i.e., coordinating 
publications program)

EVELYN FRANGAKIS (NAL)

* Been in current position since January 1997.  She is NAL's first preservation 
officer.

* Duties: plan, direct, and implement agency-wide programs for ensuring 
permanent and future accessibility of the foremost national collection of 
materials in agriculture.

* Coordinates activities with other national efforts, such as the U.S. 
Agricultural Information Network (USAIN). Established in 1988, USAIN provides a 
forum for discussion of agricultural issues, takes a leadership role in the 
formation of a national information policy as related to agriculture, makes 
recommendations to the National Agricultural Library on agricultural information 
matters, and promotes cooperation and communication among its members. NAL 
participates in implementing USAIN's preservation plan for print materials, A 
National Preservation Program for Agricultural Literature.  The USAIN 
Preservation Steering Committee, on which Frangakis serves, oversees this 
national cooperative plan.  Under the auspices of Cornell University, the USAIN 
plan has received two NEH grants to microfilm core national and state 
agricultural literature.  To date, 15 states are participating in these grants.  
Other components of the national plan and program include determining what are 
the important archival and manuscript collections of agricultural materials and 
what approaches can be used for their preservation.

* NAL efforts include developing their own preservation program that includes a 
traditional preservation program and digital efforts.

* Digital efforts are two-pronged: conversion of brittle paper materials into 
digital products by working with best available guidelines to implement good 
preservation practices (this digital material will be available on the web); 
develop a program to preserve USDA digital materials (i.e., materials that are 
born digitally).

* Helps develop preservation policies and analyze other policies that come to 
NAL or USDA that affect preservation of the collection.

* Time spent on different types of work at different times.  Duties fairly split 
among policy, technical, administrative, analytical areas.

* Staff consists of two assistants at present. However, NAL leverages its 
preservation resources by establishing cooperative inter-institutional 
agreements and contributing funds to sister institutions in order to further 
develop the USAIN preservation program (e.g., cooperative agreements with 
Cornell University to establish copyright clearance for core historical 
literature, developing NEH grant proposals).


2. What is the distinction between digital preservation and permanent public 
accessibility of electronic records (as it relates to, for example, the Federal 
Depository Library Program)? How long is "long-term" preservation vs. "permanent 
public accessibility?"

FRANGAKIS

* Some background from the USDA perspective: The USDA Digital Publications 
Preservation Steering Committee was established this past summer to oversee the 
implementation of the plan, A Framework for the Preservation of and Permanent 
Public Access to USDA Digital Publications.  This group met for the first time 
in October 1998. There was a discussion of definitions in order to place into 
context the universe of material covered by the Framework.  Publication was 
defined as "a data or information product prepared by the USDA in digital form 
intended to be disseminated to the public." The Framework defines preservation 
as "the act of permanently maintaining and making available data or information, 
with all original content intact."

* Other experts, such as Don Waters of the Digital Library Federation, talk 
about preserving integrity and ensuring persistence of digital information. The 
Commission's SGML report talks about preservation goals such as enhancing the 
long-term preservation of and access to information of enduring value for as 
long into the future as possible.  

* THE CPA Digital Archiving Task Force was charged to investigate the means of 
ensuring "continued access indefinitely into the future of records stored in 
digital electronic form."

* Concept of preservation in traditional preservation world examines the concept 
of permanence, but in the print world the concept of permanence relates to 
chemical inertness and mechanical durability. These concepts do not translate 
easily into a digital world.  In the digital world, we are no longer dependent 
on original copies (i.e., original copies do not have the same meaning.).

* Within NAL, they use digital preservation fairly loosely to speak about both 
digital efforts: conversion of brittle materials and USDA digital publication 
preservation efforts.  Not sure professionals in the library and preservation 
community have a common understanding of what it means, even though it is 
important to come to a common understanding.  GPO defines permanent access as 
"Government information products within the scope of the Federal Depository 
Library Program that remain available for continuous no-fee public access 
through the program." The 1996 GPO report to Congress, Study to Identify 
Measures Necessary for a Successful Transition to a More Electronic Federal 
Depository Library Program, states that "'preservation' means that official 
records of the Federal Government, including Government information products 
made available through the FDLP, which have been determined to have sufficient 
historical or other value to warrant being held and maintained in trust for 
future generations of Americans, are retained by the National Archives and 
Records Administration (NARA)."

* At NAL, the mission of its preservation program is "to preserve and ensure 
access to the intellectual content and physical composition of agricultural 
works of national and international importance indefinitely into the future."  
No timeframe is mentioned because no one at this time can say how long into the 
future information will be needed. 


SMITH

* There is no standard accepted method of ensuring long-term access to digital 
information.  She described preservation goals as permanent or persistent or 
perhaps more accurate to say that one of the primary goals of preservation is to 
set up systems that "sustain predictable levels of loss."  

* Difference between preservation in a digital world and in an analog world is 
that in a digital world, information is completely independent from the medium 
on which it is carried. In an analog world, people try to preserve the media in 
which information is recorded.  No analogy in the digital world. No concept of 
preserving the artifact as an artifact that has its own level of information.

* Problem in digital preservation is that there is no way to ensure that 
digitally stored information can move from one software-hardware configuration 
onto another through generations. Two problems:

(1) Problem of instability of media in which information is stored (don't know 
how long CDs or other media will last)

(2) More serious issue from CLIR point of view is that software/hardware 
configurations on which information is stored become obsolete so quickly that 
even when you migrate information from one system to another, much of the 
information is lost (data and functionality).

* CLIR tries not to talk about "digital preservation" but they cannot avoid it.

* Other countries may view preservation differently. England interested in 
American concept of digital archiving -preserving the integrity of data; that 
is, information is original and authentic and it can be proved that data have 
not been changed.  According to Smith, scientists say that we will solve the 
problem of authentication, but hasn't been solved thus far. 


Permanent Public Accessibility Issues (FRANGAKIS)

* Federal agencies relying on FDLP to serve an "archiving" function for 
retrospective materials.

* Question of how to preserve digital information indefinitely into the future 
has not been answered.  CLIR and NAL are discussing strategies by opening up 
dialogue and promoting research in this area.

* Digital Archiving Task Force Report discussed ensuring the integrity and long-
term availability of digital information through migration.  Information on CD-
ROM and other media is in a format that may or may not be readable into the 
future due to hardware/software obsolescence.  Even if media could be preserved, 
no guarantee that it would be accessible and functional indefinitely into the 
future.  No answers anytime soon.


3. What are the key problems associated with digital preservation?

SMITH

* Additional problems: fragility of media and platform dependence issue


(1) Two additional issues: difficulty of understanding what we can and cannot do 
under current copyright law. Latest iteration  of copyright law clarifies 
copyrighting for preservation purposes, but  still unclear for access purposes.  
Library of Congress is currently studying this.  Copyright law may not have any 
implications for Government information but many vendors create derivatives of 
Government information and copyright it. Government should never be in the 
position of depending upon the private sector to preserve some of this 
information.

(2) Any transmission link is as strong as the weakest link.  The weak link in 
the transmission of electronic information is not technology; it's human beings.  
Human infrastructure is not in place yet that would ensure permanent access.

FRANGAKIS

* Agrees that human error is far more prevalent than technology error.  Key 
problems in digital preservation are infrastructure, technology, and media.  
Humans need to learn to live, exist, and operate in a digital world. It's still 
very new to us as compared to the print world.  


4. What key policy, organizational, economic, and other non-technical issues 
need to be addressed or solved to facilitate digital preservation?

SMITH

* Digital Archiving Task Force Report met with consensus in the community.  CLIR 
has been lobbying people to pay attention to these issues.  No single community 
has stepped forward and said this is our problem and we are going to work on 
solving the problem. Therefore, one of difficulties is that organizations that 
collect, preserve, and disseminate information, as opposed to create 
information, find themselves in this digital world in which the preservation of 
that information must be thought about at the creation stage, not after the 
fact.  Need to forge partnerships with the computer science industry, 
publications industry, scholarly and scientific publishing communities to 
address some of these issues.

* One of perhaps intractable core infrastructure problems is the issue of 
creating a failsafe archives mechanism for materials that disappear from the 
web.  What happens when information is created and the people who created it do 
not have responsibility for preserving it?  Who is going to authorize a failsafe 
archive that is going to take and preserve that information for the public good?  
This would be the equivalent of libraries, but so far, it doesn't exist and no 
one has expressed interest in creating it at the Governmental level.

* CLIR's role in above: Not in a position to do much more than alert people 
about the problems. NAL and literature that agriculture creates are one of few 
examples where this failsafe archive might work because NAL is a national 
library dedicated to one type of literature.  Not the case with other 
literatures except for medicine (NLM).  CLIR is looking for partners like ARL to 
address this issue, but has not made much progress.  CLIR has been fairly 
effective in talking to National Science Foundation (NSF) in getting their 
second round of digital library initiatives grants to address the issue of 
preservation as a distinct issue.  No luck with archiving part.

FRANGAKIS

Refer to the report Framework for the Preservation of and Permanent Public 
Access to USDA Digital Publications by Paul Uhlir, November 1997 (listed in 
bibliography).

* Three areas of issues: 

(1)  management structure and organizational relationships within and outside 
USDA, 

(2)  funding of program on a permanent basis (keeping in mind the need to 
minimize costs of access and retrieval to information users), and

(3)  identification of legislative or administrative actions or policies 
required to implement a digital publications preservation program.

Sub-Issues: Needs and Considerations for USDA

* Inventory and life cycle management: a comprehensive inventory of all 
departmental digital information products and how they are being managed needs 
to be conducted.  A system for tracking the creation of each new USDA digital 
information product that is intended for public distribution needs to be 
recommended.

* Technical requirements: identification of acceptable document formats and 
media, and related standards, for long-term retention; development of processes 
for transferring all digital publications from old storage media to new media; 
establishment of one or more separate back-up facilities for all digital 
publications; review and establishment of system security protocols; review and 
establishment of system interoperability requirements; and identification and 
review of other permanent digital preservation and access initiatives.

* User access and retrieval: provide equitable access and retrieval services to 
all potential users; minimize technical, regulatory, and cost barriers to access 
and retrieval; assure the integrity of the information that is made publicly 
available; make the information as easy to find and use as possible, with 
directories and documentation (metadata), consistent with the Government 
Information Locator System, while protecting confidential or proprietary 
information; and establish a means for users to provide feedback and a mechanism 
for responding to user feedback.

* Status: Moving ahead with implementation. USDA CIO accepted the report, and 
under her guidance NAL established a national steering committee made up of 
representatives from USDA and from agribusiness, research library community, 
USAIN, Federal partners, etc. 

* Group will meet on a quarterly basis for first 2 years.

* Will establish test groups to explore issues such as inventory and life cycle 
management, technical requirements, and user access and retrieval, as well as 
funding issues.

* Hoping to get funding for a pilot project and then take entire framework and 
test it on an agency within USDA to see how manageable Framework will be for 
full-scale implementation.


5. What technological strategies or models have various organizations such as 
the Association of Research Libraries, the Digital Library Federation, National 
Archives, etc., identified to address these problems?  Evelyn, one of the NCLIS 
staff mentioned that NAL has established a structure or framework that addresses 
this problem.  Could you please talk more about that? If you have any handouts 
you can fax to us, that would also be helpful.   Abby, can you describe some of 
CILR's recent efforts to address the issue of digital preservation, including 
the survey by Jeff Rothenberg of the RAND Corporation?

SMITH

Three CLIR initiatives:

* Commissioned report by Jeff Rothenberg from RAND Corporation on emulation. 
[Emulation is the process of imitating one system with another so both accept 
the same data, execute the same programs, and achieve the same results.] Report 
complete and may be published by January 1999.  Since report is highly 
controversial, CILR will partner with National Research Council to convene a 
group of computer scientists to engage Rothenberg on issues of emulation to 
stimulate research.  Report describes the weaknesses of migration and the 
strengths of emulation and sets up a research agenda to develop emulation.

* Commissioned an analysis of migrating file formats to do a risk assessment 
associated with those file formats during migration.  Study commissioned from 
Cornell using data from the Mann Library (agricultural library) and will use 
numeric file formats and databases and text formats.  Report will be finished by 
September 1999 and will include analysis and a template that others can use for 
doing a risk assessment of migration of those file formats. Purpose: to 
stimulate further research.


* Identified a computer scientist at Carnegie-Mellon University (CMU), John 
Ockerbloom, who has developed a system of file conversion; type of migration 
that converts web-based materials to different file formats, called TOM (Typed 
Object Model). (www.cs.cmu.edu/afs/cs.cmu.edu/user/spok/www/defense/index.html).  
He developed this as part of his thesis.  Working with CMU to see if they can 
bring his concepts into fuller application to do an assessment about its 
scalability.

* Log on to publications on CLIR site, which summarize Rothenberg report. 
Water's report addresses definition of digital preservation.


6. What do we know about specific file formats or mediums that might facilitate 
digital preservation such as SGML, CD-ROM, etc.?

SMITH
Nothing to say about this

FRANGAKIS

* For conversion efforts from paper to digital images, SGML serves as an 
important descriptive markup tool.  Thinks it will be valuable to them.  CD-ROM 
serves specific functions in NAL's Preservation Program but right now has a 
limited life expectancy.  NAL is looking for things that are non-proprietary, 
platform independent, things that will allow user full access to the content of 
digital products.  They know that media will continue to change.

7. Are there any important preservation issues that we have not addressed in the 
above-listed questions?  If so, please discuss them.

No.

8. Could you please refer us to any important articles on this topic that have 
been published in the last year?

1. Margaret Hedstrom at the University of Michigan School of Information 
believes that there is a reliable way of preserving, with predictable levels of 
loss, migration of digital information, through ASCII.  It happens now. She is a 
leading authority in the field.

2. Check ARL's website.

3. Coalition of Networked Information - Dedicated to computer use in education. 
www.cni.org

4. White paper on access authorization. Developing infrastructure for digital 
libraries. 

5. www.RLG.org/preserv - (includes information on Hedstrom's research).

6. Reference Model for Open Archival Information Systems: 
http://ssdoo.gsfc.nasa.gov/nost/isoas/ref--model.html
White Book, issue #4, Sept. 1998- preservation of digital information; technical 
recommendation for use in developing consensus on what's required of any archive 
to provide permanent preservation.  Hoping to turn this into an ISO standard, 
but now just in draft form.

           Interview Questions for Information Resources Specialist:
                                  John Bertot
                              (November 18, 1998)


1. What are the primary obstacles to successful information resources management 
(IRM) practices in the Federal Government, in priority order?  What changes 
should occur to eliminate or alleviate the barriers?


2. In your article on the impact of Federal IRM on agency missions, you mention 
the reinvention of IRM to be the key link between agency information and agency 
performance.  Can you describe a small and a large Federal agency that currently 
meet this goal in spite of the lack of a concentrated, coordinated Federal IRM 
policy? Why are they more effective than other agencies?


3. We have found in our interviews with Federal agency personnel that many 
agencies have not come to terms with two important issues: information life-
cycle management, and the concept of permanent public availability of electronic 
Government information.  What are some of the larger policy issues that have 
prevented agencies from addressing these important issues? 


4. In our site visits to Federal depository libraries in the D.C. metropolitan 
area, we are keenly aware that the problems and issues faced by FDLs here are 
different than they might be for FDLs located in more isolated, rural areas.  
Based on your survey of public libraries connected to the Internet, what are 
some of the key concerns or problems faced by users who want to access 
electronic Government information who live in small, isolated communities with 
limited resources? 


5. Based on your experience in working with Federal Government agencies that are 
analyzing their web usage, what are the key questions they want to answer and 
how are they using the data?  Are they analyzing the  websites for technical or 
content-related purposes?  What techniques, other than log file analysis, are 
being used?  What agency units or departments (e.g., IT, program areas, CIO) are 
involved in the process?  



                Interview Questions for Information Resources Specialist:
                                    Charles McClure
                                  (November 24, 1998)


1. What is the current status of IRM policy since your 1994 article, "Federal 
Information Resources Management: New Challenges for the Nineties?"  
Specifically, has OMB or another appropriate agency begun addressing the issue 
of developing a broad vision that reflects the evolving role of IRM within the 
Government with general guidelines and standards for all Federal agencies to 
better manage the life cycles of information?  If so, how? 


2. We have found in our interviews with Federal agency personnel that many 
agencies have not dealt with two important issues: information life-cycle 
management, and permanent public accessibility of electronic Government 
information.  What are some of the larger planning, policy, and organizational 
issues that are preventing agencies from addressing these important problems? 


3. Could you talk a little more about the design-based assessment for evaluating 
Government websites that you described in the 1997 Proceedings of the 60th ASIS 
annual meeting?  What were the technical and policy problems in the design-based 
assessment?  What specific policy issues did you assess?


4. What is the status of electronic record management (ERM) guidance for Federal 
agencies since the 1998 conference?  Is there another conference planned next 
year?  What specific guidance are the NARA Working Group and other agencies 
planning?


5. What would you say are the top three Federal IRM challenges in the next 
decade? 



                                   Summary of Notes
                                    Interview with
                     John Bertot, Associate Professor, SUNY/Albany


1. What are the primary obstacles to successful IRM practices in the Federal 
Government, in priority order?  What changes should occur to eliminate or 
alleviate the barriers?

* IRM is not on the radar for top-level agency managers and it will never be 
raised up to the point of where it matters.

* IRM has been lost in the transition to the CIO.  CIO is the next iteration of 
IRM.  There has been 20 years worth of talking and it has never seemed to make 
it out of the administrative trenches of the agencies.  Typically, IRM has been 
a low-level position located within the printing, reprographic, or records 
management units of agencies.  IRM is not viewed as strategic or long-range 
function.


2. In your article on the impact of Federal IRM on agency missions, you mention 
the reinvention of IRM to be the key link between agency information and agency 
performance.  Can you describe a small and a large Federal agency that currently 
meets this goal in spite of the lack of a concentrated, coordinated Federal IRM 
policy? Why are they more effective than other agencies?


The article is based on Bertot's dissertation.

The purpose of the survey was to get an internal assessment on what IRM is 
trying to do, and to get an external assessment of what IRM is doing.  Bertot 
tried to compare the two in relation to strategic planning.

Generally, those agencies that understood IRM tended to be the smaller agencies. 
There is a scale factor, and much more attention was given to IRM in the smaller 
to mid-sized agencies.  FDIC and the Peace Corps were doing some interesting 
things.

Other factors relating to size:

* Small to medium-sized agencies have fewer programs and staff; with  fewer 
administrative layers, there are fewer communication and organizational 
barriers.
* One can more easily work collaboratively in a small organization. 
* Top administrators are not as removed from day-to-day operations and can 
ideally participate more in implementing new initiatives because they have a 
vested interest in the projects' working. 
* There is less oversight from OMB and Congress for smaller agencies. The 
smaller agencies tend to have less mandated legislation that can interrupt work, 
so they tend to have higher motivation to finish projects.
* Larger agencies tend to have the greater expertise.  Smaller agencies may have 
better levels of management, but they do not always have the experts.  

Models
* As far as larger agencies were concerned, Treasury was moving along.  However, 
many have a central agency component that is not very powerful, although 
subagencies might be very powerful. For example, Treasury has IRS and the FBI, 
and they are pretty powerful players. 
* Another agency that has done a great deal of work with IRM is EPA.  EPA, along 
with AID, are strange models.  They have large data shops, but they are all 
contractors.  The model that is adopted for information and information 
technology management makes a difference.  Whether IRM is in-house or out-
sourced has a real impact on how it is implemented inside the agency, and the 
choice of contractor really matters.  Another aspect that makes EPA unique is 
that a large portion of their system management function occurs in North 
Carolina.  


3. We have found in our interviews with Federal agency personnel that many 
agencies have not come to terms with two important issues: information life 
cycle management, and the concept of permanent public availability of electronic 
Government information.  What are some of the larger policy issues that have 
prevented agencies from addressing these important issues? 

* The biggest barrier to successful implementation of IRM is that agencies do 
not view information as a resource. There is little or no understanding of the 
concept of information as a life cycle; it's not linear. 
* IRM policy initiatives and legislation do not fully address the life cycle of 
information.  It is mentioned in some of the policies developed within the last 
20 years. but not adequately addressed.  Most policy initiatives focus on the 
technology side of the issue, probably because it is tangible. 
* The web has created problems that have not been handled.  Many agencies 
believe that if it is up on the web, it has been published.  Along with the 
pressure over Title 44 Reform, there is no discussion of preservation and public 
accessibility.  Should we move to an electronic FDL program?  What does that 
mean and how will that work? 
* GPO is under attack for being deficient in distributing Government information 
to the public.  One reason is because GPO (centralized print environment) is so 
slow, and the technology allows distribution to be handled more efficiently 
(decentralized, electronic environment).  Agencies are under the gun to cut 
costs, so by putting information on their websites and contracting printing jobs 
with outside sources, they don't have to go through GPO. 
* Going electronic does offer potential.  The back end means, however, that 
anyone with access is a vehicle for getting Government information.  Putting 
information up on the web does cut costs, but we have not figured out a 
systematic way of distributing Government information to the public, making sure 
it is preserved for posterity and provided to the public on a long-term basis. 

4. In our site visits to Federal Depository Libraries in the D.C. metropolitan 
area, we are keenly aware that the problems and issues faced by FDLs here are 
different than they might be for FDLs located in more isolated, rural areas.  
Based on your survey of public libraries connected to the Internet, what are 
some of the key concerns or problems faced by users who want to access 
electronic Government information who live in small, isolated communities with 
limited resources? 

Recently Bertot conducted some research in rural Pennsylvania to study public 
libraries.  Public libraries in rural communicates face large problems with 
access and technology.  

* These areas are composed of populations that tend not to have computers in the 
home, so they rely totally on the library for Internet access. 
* The public libraries tend to only have one station in these rural areas. 
* Computers are slow; libraries have 56 K modems, but they do not necessarily 
have access to a 56 K Internet provider. 
* Patrons can only reserve computers in half-hour time slots. 
* Libraries may or may not have access to print equipment. 
* Staff training is minimal due to cost and little access to computers. (Models 
like GPO Access have been useful, but patrons and librarians still need one site 
for access to all Government agencies rather than many individual websites. Most 
agencies have more than one site.) 
* Staff are competing with patrons for access because there are only one or two 
computers. 
* Libraries cannot always pay for printing services. Some are passing the cost 
to the user, but they are trying to avoid that approach (e.g., first 5 pages are 
free and then it is 10 cents per page). 

5. Based on your experience in working with Federal Government agencies that are 
analyzing their web usage, what are the key questions they want to answer and 
how are they using the data?  Are they analyzing the websites for technical or 
content-related purposes?  What techniques, other than log file analysis, are 
being used?  What agency units or departments (e.g., IT, program areas, CIO) are 
involved in the process?  

* Bertot has not seen agencies doing very much with their web statistics.  Part 
of the reason is that they do not want to make the information public. For 
example, one agency had a request for all of their agency log file records.  
They panicked and rejected the request on the condition of privacy.  If one can 
access the log, one can get IP addresses, and they were afraid that someone 
would use this information as a reverse directory mailer. 
* A second reason is that sometimes it is difficult to get the statistics if a 
different administrative unit within the agency is managing the website.  They 
won't necessarily turn them over to the unit that needs the statistics because 
it crosses administrative barriers. 
* A third reason relates to records management of log files.  Should we 
"schedule" log files for NARA?  NARA doesn't want this to happen either.  They 
would then have to schedule the information for retention.  However, this raises 
the question of whether the logs are public information.
* It was not until recently that there was a demand to look at web statistics as 
a management and strategic decision-making tool. Managers are still learning how 
to use them. Right now, these statistics are used primarily by network and 
system administrators. 
* Bertot wonders how many agencies are doing web analysis and evaluation given 
Circular 130-A, which cautions agencies not to do so if it creates a paperwork 
burden for them. 

                                  Summary of Notes
                         Interview with Charles McClure, 
    Distinguished Professor, School of Information Studies, Syracuse University


1. What is the current status of IRM policy since your 1994 article, "Federal 
Information Resources Management: New Challenges for the Nineties"?  
Specifically, has OMB or another appropriate agency begun addressing the issue 
of developing a broad vision that reflects the evolving role of IRM within the 
Government with general guidelines and standards for all Federal agencies to 
better manage the life cycles of information?  If so, how? 


IRM policy came and went and no one noticed:

* GSA is mounting its CIO university effort to provide education and training to 
CIOs. 
* IRM in Government policy is now whole world to CIO. 
* Many agencies do not now know what to do with IRM. 

* ITMRA (Information Technology Management Reform Act of 1996)-McClure thought 
this policy would strengthen IRM, but in reality, it took responsibility away 
from existing IRM people and gave it to CIO; it gave more attention to 
technology management. 
* A few agencies don't know what to do with IRM staff since CIO is on board; in 
other agencies there is conflict between IRM and CIO functions. 
* Eighty-two percent of technology efforts in agencies are currently focused on 
Y2K efforts. 


2. We have found in our interviews with Federal agency personnel that many 
agencies have not dealt with two important issues: information life cycle 
management, and permanent public accessibility of electronic Government 
information.  What are some of the larger planning, policy, and organizational 
issues that are preventing agencies from addressing these important problems? 

* There is no staff or time to devote to standards and interoperability. 
* Even if agencies had staff and time, staff need to upgrade skills and 
knowledge. 
* Information life cycle and permanent public accessibility are not priorities 
for agencies; they don't seem to understand the issues. 
* For example, GPO Reform Bill is dead in the water; no one in Congress cared 
about it. 
* No one is concerned about long-term accessibility. 


3. Could you talk a little more about the design-based assessment for evaluating 
Government websites that you described in the 1997 Proceedings of the 60th ASIS 
annual meeting?  What were the technical and policy problems in the design-based 
assessment?  What specific policy issues did you assess?


They are now using more advanced website methodology (i.e., a 4-legged 
approach):

* User-based: Usability testing; simulates user searching that is videotaped.
  With fairly sophisticated graduate students, they use scripted search analysis 
with a range of criteria. System and design staff are showing videotapes to so 
they can see the problems with searching information on specific sites.  
Agencies have used different audiences to do testing based on objectives and 
purpose of sites. 
* Log analysis: Using in-house scripts beyond WebTrends and Log Tracker that 
allow them to do cross-file analysis with access vs. error and browser files. 
Perl scripts allow them to dump selected variables in log files into SASS or 
SPSS.  Commercial products do not do cross-log analysis well. 
* Policy analysis: Internal policies (who's in charge) and external policies 
(e.g., Freedom of Information Act, public access, privacy issues, etc.). 
* Management assessment:  How is agency department organized for web maintenance 
and evaluation?  Information is gathered through interviews and focus groups 
with managers. 

4. What is the status of electronic records management (ERM) guidance for 
Federal agencies since the 1998 conference?  Is there another conference planned 
next year?  What specific guidance are the NARA Working Group and other agencies 
planning?

* Update: McClure and Tim Spreche are working on a new project, PARS- Public 
Access Rating System.  The purpose of this project is to create a core set of 
performance measures and indicators (now being developed for four Government 
agencies) with public access criteria to help agencies rate their websites.  
Agencies will be able to determine the degree to which the site is publicly 
accessible.  It's difficult to sell ERM by itself, so they are taking a public 
access approach. ? National Archives hasn't done much.  Court ruling delayed 6-8 
months. 


5. What would you say are the top three Federal IRM challenges in the next 
decade? 

* IT management policy development is on hold due to Y2K.  No one currently 
knows how good or bad preparation is for this. 
* How best to integrate and coordinate IT and IT management. Agencies do not 
have a good handle on this yet. 
* Issues of interoperability and standards that cut across all agencies. Need to 
be able to access Government information horizontally rather than vertically. 
(For example, for public access sites; GILS, gov.doc l, and one more; no way 
public can access specific information from one point of entry.  GILS does not 
work the way it was originally conceived.) 
* Lack of money for training and education.  IRM graduate students' degrees are 
useful for about 1-2 years. After that, their skills are 50 percent out of date.  
Government agencies have well meaning people who don't have the knowledge and 
skills to implement policies.  For example, agencies say they don't need to send 
copies of all products to GPO because they are on their website.  Then you ask 
them will it be there in 6 months and they have given no thought to this issue. 

----

Appendix I

Sample Agency Meeting Agenda
Electronic Government Information Products Assessment


                          Sample Agency Meeting Agenda
             Electronic Government Information Products Assessment


                Meeting with Health and Human Services Agency 
                    Representatives and Product Respondents

                         Tuesday, September 15, 1998
                                  9:15 a.m.



9:15  Introductions                            Westat, NCLIS, and HHS Personnel


9:25  Background and Purpose of the Meeting    Westat and NCLIS


9:30  General Questions and Answers            Westat, NCLIS, HHS Personnel
      * Sections A and B on the Questionnaire


10:00 Q and A to Sections C and D              NCLIS, Westat, HHS Personnel
      * Review Data Collection Procedures


10:30 Break


10:45 Agency Discussion Questions              Westat and HHS Personnel
      (Discussion questions posted on NCLIS web site)


11:45 Adjournment

----

Appendix J

Task 16

Assessment of Electronic Government Information Products

Statement of Work



                      Section J:  Background and Objectives

Background

An "Assessment of Electronic Government Information Products" (hereinafter 
referred to as the "Assessment") is a research study being conducted through an 
interagency agreement between the National Center for Education Statistics 
(NCES) and the U. S. National Commission on Libraries and Information Science 
(NCLIS) on behalf of the Superintendent of Documents, U.S. Government Printing 
Office (GPO), pursuant to an Interagency Agreement between NCLIS and GPO, 
approved by the Joint Committee on Printing (JCP).

Information gathered from this assessment is to be used by the Superintendent of 
Documents to facilitate improved public access to Federal Government information 
made available to Federal depository libraries and the general public through 
the Federal Depository Library Program (FDLP).  The assessment will 

(1) identify medium and format standards that are the most appropriate for 
permanent public access;
(2) assess the cost-effectiveness and usefulness of various alternative medium 
and format standards; and
(3) identify public and private medium and format standards that are, or could 
be used for products throughout their entire information life cycle, not just at 
the dissemination or permanent public access phase.


Objectives
 
The contractor shall undertake research and data collection from Federal 
agencies in all three Branches, as well as solicit the opinions of selected 
knowledgeable experts.  The contractor shall also complete an analysis of the 
data and opinions for the purpose of interpreting their general meaning and 
significance, including identifying broad emerging trends and patterns, and 
documenting findings, conclusions and recommendations in a deliverable final 
report.
 
More specifically, for a cross-section of Government information products, the 
goals are to: (1) determine in which format(s) and medium(s) such products are 
now produced, using which standard(s) if any (whether promulgated by official 
standards-setting bodies such as ISO, NISO, ANSI or FIPS, or voluntarily adopted 
through common agency practice, such as the use of file formats such as PDF or 
TIFF, or Microsoft Word or dBase); and (2) assess agency future plans for new or 
changed products, including the medium(s) and format(s) in which they will be 
disseminated for permanent public access.
 
The Superintendent of Documents will use the results of this work effort to 
continue to plan and implement the transition to a more electronic FDLP.  The 5 
major specific objectives are:


* First, with respect to electronic publishing practices and plans of Federal 
agencies (including ways in which the FDLP can best accommodate them), the 
objective is to provide an analysis of current practices as well as future plans 
for creating, disseminating, and providing permanent public accessibility to 
electronic information products, and to identify the standards for software, and 
electronic mediums and formats that are or will be used throughout the product's 
information life cycle, from creation to archiving, but especially by the time 
of the permanent public accessibility phase; 

* Second, with respect to cost effectiveness of various dissemination mediums 
and formats that are, or could be, utilized, the objective is to gather 
information on standards (whether mandated or consensual) that will assist the 
FDLP in making near-term decisions regarding the cost-effectiveness of 
alternative mediums and formats for all FDLP participants.  This information 
should also assist participants in long-term planning for permanent public 
accessibility, and the collection and analysis of overall information life-cycle 
costs;

* Third, with respect to the practical utility of various electronic mediums and 
formats to depository libraries and the public, the objective is to identify 
preferred standards used in various mediums and formats that depository 
libraries will need to support;

* Fourth, with respect to utilizing standards employed in mediums and formats 
that can be used throughout all stages of the information life cycle (including 
creation, composition, computer terminal display, encryption, secure digital 
signature with non-repudiation and secure transmission capabilities), electronic 
dissemination, but especially permanent public accessibility, the objective is 
to assess standards for basic security services in order to provide for secure 
and reliable transmission and document interchange; and

* Fifth, with respect to standards that are being developed and used in the 
private sector, the objective is to identify existing and planned standards for 
the purpose of determining what the FDLP must do to accommodate their adoption 
by the agency in terms of hardware/software requirements, staff and user 
education and training, and budgetary impacts.

For the purposes of this survey an agency Web site is considered an electronic 
information service, and one or more products may reside on the service.  Web 
sites, per se, are not considered individual Government information products.

                              Section II:  Subtasks

The contractor will be provided background materials to assist in conducting the 
required work, including a suggested data collection instrument, examples of 
completed forms, selection criteria for finalizing the list of agencies and 
products to be surveyed, and additional materials (see Appendix C).  Working 
with representatives of NCLIS and GPO, the contractor shall undertake the 
following specific activities and complete them by the scheduled dates shown 
below.  A detailed explanation of each activity appears in Appendix C:

Activity No.    Activity Title
1               Prepare a Plan of Action
2               Develop a Data Collection Plan identifying preliminary agency
                and product selections, and a schedule of interviews, focus 
                group meetings and site visits, using Appendix B as a guide
3               Conduct interviews, focus group meetings and site visits; revise 
                preliminary agency and product selections if necessary
4               Pre-test the Data Collection Instrument (Appendix A); make 
                changes if necessary
5               Create a Baseline Inventory Products Data Base ready to populate  
                with actual data
6               Collect the data from Federal agencies, experts, and literature 
                references
7               Prepare tabulations and summaries based on the populated data   
                base and analysis
8               Recollect and retabulate if/as necessary
9               Prepare a draft Report documenting findings, conclusions and 
                recommendations, and provide oral briefings on request; obtain  
                feedback from Government
10              Prepare a final Report and submit to the Government



                Section III.  Deliverables And Key Events Schedule

Deliverables, in terms of their delivery in weeks after award, are shown below.  
For each of the deliverables, the contractor shall provide six copies to the 
COTR.  [This is the original schedule and does not reflect the changes that were 
later negotiated between NCLIS and Westat.]

Deliverable                                                    Weeks After Award
Government-contractor planning meeting                                 1
Plan of Action approved                                                2
Approval of prelim. agency/contact list                                3
Completion of interviews/meetings/site visits                         12
Approval of Data Collection Plan                                      12
Agency data collection instruments transmitted to agencies            13
Actual data collected                                                 16
First oral briefing                                                   17
Tabulations/summaries completed                                       17
Recollection and retabulation completed                               19
Second oral briefing                                                  19
Draft Report submitted                                                20
Final oral briefing                                                   22
Final Report submitted                                                25



            Section IV - Management Reporting Requirements

The contractor shall provide the following reports for the purpose of 
maintaining a detailed record of work ordered and funds spent on this task 
order:

-  Cost Proposal;
   Schedule of Deliverables
-  Task Order "Log" or menu [monthly];
-  Monthly Cost Report;
-  Task Invoices; and
-  Reimbursement Report [monthly].

Details pertaining to these and ancillary reporting activities follow. 

Cost Proposals

The contractor shall provide a cost proposal to accompany the contractor's 
response to the work identified.  The cost proposal will be reviewed and 
approved by the COTR and the contracting officer, who will communicate the 
approval to the contractor. [Should there be a need to modify the cost proposal, 
the COTR and the contracting officer will communicate that instruction to the 
contractor's Project Director, who shall submit a revised cost proposal for 
approval.]

Task Order "Log"

The contractor shall provide a monthly report indicating the tasks ordered by 
the COTR and the contracting officer to date and showing the internal (to the 
contractor) tracking number assigned to each.  The contractor shall insure that 
this report includes such summary information as: the contractor staff member 
assigned as task leader; date(s) associated with the task, including 1) date 
assigned to the contractor; 2) date scheduled for completion of the task; and 3) 
upon completion, date of invoice(s) for services rendered in support of the 
task. 

Monthly Cost Report

The contractor shall provide a monthly report to the cotr and the contracting 
officer as specified under the terms and conditions of the contract and detailed 
in the proposal.  As part of the monthly report, the contractor shall include a 
list of deliverables and their anticipated due dates.

Reimbursement Report

The contractor shall provide the COTR and the contracting officer with a status 
report on reimbursement activities.  The contractor shall insure that the report 
includes the names of the consultants and participants and any others who are to 
be reimbursed for expenses incurred, as applicable and approved by the COTR and 
the contracting officer.  The contractor shall include:  1) the subtask for 
which the person is subject for reimbursement; 2) the amount subject to 
reimbursement for each subtask in which the person is a participant; 3) the date 
each participant submitted the necessary reimbursement form for each subtask; 4) 
the date the participant is scheduled to be issued reimbursement for each 
subtask; and 5) cumulative reimbursement totals to date, updated.


Period of Performance

The period of performance for this task is eight months from the date the award 
is signed.

----