Information Management: Dissemination of Technical Reports
(18-MAY-01, GAO-01-490).
This report reviews the Department of Commerce's National
Technical Information Service (NTIS), which is a permanent
repository and principal disseminator of scientific, technical,
engineering, and business-related information. NTIS acquires
research reports primarily from federal agencies and their
contractors and grantees as well as from international sources.
GAO discusses (1) the various functions of NTIS; (2) the
quantity, age, and demand trends of reports in NTIS' repository;
(3) the extent to which the reports in NTIS' repository are
readily available from other public sources; and (4) whether
federal agencies are sending their reports to NTIS for sale to
the public, as required by law. GAO found that NTIS provides its
basic statutory clearinghouse repository function of collecting
and disseminating full-text paper reports, and various other
fee-based services for agencies. These include brokerage
services, distribution services, and Web services. NTIS has about
2.5 million reports in its repository that are to be retained
permanently. About 75 percent of the reports are more than 12
years old, and NTIS has sold one or more copies of about eight
percent of its 2.5 million reports. Of the 1.8 million reports
more than 12 years old, only about one percent has sold since
1995. About 19 percent of NTIS reports were readily available
from one or more of the four sources at the time GAO searched.
Agencies often did not send their reports to NTIS as required by
law. The reasons agencies cited for not sending their reports
included that they (1) were unaware of the law; (2) would incur
additional costs and duplication of effort to format and transfer
information to NTIS that is free on the agency's Web site; or (3)
did not believe their reports were covered because, in their
view, their reports did not contain scientific, technical, or
engineering information.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-01-490
ACCNO: A01005
TITLE: Information Management: Dissemination of Technical
Reports
DATE: 05/18/2001
SUBJECT: Research reports
Government information dissemination
Federal records management
Noncompliance
Internet
Reporting requirements
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Testimony. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-01-490
Report to Congressional Requesters
United States General Accounting Office
GAO
May 2001 INFORMATION MANAGEMENT
Dissemination of Technical Reports
GAO- 01- 490
Page i GAO- 01- 490 NTIS Technical Reports Letter 1
Appendix I Scope and Methodology 28
Appendix II Comments From the Department of Commerce 32
Table
Table 1: NTIS Sales, by Customer Segment 12
Figures
Figure 1: Percentage of Reports, by Year of Publication 9 Figure 2: Year of
Publication for Full- Text Reports in NTIS
Repository 10 Figure 3: Products Distributed 11 Figure 4: Estimated
Percentage of Reports Sold in Each Stratum 13 Figure 5: Estimated Report
Availability at Different Sources 16 Figure 6: Estimated Reports Available,
by Year of Publication 17 Figure 7: Estimated Percentage of Reports
Available Free From
Other Sources 18 Figure 8: Estimated Availability of Reports Sold Since 1995
19
Abbreviations GPO Government Printing Office IRS Internal Revenue Service
NCLIS National Commission on Libraries and Information Science NTIS National
Technical Information Service ONDCP Office of National Drug Control Policy
Contents
Page 1 GAO- 01- 490 NTIS Technical Reports
May 18, 2001 The Honorable Thomas M. Davis Chairman, Subcommittee on
Technology and
Procurement Policy Committee on Government Reform House of Representatives
The Honorable Vernon J. Ehlers Chairman, Subcommittee on Environment,
Technology, and Standards Committee on Science House of Representatives
The Honorable Constance A. Morella House of Representatives
This report conveys the results of our review of the Department of
Commerce?s National Technical Information Service (NTIS). As you know, NTIS
is a permanent repository and principal disseminator of scientific,
technical, engineering, and business- related information. NTIS acquires
such research reports primarily from federal agencies and their contractors
and grantees as well as from international sources.
As requested, our objectives were to determine (1) NTIS? various functions;
(2) the quantity, age, and demand trends of reports in NTIS? repository; (3)
the extent that reports in NTIS? repository are readily available from other
public sources; and (4) whether federal agencies are sending their reports
to NTIS for sale to the public, as required by law, and, if not, the reasons
for this lack of compliance.
To determine NTIS? functions and the quantity, age, and demand trends for
reports in its repository, we interviewed agency officials and analyzed
various reports. To determine the extent that research reports in NTIS?
repository were readily available from other public sources, we randomly
selected 600 reports from the repository and searched for them on World Wide
Web sites at the issuing organization and the Government Printing Office
(GPO). 1 We also searched at FirstGov. gov and at a private search
1 Includes GPO Access, the Federal Depository Library Program, and the GPO
Bookstore.
United States General Accounting Office Washington, DC 20548
Page 2 GAO- 01- 490 NTIS Technical Reports
engine called Google. com. We used our sample results to estimate what
portion of the total population of reports in the repository was available
at these Web sites. All estimates reported from the sample have sampling
errors of plus or minus 10 percentage points or less. To determine whether
agencies sent their reports to NTIS as the law requires, we decided that it
was not practical for us to perform compliance audits at every major
department and agency. Rather, we relied on a proxy for identifying agency
noncompliance. We used reports that NTIS identified and acquired
electronically from the issuing agencies? Web sites during 2000 after the
agencies did not provide NTIS with the reports in a timely manner. 2 NTIS
refers to this process as ?harvesting? reports. In addition, we asked
officials at the seven agencies with the most harvested reports why they had
not sent their reports to NTIS. Appendix I contains a more detailed
discussion of our scope and methodology.
We conducted our review from September 2000 to April 2001, in accordance
with generally accepted government auditing standards.
In addition to its basic statutory clearinghouse repository function of
collecting and disseminating full- text paper reports, NTIS provides various
other fee- based services for agencies. These include brokerage services
such as billing and collecting for other agencies that charge for products
and services; distribution services such as disseminating another agency?s
products or operating the Internal Revenue Service?s (IRS) Web site; and Web
services such as FedWorld (an on- line information network).
We estimate that NTIS has about 2.5 million reports 3 in its repository that
are to be retained permanently. In recent years, NTIS? repository has been
growing at a slower pace. For example, the repository contains 56,000
reports published in 1995 while only 10,000 reports published in 2000 had
been added to the repository as of mid- November 2000.
The newest reports in NTIS? repository- those published in 2000- accounted
for less than 1 percent of all research reports in its repository. We
estimate that about 75 percent (1.8 million) are over 12 years old. Since
1995, we estimate that NTIS has sold 1 or more copies of about 8 percent
2 We did not evaluate whether each report that NTIS harvested was required
to be reported to NTIS by law. 3 Includes reports that NTIS asks other
agencies to provide directly to customers (i. e., source fills). Results in
Brief
Page 3 GAO- 01- 490 NTIS Technical Reports
(187,000) of its 2.5 million reports. Of the 1.8 million reports over 12
years old (pre- 1989), we estimate that only about 1 percent has sold since
1995.
We estimate that about 19 percent (470, 000) of NTIS? reports were readily
available from one or more of the four sources at the time we searched. Of
these reports, we estimate about one- third could be obtained free from the
Internet. The remaining 2 million reports (81 percent) were only readily
available from NTIS. We estimate that reports published in 2000 were over
five times more likely to be available at these other sources than those
over 12 years old.
Agencies often did not send their reports to NTIS as required by 15 U. S. C.
3704b- 2. In 2000, officials in NTIS? Information Services Division
harvested over 1,100 reports from agency Web sites that they believed should
have been sent to NTIS but were not. These included reports from the
Departments of Justice and Commerce, NTIS? parent organization. The reasons
agencies cited for not sending their reports included that they (1) were
unaware of the law; (2) would incur additional costs and duplication of
effort to format and transfer information to NTIS that is free on the
agency?s Web site; or (3) did not believe their reports were covered
because, in their view, their reports did not contain scientific, technical,
or engineering information.
The increasing availability of scientific, technical, and engineering
information on the Web is raising fundamental policy questions about the
future direction of NTIS. Therefore, we are presenting matters for
consideration to the Congress on determining the best approach for defining,
retaining, and disseminating government research in the Internet era. We are
also making recommendations to the Secretary of Commerce to improve the
extent to which agencies send their reports to NTIS, as required by law.
In commenting on a draft of our report, the Secretary of Commerce agreed
with our identification of the key policy issues and concurred with the
thrust of our recommendations. The Secretary said he would determine how
best to implement our recommendations and ask the Acting Under Secretary for
Technology to determine if there are any practical ways, consistent with the
statutory requirement that NTIS be self- sustaining, to make NTIS?
information collection more widely available to the general public.
Page 4 GAO- 01- 490 NTIS Technical Reports
Under statutory authority enacted in 1950, 4 NTIS is directed to collect
scientific, technical, and engineering information from both domestic and
foreign sources and to make the information available to (1) business and
industry, (2) state and local governments, (3) other federal agencies, and
(4) the general public to increase U. S. competitiveness in the global
economy. The statute does not define scientific, technical, and engineering
information. However, a 1954 Comptroller General Decision found the
Secretary of Commerce?s position reasonable that the statute included ?all
types of information which have a more or less direct bearing on business
and industry generally.? 5 The Secretary specifically described such
information as including ?economic information, market information, and
related information so long as it is reasonably specific and bears some
direct relationship to the organization and operation of industrial or
business enterprise.?
Agencies? transfer of research information to NTIS is generally governed by
15 U. S. C. 3704b- 2, which states that
?The head of each Federal executive department or agency shall transfer in a
timely manner to the National Technical Information Service unclassified
scientific, technical, and engineering information which results from
federally funded research and development activities for dissemination to
the private sector, academia, State and local governments, and Federal
agencies. Only information which would otherwise be available for public
dissemination shall be transferred under this subsection. Such information
shall include technical reports and information, computer software,
application assessments generated pursuant to section 3710( c) of this
title, and information regarding training technology and other federally
owned or originated technologies.?
This statute is implemented by NTIS regulation 15 C. F. R. Part 1180 (2000).
The regulation, in general, provides for transfer to NTIS of (1) at least
one copy of every final research product resulting from an agency?s
federally funded research and development activities and (2) a summary of an
agency?s new and ongoing research that is likely to result in a final
product.
NTIS? basic statutory function is to collect research reports from agencies,
maintain a bibliographic record and permanent repository of reports, and
disseminate these reports to the public as required. NTIS charges user fees
4 15 U. S. C. 1152. 5 34 Comp. Gen. 58 (1954). Background
Page 5 GAO- 01- 490 NTIS Technical Reports
for the sale of its products and services to the public. NTIS? enabling
legislation requires that, to the fullest extent feasible, all of the
services and functions it provides be self- sustaining. 6 From fiscal years
1988 through 1994, NTIS showed a net profit in all years except 1 and
generated a cumulative net profit of $5.8 million. However, over the next 5-
year period of fiscal years 1995 through 1999, NTIS had a cumulative net
loss of about $4.2 million.
In August 1999, the Secretary of Commerce proposed closing NTIS by September
30, 2000, because he believed that declining sales revenues soon would not
be sufficient to recover all of NTIS? operating costs. The Secretary partly
attributed this decline to other agencies? practice of making their research
results available to the public for free through the Web. He also proposed
transferring NTIS? research report archives to the Library of Congress and
requiring federal agencies to give the public free on- line access to new
research reports.
In June 2000, we issued a report noting that NTIS may be able to remain
self- sustaining for a few years, but that its solvency beyond fiscal year
2003 remained in doubt. 7 Since the issuance of our June report, however,
NTIS? Deputy Director said NTIS has significantly reduced costs and shown a
profit of $2.2 million for fiscal year 2000, and the Deputy Director
projects a profit of $1.5 million for fiscal year 2001.
In addition to NTIS? basic statutory function of collecting and
disseminating technical reports, NTIS also provides various other feebased
services for agencies. According to NTIS, these include brokerage services
such as billing and collecting for other agencies that charge for products
and services; distribution services such as disseminating other agencies?
products or operating the IRS? Web site; and Web services such as FedWorld
(an on- line information network). According to NTIS financial data, revenue
from all functions totaled $35 million in fiscal year 2000.
NTIS? four major functions and the revenue they generated are described
below with some examples:
6 15 U. S. C. 1153. 7 Information Policy: NTIS? Financial Position Provides
an Opportunity to Reassess Its Mission (GAO/ GGD- 00- 147, June 30, 2000).
NTIS Provides a
Variety of Services in Addition to Its Statutory Function of Collecting and
Disseminating Research Reports
Page 6 GAO- 01- 490 NTIS Technical Reports
Clearinghouse repository function. This is NTIS? basic statutory function of
collecting and disseminating technical reports, and the function includes
seven product lines. According to NTIS, these product lines generated a
total of almost $19 million (55 percent of NTIS? total revenue) in fiscal
year 2000:
Full- text products. This product line refers to NTIS? basic document
sales program. It includes reports that agencies and foreign sources provide
either in hard copy or microfiche (retail); bulk sales to resellers
(wholesale); and sales from special collections, such as military
publications. It generated a reported revenue of about $6 million in fiscal
year 2000.
Announcement products. This product line refers to information about new
research reports available from NTIS. Included in this line are NTIS?
bibliographic database, which is disseminated to the public for a fee by
vendors who share the resulting revenue with NTIS, and ?NTIS Alerts,? a
biweekly product announcing the availability of new products in specific
subject categories. It generated a reported revenue of about $1.2 million in
fiscal year 2000.
Computer products. This product line includes all sales of software,
CDROMs, and diskettes- containing such items as export regulations, IRS
forms, and IRS? Small Business Resource Guides. It generated a reported $5
million in fiscal year 2000.
Paper subscription products. These are requests by users for a specific
information product that is issued or updated on a regular, predictable
basis and in hard copy. Examples of these products include Foreign
Agricultural Service publications and EPA Quarterly Abstracts. The product
line generated a reported $890,000 in fiscal year 2000.
Electronic subscription products. These are products that are disseminated
electronically on a predictable basis, such as World News Connection (an on-
line foreign news service), Federal Research in Progress, and Davis- Bacon
Act wage determinations. The product line generated a reported $3 million in
fiscal year 2000.
Standing order products. These are similar to subscriptions, except that
the product is issued or updated on an irregular basis, such as National
Transportation Safety Board Accident Reports. The product line generated a
reported revenue of almost $2 million in fiscal year 2000.
Page 7 GAO- 01- 490 NTIS Technical Reports
Audiovisual products. In addition to paper and electronic formats, NTIS
sells a number of audiovisual products, such as videocassettes, that contain
training, health and safety, environmental, and other forms of information.
These products are collected by NTIS? audiovisual center, which was
transferred to NTIS from the National Archives and Records Administration.
The product line generated a reported $590,000 in fiscal year 2000.
Brokerage services. NTIS provides billing and collection services on a
reimbursable basis to other agencies that, like NTIS, charge for products
and services, but lack the necessary financial infrastructure to do their
own billing and collecting. The product line generated about $340,000 in
fiscal year 2000, according to NTIS officials.
Distribution services. As an outgrowth of its own activities to disseminate
research reports, NTIS will also produce and/ or disseminate specific
information products to an agency?s constituents under a reimbursable
agreement with the agency, such as operation of the IRS? Web site. These
products may be distributed in hard copy, on videocassette, on CD- ROM, or
via facsimile or the Web. The agency may supply a mailing list or direct its
constituents to call NTIS. NTIS may provide related ?help desk? services to
assist those constituents in using the material. At times, NTIS will develop
multiformat packages for an agency, such as videocassettes and printed
instructional material for use in training courses. In addition, NTIS offers
agencies comprehensive services to integrate hardware and software in the
design of systems that allow agencies to conduct transactions over the
Internet. According to NTIS, distribution services generated about $15
million in fiscal year 2000.
Web services. NTIS? statutory charter calls for it to develop new methods or
media for disseminating information; in 1992, the statute was amended
specifically to include electronic media. In response, according to NTIS
officials, NTIS developed the FedWorld platform as a means of handling high-
volume traffic over the Internet in a secure environment- with sufficient
capacity to allow NTIS to offer it for a fee to other agencies having a
similar need. Web services generated a reported revenue of about $660,000 in
fiscal year 2000.
According to NTIS, in addition to the four major functions, it provides
other miscellaneous services, such as experimenting with new ways to help
agencies reach their constituents. For several years, NTIS has offered CD-
ROM mastering and premastering services. Recently, NTIS and the Department
of the Navy developed a program that provides sophisticated
Page 8 GAO- 01- 490 NTIS Technical Reports
satellite broadcast services to agencies that need to reach employees or
constituents around the country or the globe.
We estimate that NTIS? repository has about 2.46 million reports- 2.04
million recorded in a database 8 and 420,000 older reports that were
manually recorded on index cards. Of these estimated 420,000 older reports,
we estimate that 200, 000 were maintained and disseminated by other agencies
on behalf of NTIS.
Our estimate of 2.46 million reports is somewhat lower than the nearly 3
million reports NTIS stated in its Web site and financial audit report for
fiscal year 2000. NTIS overstated its holdings because it (1) double counted
reports that were in both its database and on index cards and (2) overstated
the number of reports maintained and disseminated by other agencies on
behalf of NTIS (i. e., ?source fill? reports).
As shown in figure 1, we estimate that about 75 percent of the reports in
NTIS? repository were over 12 years old (i. e., published in 1988 or
earlier). The most recent reports- those published in 2000- accounted for
less than 1 percent of all reports in NTIS? repository. Specifically, there
were
10,000 reports published in 2000,
97,000 reports published from 1997 to 1999,
516,000 reports published from 1989 to 1996, and
an estimated 1, 835,000 9 reports published in 1988 or earlier. 8 The
automated file we analyzed was as of mid- November 2000. 9 This figure is an
estimate because it includes our estimate of the number of reports recorded
on index cards plus the number recorded in the database over 12 years old.
Quantity, Age, and
Demand Trends of Reports in NTIS? Repository
Page 9 GAO- 01- 490 NTIS Technical Reports
Figure 1: Percentage of Reports, by Year of Publication
Source: NTIS.
As shown in figure 2, in recent years, NTIS? repository has been growing at
a slower pace- from about 56,000 reports published in 1995 to just over
10,000 published in 2000 (as of mid- November 2000). 10 However, the
Assistant to the Director of NTIS said that since about 75 to 80 percent of
the reports in NTIS? collection for a given year are added in the years
following the year of publication, the final number of reports for 2000
might increase significantly. Nevertheless, the Assistant to the Director of
NTIS acknowledges that the total number of reports added to the repository
in recent years has been declining.
10 These data do not include reports in which NTIS (1) has only the
bibliographic record (not the report) and (2) refers customers to the source
agency for copies. For example, NTIS began making available military
specifications and standards in 1997 and 1998 as part of a joint venture
with the private sector, but NTIS is no longer providing these data to
customers.
4%
1988 or earlier 2000
1989- 96 1997- 99
0.4% 75% 21% A
Page 10 GAO- 01- 490 NTIS Technical Reports
Figure 2: Year of Publication for Full- Text Reports in NTIS Repository
Source: NTIS.
Although total demand for NTIS reports has been increasing since fiscal year
1996, the public?s demand for NTIS paper reports has declined, while demand
for electronic products increased, as shown in figure 3.
0 10000
20000 30000
40000 50000
60000 70000
80000 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 Number of
reports
Calendar year
Page 11 GAO- 01- 490 NTIS Technical Reports
Figure 3: Products Distributed
Source: NTIS.
Demand for products was primarily from business and industry sources. As
shown in table 1, nearly three- fourths of NTIS? sales revenues during the
last 3 fiscal years from the clearinghouse (except subscription sales) came
from business and industry. The next largest customer segments were schools/
universities, the federal government, and state and local or foreign
governments.
0 1
2 3
4 5
6 7
8 9
10 Numbers in millions
Fiscal year 1998 1999 2000 1997 1996
Paper Electronic
A
Page 12 GAO- 01- 490 NTIS Technical Reports
Table 1: NTIS Sales, by Customer Segment
Dollars in thousands
Customer segment FY 1998 FY 1999 FY 2000 3- year average
as percentage of total revenue
Business/ Industry $11,045 $10,813 $10,257 74% Schools/ Universities 969 864
857 6 Federal government 755 1065 538 5 State, local, and foreign
governments
883 785 679 5 Individuals 671 710 912 5 Foreign dealers 293 274 279 2
Nonprofit 242 254 248 2
Total $14,858 $14,765 $13,770 99% a
a Does not add to 100 percent due to rounding. Source: NTIS.
Since 1995, we estimate that NTIS has sold one or more copies of about
187,000 (8 percent) of its 2. 46 million reports. 11 As shown in figure 4, a
higher percentage of reports published from 1989 to 1999 have sold than the
newest reports or the oldest reports. Of the 1.8 million reports over 12
years old (pre- 1989), we estimate that only about 1 percent has sold since
1995.
11 This is the number of reports (i. e., titles) that sold one or more
copies, not the total number of copies sold, which would be greater. We used
1995 because that is the year NTIS began recording sales data in its
database. A report that was published in 1995 has had 6 years to sell while
a report published in 2000 has had 1 year to sell. Fewer Than One in Ten
Reports in Repository Has Sold Since 1995
Page 13 GAO- 01- 490 NTIS Technical Reports
Figure 4: Estimated Percentage of Reports Sold in Each Stratum
Source: GAO estimates from NTIS data.
The Assistant to the Director of NTIS described ?best sellers? as products
that have a broad audience appeal or that sometimes are targeted for a
particular audience. These are products for which NTIS does special product
development, marketing, and promotion activities to enhance distribution.
NTIS officials sometimes work with agencies and its contractors to preserve
or resurrect a popular publication that the agency can no longer afford to
produce on its own. NTIS also helps an agency customize its product for
particular audiences, such as the medical community. NTIS? Best Sellers
Comprise a Variety of Reports on Various Topics
0 10
20 30
40 50
60 70
80 90
100 Percentage
1989- 96 1997- 99 2000
Sold Not sold
Year of publication 1988 and earlier
Page 14 GAO- 01- 490 NTIS Technical Reports
According to NTIS data, revenue from the sale of only the best sellers in
fiscal year 2000 totaled over $1 million 12 -about 6 percent of the total
$19 million in revenue from the sale of all paper and electronic products in
the repository that year.
NTIS? best- selling products comprise a variety of reports on various
topics. The following report titles are examples of those that appeared as
best sellers on NTIS? Web site in January 2001:
?Any PIA (Primary Insurance Amount) - A Program to Estimate Social Security
Monthly Benefit Payments?- Explains how much Social Security benefits will
be at retirement.
?Federal Tax Products from the IRS?- Includes forms, publications, and
bulletins.
?Statistical Abstract of the United States?- Reference book containing vital
statistics on the United States.
?US Industry and Trade Outlook?- Business forecasts for over 200 industries.
?Armed Forces Recipe Cards?- 1,300 recipes for groups of 100 that can be
adjusted up or down.
According to the Assistant to the Director of NTIS, these best sellers are
usually sold in one or more of the following formats:
Published books. These books are either printed at GPO (although not
necessarily distributed by GPO) and purchased by NTIS or developed by NTIS
and a joint venture partner. According to NTIS records, these books
generated almost $700,000 in sales in fiscal year 2000.
CD- ROM. In most cases, these NTIS products are developed from published
books as an alternative format and are available only from NTIS. The one
major exception is the IRS CD- ROM. According to NTIS records, these
products generated over $200,000 in sales in fiscal year 2000.
Audiovisual . These products cover a variety of topics. For example, NTIS
sells a 21- minute tape called ?Ergonomic Programs That Work? for $55.00
12 In addition, sales of IRS CD- ROM tax products (forms, publications, and
bulletins) were $1.3 million. We did not include these sales in the best-
seller totals because NTIS officials said this is a unique product that NTIS
bids for and may not always be able to offer.
Page 15 GAO- 01- 490 NTIS Technical Reports
from the Occupational Safety and Health Administration. According to NTIS
records, these generated about $136,000 in sales in fiscal year 2000.
We estimate that about 470,000 of NTIS? 2.46 million reports (19 percent)
could be readily obtained through one or more of the four sources at the
time that we searched. 13 The remaining 2 million reports (81 percent) could
be readily obtained only from NTIS.
As shown in figure 5, the same NTIS reports often were available at more
than one of the four Web sites. For example, of the 10,000 reports published
in 2000, we estimate that 56 percent were available from the issuing
organization, 40 percent from sources identified through Google. com, 17
percent from sources identified through FirstGov. gov, and 6 percent from
GPO. For reports in all age categories, the source with the greatest
probability of having the NTIS report was the issuing organization, and the
source with the least probability was GPO. For reports published from 1989
to 2000, Google. com was able to locate more than twice the percentage of
reports than either FirstGov. gov or GPO?s Web site search engines could
locate.
13 For this estimate, we counted reports available at more than one Web site
only once. Had we not known the complete title of the report and its date of
publication as shown in NTIS? database, we might not have found as many
reports. Nineteen Percent of
the Reports in NTIS? Repository Were Available From Other Public Sources
Page 16 GAO- 01- 490 NTIS Technical Reports
Figure 5: Estimated Report Availability at Different Sources
Source: GAO estimates from Web site data.
As shown in figure 6, reports published in 2000 were over five times more
likely to be available elsewhere than reports published in 1988 or earlier,
and the percentage not readily available from these sources increased with
the age of the report. For example, we estimate that only 12 percent of the
1.8 million reports published in 1988 or earlier were readily available from
the other sources.
0 10
20 30
40 50
60 1988 and earlier 1997- 99 1989- 96 2000 Year of publication Percentage
available
GPO FirstGov. gov Google. com Agency Web site
Page 17 GAO- 01- 490 NTIS Technical Reports
Figure 6: Estimated Reports Available, by Year of Publication
Source: GAO estimates from Web site data.
We estimate that about 173,000 (37 percent) of the nearly 470,000 NTIS
reports available elsewhere could be obtained for free from one or more of
the four Internet sources we searched. For example, the following reports
available for a charge from NTIS, were free from the issuing organization?s
Web site:
?Arizona - 1997 Census of Agriculture State and County Data. Volume 1.
Geographic Area Series Part 3,? March 1999, National Agricultural Statistics
Service, 328 pages. (At NTIS, a paper copy costs $25, microfiche $23.)
?Packaging - and Transportation- Related Occurrence Reports - 1993 Annual
Report,? June 1994, Oak Ridge National Laboratory, Department of Energy, 116
pages. (At NTIS, a paper copy costs $36.50, microfiche $21.)
?Consumer Use of Information - Implications for Food Policy,? June 1999,
Economic Research Service, 32 pages. (At NTIS, a paper copy costs $14,
microfiche $12.) Thirty- Seven Percent of
NTIS? Reports Available Elsewhere Were Free From the Internet
0 10
20 30
40 50
60 70
80 90
100 1988 and earlier 1997- 99 1989- 96 2000 Percentage
Available Not available
Year of publication
Page 18 GAO- 01- 490 NTIS Technical Reports
?Summary Travel Characteristics, District of Columbia. 1995 American Travel
Survey,? October 1997, Bureau of Transportation Statistics, 26 pages. (At
NTIS, a paper copy costs $23, microfiche $12.)
As shown in figure 7, reports published from 1997 to 2000 were considerably
more likely to be available free on the Internet than were reports published
in other years.
Figure 7: Estimated Percentage of Reports Available Free From Other Sources
Source: GAO estimates that were based on Web site data.
As shown in figure 8, we found reports that NTIS had sold since 1995 were
more likely to be available elsewhere than reports that had not sold.
Specifically, we estimate that 44 percent of the reports that sold in all
age strata since 1995 were readily available elsewhere, compared with only
17 percent of the reports that did not sell since 1995. Reports That NTIS
Had
Sold Since 1995 Were More Readily Available Elsewhere Than Those Not Sold
0 5
10 15
20 25
30 35
40 45
50 Year of publication Percentage
1988 and earlier 1989- 96 1997- 99 2000 A
Page 19 GAO- 01- 490 NTIS Technical Reports
Figure 8: Estimated Availability of Reports Sold Since 1995
Source: GAO estimates that were based on Web site data.
Executive branch agencies are required by 15 U. S. C. 3704b- 2 to transfer
to NTIS unclassified scientific, technical, and engineering information
resulting from federally funded research and development activities. NTIS?
implementing regulation 15 C. F. R. Part 1180 (2000) 14 defines scientific,
technical, and engineering information as (1) basic and applied research
that results from the efforts of scientists and engineers in any medium
(including new theory and information obtained from experimentation,
observation, instrumentation, or computation in the form of text, numeric
data, or images) and (2) information that bears on business and industry
generally, such as economic information and market and related information,
if the agency determines such information would be of value
14 NTIS? regulation defines the term ?agency? as used in the Stevenson-
Wydler Act, which specifically includes legislative branch agencies. Section
3704b- 2, however, is applicable only to ?each federal executive department
or agency,? excluding legislative branch agencies. Accordingly, we brought
this issue to the attention of NTIS officials, and they told us they would
consider an appropriate amendment to the regulation. Agencies Did Not
Always Send Their Reports to NTIS, as Required by Law
0 10
20 30
40 50
60 70
80 90
Sold Not sold
Available Not available
Percentage
Page 20 GAO- 01- 490 NTIS Technical Reports
to consumers of the information described in the first part of this
definition. The regulation requires agencies to send NTIS at least one copy
of every final research product and further provides that agencies can enter
into agreements with NTIS to transfer reports using alternative means. Such
alternative means include providing the location for reports on the agency?s
Web site so that NTIS can review the reports and select ones it believes
meet the regulation?s requirements.
However, according to NTIS, agencies are not providing all of the required
reports. To obtain more of those reports, in 2000, NTIS began harvesting 15
electronic versions of reports from agency Web sites. Although it would be
difficult to identify the universe of federal reports that agencies should
have provided to NTIS, harvesting provides some perspective on the extent to
which agencies are complying with the regulation?s intent. For example, in
2000, an employee in NTIS? Information Services Division harvested 1,127
reports 16 from agency Web sites that NTIS believes fit the definition in
their regulation and should have been provided. NTIS harvested 10 or more of
the 1,127 reports from each of the following agencies? Web sites:
Federal Reserve System (169 reports)
International Trade Commission (151)
Office of National Drug Control Policy 17 (108)
Commerce?s International Trade Administration (95)
Department of Transportation?s Federal Highway Administration (61)
Commerce?s Bureau of the Census (52)
Department of Justice (41)
Federal Emergency Management Agency (24)
Federal Communications Commission (20)
Department of Energy?s Energy Information Administration (18)
Agency for International Development (18)
The White House (13)
Department of the Interior?s Fish and Wildlife Service (11)
Department of Agriculture?s Forest Service (10) 15 Harvesting is NTIS?
process of searching agency Web sites and downloading reports that it
believes should have been submitted by the agencies for inclusion in NTIS?
repository. 16 These reports included 19 GAO reports. GAO, as a legislative
branch agency, is not required to submit reports to NTIS. 17 Office of
National Drug Control Policy (ONDCP) officials questioned whether ONDCP- as
an agency of the Executive Office of the President- is covered by the law.
Page 21 GAO- 01- 490 NTIS Technical Reports
Commerce?s National Institute of Standards and Technology (10)
Federal Trade Commission (10) The seven agencies with the most harvested
reports had one or more of three general reasons for not sending their
reports to NTIS:
they were not aware of the law (2 agencies);
they would incur additional costs and duplication of effort to format and
transfer the information to NTIS that is free on the agency?s Web site (4
agencies); and/ or
they thought that a few of the reports harvested by NTIS were not
scientific, technical, or engineering reports and, thus, did not fall within
the law?s scope (3 agencies).
For example, Justice officials did not think that their harvested reports
School Shooter: A Threat Assessment Perspective and Reducing Illegal
Firearms Trafficking: Promising Practices and Lessons Learned were covered
because they were not scientific, technical, or engineering reports. These
officials said that additional guidance was needed on the type of reports
covered under the law. Similarly, Transportation did not think that its
Inspector General?s strategic plan was covered under the law. It is unclear
whether reports like these fall under the definition and should be submitted
to NTIS. However, our cursory review of the titles of the 1,127 harvested
reports found that most, although not all, appeared to contain scientific,
technical, engineering, or business- related information.
NTIS has taken steps to improve compliance and thereby reduce the need for
harvesting. The Assistant to the Director of NTIS said that its Office of
Agency Liaison (1) helps agencies identify the kind of material they should
be sending and (2) reminds agencies of their legal requirements, although
NTIS? efforts are limited by available resources. In addition, at one time,
NTIS and Energy had an agreement- now expired- that NTIS would harvest those
reports it required directly from Energy?s Web site, rather than Energy?s
sending NTIS the reports. The Assistant to the Director of NTIS said that
this arrangement saved NTIS the costs of scanning paper reports and
simplified the agency?s responsibilities under the regulation. He also noted
that Energy made harvesting easy by systematically collecting reports from
the various Energy offices and components at one location on its Web site.
This agreement was not renewed, however, when NTIS and Energy could not
agree on the specific terms under which they would continue this
arrangement.
Page 22 GAO- 01- 490 NTIS Technical Reports
NTIS, however, has not pursued other ways of improving compliance. Although
NTIS found the agreement with Energy to be beneficial, NTIS has no
additional agreements in place. NTIS does provide instructions to agencies
on alternative means for transferring reports- such as identifying the
location of reports on agency Web sites or submitting by other electronic
formats- but only when requested. Further, the Assistant to the Director of
NTIS stated that NTIS does not routinely notify agencies that it harvested
their reports and/ or discuss with them why these reports were harvested.
As a way of keeping the Congress informed on whether agencies are sending
their technical reports to NTIS, the Secretary of Commerce is required,
under subsection (b) of 15 U. S. C. 3704b- 2 to report annually to the
Congress ?on the status of efforts under this section to ensure access to
federal scientific and technical information by the public.? The Assistant
to the Director of NTIS said that (1) they have not submitted any of these
required reports because they understood the Congress and the executive
branch were trying to eliminate surplus reporting requirements and (2) NTIS
had erroneously assumed that their annual financial statement audit reports
would include the necessary information to meet this requirement.
Attempting to achieve greater compliance with the requirement for agencies
to send research reports to NTIS is appropriate under the government?s
current strategy for disseminating scientific, technical, and engineering
information. However, the rising demand for electronic products, coupled
with increasing availability of this information on the Web, raises more
fundamental issues about how the information should be collected, stored,
and disseminated- and, specifically, about the future of NTIS itself. These
issues include whether the government?s current approach- a central
repository to provide permanent public access on a self- sustaining basis-
is still appropriate in the current environment.
In recognition of this and in response to a request from several Members of
Congress, the National Commission on Libraries and Information Science
(NCLIS)- an independent agency that provides advice to the President and the
Congress on information issues- conducted a study of Improving
Compliance Would Not Diminish the Need to Address Fundamental Policy Issues
Page 23 GAO- 01- 490 NTIS Technical Reports
public information dissemination, which included an assessment of NTIS.
NCLIS? final report was issued in January 2001. 18
Specifically, the policy issues raised by NCLIS and our studies include the
following:
How should ?scientific, technical, and engineering information? be defined
in the current environment? NTIS has defined scientific, technical, and
engineering information broadly to include information directly related to
business and industry that results from the efforts of scientists and
engineers. NTIS, as the agency responsible for administering the law, is
given discretion to define such terms. However, this broad definition has
led to the agency?s acquiring and disseminating publications that appear to
bear little relationship to business and industry, such as some of the ?best
sellers.? This raises the question of whether policymakers believe that the
mission of NTIS should be to disseminate this variety of documents or
whether a narrower interpretation of scientific, technical, and engineering
information would be more appropriate in the current environment.
NCLIS also addressed this issue. According to NCLIS, NTIS- in an effort to
find revenue to support its operations- has expanded the scope of its
collection well beyond its primary mission of collecting, organizing, and
disseminating scientific and technical information. NCLIS concluded that
NTIS? scope should not be restricted to science and technology narrowly
defined. However, they added that the scope should not include general
public information that does not have a strong and direct relationship with
business, industry, or technology.
Is there a need for a central repository to disseminate scientific,
technical, and engineering information? The increasing availability of newer
publications from sources other than NTIS, along with the relatively low
demand for older reports, may raise the question of whether a central
repository for scientific, technical, and engineering information is still
18 A Comprehensive Assessment of Public Information Dissemination, National
Commission on Libraries and Information Science, January 26, 2001.
Page 24 GAO- 01- 490 NTIS Technical Reports
needed. 19 Answering this question requires knowing (1) the extent to which
agencies could be relied upon to provide permanent public access to
publications and (2) in the absence of a central repository, what type of
bibliographic information and control over government publications are
needed to ensure that reports can be identified and located indefinitely.
In its study, NCLIS stated that the notion of a central source for
government technical information to the public should not be abandoned.
NCLIS, however, recommended consolidating the missions, functions, and
programs of NTIS- along with other government entities responsible for
disseminating government information to the public- into new independent
agencies in each branch of the government.
What obligations should a central repository have to retain all reports
indefinitely? We estimate that only 1 percent of the NTIS reports over 12
years old has sold since 1995. This information suggests that it may be
useful to consider whether all scientific, technical, and engineering
information should be retained permanently. How long to retain such
information, however, would have to be based on demand patterns for older
documents and on the costs and benefits of retaining documents of various
age strata. This decision should also be based on a policy determination
about permanent access and the need to maintain a central repository, as
opposed to reliance on other repositories such as libraries.
The NCLIS study also addressed the issue of permanent access to federal
information. The study concluded that permanent public availability of and
access to government information are critical parts of the overall strategy
to meet the public information needs of the public and need to be
strengthened. Further, the study concluded that NTIS is the only fail- safe
source to make research results available when they are no longer available
from the originating agency.
19 In our report Major Management Challenges and Program Risks: A
Governmentwide Perspective (GAO- 01- 241, January 2001), we suggested that
the broad- based and rapid changes facing government today call for a
fundamental review of existing programs and operations that address the
following questions: (1) Do the conditions that caused the program to be
created still exist? (2) Would we design the same program in the same way if
we were designing it today? (3) Are there other government programs or
policies that are not coordinated with or are working at cross- purposes to
this program? (4) Is the program operating in the most economical,
efficient, and effective manner? (5) Is the program affordable and
sustainable both now and over the longer term? and (6) Is the program
targeted appropriately to the beneficiaries most in need of federal
assistance?
Page 25 GAO- 01- 490 NTIS Technical Reports
If NTIS continues to exist as a central repository, what should be its
business model? Issues concerning the financial viability of NTIS and the
availability of research reports on agency Web sites raise questions about
the appropriateness of NTIS? business model. NTIS currently is required to
be self- sustaining while maintaining permanent public access to scientific,
technical, and engineering information in their repository.
In its report, NCLIS recommended one such alternative business model.
Specifically, NCLIS identified a number of NTIS functions that they believe
generally benefit the public and recommended that these functions be funded
with appropriations. These so- called ?public good? functions included (1)
collection or acquisition of reports, (2) cataloging and indexing, and (3)
offering the NTIS database and full text of reports- to the extent that they
are not available on agency Web sites- on NTIS servers for free public
access.
In addition to its statutory function of being a permanent repository and
disseminator of scientific, technical, and engineering information, NTIS
provides a variety of other fee- based services for agencies. In NTIS?
permanent repository of about 2.5 million reports, the number being added
has been declining in recent years, and about 75 percent of the reports are
now over 12 years old. While demand for electronic NTIS products is rising,
research reports and other scientific, technical, and engineering
information maintained by NTIS are also increasingly available on agency Web
sites and through other public sources- often at no cost.
Further, although NTIS has taken action to encourage agency compliance with
the requirement to send it their research reports, agencies are often
unaware of the law or confused about its application. NTIS has not been
successful in pursuing agreements with agencies to facilitate the transfer
of required documents. Finally, NTIS has not submitted its required annual
report on the status of its efforts to ensure public access to scientific
and technical information. As a result, NTIS has missed an opportunity to
inform the Congress of the challenges that it faces.
Broader policy questions remain to be answered about how government
information should be disseminated in this Internet era and about NTIS?
role, if any, in that effort. Conclusions
Page 26 GAO- 01- 490 NTIS Technical Reports
In light of developments in report dissemination through the Internet, the
Congress may wish to reexamine how scientific, technical, and engineering
information is disseminated to the public. In this reexamination, the
Congress could specifically look at how this information is defined; whether
there is a need for a central repository of this information, such as at
NTIS; and, if a central repository is maintained, whether all information
should be retained permanently, and what business model should be used to
manage it.
In the interim, while the Congress is reexamining the broader policy issues
of the dissemination of government research information, we recommend that
the Secretary of Commerce instruct the Director, NTIS, to
contact the major departments and agencies to outline the law?s
requirements for sending reports to NTIS and offer assistance in helping
these organizations comply;
pursue agreements with agencies to facilitate harvesting reports from
agency Web sites as an alternative method of achieving compliance;
establish, in consultation with the President?s Council on Integrity and
Efficiency, a method for monitoring future compliance, which could involve
the participation of agency inspectors general; and
submit annual reports to the Congress, as required, on the status of
actions taken to ensure access by the public to federal scientific,
technical, and engineering information, including any action taken to
implement these recommendations.
We provided a draft of this report to the Secretary of Commerce for review
and comment. The Secretary?s comments are reprinted as appendix II.
In a letter dated April 26, 2001, the Secretary of Commerce agreed with our
identification of the key policy issues- how to define scientific and
technical information, whether there is a need for a central repository, how
long reports should be maintained, and what is the proper business model for
NTIS. The Secretary stated that, of these issues, the key question is
whether there is a need for a central repository in view of the increasing
availability of newer publications from sources other than NTIS. The
Secretary also noted the need for a central repository depends on whether
the information will be permanently maintained by agencies and whether it
will be easy to locate absent the kind of bibliographic control NTIS
provides.
The Secretary concurred with the thrust of our recommendations and said he
would determine how best to implement them. He said he will ask the Matters
for
Congressional Consideration
Recommendations for Executive Action
Agency Comments
Page 27 GAO- 01- 490 NTIS Technical Reports
Acting Under Secretary for Technology to determine if there are any
practical ways, consistent with the statutory requirement that NTIS be self-
sustaining, to make NTIS? information collection more widely available to
the general public.
As agreed with your offices, unless you publicly announce the contents of
this report earlier, we will not distribute it until 30 days from the date
of this report. At that time, we will send copies to the Honorable Donald L.
Evans, the Secretary of Commerce; the Honorable Mitchell E. Daniels, Jr.,
Director, OMB; the Honorable Michael F. DiMario, the Public Printer; Mr.
Ronald Lawson, Director, NTIS; and other interested congressional
committees. Copies will also be available at our Web site at www. gao. gov.
Should you have any questions on matters contained in this report, please
contact me at (202) 512- 6240 or Alan Stapleton, Assistant Director, at
(202) 512- 3418. We can also be reached by e- mail at koontzl@ gao. gov or
stapletona@ gao. gov, respectively. Other key contributors to this report
included Wendy Ahmed, William Isrin, Michael Jarvis, Frank Maguire, Warren
Smith, and Ray Wessmiller.
Linda D. Koontz Director, Information Management Issues
Appendix I Scope and Methodology
Page 28 GAO- 01- 490 NTIS Technical Reports
To identify the functions that NTIS performs, we interviewed National
Technical Information Service (NTIS) officials and reviewed various reports
and studies, including NTIS? handbook NTIS Services for Federal Agencies.
To determine the quantity of NTIS? research holdings, we added the number of
newer reports recorded in NTIS? ?STAR? database to the estimated number of
older reports recorded on index cards in NTIS? file drawers. To estimate the
number of reports recorded on index cards, we multiplied the number of
drawers times the average number of cards in selected drawers to arrive at
our initial estimate of 1.5 million reports.
To determine if the data in STAR were valid and reliable, we
randomly selected 59 accession numbers from the total recorded in STAR and
attempted to locate these reports in NTIS to verify that the title and date
of issuance were accurate and
randomly selected 59 reports that were located near (above, below, to the
right, or to the left) of the 59 reports to verify that they were recorded
in STAR or in the index card drawers, depending on when the reports were
issued.
NTIS officials said that old reports were recorded on index cards located in
hundreds of drawers. To verify that NTIS had the reports recorded on these
cards and that the titles and dates were accurate, we randomly selected 75
cards and asked NTIS to attempt to locate these reports. These same 75 cards
were part of our random sample of 150 reports 12 years or older.
We found that NTIS? computer generated data were sufficiently valid,
complete, and reliable for purposes of our review, but the data in the index
card drawers were not reliable. In particular, we found that
the reports were available and the titles were accurately recorded in STAR
for all 59 randomly selected accession numbers; the year of the report?s
issuance was accurately recorded for 54 of the 59 accession numbers, but the
5 remaining dates did not match exactly; and
nearly all of the reports- 57 of 59- that we randomly selected from NTIS?
warehouse shelves were accurately recorded either in STAR or in the index
card drawers.
However, 40 of the 75 reports randomly selected from the drawers were also
found in STAR. This duplication was not expected because NTIS Appendix I:
Scope and Methodology
Appendix I Scope and Methodology
Page 29 GAO- 01- 490 NTIS Technical Reports
officials said they make their entries into the database directly from the
document and no longer create an index card. We did not anticipate finding
cards in the drawers that were evidently created after 1964, which is the
year NTIS began the database that was later incorporated into STAR. However,
an NTIS official in the Information Services Division said that during the
period of 1964- 80, NTIS created cards as well as an electronic record. They
did this because at that time not all employees who needed to work with the
database had the easy electronic access to it that is now routinely provided
by personal computers. This double- counting of reports in the two systems
means that our earlier estimate of 1.5 million reports recorded in the index
card drawers was overstated by about 53 percent. Therefore, we reduced our
estimate of the number of reports recorded on index cards accordingly, from
1.5 million to 700,000.
Of the remaining 35 unduplicated reports, NTIS was not able to locate 14 of
them. Thus, we reduced by 40 percent the estimated number of reports that do
not duplicate the STAR file and that NTIS can find and provide to customers-
from 700,000 to 420,000 reports. This does not mean, however, that NTIS lost
nearly 300,000 reports from its repository, since 10 of the 14 cards in the
sample (representing an estimated 200,000 cards in the entire index file)
were ?source filled?- that is, NTIS retains only a bibliographic record (not
the report) and asks the issuing organization to send the report directly to
the customer, if it is available. The Information Services Division official
said that these reports may no longer be available from the sources that
originally issued them.
Adding the 420,000 reports from the index cards to the 2.04 million reports
recorded in STAR, we estimate that NTIS has about 2.46 million reports that
it can find in the repository or at other agencies. This total is
understated to the extent that subscriptions are entered annually in STAR
even though a subscription actually represents the dissemination of more
than one document over the course of a year.
To determine the age of NTIS? research holdings, we used the reports? date
of publication in the STAR database as of mid- November 2000 to sort the
reports into four age strata: 2000; 1997 to 1999; 1989 to 1996; and 1988 or
earlier. For the 1988 or earlier category, we also added the estimated
number of reports recorded on index cards.
To determine the demand for NTIS? research holdings, we analyzed various
NTIS statistical and financial data on sales of reports, including the
financial statement audit reports for fiscal year 1997 through the draft
Appendix I Scope and Methodology
Page 30 GAO- 01- 490 NTIS Technical Reports
report for fiscal year 2000. Sales data came primarily from NTIS? database
called ?CIS. PUB,? which has records of all sales of reports since May 1995.
To determine if the reports in NTIS? repository were readily available
elsewhere, we first randomly selected 600 report accession numbers from the
repository- 525 recorded in STAR and an additional 75 accession numbers from
the index card drawers. These 600 reports consisted of 150 randomly selected
from each of our 4 age strata. The 150 in the oldest category (1988 or
earlier) consisted of 75 randomly selected from the approximately 1. 4
million reports in that age category in STAR plus 75 randomly selected from
those recorded in index card drawers.
Because we selected a sample of reports, all of the results from this sample
are estimates that are subject to sampling errors. These sampling errors
measure the extent to which samples of this size and structure are likely to
differ from the population they represent. Each of the sample estimates is
surrounded by a 95- percent confidence interval, indicating that we can be
95 percent confident that the interval contains the actual population value.
Unless otherwise noted, 1 the 95- percent confidence intervals for
percentage estimates throughout this report do not exceed plus or minus 10
percentage points around the estimate.
Next, we searched the Internet for each of the 600 reports in our sample,
using the following steps (in this order): (1) on the issuing organization?s
Web site and, if unavailable there, through e- mail to its Office of Public
Information or equivalent; (2) on the Government Printing Office?s Web site;
(3) on the FirstGov. gov Web site; and (4) on Google. com?s Web site. 2
Because organization officials may be more responsive to an e- mail request
from us than they would be from the public, we did not use our computers to
send the e- mail requesting reports. Thus, the agencies? responses to our e-
mails should be similar to what any citizen would expect who inquires about
how to obtain a report. If an official at any of these sources told us the
report was available for a fee, we assumed it was available and did not
order the report because of the time and expense involved. If we found in a
previous step (e. g., searching an organization?s
1 For our estimate that 44 percent of reports that sold in all age strata
since 1995 were available elsewhere, the 95- percent confidence interval is
plus or minus 15 percentage points around the estimate.
2 Google. com was selected (rather than another search engine) after
discussions with the requesters? offices.
Appendix I Scope and Methodology
Page 31 GAO- 01- 490 NTIS Technical Reports
Web site) that the report was available, we still attempted to find the
report at all remaining sources (except the Office of Public Information).
Finally, we used our sample results to estimate what portion of the total
population of NTIS reports was available through the four different sources
discussed above in each age category. We performed a separate analysis of
our sample results to show whether reports that had sold one or more copies
since 1995 were more or less readily available through the same four
sources, compared with reports that had not sold.
To determine whether federal agencies were sending their research reports to
NTIS for sale to the public during calendar year 2000 as required by law, we
asked NTIS officials to provide automated data from the STAR database that
would allow us to identify all of the reports in their repository that they
had harvested from the issuing agency?s Web site during that period. Then,
to determine why agencies had not provided the reports, we asked officials
at the seven agencies with the most harvested reports why they had not sent
their reports to NTIS. These seven agencies represented more than half of
all harvested reports. Although we did not review the content of all 1,127
harvested reports to determine if they should have been sent to NTIS, we
reviewed the report titles to determine if they appeared to be scientific,
technical, engineering, or business- related information. We did not
evaluate whether each of the harvested reports was covered under the law.
We conducted our review from September 2000 to April 2001, in accordance
with generally accepted government auditing standards.
Appendix I Scope and Methodology
Page 32 GAO- 01- 490 NTIS Technical Reportss
Appendix II: Comments From the Department of Commerce
(512025)
The first copy of each GAO report is free. Additional copies of reports are
$2 each. A check or money order should be made out to the Superintendent of
Documents. VISA and MasterCard credit cards are also accepted.
Orders for 100 or more copies to be mailed to a single address are
discounted 25 percent.
Orders by mail:
U. S. General Accounting Office P. O. Box 37050 Washington, DC 20013
Orders by visiting:
Room 1100 700 4 th St., NW (corner of 4 th and G Sts. NW) Washington, DC
20013
Orders by phone:
(202) 512- 6000 fax: (202) 512- 6061 TDD (202) 512- 2537
Each day, GAO issues a list of newly available reports and testimony. To
receive facsimile copies of the daily list or any list from the past 30
days, please call (202) 512- 6000 using a touchtone phone. A recorded menu
will provide information on how to obtain these lists.
Orders by Internet
For information on how to access GAO reports on the Internet, send an email
message with ?info? in the body to:
Info@ www. gao. gov or visit GAO?s World Wide Web home page at: http:// www.
gao. gov
Contact one:
Web site: http:// www. gao. gov/ fraudnet/ fraudnet. htm
E- mail: fraudnet@ gao. gov
1- 800- 424- 5454 (automated answering system) Ordering Information
To Report Fraud, Waste, and Abuse in Federal Programs
*** End of document. ***