Federal Research: Interim Report on the Pilot Technology Access Program
(Letter Report, 03/07/94, GAO/RCED-94-75).
As part of the federal effort to bolster U.S. industries'
competitiveness, the Pilot Technology Access Program provides small
businesses with access to: (1) computerized databases containing
technical and business information that they typically are unaware of,
or cannot afford; and (2) experts knowledgeable in a wide range of
technical fields. The program is being implemented at several small
business development centers, which provide counseling, training, and
research assistance to small businesses nationwide. The centers are run
by the Small Business Administration. In 1991, six centers in Maryland,
Missouri, Oregon, Pennsylvania, and Wisconsin were chosen to implement
the program. This report discusses the status, implementation, and
evaluation of the program. GAO also includes the views of the
participating centers' directors on the program's effect on small
businesses' productivity and innovation.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: RCED-94-75
TITLE: Federal Research: Interim Report on the Pilot Technology
Access Program
DATE: 03/07/94
SUBJECT: Small business assistance
Grants to states
Federal/state relations
Grant monitoring
Technical assistance
Program evaluation
Information centers
Reference service operations
Surveys
Project monitoring
IDENTIFIER: Maryland
Oregon
Missouri
Pennsylvania
Wisconsin
Texas
Small Business Innovation Research Program
NIST Pilot Technology Access Program
NIST Manufacturing Extension Partnership
**************************************************************************
* This file contains an ASCII representation of the text of a GAO *
* report. Delineations within the text indicating chapter titles, *
* headings, and bullets are preserved. Major divisions and subdivisions *
* of the text, such as Chapters, Sections, and Appendixes, are *
* identified by double and single lines. The numbers on the right end *
* of these lines indicate the position of each of the subsections in the *
* document outline. These numbers do NOT correspond with the page *
* numbers of the printed product. *
* *
* No attempt has been made to display graphic images, although figure *
* captions are reproduced. Tables are included, but may not resemble *
* those in the printed version. *
* *
* A printed copy of this report may be obtained from the GAO Document *
* Distribution Facility by calling (202) 512-6000, by faxing your *
* request to (301) 258-4066, or by writing to P.O. Box 6015, *
* Gaithersburg, MD 20884-6015. We are unable to accept electronic orders *
* for printed documents at this time. *
**************************************************************************
Cover
================================================================ COVER
Report to Congressional Committees
March 1994
FEDERAL RESEARCH - INTERIM REPORT
ON THE PILOT TECHNOLOGY ACCESS
PROGRAM
GAO/RCED-94-75
Federal Research
Abbreviations
=============================================================== ABBREV
BIAS - Pennsylvania Business Intelligence Access System
GAO - General Accounting Office
MOTAP - Missouri Technology Access Program
NASA - National Aeronautics and Space Administration
OIC - Oregon Innovation Center
OTAP - Oregon Technology Access Program
SBA - Small Business Administration
SBDCs - Small Business Development Centers
TAPP - Pilot Technology Access Program
TAP/Texas - Texas Technology Access Program
Teltech - Teltech Resource Network Corporation
TEN - Maryland Technology Expert Network
Cover
================================================================ COVER
Letter
=============================================================== LETTER
B-255654
March 7, 1994
Congressional Committees
Several federal programs exist or are being proposed to bolster U.S.
industries' competitiveness. One ongoing program directed to help
small businesses compete was created when the Congress amended the
Small Business Act in 1991 to establish the Pilot Technology Access
Program (commonly referred to as TAPP).\1 TAPP provides small
businesses with access to (1) computerized data bases containing
technical and business information that they typically are not aware
of, or cannot afford, and (2) experts knowledgeable in a wide range
of technical fields.
The law specified that TAPP be implemented at several small business
development centers that are administered by the Small Business
Administration (SBA). The centers deliver counseling, training, and
research assistance to small businesses through a network of 56
centers--one or more in each of the 50 states--and more than 700
service locations. Although the SBA provides funding for TAPP, the
National Institute of Standards and Technology (the Institute)--an
agency within the Department of Commerce's Technology
Administration--manages and monitors the program. In 1991, the
Institute selected six centers located in Maryland, Missouri, Oregon,
Pennsylvania, and Wisconsin to implement TAPP. Public Law 102-140
requires us to submit an interim report on the implementation of TAPP
to several congressional oversight committees. The law also requires
us to submit in September 1994 a final report evaluating the
effectiveness of the program.\2
Accordingly, this interim report provides information on the status,
implementation, and evaluation of TAPP. In addition, this report
includes the judgments of the participating centers' directors on
TAPP's effect on small businesses' productivity and innovation.
--------------------
\1 TAPP was originally established by section 232 of the Small
Business Administration Reauthorization and Amendments Act of 1990,
which was repealed by section 609 of Public Law 102-140 and replaced
with the current program.
\2 As discussed in this report, the data needed to evaluate the
effectiveness of the program are not yet available and may not be
available for some time. Accordingly, we are discussing with
congressional oversight committees an appropriate date for the final
report.
RESULTS IN BRIEF
------------------------------------------------------------ Letter :1
During the first year of TAPP, the Institute assumed a "hands off"
approach, allowing the six centers to have autonomy and control over
the development of their pilot programs. During this time, the
centers focused on program development. By the end of the first
year, all of the centers were providing some services to their small
business clients, although the Institute considered two programs--one
in Oregon and one in Wisconsin--nonperformers because of the centers'
lack of progress in developing and implementing their programs. By
the end of the second year, Institute officials told us that
Wisconsin's program had improved but that Oregon's had to drop out of
TAPP because it could not obtain matching state funding.
While each program director views TAPP as a valuable tool for
educating small businesses on the value of computerized information,
only a limited number of clients were served by the program during
the first year, which began in September 1991. The number of clients
who used the program at five of the six centers ranged from 1 to 13
percent of the total number of clients counseled at the centers.
(Texas did not identify the number of clients who used the program
for this period.) However, program use has generally picked up, with
several of the programs exceeding the number of first-year users
during the first three quarters of the second year. Furthermore, the
centers have experienced relatively low demand for technical experts.
In addition, the centers initially had some problems marketing the
benefits of using the program to their counselors and clients;
however, during the second year, the centers devoted more time to
marketing the program.
Because the program is new, it is too early to make any overall
judgment about its effectiveness. Furthermore, the program did not
begin collecting uniform data from the centers until March 1993--well
into the program's second year. In addition, no uniform data have
been collected from the centers on client satisfaction or program
impact. Only recently did the centers develop uniform questionnaires
to assess program impact. Although Institute officials believe that
the questionnaires support their program management needs, we believe
that in their present form the questionnaires will not generate the
accurate, valid data needed to evaluate the impact of the program
because they lack clarity and precision.
BACKGROUND
------------------------------------------------------------ Letter :2
TAPP was modeled on a Minnesota state program established in
1989--Minnesota Project Outreach--which provides small businesses
with access to technical experts and computerized data bases with
technical and business information. Minnesota selected Teltech
Resource Network Corporation (Teltech)--a national supplier of
technical and business knowledge--to develop and deliver these
services. The perceived success of this program in providing
user-friendly services to small businesses that would not otherwise
have the means or the ability to obtain needed technical information
created the stimulus for the TAPP legislation. (App. II describes
Minnesota Project Outreach.)
Section 232 of Public Law 101-574, the Small Business Administration
Reauthorization and Amendments Act of 1990, in establishing TAPP,
stipulated that SBA work in consultation with the Institute and the
National Technical Information Service.\3 In early 1991, the
Institute and SBA signed a memorandum of understanding that resulted
in the Institute's implementing TAPP on behalf of and in close
cooperation with SBA. In October 1991, section 609 of Public Law
102-140 repealed section 232 of Public Law 101-574 and replaced it
with the current program.
Officials from the Institute's Manufacturing Extension Partnership
manage and monitor the program. These officials view the program as
contributing to their objective of coordinating statewide technical
assistance programs to small businesses and as a vehicle that
provides scientific and technical information to all types of
businesses. In managing the program, Institute officials visit all
of the centers at least once a year to review their progress in
developing their programs. The Institute also requires that the
centers' directors or program administrators attend quarterly
meetings to discuss their progress with each other. The centers must
also submit to the Institute quarterly and annual reports that
characterize their progress toward program goals. An SBA official
receives copies of all reports, participates in site visits and
directors' meetings, and is involved in making all decisions about
TAPP. SBA officials see the program as a way to expand the range of
services to small businesses and attract a broader range of small
businesses to the small business development centers.
Small businesses typically visit centers for business management
assistance. When a small business owner visits a TAPP-funded small
business development center, a counselor determines whether the
client also needs technical or scientific information. If the client
needs these services, the counselor will provide access to TAPP data
bases. TAPP can also provide a list of experts in a wide range of
technical and scientific fields so that a client can identify who he
or she can consult with on a particular subject.
--------------------
\3 Both the National Technical Information Service and the Institute
are components of the Department of Commerce. Within the Department,
the Institute was the agency that undertook the consultative role.
PROGRAM STATUS
------------------------------------------------------------ Letter :3
In September 1991, six small business development centers were
selected to participate in TAPP on the basis of their ability to (1)
increase access by small businesses to online data base services that
provide business and technical information relevant to the needs of
industries in their areas, (2) integrate TAPP with existing federal
and state technical and business assistance resources, and (3)
continue providing technology access after the pilot program which is
authorized through fiscal year 1995, is terminated. As a
prerequisite to receiving funding, each center had to obtain matching
state funds.
FIRST YEAR OF OPERATION
FOCUSED ON PROGRAM
DEVELOPMENT
---------------------------------------------------------- Letter :3.1
During the first year of the program, which began in September 1991,
each of the TAPP centers received $200,000, with the exception of
Maryland, which received $50,400. According to Institute officials,
during the first year of TAPP, the Institute assumed a hands off
approach in order to allow the centers to develop unique pilot
programs tailored to the specific environments of their states.
Furthermore, the program was short-staffed; only one person was
available part-time to manage the program until July 1992, when the
State Technology Extension Program (now part of the Manufacturing
Extension Partnership) hired a manager whose responsibilities
included managing TAPP.
Because the TAPP centers spent much of the first year developing
their programs, and because Institute managers did not want to
prescribe an information collection system before the programs were
established, the Institute did not obtain uniform data from the
centers on what type of clients they were serving or what type of
information the centers were providing to clients. For example, in
their first-year annual reports, the centers provided client usage
data using different terms such as "log-ins," registered users,
and/or research projects. In addition, data base searches were
categorized differently by each of the programs. Nevertheless, all
of the programs were providing some TAPP services to small businesses
by the end of the first year.
After receiving first-year reports from the centers, Institute
officials expressed concern that some of the programs were providing
market information to the exclusion of technical information. The
Institute defines nontechnical information as marketing, management
planning, trademark, and vendor information and technical information
as product/process, development/modification, and patent or
regulatory information. Furthermore, the officials were concerned
that many of the clients receiving this information were not
technical or manufacturing concerns but rather small local
retail-type businesses. (TAPP data bases contain "nontechnical"
information such as demographics as well as technical data.) The
officials cautioned the centers that there must be a balance of
information provided because they believe that the program should
also provide scientific and technical information to small businesses
as prescribed by Public Law 102-140; the officials also believe that
interactive data bases are not the most cost-effective means of
providing marketing information.
Although the Institute has no official policy concerning the specific
percentage of program activity that should be devoted to providing
scientific and technical information, several Institute officials
told us that they hope that eventually 50 percent of the information
provided to TAPP clients will be technical and/or scientific.
However, three of the centers' directors told us that they were
concerned that directing the program to providing technical
information would narrow the number of interested participants. In
that regard, some directors expressed concern that they not be
limited to providing scientific and technical information under the
program. The directors believe that clients benefit from marketing
information obtained from data base searches.
The following is an example (from Pennsylvania's 1991-92 annual
report) of how a client used marketing data obtained from TAPP:
A 30-year-old company that manufactures fiber optic cables and
local area network cable systems needed to obtain information on
market size, potential areas of growth, and competitors in order
to analyze which European countries to target in future
marketing efforts. A small business development center
counselor retrieved relevant information from a TAPP data base.
With this information as part of its marketing plan, the company
was chosen to participate in a Pennsylvania program that
subsidized attendance in European trade shows for promising
state firms. Subsequently, the company's president submitted a
quote for a cable order to a European firm.
TAPP'S SECOND YEAR FOCUSED
ON IMPROVING PROGRAM
OPERATIONS
---------------------------------------------------------- Letter :3.2
During the second year of the program, the Institute awarded each of
the six centers $190,400, except Maryland, which received $50,000,\4
and the Institute encouraged the centers to analyze whom they
provided information to and what type of information they were
providing. In so doing, the Institute provided the centers in March
1993 with a TAPP client tracking format, developed in cooperation
with the centers' directors, and required that they use it to prepare
client usage statistics for quarterly reports. This format requires
the centers to categorize clients by type of business, Standard
Industrial Classification Code (developed by Commerce) category, age
of firm, and type of information received as scientific/technical or
nontechnical. The first reports using the new format were submitted
in April 1993. Quarterly reports also describe progress toward
program goals, characterize businesses that use the program, and
include case studies of those that use the program.
--------------------
\4 Institute officials told us that Maryland received as much funding
as it requested for the first and second years of the program.
RECOMPETITION FOR PROGRAM
FUNDING
---------------------------------------------------------- Letter :3.3
In November 1992, the Institute and SBA officials notified the TAPP
centers that they had decided to require the centers to submit
proposals to compete for third-year TAPP funding. According to the
officials, this decision was made because (1) two states (Oregon and
Wisconsin) were not operational 1-1/2 years into TAPP--that is, they
were not operating according to the statement of work included in
their proposal; (2) total program funding for the third year was
being cut and it would therefore be better to consolidate the number
of states receiving funding rather than cut the amount of funding to
all six states; and (3) the Institute wanted to ensure that states
with operational programs were chosen for funding since they would be
able to better use the funding. To further preserve the integrity of
TAPP, Institute and SBA officials agreed that the recompetition would
be limited to the existing six programs. After the recompetition,
the Institute awarded $170,000 each to Maryland, Missouri,
Pennsylvania, Texas, and Wisconsin in September 1993. (Oregon
decided not to submit a proposal during the recompetition because it
could not obtain matching state funding.)
According to the Institute, the recompetition for TAPP funds caused
the centers to be more responsive to the Institute's requests.
Institute officials said that earlier the centers did not provide
them with requested information, such as business plans, and that as
part of evaluating their programs, they had yet to ascertain customer
satisfaction. The officials also said that the centers' reluctance
to provide them with information could in part be attributable to
extensive data requirements already imposed by SBA. By the later
part of TAPP's second year, the centers responded to the Institute's
request that they monitor client satisfaction and developed a
standard client evaluation methodology. However, two states said
that the recompetition process inhibited their ability to share
information with other centers because they did not want to give away
any competitive advantage.
THE INSTITUTE'S PERSPECTIVE
ON THE STATUS OF TAPP
---------------------------------------------------------- Letter :3.4
Currently, Institute officials are generally satisfied with the
centers' progress in developing their programs. In particular, for
the first 1-1/2 years of TAPP, Institute officials believe that Texas
and Missouri had the strongest programs because these states
initially made the most progress in developing their programs. For
example, Missouri integrated its program into the state small
business support infrastructure, which was an explicit part of the
legislation creating TAPP, and attempted to provide a mix of
technical and business information to its clients. However,
Institute officials believe that the distinction between Texas and
Missouri and the other centers has become less pronounced as they
have developed and focused their programs. For example, the
officials note that Wisconsin--one of the original nonperformers--has
progressed under the direction of a new program administrator who
developed a technical expert network and a manual for counselors that
facilitated implementation of the program. Furthermore, Institute
officials note that they had concerns that Oregon's program
development was behind schedule and that it was not providing a mix
of technical and business information to clients. Nevertheless, an
Institute official said that Oregon made progress in developing its
program in the last 6 months before it dropped out of TAPP because it
could not obtain matching state funding.
THE SIX TAPP CENTERS HAVE
DIFFERENT APPROACHES BUT
SIMILAR OPERATING EXPERIENCES
------------------------------------------------------------ Letter :4
The Institute and SBA view TAPP as a pilot project and as a result
provided limited direction to the six states as they established
their programs. Consequently, the recipient centers used their
grants to create diverse, unique programs. However, the programs are
similar in that each of the six states has contracted with a computer
vendor or a supplier of technical and business information to provide
access to data bases that contain scientific, technical, and
marketing information.
Typically, the technical and business information companies provide
specialists who give technical support to TAPP counselors and
clients. These specialists are generally available to search data
bases for clients, to help counselors conduct data base searches, to
provide training for counselors and clients on how the program works,
and in some cases, to promote TAPP. Missouri is an exception because
it provides in-house specialists to conduct data base searches.
Other centers either rely entirely on the information specialists
provided by the information suppliers for data base searches or use
them as a backup to help counselors conduct searches. Some centers
allow clients to do their own data base searches directly.
Counselors, information specialists, or clients can access
information from data base searches on a wide range of subjects. For
example, clients can receive patent information, information on their
competitors, or addresses of potential distributors of their product.
Some of the technical information suppliers have data bases with the
names of technical experts that clients can also access. In other
cases, states use other sources for access to technical experts. For
example, Pennsylvania uses the Ben Franklin Institute--a state-funded
nonprofit corporation that provides financial and technical
assistance to businesses--to provide access to data bases but uses
PENNTAP, a statewide information and technology access program, to
find technical experts.
Although the TAPP centers have some broad program similarities, they
also have many differences. For example, the centers vary in the way
they (1) assign roles to counselors, (2) provide services, (3) target
different audiences to market the program to, and (4) require payment
from users. For example, under Pennsylvania's program, counselors
conduct data base searches for clients, whereas in Missouri the
counselors refer clients to technical experts for marketing
information or data base searches. Furthermore, Missouri's program
has a very integrative approach to clients, which can take them from
an initial idea through product development and marketing, while
other programs, such as Oregon's, refer clients to an outside vendor
for data base searches and technical experts.
The centers also market and charge for their TAPP services
differently. For example, Maryland targets participants in the Small
Business Innovation Research program (an SBA program established to
strengthen the research and development role of small innovative
companies), while Texas targets manufacturers and technology-oriented
companies in different industries. Each program has established fees
for data base searches, and small businesses often have to pay some
amount for consultations with experts. These fees vary from center
to center and are largely dependent upon the amount of time spent on
a data base search and the amount of subsidization each program has
decided to provide. In Wisconsin, for example, clients get their
first data base search free and are billed directly by the
information vendor for subsequent usages, while Texas bills clients
50 percent of the cost of additional assisted searches after an
initial free search. (For a detailed description of each program,
see app. III.)
SMALL PERCENTAGE OF CENTERS'
CLIENTS USED TAPP DURING THE
FIRST YEAR OF THE PROGRAM,
BUT PRELIMINARY CLIENT
FEEDBACK IS POSITIVE
---------------------------------------------------------- Letter :4.1
In general, uniform client data on the program were unavailable
during TAPP's first year, which began in September 1991. For
example, Texas did not collect data on the number of clients who used
the program until the third-quarter of the second year of the
program. On July 22, 1993, as part of the recompetition process,
Institute staff asked each center to go back and tabulate program
activity in terms of the number of clients who actually used TAPP
services. Available information indicates that during the first year
of the program, between 78 and 224 clients used TAPP, or 1 to 7.2
percent of all of the clients counseled within each of the four small
business development centers located in Maryland, Missouri, Oregon,
and Pennsylvania. For this same period, 238 clients, or about 13
percent of Wisconsin's small business development center's clients,
used TAPP services. (Texas did not calculate the number of clients
counseled.) During the first year of the program, the five programs
served an average of about 163 clients. However, as table 1
illustrates, program use has generally picked up, with several of the
programs exceeding first-year usage numbers in the first three
quarters of the second year of the program.
Table 1
TAPP Clients
No. of clients in
No. of clients first three
in first year of quarters of second
Program location program year of program
---------------------- ---------------- ------------------
Maryland 224 139
Missouri 146 201
Oregon 78 \a
Pennsylvania 130 236
Texas \b 171
Wisconsin 238 267
------------------------------------------------------------
Note: We did not verify the accuracy of the data in this table,
which we obtained from Oregon's 1991-92 annual report and the other
five programs' 1994 proposals for funding.
\a Information unavailable.
\b Texas collected data only on the number of TAPP searches for this
period.
Although all TAPP clients have not been surveyed and each of the
programs had its own approach to requesting feedback, preliminary
information from all of the centers indicates that clients view the
program favorably. For example, one client who used the Missouri
program credits it with creating his business, while another client
said that the Pennsylvania program helped her make a good decision
about allocating her resources. A client from the Wisconsin program
said that TAPP saved him time in obtaining needed information.
All of the centers' directors agree that TAPP's purpose is to make
small businesses more competitive by educating them on the value and
use of online information systems. Furthermore, all of the
participating program administrators and center directors believe
that the program is a positive addition to small business development
centers' services and has a positive impact on clients' productivity
and innovation. In particular, directors told us that the program
has saved clients time, provides timely information, and helps
clients position themselves against their competition.
LOW DEMAND FOR TECHNICAL
EXPERT NETWORK SEARCHES
---------------------------------------------------------- Letter :4.2
Several of the centers told us that there is little client demand for
technical expert searches. In addition, we reviewed the other
centers' third-quarter second-year reports to see how many requests
for searches of the expert network were made and found that Texas had
8 requests during this period out of a total of 93 data base
searches, while Wisconsin conducted 108 searches, but only 16 of
these were searches for technical experts. One center's director was
not sure if the low use is attributable to not marketing the program
well or if no market exists for these services. Another director
attributed the low demand to clients' concerns over the potential
expense of these services or concerns about experts' stealing ideas.
In addition, several of the directors said that it is difficult to
develop technical expert networks--developing a network can be
time-consuming and challenging with respect to identifying such
experts.
TAPP HAS POSED MARKETING
CHALLENGES FOR CENTERS
---------------------------------------------------------- Letter :4.3
Several of the centers' directors said that the program initially was
tough to "sell" to their counselors--a key factor in terms of
providing TAPP services to potential clients. Centers' counselors
typically work with small businesses that have cash flow problems or
companies that need to obtain financing. Generally, the counselors
are not familiar with science or technical areas or computer data
bases. One counselor said that at the beginning of the program, she
had a learning curve to overcome in order to understand the data
bases and was not sure what the program could deliver. As a result,
counselors needed to be educated on what the program does and what
the program is about. All of the centers have provided training for
counselors, and two centers told us that they have created reference
materials that describe the program or describe key processes to help
counselors provide services. For example, Wisconsin's program has
developed a description of all of its services, which it gives to
each counselor. In addition, counselors are generally not accustomed
to charging fees for services because most services from small
business development centers are free. An information specialist who
provides support for Pennsylvania's program said that it was
initially difficult for counselors to charge clients for program
services but that most counselors became accustomed to doing it.
According to several centers' directors, their programs have also had
problems marketing TAPP to clients. The directors have had trouble
convincing clients about the benefits of data bases or have clients
who have never used online services. For example, one counselor said
that clients are reluctant to use the program because they are afraid
of the cost and do not understand the technology. One center
director summed up his concerns about marketing by saying that there
are a lot of up-front costs in developing demand.
As a result, some of the programs have targeted specific clients for
TAPP services. For example, as previously noted, Maryland is
targeting Small Business Innovation Research program participants to
provide them with information that will move them from Phase I
(designed to determine the scientific and technical merit and
feasibility of a proposed idea) to Phase II (designed to further
develop the idea) of the Small Business Innovation Research program.
Texas is targeting manufacturers and technology-oriented firms.
Wisconsin is concentrating on working with existing manufacturers and
technology-based companies. In addition, some of the programs are
conducting demonstrations for interested businesses. For example,
Texas conducts a workshop called "Technology at Your Fingertips," in
which it asks participants (potential clients) for search ideas and
demonstrates the program's searching capabilities.
PROGRAM EVALUATION WEAKNESSES
AND OTHER CONCERNS
------------------------------------------------------------ Letter :5
Evaluating the impact of TAPP on small businesses is complicated by
two factors: (1) The Institute did not develop an evaluation
methodology or collect uniform data during the first year of the
program, and (2) it is hard to link the information provided to small
businesses to increased productivity. Furthermore, the diversity and
uniqueness of each of the programs will complicate any evaluation.
The Institute did not develop a method for evaluating TAPP before
making its initial awards in September 1991 or during the first year
of the program. The program's first year was highly developmental;
that is, each center was structuring its program and choosing data
bases and technical expert networks. Each center tracked program use
differently, using its own definitions and techniques. As a result,
first-year data cannot be compared across programs. During the
second year of the program, in March 1993, the Institute provided the
centers with a specific format for tracking the number of clients who
used the program and gathering preliminary data on client
satisfaction. However, several centers still had not conformed to
the assigned format in their reports submitted in April 1993.
Furthermore, the centers have not widely or uniformly surveyed
clients to determine their satisfaction with TAPP or received
sufficient response rates from clients. For example, Missouri
surveyed all 146 of its clients and received only 42 responses, while
Wisconsin using a different questionnaire surveyed all of its clients
and received a 58-percent response rate. Pennsylvania attempted to
telephone survey its clients but was unsuccessful because it had
problems getting in touch with them.
In March 1993, the Institute began requiring each center to conduct a
postcard survey of its clients similar to an earlier survey conducted
by the Maryland center. During the first year of the program,
Maryland asked its clients (1) if they received information that they
needed, (2) if they used the information to make a business decision,
(3) what type of information was most useful, (4) if they would be
willing to use the program in the absence of a subsidy, and (5) what
prices they would consider paying for TAPP services. Only 60
companies were surveyed and 47 responded. Eighty-three percent of
the respondents said that Maryland's program, the "Technology Expert
Network," provided them with information that they needed, and 66
percent said that it provided them with information that was used to
make a business decision.
According to the Institute, three additional centers (Missouri,
Oregon, and Texas) have conducted postcard surveys of their small
business clients. These surveys did not ask the same questions, so
comparisons cannot be made across programs. Furthermore, Missouri
sent its survey to clients during the second quarter of the program's
second year, and Texas and Oregon sent their surveys to clients
during the third quarter of the program's second year. Oregon's
preliminary results, with only 17 clients responding of 58 polled,
indicate that 11 respondents found the overall quality of TAPP to be
excellent.
As part of their fiscal year 1994 proposals for funding, Institute
and SBA officials encouraged the centers to develop a standard client
evaluation methodology. The methodology includes (1) a questionnaire
on client satisfaction that will be distributed to clients directly
following completion of the first data search and dissemination of
information, (2) a questionnaire on client impact that will be sent
to clients 6 months after receiving the information from the initial
data search, (3) and a competitive position survey that will be
mailed to clients 1 year after the initial data search to find out
what impact the program has had on the company's competitive
position.
We believe that these questionnaires, in their present form, will not
generate reliable and valid data for a number of reasons. First, the
questionnaires lack precision and clarity. For example, the
questionnaires do not provide definitions of terms or instructions
that will help respondents understand how to answer questions.
Second, questions in the instruments do not make a direct connection
between the program's impact on clients and their businesses'
productivity. For example, the client behavior section of one of the
questionnaires does not ask the respondent if he or she was impacted
by the program and then ask the degree to which he or she was
impacted by the program. Instead, the questionnaire lists factors
that could have occurred without TAPP and does not give clients the
ability to state the degree to which the program had an influence on
them. Finally, the questionnaires fail to ask clients basic
questions, such as if they were satisfied or dissatisfied with the
program, or request them to identify key factors that led them to be
satisfied or dissatisfied with the program.
Because of these concerns, we have little confidence that the
questionnaires in their present form will generate data that will
allow the Institute to gauge client satisfaction and accurately
represent the impact of the program on its small business clients.
Furthermore, it is possible that low response rates to these
questionnaires will preclude the Institute from obtaining valid data.
On November 1, 1993, we met with Institute officials to discuss these
concerns and suggested that they address them so that the
questionnaires will yield valid, accurate data.
In November 1993, the TAPP Program Manager reviewed the
questionnaires with the program directors at the quarterly directors'
meeting and provided them with input from the Manufacturing Extension
Partnership's Associate Director for Program Quality. Our concerns
with the questionnaires were discussed at this meeting, and Institute
officials told us that the directors would attempt to "tighten up"
the questionnaires. In addition, in September 1993, the Associate
Director for Program Quality told us that her future role will be to
develop evaluation methodologies for all partnership programs,
including TAPP.
In December 1993, we met with Institute officials, who told us that
the centers have implemented the first questionnaire on client
satisfaction and have since made some modifications based on our
comments. We reviewed the revised questionnaire and determined that
only minor changes had been made, and most of our initial concerns
had not been resolved. The officials also told us that the centers'
directors are modifying the two other questionnaires in response to
our comments but do not know when they will be complete. The
officials said that the questionnaires meet their program management
needs and that they view them as only one tool that can be used to
evaluate TAPP.
Although uniform data on program effectiveness are not available,
each of the centers has collected some anecdotal information from
clients that generally indicates that small businesses are satisfied
with TAPP. The following examples were obtained from TAPP annual
reports:
A research and development company involved in the electrochemical
treatment of wastewater and the treatment and preservation of
certain highly perishable food products contacted TAPP to obtain
patent information, current and historical literature on
relevant technology and its applications in environmental
industries, and the location of important but difficult to find
books pertaining to the technology. The company received this
information and used it to respond to several large nationally
issued requests for proposals for services.
A small high-technology firm that specializes in research,
development, and marketing a group of proprietary chemical
formulations--especially for the microelectronics manufacturing
industry--contacted TAPP. The firm requested hard-to-get
information on the research, development, patenting, and
marketing of its products. TAPP provided the firm with this
information, which was incorporated in Small Business Innovation
Research program proposals and used to better understand the
potential for growth in areas targeted for its new products.
Finally, even when more impact data are collected, it will be
difficult to link the information that the program provides to an
increase in small businesses' productivity. Centers' directors and
administrators agree that the program's impact is and will be
difficult to measure. For example, one director said that he was
unable to identify data on program impact because it takes a long
time between when a company gets a piece of information and sees a
result of increased productivity. Another director volunteered that
it is difficult to correlate literature searches and counseling to
productivity and innovation.
OTHER COMPETITIVENESS
INITIATIVES
---------------------------------------------------------- Letter :5.1
Since TAPP was enacted, other initiatives for bolstering U.S.
industries' competitiveness have been proposed or are already
implemented. For example, over the next several years, the
Department of Defense's Advanced Research Projects Agency will be
providing funding for defense technology conversion, reinvestment,
and transition assistance. Also, the National Aeronautics and Space
Administration has a National Technology Transfer Center that can
give businesses access to data from federal laboratories for specific
technologies that can be transferred to the private sector. In
addition, states, such as Washington, have developed technical
assistance programs for small businesses without TAPP funding.
Institute officials agreed that there needs to be coordination
between the existing and proposed programs designed to bolster
competitiveness to avoid duplication and to make use of existing
resources. An Institute official told us that he has started work on
a resource map that geographically represents a comprehensive listing
of industry assistance programs available to small businesses. When
completed, this map will enable policymakers to better direct
resources across the country and give service providers knowledge
about programs that they can tell clients about. However, this
effort is projected to take 1 year to complete.
CONCLUSIONS
------------------------------------------------------------ Letter :6
The TAPP centers have created diverse and somewhat unique programs.
By the end of the first year, all of the programs were providing some
TAPP services to small business clients. On the basis of comments
centers' directors and counselors made about their experiences to
date, it appears that the key to the program's development is each
center's ability to market TAPP services to potential clients,
provide counselor training, and provide good technical support
services for clients. These activities have a direct impact on
promoting small businesses' awareness and usage of the program,
improving counselors' comfort level with using the program, and
ensuring that clients receive information that they can use and need.
These activities are crucial to being able to attract and maintain
demand for the program. It is also important that the centers
communicate with one another to refine their programs, share ideas,
and help resolve common problems and concerns.
It is too early, however, to make any overall judgment about the
effectiveness of TAPP. The program is new, and no uniform data have
been collected from the centers on client satisfaction or client
impact. Only recently did the centers develop uniform questionnaires
to assess program impact. Although Institute officials said that
these questionnaires adequately support their management needs, we
believe that in their present form they will not generate accurate,
valid data needed to evaluate the impact of the program. To
facilitate the collection of such data, we have met with Institute
officials and provided them with feedback on the questionnaires.
Although it was beyond the scope of our mandate to review similar
programs aimed at increasing the competitiveness of U.S. industry,
the government should be aware of all such existing and proposed
programs, including TAPP, and make sure that they are coordinated to
avoid duplication.
RECOMMENDATION TO THE SECRETARY
OF COMMERCE
------------------------------------------------------------ Letter :7
Because an accurate determination of the effectiveness of the Pilot
Technology Access Program will be critical to future decisions about
the program, we recommend that the Secretary of Commerce direct the
Administrator of the Technology Administration to take steps to
improve the precision and clarity of the questionnaires that managers
use to collect client feedback on the program. The questionnaires
should provide definitions where appropriate and ask direct questions
that will determine client satisfaction and program impact.
AGENCY COMMENTS
------------------------------------------------------------ Letter :8
We discussed the factual contents of this report with Institute
officials from the Manufacturing Extension Partnership who manage
TAPP and SBA's Associate Administrator of Small Business Development
Centers. SBA officials generally agreed with the fact sheet that we
provided to them. Institute officials, however, raised concerns
about some of the language and word choices in the fact sheet that
they believed overstated or oversimplified the program's problems.
The officials provided us with detailed comments that have been
incorporated into the report where appropriate. The officials also
maintain that existing evaluation tools are sufficient to meet the
information and internal management needs of the individual state
directors and Institute and SBA officials. However, our
recommendation goes beyond what the Institute says it needs for
internal management purposes to ensure that an accurate evaluation of
client satisfaction and program impact can be made. The Congress
needs accurate program impact information in order to make an
informed decision on whether to extend the program. As requested, we
did not obtain written agency comments on a draft of this report.
---------------------------------------------------------- Letter :8.1
We conducted our review between March and December 1993 in accordance
with generally accepted government auditing standards. (App. I
provides a detailed description of our scope and methodology.) We are
sending copies of this report to appropriate congressional
committees; the Secretary of Commerce; the Administrator of SBA; and
the Director, Office of Management and Budget.
This review was conducted under the direction of Victor S. Rezendes,
Director, Energy and Science Issues, who may be reached at (202)
512-3841 if you have any questions. Major contributors to this
report are listed in appendix IV.
Keith O. Fultz
Assistant Comptroller General
List of Committees
The Honorable Dale L. Bumpers
Chairman
The Honorable Larry Pressler
Ranking Minority Member
Committee on Small Business
United States Senate
The Honorable Ernest F. Hollings
Chairman
The Honorable John C. Danforth
Ranking Minority Member
Committee on Commerce, Science
and Transportation
United States Senate
The Honorable John J. LaFalce
Chairman
The Honorable Jan Meyers
Ranking Minority Member
Committee on Small Business
House of Representatives
The Honorable George E. Brown
Chairman
The Honorable Robert S. Walker
Ranking Minority Member
Committee on Science, Space,
and Technology
House of Representatives
SCOPE AND METHODOLOGY
=========================================================== Appendix I
To obtain information on the Pilot Technology Access Program (TAPP),
we spoke with officials from the National Institute of Standards and
Technology's (the Institute) Manufacturing Extension Partnership in
Gaithersburg, Maryland, and the Associate Administrator from the
Small Business Administration's (SBA) Small Business Development
Centers (SBDC) in Washington, D.C. We reviewed the Institute's files
on each of the six centers selected to implement TAPP. The centers
are located in Maryland, Missouri, Oregon, Pennsylvania, and
Wisconsin. We also reviewed TAPP annual and quarterly reports.
We interviewed Directors and Program Administrators from each of the
six participating centers to obtain their perspectives on their
programs and visited the programs in Maryland, Pennsylvania, and
Wisconsin. We also spoke with SBDC counselors and clients at each
center who participate in and use the program, whose names were
provided by the centers. We obtained the Association of Small
Business Development Center's perspective on TAPP from its legal
representative as well as perspectives from officials from Washington
State and Maine involved in technology access programs that are not
funded by TAPP. We also obtained information on the Minnesota
Project Outreach from a Teltech Resource Network Corporation
(Teltech) official whose company operates the project and delivers
the service.
DESCRIPTION OF THE MINNESOTA
PROJECT OUTREACH PROGRAM
========================================================== Appendix II
Minnesota Project Outreach was adopted in 1989 by the Minnesota
Legislature, whose goal was to provide small manufacturing firms in
Minnesota with access to technical and business knowledge so that
these firms could be competitive in the global economy. The program
targets up to 400 manufacturing firms for service directly.
Eligibility is restricted to companies with annual sales of less than
$10 million. These firms have use of computer software enabling on
demand access to Project Outreach services. Teltech was selected to
develop and deliver this service. Teltech is a national supplier of
technical and business knowledge, including its network of experts,
literature search, and vendor locating services. During the first 2
years of service, the state provided $4.4 million for Project
Outreach.
According to a Teltech official, Teltech has learned several lessons
thus far. First, small businesses attach little value to free
services, with a consequence that they are not used well or at all.
Thus, during the project's third year, the structure of the program
was changed to require that all firms receiving service pay an
initiation fee along with a fee each time the service was used.
Second, clients want to be able to get their question answered
quickly and efficiently with little loss of time and at affordable
rates. Also, clients have a great need for marketing information to
back up their technology idea or product. Last, the importance of
providing a variety of important services through a single point of
contact has been confirmed by clients. For example, sometimes
clients request to speak with an expert and end up doing a patent
search.
DESCRIPTION OF SIX PROGRAMS
========================================================= Appendix III
MARYLAND TECHNOLOGY EXPERT
NETWORK
------------------------------------------------------- Appendix III:1
The Maryland Technology Expert Network (TEN) contracts with
Teltech,\1 a private information vendor, to provide online data base
and expert network services to its small business clients. TEN
provides these services at five regional subcenters throughout the
state. SBDC counselors can conduct data base searches with clients
or request that Teltech conduct such searches or clients can access
the Teltech system directly. Teltech provides access to more than
1,000 worldwide data bases on public and private technology,
services, and expertise and more than 10,000 experts in science and
technology. The Maryland program targets Small Business Innovation
Research phase one award recipients as clients for its services.
--------------------
\1 Some of the SBDCs that currently use Teltech have indicated that
its cost may result in them changing vendors after their contract has
ended.
MISSOURI TECHNOLOGY ACCESS
PROGRAM
------------------------------------------------------- Appendix III:2
The Missouri Technology Access Program (MOTAP) is a complex support
system in which small businesses can gain access to a wide range of
technology and marketing information through caseworkers, technical
and marketing specialists, experts, and data bases. MOTAP services
can be accessed at 12 regional locations, 2 specialized centers, and
a host office that are geographically distributed across the state.
Clients can also use a toll free number for assistance.
Caseworkers identify client problems and refer them to the
appropriate specialists. The program uses data bases such as Dialog,
NASA Recon, and resources available through Teltech to conduct
searches. The program is structured to provide fully integrated
information and counseling efforts across a broad range of technical
areas using primarily "in-house" expertise and by accessing state
resources.
OREGON TECHNOLOGY ACCESS
PROGRAM
------------------------------------------------------- Appendix III:3
The Oregon Technology Access Program (OTAP) contracts with the Oregon
Innovation Center (OIC)--a technology assistance organization--to
conduct all data base searches and referrals to technical experts for
clients. Clients can access OIC by calling a 1-800 number, using an
electronic bulletin board or via public access sites, including 20
locations throughout Oregon. OIC provides access to data bases such
as CompuServe and Dialog and Teltech's technical expert data bases.
OTAP has targeted certain key industries and SBIR recipients as
clients for its services.
TECHNOLOGY ACCESS PROGRAM FOR
TEXAS
------------------------------------------------------- Appendix III:4
The Technology Access Program for Texas (TAP/Texas) contracted with
an in-state resource, Texas Information Network System, to provide
online data base searching capability and access to technical
experts. Clients can also access Teltech, Knowledge Express, and
NASA Midcontinent Technology Transfer Center data bases by contacting
56 locations throughout the state. In addition, clients can access
these resources directly. The TAP/Texas program has targeted
manufacturers and technology-oriented companies in different
industries as clients for its services.
WISCONSIN TECHNOLOGY ACCESS
PROGRAM
------------------------------------------------------- Appendix III:5
The Wisconsin Technology Access Program (WisTAP) contracts with
Teltech, Dialog, TechSearch (at the engineering library at the
University of Wisconsin at Madison), the Office of Industrial
Research (at the University of Wisconsin at Milwaukee), and several
other technical consultants to provide data base searches and
technical counseling. SBDC counselors, at 10 subcenters throughout
Wisconsin, conduct online data base searches for clients
interactively (on-site) or remotely (telephone requests to WisTAP
office or TechSearch). In addition, the Applied Population
Laboratory can provide clients with demographic and economic data for
specific geographic areas. WisTAP also provides clients with access
to the Wisconsin Expert Network, which includes nearly 4,000 local
technical experts.
PENNSYLVANIA BUSINESS
INTELLIGENCE ACCESS SYSTEM
------------------------------------------------------- Appendix III:6
The Business Intelligence Access System (BIAS) contracts with the Ben
Franklin Technology Center's Business Information Center to provide
online and other research training to counselors and assist clients
and counselors on complex data base searches. Otherwise BIAS
counselors access Telebase to conduct searches for clients at 14
locations throughout Pennsylvania. BIAS offers clients access to
over 850 data bases that include information on trade and specialized
industry newsletters, articles from business sections of local papers
across the country, reports from financial analysts, and summaries of
current university- and government-funded research projects. This
system notifies the user of the cost of each search, thereby allowing
counselors to gauge the cost of searches. SBDC contracts with
PENNTAP, a statewide information and technology access program, to
provide technical experts to interested clients.
MAJOR CONTRIBUTORS TO THIS REPORT
========================================================== Appendix IV
RESOURCES, COMMUNITY, AND ECONOMIC
DEVELOPMENT DIVISION, WASHINGTON,
D.C.
Jim Wells, Associate Director
Robert E. Allen, Jr., Assistant Director
Ronald E. Stouffer, Assignment Manager
Elise B. Bornstein, Evaluator-in-Charge
Alice G. Feldesman, Supervisory Social Science Analyst