Federal Research: Preliminary Information on the Small Business
Technology Transfer Program (Chapter Report, 01/24/96, GAO/RCED-96-19).
The nation's research institutions--its universities, federal
laboratories, and nonprofit research organizations--account for about a
quarter of all the scientists and engineers in the United States. To
help move new knowledge from research institutions to industry, Congress
authorized the Small Business Technology Transfer Pilot Program for
three years, beginning in fiscal year 1994. Federal agencies rated the
quality and commercial potential of winning proposals favorably in the
program's first year. Technical experts, however, were cautious about
the commercial potential, possibly because of the newness of the
program. Agencies have taken steps to avoid potential conflicts of
interest that might arise because of the close connection between
federal agencies and research and development centers. In addition, the
Departments of Defense and Energy, which accounted for almost all of the
awards involving such centers, have taken steps to prevent the centers
from using privileged information in preparing program proposals. Agency
officials expressed differing views on the Technology Transfer Program's
effect on the Small Business Innovation Research Program and other
agency research and development, although none of them suggested any
negative effects, such as competition between the two programs for
quality proposals. The similarity of the two programs, however, raised
questions about the need for a new program.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: RCED-96-19
TITLE: Federal Research: Preliminary Information on the Small
Business Technology Transfer Program
DATE: 01/24/96
SUBJECT: Technology transfer
Research and development contracts
Non-profit organizations
Conflict of interest
Small business assistance
Disadvantaged persons
Scientific research
Technical proposal evaluation
IDENTIFIER: Small Business Technology Transfer Pilot Program
Small Business Innovation Research Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to Congressional Committees
January 1996
FEDERAL RESEARCH - PRELIMINARY
INFORMATION ON THE SMALL BUSINESS
TECHNOLOGY TRANSFER PROGRAM
GAO/RCED-96-19
Federal Research
(307732)
Abbreviations
=============================================================== ABBREV
AFOSR - Air Force Office of Scientific Research
ARPA - Advanced Research Projects Agency
BMDO - Ballistic Missile Defense Organization
DOD - Department of Defense
DOE - Department of Energy
FAR - Federal Acquisition Regulation
GAO - General Accounting Office
NASA - National Aeronautics and Space Administration
NIH - National Institutes of Health
NSF - National Science Foundation
ONR - Office of Naval Research
R&D - research and development
SBA - Small Business Administration
SBIR - Small Business Innovation Research Program
STTR - Small Business Technology Transfer Pilot Program
Letter
=============================================================== LETTER
B-270359
January 24, 1996
The Honorable Christopher S. Bond
Chairman
The Honorable Dale Bumpers
Ranking Minority Member
Committee on Small Business
United States Senate
The Honorable Jan Meyers
Chairman
The Honorable John J. LaFalce
Ranking Minority Member
Committee on Small Business
House of Representatives
This report was mandated by the Small Business Research and
Development Enhancement Act of 1992. The report focuses on the
implementation of the Small Business Technology Transfer (STTR) Pilot
Program, which was established by the act. In particular, the report
discusses (1) the quality and commercial potential of the STTR
Program's research as shown by technical evaluations of the winning
proposals in the first year of the STTR Program, (2) how agencies
addressed potential conflicts of interest resulting from the
involvement of federally funded research and development centers in
the program, and (3) agencies' views on the effects of and need for
the STTR Program in view of its close similarity to the Small
Business Innovation Research Program.
Please contact me at (202) 512-3841 if you or your staff have any
questions. Major contributors to this report are listed in appendix
I.
Victor S. Rezendes
Director, Energy and
Science Issues
EXECUTIVE SUMMARY
============================================================ Chapter 0
PURPOSE
---------------------------------------------------------- Chapter 0:1
The nation's research institutions--its universities, federal
laboratories, and nonprofit research institutions--account for about
a quarter of all the scientists and engineers in the United States.
In an effort to move new knowledge from research institutions to
industry, the Congress authorized the Small Business Technology
Transfer (STTR) Pilot Program for 3 years, beginning in fiscal year
1994. The Congress also required that GAO report on the
implementation of the program. For this report, GAO reviewed (1) the
quality and commercial potential of the STTR Program's research as
shown by technical evaluations of the winning proposals in the first
year of the program, (2) how agencies addressed potential conflicts
of interest resulting from the involvement of federally funded
research and development centers in the program, and (3) agencies'
views on the effects of and need for STTR in view of its close
similarity to the Small Business Innovation Research (SBIR) Program.
BACKGROUND
---------------------------------------------------------- Chapter 0:2
As part of the legislation (P. L. 102-564, Oct. 28, 1992)
reauthorizing the SBIR Program, the Congress established the STTR
Program to (1) stimulate technological innovation, (2) use small
businesses to meet federal research and development (R&D) needs, (3)
foster and encourage socially and economically disadvantaged persons'
participation in technological innovation, and (4) increase the
private sector's commercialization of innovations derived from
federal R&D.
Five agencies participate in the STTR Program, including the
Departments of Defense (DOD), Energy (DOE), and Health and Human
Services' National Institutes of Health (NIH); the National
Aeronautics and Space Administration (NASA); and the National Science
Foundation (NSF). Each agency manages its own program, while the
Small Business Administration (SBA) plays a central administrative
role, issuing policy directives and annual reports for the program.
The legislation authorized each agency having an external R&D budget
in excess of $1 billion to set aside not less than 0.05 percent of
that budget for the STTR Program in fiscal year 1994, not less than
0.1 percent in fiscal year 1995, and not less than 0.15 percent in
fiscal year 1996. In the STTR Program's first year, the agencies
expended about $20 million and estimated that funding will triple to
$60 million in the third and last year of the program.
The STTR Program provides funding for phase I and phase II awards.
Work in phase I is intended to determine the scientific, technical,
and commercial merit and feasibility of ideas; the work is generally
not to exceed 1 year. Work in phase II further develops the proposed
ideas and is generally not to exceed 2 years. The statute generally
limits the size of awards in phases I and II to $100,000 and
$500,000, respectively.
The STTR Program is closely modeled on the SBIR Program, which was
established in 1982. The two programs share the same four goals and
other basic features, including participation by many of the same
agencies, the use of a percentage of the external budget for funding,
and a three-phase approach.
However, the two programs differ in one important respect. In order
to be eligible for an STTR award, a small business must collaborate
with a nonprofit research institution, such as a university, a
federally funded research and development center, or other entity.
This collaboration is permitted under the SBIR Program but is not
mandatory. This special STTR requirement, according to a 1992
report,\1 was to provide a more effective mechanism for transferring
new knowledge from research institutions to industry.
The Congress expressed concern about potential conflicts of interest
resulting from the role of research and development centers in the
program. For example, a conflict could arise if a center and a small
business submitted an STTR proposal as partners and, at the same
time, the center helped the agency judge its own and other proposals.
As a result, the Congress required agencies to take steps to avoid
these potential conflicts of interest.
--------------------
\1 H.R. Rep. No. 554, 102nd Cong., 2nd Sess., pt. 1 (1992). The
report accompanied H.R. 4400, a predecessor to the bill (S. 2941)
that was enacted.
RESULTS IN BRIEF
---------------------------------------------------------- Chapter 0:3
Federal agencies rated the quality and commercial potential of the
Small Business Technology Transfer Program's winning proposals
favorably in the first year of the program. While the quality of the
proposals was rated highly, technical experts were somewhat cautious
about the commercial potential, possibly because of the newness of
the program.
Agencies have taken steps to avoid potential conflicts of interest
that might arise because of the close connection between federal
agencies and research and development centers. In addition, the
Departments of Defense and Energy, which accounted for almost all of
the awards involving such centers, have taken steps to prevent the
centers from using privileged information in preparing Small Business
Technology Transfer Program proposals.
Agency officials expressed differing views on the Technology Transfer
Program's effect on the Small Business Innovation Research Program
and other agency research and development, although none of them
indicated any negative effects such as competition between the two
programs for quality proposals. The similarity of the two programs,
however, raises an issue about the need for the new program.
PRINCIPAL FINDINGS
---------------------------------------------------------- Chapter 0:4
EVALUATIONS WERE FAVORABLE,
BUT THE REVIEW PROCESS
VARIED
-------------------------------------------------------- Chapter 0:4.1
Evaluations of the STTR Program's proposals by technical experts
showed a generally high degree of confidence in the quality of the
research and commercial potential. As one example, DOE rated the
quality of proposed research for all of its winning proposals as high
as that for those proposals in the top 10 percent of research in the
Department. Evaluations of the commercial potential were also
favorable but occasionally expressed concern about the cost of the
product that might result or the limited size of its potential
market. Such reservations may be due to the newness of the program
and the risk associated with many of the projects.
The evaluation process, upon which these findings depended, varied
greatly. In NIH, NASA, NSF, and DOE, the selection of winning
proposals relied on consensus among several technical reviewers. In
DOD, by contrast, a single reviewer was frequently responsible for
the technical evaluation of proposals. Of DOD's 105 winning
proposals, 47 received only one technical review that, in 37 cases,
provided only limited analysis to support the ratings.
AGENCIES HAVE TAKEN STEPS TO
AVOID CONFLICTS OF INTEREST
-------------------------------------------------------- Chapter 0:4.2
In general, the five agencies have taken steps to prevent conflicts
of interest from occurring as a result of involvement by research and
development centers in the STTR Program. DOD, DOE, and NIH developed
specific policy guidance concerning potential conflicts of interest.
NASA and NSF relied on general procedures in their procurement
guidelines to avoid conflicts of interest. DOD and DOE have taken
steps to prevent research and development centers from gaining an
unfair advantage in preparing proposals. DOD has carefully
restricted participation by its own laboratories, and DOE has
prohibited agency officials from requesting or receiving assistance
from personnel in research institutions in the preparation of
research topics. These two agencies accounted for all but 3 of the
32 awards involving research and development centers as research
partners during the first year of the pilot program. NIH, NASA, NSF,
and DOE also made use of outside peer review; by contrast, DOD tended
to rely on technical expertise within the Department.
VIEWS DIFFER ON EFFECTS OF
AND NEED FOR THE PROGRAM
-------------------------------------------------------- Chapter 0:4.3
Agency officials expressed differing views on the effect of the STTR
Program on the agencies' other R&D, including the SBIR Program.
However, the agencies provided no evidence at this early point in the
program to suggest that the STTR Program was competing for quality
proposals with the Innovation Research Program or reducing the
quality of the agencies' R&D in general. Instead, some officials
noted some potentially beneficial effects such as greater
collaboration between small businesses and research institutions in
the Innovation Research Program.
The similarity of the two programs, however, raises an issue about
the need for the pilot program. One way to assess the need for the
program is to determine its effectiveness in transferring technology
from research institutions to the marketplace. Such information will
not be ascertainable for at least several years because of the time
needed to turn an initial concept into a marketable technology.
AGENCY COMMENTS
---------------------------------------------------------- Chapter 0:5
STTR program managers and other officials from DOD, DOE, NASA, NIH,
NSF, and SBA reviewed a draft of this report and provided oral
comments. In general, these officials regarded the report as
accurate and balanced. DOD officials said that even a single
technical evaluation of a proposal may be adequate because of the
reviewer's level of expertise. Agency officials also provided minor
technical corrections that have been incorporated.
INTRODUCTION
============================================================ Chapter 1
The Small Business Research and Development Enhancement Act of 1992
established the Small Business Technology Transfer (STTR) Pilot
Program and authorized it for 3 years, beginning in fiscal year 1994.
The act also reauthorized the Small Business Innovation Research
(SBIR) Program, which was established in 1982. The STTR Program was
modeled closely on the SBIR Program. Both of the programs, for
example, share the same four major goals:
Stimulating technological innovation.
Using small business to meet federal research and development (R&D)
needs.
Fostering and encouraging participation by minorities and
disadvantaged persons in technological innovation.
Increasing private-sector commercialization of innovations derived
from federal R&D.
The two programs include many of the same agencies, and the same
agency offices administer the programs.
In spite of the numerous points in common, the two programs differ in
one important respect. The STTR Program requires a small business to
collaborate with a U.S. nonprofit research institution, such as a
university; a contractor-operated, federally funded research and
development center; or other entity, in order to be eligible for an
award. This collaboration is permitted under the SBIR Program, but
it is not mandatory.
THE ADMINISTRATION OF THE STTR
PROGRAM
---------------------------------------------------------- Chapter 1:1
Agency participation in and funding for the STTR Program followed the
general approach established for the SBIR Program. Agencies having a
budget of more than $1 billion annually in fiscal years 1994, 1995,
or 1996 for external R&D are authorized to set aside a percentage of
this amount for the STTR Program. The act sets the percentage at not
less than 0.05 percent in fiscal year 1994, not less than 0.1 percent
in fiscal year 1995, and not less than 0.15 percent in fiscal year
1996. Five agencies--DOD, DOE, NASA, NIH, and NSF--currently
participate in the program. In the SBIR Program, by contrast, the
same five agencies and six additional agencies with smaller budgets
for external R&D were authorized to set aside a significantly higher
percentage and amount of money. When the STTR Program reaches its
highest authorized funding percentage in fiscal year 1996, it will
receive about $60 million, according to Small Business Administration
(SBA) calculations; when the SBIR Program reaches its maximum funding
percentage in fiscal year 1997, it will receive about $1 billion.
As with SBIR, the legislation required agencies to solicit proposals
for R&D projects. The solicitation lists and describes the topics to
be addressed by company proposals and invites companies to submit
proposals for consideration. Each agency is responsible for
targeting research areas and administering its own STTR funding
agreements. Such agreements include a contract, grant, or
cooperative agreement entered into between a federal agency and a
small business for the performance of experimental, developmental, or
research work funded in whole or in part by the federal government.
The legislation also required SBA to issue a policy directive for the
general conduct of STTR Programs within federal agencies. The
directive was to include such features as simplified, standardized,
and timely SBIR solicitations; a simplified, standardized funding
process; and minimization of the regulatory burden for small
businesses participating in the program. The STTR policy directive
was issued in August 1993 and remains in effect. Federal agencies
are also required to report key data to SBA.
To be eligible for an STTR award, small businesses must be
independently owned and operated,
other than the dominant firms in the field in which they are
proposing to carry out STTR projects,
organized for profit,
the employer of 500 or fewer employees (including its affiliates),
and
at least 51 percent owned by U.S. citizens or lawfully admitted
permanent resident aliens.
The law established a three-phase structure for the program. The
first phase, not to exceed 1 year, is designed to determine the
scientific, technical, and commercial merit and feasibility of a
proposed idea. The second phase, not to exceed 2 years, is designed
to further develop the idea. The statute established $100,000 and
$500,000 as the general limits for phases I and II awards,
respectively. The third phase is somewhat more flexible and
difficult to define. In general, this phase is expected to result in
commercialization or further continuation of R&D. No STTR funding is
provided for phase III. Unlike phases I and II, phase III has no
general time limits. In addition, phase III can include not only
federal non-STTR funds but private-sector funds.
For selection of phase I proposal awards, SBA's 1993 policy directive
stated that agency criteria shall give primary consideration to
scientific and technical merit along with the potential for
commercialization. According to the directive, funding for phase II
shall be based upon the results of phase I and the scientific and
technical merit and commercial potential of the phase II proposal.
THE RATIONALE FOR THE STTR
PROGRAM
---------------------------------------------------------- Chapter 1:2
Given the similarities between the SBIR and STTR Programs, the
question of the rationale for the STTR Program naturally presents
itself. In the 1992 report cited above, the Chairman of the House
Committee on Small Business provided two basic answers to this
question when the STTR Program was under consideration. The report
states that the program addresses a core problem in U.S. economic
competitiveness and that the existing SBIR Program does not provide a
direct mechanism for technology transfer.
In discussing the first point, the report noted that the nation's
research institutions--its universities, federal laboratories, and
nonprofit research institutions--contain enormous scientific and
technical resources. The report also noted that the institutions
employ one in four scientists and engineers in the nation and perform
nearly $40 billion in R&D each year, or one-quarter of all R&D
conducted in the United States. In addition, the report stated that
perhaps the core of the U.S. competitiveness problem is the
inability to translate its worldwide leadership [in science and
engineering] into technology and commercial applications that benefit
the economy. The report concluded that what is needed is an
effective, systematic "technology transfer" mechanism to move new
knowledge from the research institution to industry, where it can be
exploited for the national good.
In discussing the second point, the report noted that SBIR has turned
out to be a remarkably effective mechanism for commercializing ideas
in the small business community. However, according to the report,
SBIR is less effective at fostering commercialization of ideas that
originate in universities, federal laboratories, and nonprofit
research institutions. STTR would provide a strong incentive for
small companies and researchers at universities, federal
laboratories, and nonprofit research institutions to find each other
and work together because the only way they can access STTR funding
is by collaborating.
Thus, STTR was envisioned primarily as a technology transfer program,
in which promising concepts originating in the nonprofit research
community would move toward commercialization. In accomplishing this
objective, researchers in nonprofit research institutions would ally
themselves with small businesses and play a greater role in STTR
projects than they could play in SBIR projects.
AWARDS MADE DURING THE FIRST
YEAR OF THE STTR PROGRAM
---------------------------------------------------------- Chapter 1:3
Table 1.1 summarizes basic information about the phase I awards made
during the first year (fiscal year 1994) of the STTR Program.
Table 1.1
An Overview of STTR Awards, Fiscal Year
1994
((Dollars in millions))
Number
of Phase R&D
proposal I Universi center
Agency s awards Amount ties s Other
------------------ -------- ------ ------ -------- ------ ------
DOD 911 105 $9.8 85 15 5
NIH 283 48 4.7 41 2 5
NASA 159 21 2.1 17 1 3
NSF 110 11 1.1 11 0 0
DOE 487 21 2.1 6 14 1
======================================================================
Total 1,950 206 $19.8 160 32 14
----------------------------------------------------------------------
As shown in the table, agencies made 206 phase I awards in the first
year of the program, with DOD accounting for about half of them.
Among the small business affiliates, universities constituted the
majority (78 percent) of the research partners. DOE was the only
agency in which the awardees formed partnerships with research and
development centers in a majority of cases.
Among other significant points about awardees during this period,
according to SBA officials, almost two-thirds of the awards went to
companies that had previously received awards from the SBIR Program.
In addition, the officials told us that the proposed allocation of
award money between the small businesses and the research
institutions showed the former receiving about 61 percent of the
funding and the latter receiving about 39 percent.
OBJECTIVES, SCOPE, AND
METHODOLOGY
---------------------------------------------------------- Chapter 1:4
In addressing the quality and commercial potential of STTR research,
we obtained and reviewed all of the agencies' technical evaluations
for 206 proposals that received phase I awards on the basis of fiscal
year 1994 solicitations. The technical evaluations were prepared by
experts in federal agencies or the private sector, if an agency
relied on outside peer review. The evaluations played the primary
role in determining which proposals were selected for funding. The
number of evaluations for each award ranged from one to as many as
six.
We reviewed the technical evaluations of project proposals because
they were the only systematic source of information on the quality
and commercial potential of the research. At this time, the actual
results of these awards cannot be assessed. We restricted our work
to the 206 awards because they were the only complete set of awards
made under the pilot program at the time that we conducted our
review. The results of the first round of the phase II award process
were not available for all agencies until late in 1995; in fact, only
a few phase II awards were made during fiscal year 1994.
Because our conclusions about the quality of research and commercial
potential were drawn from these technical evaluations of proposed
projects, we gave additional attention to the evaluation process used
by each of the agencies. In our view, greater confidence could be
placed in evaluations resulting from a thorough and well-documented
evaluation process. In particular, we noted the number of reviewers
per proposal, the number of proposals per reviewer, and the level of
analysis resulting from the reviews. Our findings were
agency-specific for DOD, NIH, NASA, NSF, and DOE. In addition to
reviewing the technical evaluations, we obtained further information
about the evaluation process from discussions with program officials
in each of the STTR agencies.
To address compliance by agencies and research and development
centers with a requirement of the act to avoid conflicts of interest,
we obtained relevant documents from and conducted interviews with
STTR Program officials. The act required each federal agency with an
STTR Program to develop, in consultation with the Office of Federal
Procurement Policy and the Office of Government Ethics, procedures to
ensure that research and development centers (1) are free from
organizational conflicts of interests relative to the STTR Program;
(2) do not use privileged information gained through work performed
for an STTR agency or private access to STTR agency personnel in the
development of an STTR proposal; and (3) use outside peer review, as
appropriate.
To address the effect, if any, of STTR on SBIR and other agency R&D,
we interviewed STTR Program officials at SBA and the five agencies
that have STTR Programs.As agreed with the Committees, we also looked
at the need for the STTR Program. The STTR Program, as mentioned
earlier, is modeled closely on the SBIR Program. The issue then
arises whether there is a need for two separate programs. In
addressing this concern, we reviewed a 1992 report by the House
Committee on Small Business. The report helped us to identify
questions that are relevant in determining the need for the STTR
Program.
We performed our audit work between June and December 1995 in accord
with generally accepted government auditing standards.
EVALUATIONS OF STTR PROPOSALS FOR
RESEARCH QUALITY AND COMMERCIAL
POTENTIAL WERE FAVORABLE, BUT THE
REVIEW PROCESS VARIED GREATLY
============================================================ Chapter 2
Agencies generally rated the quality of the proposed research and
commercial potential in STTR proposals highly. For example, DOE
rated the quality of research in all 21 of its winning proposals as
being among the top 10 percent of all research in the agency.
Evaluations of the commercial potential were also favorable but
somewhat more cautious. For example, in some cases there were
concerns about the cost of the product that might result or the
limited size of its potential market. Such reservations were
understandable in view of the newness of the program and the
innovation or risk associated with many of the proposed projects.
The evaluation process, upon which these findings depended, varied
greatly. In agencies other than DOD, the selection relied on input
and consensus among several (generally three or four) technical
reviewers. In DOD, by contrast, a single reviewer was frequently
responsible for the technical evaluation.
EVALUATION PROCESSES AND THEIR
RESULTS
---------------------------------------------------------- Chapter 2:1
Table 2.1 provides a brief overview of the evaluation processes used
by the five agencies and the agencies' assessment of the research
proposals.
Table 2.1
An Overview of the STTR Evaluation
Process, Quality of Proposed Research,
and Commercial Potential, Fiscal Year
1994
Sing
le Quality of
Total Reviews per revi proposed Commercial
Agency awards proposal ews research potential
---------------------- -------- ------------ ---- ------------ ------------
NIH 48 4 0 a a
DOD (total) 105 1 to 6 47 highly rated highly rated
Navy 22 1 to 3 18 10 of 22 10 of 22
perfect perfect
scores scores
Army 21 3 0 cutting edge highly rated
research
Air Force/Wright 18 1 to 3 13 excellent excellent
Laboratory
Air Force Office of 13 3 to 6 0 a a
Scientific Research
Ballistic Missile 15 2 to 6 0 a a
Defense Organization
Advanced Research 16 1 16 all 16 14 perfect
Projects Agency perfect scores
scores
NASA 21 3 0 above above
average/ average/
top 10 top 10
percent percent
NSF 11 3 0 excellent excellent
DOE 21 3 0 top 10 highly rated
percent
--------------------------------------------------------------------------------
\a NIH, the Air Force Office of Scientific Research (AFOSR), and the
Ballistic Missile Defense Organization (BMDO) provided only
qualitative, not quantitative, evaluations for this criterion. At
NIH, each project was given a score on an overall basis that included
research quality and commercial potential. In general, NIH's
proposals were judged as either outstanding or excellent. At AFOSR,
the research quality and commercial potential were highly rated. At
BMDO, it was difficult to draw conclusions because of the diversity
of views expressed in the evaluations.
The following sections summarize (1) the agencies' evaluation
processes and (2) the quality of research and commercial potential
identified in the technical evaluations.
NIH'S EVALUATION PROCESS AND
ITS RESULTS
-------------------------------------------------------- Chapter 2:1.1
In its first (fiscal year 1994) solicitation for STTR proposals, NIH
identified seven review criteria, including the soundness and
technical merit of the proposed approach and commercial potential,
which were used to select proposals for funding. These criteria were
not given specific weight in the evaluation process.
In the process of evaluation, NIH assigned two reviewers (who
provided written evaluations) and two additional readers to each
proposal. A peer review panel consisting of 10 or more experts was
convened. The four reviewers and readers began the discussion of
each proposal by presenting a specific, numerical score to the panel.
The two reviewers also presented their written statements. The panel
then discussed the proposal in further detail to assist in the
development of a final summary statement. Each member of the panel
also provided a score, which ranged between 100 for the best and 500
for the worst. The level of quality was defined as follows:
100 to 150: Outstanding.
150 to 200: Excellent.
200 to 250: Very good.
250 to 350: Good.
350 to 500: Acceptable.
For the 48 winning proposals, the average score was 165 (or
"excellent"). Specifically, 14 proposals were judged outstanding, 31
excellent, 2 very good, and only 1 good. There were none in the
acceptable category. This overall result compared favorably with
NIH's SBIR Program, in which the average score for all SBIR phase I
projects with awards in fiscal year 1994 was 187. About 94 percent
of STTR's awards were judged as outstanding or excellent, which
compared very favorably with SBIR's 66 percent.
In contrast to the practice at other agencies, no separate score was
assigned for the quality of research and commercial potential. The
Chief of the Technology and Applied Sciences Section said that the
importance of individual criteria may vary from one proposal to
another. In his view, an overall score allows for greater
flexibility than the sum of a series of specific scores, each of
which represents a fixed percentage of the total potential score.
The Chief added, however, that NIH has organized a committee to
evaluate its entire scoring system.
Although the panels did not provide scores for each criterion, they
provided detailed written analyses of the proposals. The summaries
for NIH's winning proposals include, among other points, a resume of
the research, a critique of strengths and shortcomings in terms of
research, and an assessment of its commercial potential.
NIH's evaluations of the quality of research were generally very
favorable. One evaluation, for example, described a proposal as
addressing a well worthwhile problem involving a new method of
enhancing the power of X-ray tubes. The evaluation stated that the
proposed approach was sound, had high technical merit, and was
supported by the theoretical calculations presented. Statements
about commercial potential were similarly favorable but sometimes
contained expressions of concern about costliness or other potential
drawbacks. In one instance, involving a new approach to X-ray
mammography, the evaluation found that the proposed approach likely
will result in an expensive technology, but prospects are good for a
result that improves on current technology and should enjoy a very
high market potential. In another case, the commercial potential for
a product that would help prevent geriatric wandering was described
as good if the cost of hardware and system operation could be kept
low. The evaluation raised concerns about the potential costliness
and sophistication of the system.
DOD'S EVALUATION PROCESS AND
ITS RESULTS
-------------------------------------------------------- Chapter 2:1.2
DOD identified four criteria in making its phase I awards, including
(1) the soundness and technical merit of the proposed approach and
(2) the potential for commercial (government or private sector)
application and the benefits expected to accrue from this
commercialization. DOD specified that each of the four criteria was
worth 25 points. Only one DOD agency, the Ballistic Missile Defense
Organization (BMDO), developed a different set of basic criteria and
provided no quantitative scores.
DOD officials pointed out that there was a great deal of variation
from one DOD agency to another in the evaluation process. In
particular, the agencies varied in the number of reviews that they
require to make an award. In the Navy, Air Force, and Advanced
Research Projects Agency (ARPA), 47 of the winning proposals received
only one technical review; in BMDO, by contrast, the number ranged
between two and six. Several DOD program managers noted that the
number and quality of the reviews were time-dependent and that time
pressures may lead to fewer or less thorough reviews. In fact, of
the 47 winning proposals with only one review, 37 were limited in the
analysis used to support the award.
Generally speaking, the DOD evaluations rated the quality of research
as favorable for most of the winning proposals; as with NIH, the
evaluations of commercial potential were frequently positive but
accompanied by caveats.
THE NAVY
------------------------------------------------------ Chapter 2:1.2.1
Our work identified a number of concerns which, in combination, may
weaken the reliability of the Navy's evaluation process. These
concerns included the limited number of reviewers per proposal, the
heavy reliance on only a few reviewers for all of the proposals, and
the brevity of their evaluations. The process itself resulted in
very favorable findings about the quality of research and the
commercial potential.
In general, the Navy's evaluation process and its conclusions
depended primarily on the views of only one person. Of the 22
awards, 18 were made on the basis of a single review. Two of the
remaining four received two evaluations and the other two received
three evaluations.
In addition to the large preponderance of single reviews, the Navy
relied heavily on relatively few individuals for its evaluations.
For example, one individual in the Office of Naval Research performed
seven of the evaluations for winning proposals. The Navy's program
manager told us that this reviewer was also responsible for
evaluating 13 unsuccessful proposals.
As a third concern, comments on most of the winning proposals were
typically very brief. Single word comments such as "superb" and
"outstanding" were used to justify scores on winning proposals that
received only one review. In the case of one proposal, for example,
"excellent," "very adequate," and "top notch" sufficed for the award
of nearly perfect scores in three of the four individual categories;
a brief technical comment (less than a sentence) addressed the
soundness and merit of the approach. No other written narrative or
corroborating review was available to support the scores.
We asked the Navy's SBIR/STTR Program manager whether the conclusions
and scores recorded on the worksheets resulted from previous
discussions with other reviewers and represented a consensus by more
than the one reviewer signing the worksheet. The program manager
told us that the reviewer was presenting only his own assessment; no
consensus was involved. However, he added that generally two
additional officials, usually he and one other person, checked the
individual evaluations. In no cases were changes made as a result of
these subsequent checks.
In our view, the absence of detail and a broader consensus may limit
the reliability of the results. Other DOD and civilian agencies used
more than one reviewer per proposal and arrived at a consensus. The
Navy program manager said that, although the number of technical
reviews may remain limited, he has already taken steps to improve the
thoroughness of the review(s) that the proposals receive. These
steps include a greater amount of analysis in the technical review(s)
and greater involvement in the subsequent review process by higher
level scientists within the Navy.\2
We found that 10 of the 22 winning proposals received perfect scores
(25 out of 25 points) for soundness and merit of approach. The
lowest score was a 20 (whose meaning was not defined). Generally
speaking, the Navy reviews rated the commercial potential of the
winning proposals favorably. Ten of the proposals also received
perfect scores regarding the potential for and benefit of
commercialization. The lowest score in this area was a 15; the
reviewer's main concern was the relatively high risk of the venture.
--------------------
\2 During our meeting with DOD officials in December 1995 to discuss
our draft report, the Navy's STTR program manager noted a key reason
for the limited number of technical reviewers used by the Navy in
evaluating STTR proposals. He said he has relied exclusively on
scientists within the Office of Naval Research for technical reviews
of STTR proposals. He has not included scientists from other DOD
laboratories because he hopes that these laboratories may eventually
be included as research partners in STTR work; their involvement in
the evaluation process would create a conflict of interest with their
potential role in STTR-related research.
THE ARMY
------------------------------------------------------ Chapter 2:1.2.2
Each of the Army's proposals received three reviews. A core group of
six scientists at the Army's Research and Technology Integration
Office, which moves Army-funded technology from universities to the
marketplace, oversaw the process. Each of the Army's proposals was
assigned to a staff scientist with relevant expertise at the Army
Research Office; this scientist was responsible for evaluating
assigned proposals. After evaluating a proposal, it was given to one
or more Army scientists. The evaluations were returned to the
original reviewer, who would identify and resolve any discrepancies
and then assign a numerical score to each of the four DOD criteria on
a scale of zero to 25.
The Army also defined scoring guidelines that assigned specific
meanings to numerical scores to keep evaluators consistent with each
other and ensure that low scores would be assigned if deserved. The
guidelines provide clear definitions to differentiate the various
levels of value in the proposals. Soundness and technical merit, for
instance, were defined as follows:
Nobel-quality research--25 points.
Cutting-edge research; good chance of major breakthrough--20
points.
Should lead to important product improvements--15 points.
Should lead to important but widely anticipated product
improvements--10 points.
Should lead to minor product improvements, at best--5 points.
Unsound and/or of negligible technical merit--0 points.
Commercial potential was defined as follows:
Potential to create a major industry--25 points.
Potential to redirect a major industry--20 points.
Potential to permit small business to grow to medium size, over
time and with expected follow-ons--15 points.
Potential to launch a useful new product or product line with a
significant market (private and/or government)--10 points.
Potential to sustain a small business for a limited time--5 points.
Little commercialization potential--0 points.
The scores for soundness and technical merit of the approach ranged
between 17 and 23 for the 21 winning proposals. Thus, according to
the Army's definitions, Army reviewers regarded all of the winning
proposals as leading at the very least to important product
improvements. Thirteen of the awards scored 20 or better, meaning
that they were regarded as cutting-edge research with a good chance
of a major breakthrough. Overall, the average score of the winning
proposals was 19.
For commercial potential, the scores ranged between 12 and 23. At
the very least, Army reviewers regarded all but two of the winning
proposals as having the potential to permit a small business to grow
to medium size. Thirteen of the awards scored 20 or better, meaning
that they had the potential to redirect a major industry. The award
that scored 23 was midway between the potential to redirect a major
industry and the potential to found a major industry. Overall, the
average score was about 19.
According to the Army's STTR Program manager, STTR proposals are
unquestionably higher in quality than SBIR because researchers from
generally outstanding institutions were willing to include their
names in proposals only if the research itself was excellent; by
requiring the involvement of research institutions, STTR helped to
improve the quality of research. In addition, the small size of the
STTR Program helped to foster quality; the proposals were more
"scrubbed" in the selection process. The Army made only 21 phase I
awards out of 350 proposals in the first year of the program.
THE AIR FORCE
------------------------------------------------------ Chapter 2:1.2.3
Two laboratories (Wright Laboratory and AFOSR) accounted for all the
technical evaluations and awards for 31 proposals in the Air Force's
STTR Program. Wright Laboratory evaluated 18 winning proposals, and
AFOSR evaluated the other 13.
The program manager at the Aeronautical Systems Center, which
includes Wright Laboratory at the Wright-Patterson Air Force Base,
said that generally only one person performed the technical
evaluation. There was only one technical review for each of 13 of
the laboratory's 18 winning proposals; two or three reviewers
evaluated the other winning proposals. The program manager added
that each proposal received subsequent review at three successively
higher levels by the branch, division, and directorate. In some
cases, the technical reviews were thorough critiques in which the
individual reviewer carefully documented the basis for the scores
assigned to the proposals. In other cases, there was very little
documentation to support the findings of the evaluation.
Reviewers rated the soundness and technical merit of all 18 winning
proposals as "excellent." In addition, all but 1 of the 18 were
excellent for commercial potential.
The Director of Plans at AFOSR--the second laboratory involved in the
STTR Program--said that the laboratory's policy is to have concurrent
review for each proposal. The technical managers were expected to
perform an evaluation and also send it to other reviewers. Then, the
managers were supposed to distill the overall findings in their
reviews and submit the results to higher officials for funding
decisions.
AFOSR officials told us that between three and six reviewers
participated in the laboratory's evaluations. For example, one of
the technical managers, who had signed 4 of the 11 single evaluation
sheets, described his approach to the evaluation process. The
manager said that he had personally reviewed 23 proposals in his
technical area (packaging for high temperature electronics),
eliminated 18, and focused on 5 with the greatest merit. Reviews
were then obtained from five other experts for each of these five
proposals. Four of the five proposals received awards. According to
the technical manager, the conclusions for each proposal represented
the consensus of six concurrent reviewers (including himself).
AFOSR's conclusions about the scientific/technical quality and the
commercial potential of its research were very favorable. For
example, the technical manager for the four awards involving
packaging for high temperature electronics told us that his projects
had made unusually rapid progress toward commercialization. The
technical manager said that a total of eight automotive companies,
aircraft companies, and geothermal well-drilling companies expressed
strong interest in the technologies being developed.
BMDO
------------------------------------------------------ Chapter 2:1.2.4
BMDO devised its own review form, which asked five questions: (1)
How good is the science and engineering? (2) How is the innovation
exciting? (3) Are the principal investigator, firm, and research
institution qualified? (4) How will BMDO benefit after phase II?
(5) How will phase I get to a phase II decision?
None of these criteria referred to the potential for
commercialization, which was one of the criteria stated in DOD's
solicitation. BMDO's STTR Program manager told us that he had
omitted a reference to commercialization because, in his view, the
reviewers possess technical expertise but are not in a position to
evaluate commercial potential in the private sector. The manager
also said that he preferred to emphasize innovation rather than
commercial potential in selecting phase I awards. He added that his
goal was to have four concurrent evaluations for each proposal. In
actual practice the number varied from two to six.
In reviewing the technical evaluations of the 15 winning proposals,
we found no scores assigned to the individual criteria nor was an
overall score provided. The program manager told us that he
preferred not to rely on numerical scores. In his view, numerical
scores assigned to widely different technologies would not be
comparable. In addition, the manager noted that relatively few
proposals had the same raters, and differences in the strictness of
the ratings might arise.
Although no scores were assigned, there was a wide range of
qualitative judgments regarding the merits of the winning proposals.
For one proposal with three reviewers, the first reviewer stated that
the approach has been known for many years and was a high school
science fair demonstration in the mid-1960s. The second reviewer
described the proposal as intriguing science and engineering. The
third considered the proposal as innovative with an excellent work
plan to prove the concept.
Another winning proposal with six reviewers showed an equally wide
range of comments on the quality of the science and engineering.
Negative comments included: (1) The need for the R&D was not
documented; at least 10 laboratories were working on it. (2) The
proposal did not provide enough information to determine the quality.
Among the six reviews, the only positive comment was that both the
science and engineering were excellent. The other three reviewers
simply summarized the approach without addressing the question asked
of them: How good is the science and engineering? The proposal
received four recommendations against funding and two in favor.
In reconciling these divergent views, the program manager told us
that he made use of a panel of experts and encouraged an open debate
on the merits of a proposal. Each proposal that received at least
one recommendation for funding by a reviewer was forwarded to a panel
that included the original reviewers and additional experts. On
average, about 6 experts were included in each panel, but the number
ranged from 3 to 15. According to the program manager, the panel
generally enabled him to make a final decision on funding or
rejecting each proposal, but in some cases he remained uncertain and
obtained one further review from a recognized expert in the field.
The absence of private-sector commercialization as a criterion, the
absence of numerical scores, and the diversity of opinions expressed
in the evaluations made it difficult to draw conclusions about the
research quality and the commercial potential of BMDO's awards.
ARPA
------------------------------------------------------ Chapter 2:1.2.5
ARPA used a 20-point scale for rating its proposals. The soundness
and technical merit were worth 8 points; the other three DOD criteria
were worth 4 points apiece. ARPA required one technical evaluation
with two further reviews of that evaluation. The program manager for
each scientific area in ARPA's solicitation was responsible for
providing a written technical evaluation. Supervisors of the program
managers reviewed these evaluations and prioritized the proposals.
Then the STTR Program manager made selections mainly on the basis of
the priority ranking and program balance.
Of the 16 winning proposals, 14 received perfect scores and 2
received scores of 19. For soundness and technical merit, all 16
proposals received the maximum score of 8. For commercial potential,
14 of the projects received perfect scores; the other 2 were rated
"very good."
The narrative portion of the review consisted of brief comments on
(1) the overall technical evaluation and justification for selection
or rejection and (2) the qualifications of the nonprofit partner and
adequacy of contribution to the research project. The comments
provided little information on the quality of the research or
commercial potential. Generally, the comments consisted of only one
or two sentences. For example, one reviewer noted that there is
considerable commercial impact for the technology under consideration
as well as a significant military payoff, but the reviewer did not
elaborate on this view.
NASA'S EVALUATION PROCESS
AND ITS RESULTS
-------------------------------------------------------- Chapter 2:1.3
NASA identified four criteria for evaluating proposals and assigned
them significantly different weights. These criteria included
scientific/technical merit (worth 20 percent of the total score), the
anticipated commercial applications of the technology (worth 40
percent), and two other criteria (with a combined value of 40
percent). The great emphasis placed upon commercial applications in
the evaluation process was unique among the agencies.
NASA developed a clear statement of its selection procedures.
According to this statement, proposals were to be evaluated by a
review team consisting of three members--one from academia, one from
the private sector, and one from government. Each reviewer was to
independently review the proposals. The NASA technical manager at
the field center was required to resolve differences and obtain a
consensus on the merit of the proposal. In a further document
providing "Guidelines for Evaluators," NASA stated that a scoring
range of 90 to 100 percent should be interpreted as equivalent in
quality to the top 10 percent of all NASA proposals for comparable
R&D. A score between 80 and 89 percent signified an above average
proposal. Of the 21 winning proposals, 11 were considered above
average, and 8 were judged as being among the top 10 percent of all
NASA proposals for comparable R&D.
NSF'S EVALUATION PROCESS AND
ITS RESULTS
-------------------------------------------------------- Chapter 2:1.4
NSF identified five criteria to which approximately equal
consideration was given. These criteria included (1) the
scientific/engineering quality and (2) the potential for commercial
applications and the success of past commercialization efforts.
After an initial screening to eliminate any proposals not responsive
to the solicitation, NSF required three concurrent reviews of the
remaining proposals. The reviewers rated each of the above criteria
on a five-point scale (from poor to excellent). A "very good" score
was 20, and "excellent" (or maximum) score was 25. The reviewers
then presented their results to a panel, which developed a summary
for each proposal.
Of the 11 winning proposals, the quality of the research and its
commercial potential were consistently rated as favorable. Among the
33 reviews of the 11 projects, all but one found the quality to range
between very good and excellent. For commercial potential, the
majority of the reviews found the quality to be excellent. Only 1 of
the 33 reviews evaluated the quality as merely "good."
DOE'S EVALUATION PROCESS AND
ITS RESULTS
-------------------------------------------------------- Chapter 2:1.5
DOE identified five criteria with approximately equal consideration
given to each of them. These criteria included (1) the
scientific/technical quality of the research and (2) the anticipated
technical and/or economic benefits of the proposed research, if
successful, with special emphasis on the likelihood that the project
will attract further funding for product or process development after
the STTR support expires.
DOE's evaluation process consisted of two steps. First, each of
three reviewers provided written comments addressing the five
criteria but did not assign a specific score to the criteria.
Second, technical managers reviewed the three evaluations and
quantified the results.
Of the 21 winning proposals, 16 received perfect scores. The other
five received perfect scores on four of the five criteria. The
managers expressed unanimous agreement about the quality of the
research in particular. According to DOE's definition of quality as
used in evaluating STTR proposals, a rating of "outstanding" for
scientific/technical merit indicated that the proposal was comparable
to the top 10 percent of projects in DOE. All 21 of the winners were
rated as outstanding.
While DOE did not specifically evaluate proposals for commercial
potential, it did evaluate anticipated technical and/or economic
benefits. Twenty of the 21 proposals received outstanding ratings;
only 1 was rated in the next lower category as "significant." Many of
the evaluations on which the managers based their conclusions
contained favorable, but qualified, statements about commercial
potential. For example, one of the reviewers wrote that, if the
research is successful, the technology will be rapidly
commercialized.
One reason for the excellent results, as noted by DOE's STTR Program
manager, was the unusually large number of proposals in relation to
the number of awards available. DOE received 487 proposals, meaning
that less than 5 percent of the applicants were successful. In the
manager's view, the program was excessively competitive. As a
result, he took two steps to reduce the number of applicants. First,
a notice was included in the second solicitation to alert applicants
to the situation. The notice pointed out the ratio (1 award to 23
proposals) in the first year and concluded that only those
applications with the highest scientific/technical quality would be
competitive. Second, because broad topics tend to attract more
proposals, the topics were narrowed in the new solicitation to reduce
the number of proposals and improve the award/proposal ratio from 1
in 23 to about 1 in 10.
CONCLUSIONS
---------------------------------------------------------- Chapter 2:2
In general, technical evaluations of STTR proposals showed favorable
views of the quality of proposed research and commercial potential.
For research quality, the evaluations (1) awarded perfect scores to
many proposals, (2) rated proposals among the top 10 percent of
research in certain agencies, (3) described some proposals as
"cutting edge," and (4) generally found the quality to be excellent.
For commercial potential, the evaluations arrived at similarly
favorable conclusions, although in some cases they were somewhat more
cautious because of the newness of the program or the risk associated
with the proposals.
The evaluation processes varied greatly, ranging from several
technical reviewers to only one per proposal and from detailed
critiques to evaluation sheets that provided no analysis in support
of the ratings. Almost half (47) of DOD's winning proposals received
only one review; in 37 cases, the reviews were too brief to support
the findings. Because of their limited analysis and lack of a
broader consensus, these instances tended to reduce our confidence in
the reliability of the results.
AGENCIES HAVE TAKEN STEPS TO AVOID
CONFLICTS OF INTEREST
============================================================ Chapter 3
The five federal agencies with STTR Programs have taken steps to
avoid potential problems relating to conflict of interest with
federally funded research and development centers. In addition, DOD
and DOE, which accounted for 29 of the 32 awards involving centers
during the first year of the program, have taken steps to prevent
centers from using privileged information in preparing STTR
proposals.
AGENCIES HAVE TAKEN STEPS TO
AVOID POTENTIAL PROBLEMS
RESULTING FROM R&D CENTER
INVOLVEMENT
---------------------------------------------------------- Chapter 3:1
The legislation establishing the STTR Program required agencies to
develop procedures to ensure that R&D centers are free from
organizational conflicts of interest. Such conflicts might arise,
for example, if a center formed a partnership with a company
submitting an STTR proposal and helped a federal agency judge the
merits of its own and other proposals. A second requirement directs
each agency to develop procedures ensuring that the centers use
outside peer review, as appropriate. Under the STTR Program,
however, the agencies, not the centers, are responsible for decisions
regarding peer review; accordingly, we have focused on what the
agencies have done. In addition, the legislation required agencies
to ensure that the centers do not use privileged information gained
through work performed for an STTR agency or private access to STTR
agency personnel in the development of an STTR proposal.
AGENCIES HAVE TAKEN STEPS TO
AVOID CONFLICTS OF INTEREST
-------------------------------------------------------- Chapter 3:1.1
In general, the five agencies with STTR Programs have taken steps to
prevent conflicts of interest from occurring. DOD, DOE, and NIH have
specific policies intended to prevent such conflicts while NASA and
NSF have more general procedures to prevent such conflicts of
interest from arising.
DOD's Director of Defense Research and Engineering issued a
memorandum in mid-1994 providing policy guidance on research and
development centers' participation with industry in STTR and similar
programs. The memorandum concludes that, if a center requests
authorization to participate in such programs, the mission of the
particular center and the potential for conflict of interest must be
primary considerations in the decision process.
As a result of DOD's policy, only two R&D centers are currently
approved research partners for its STTR awardees. The Air Force had
to rescind some awards because the proposed research partners
(certain DOD laboratories) were ineligible to participate. According
to DOD's STTR Program director, future proposals will be evaluated on
a case-by-case basis to ensure that conflicts of interest do not
occur.
DOE has a policy addressing conflict of interest for the STTR
Program. According to the policy, DOE staff members should neither
request nor receive assistance from personnel in research
institutions (that are eligible to participate in the STTR Program)
in the preparation of technical topics for the STTR solicitation. In
addition, no person affiliated with a research institution may serve
as a reviewer of a grant application that names that research
institution as a participant. Furthermore, no one affiliated with a
research institution may assist technical managers with the DOE
review, evaluation, and selection process for phase I grant
applications in a particular scientific area if that research
institution is a participant on any grant application submitted to
that scientific area.
NIH has adopted a certification procedure to avoid conflicts of
interest. NIH's solicitation for proposals requires the applicant to
certify that it "(1) is free from organizational conflicts of
interests relative to the STTR Program, (2) did not use privileged
information gained through work performed for an STTR agency...and
(3) used outside peer review, as appropriate, to evaluate the
proposed project and its performance therein."
NASA relied on a peer review process for its proposals to avoid
conflicts of interest. As stated earlier, NASA's review teams
consisted of three members--one from academia, one from the private
sector, and one from the government. According to NASA's STTR
Program manager, none of the reviewers were connected with a
federally funded R&D center. The manager also mentioned that NASA's
reviewers certified that they have no conflict of interest in their
evaluations. The manager said that the only federally funded R&D
center directly associated with NASA on a regular basis is the Jet
Propulsion Laboratory. The manager said that, although the
laboratory was included as a research partner in several proposals by
small businesses, it was not involved in reviewing those proposals or
administering the program; in addition, no special information was
provided to this center.
NSF also relied on peer review procedures to avoid conflicts of
interest. Its "Proposal and Award Manual" provides guidance on the
use of peer review and states that each NSF program has one primary
method for peer review which represents the minimum evaluation
received by proposals in that program. Each of its STTR proposals
received three reviews. NSF's STTR Program manager told us that, in
fiscal year 1994, none of NSF's 11 awards involved a small business
in a partnership with a research and development center.
In general, four of the five agencies with STTR programs used peer
review in evaluating STTR proposals. NIH, NASA, NSF, and DOE relied
on specified numbers of outside reviewers. NIH assigned four
technical reviewers to each proposal and provided additional input
through its peer review panels. NSF, NASA, and DOE used three. DOD
was the only agency that relied mainly on technical expertise within
the Department rather than on outside reviewers. This approach
followed DOD's usual policy in evaluating proposals.
DOD AND DOE HAVE TAKEN STEPS
TO PREVENT RESEARCH AND
DEVELOPMENT CENTERS FROM
USING PRIVILEGED INFORMATION
-------------------------------------------------------- Chapter 3:1.2
DOD and DOE, which accounted for 29 of the 32 awards involving R&D
centers as research partners, have policies to prevent centers from
using privileged information. DOD's policy of carefully restricting
participation by its own laboratories helped in preventing the
centers from using inside knowledge in preparing proposals. As
mentioned earlier, only two of DOD's centers were eligible to
participate in the program. The exclusion of the other centers,
which would be the main sources of privileged information regarding
DOD's research needs, avoided the potential problem raised by this
issue.
DOE's policy prohibits agency staff members from requesting or
receiving assistance from personnel in research institutions (that
are eligible to participate in the STTR Program) in the preparation
of technical topics for the STTR solicitation. This policy is
intended to prevent research institutions from using their expertise
to influence DOE's choice of STTR solicitation topics. Otherwise,
research institutions could acquire a significant advantage by
designing topics to match their expertise and then preparing a
proposal in the same area.
DIFFERING VIEWS ON THE EFFECT OF
AND NEED FOR THE STTR PROGRAM
============================================================ Chapter 4
Agency officials expressed differing views regarding the effect of
STTR on SBIR and other agency R&D. Furthermore, none of the
officials indicated any specifically negative effects such as a
competition between the two programs for quality proposals. However,
conclusive information concerning the effect, if any, of STTR on SBIR
and other agency R&D was not available because of the program's
newness and smallness. The similarities between the STTR and SBIR
Programs raise a broader issue about the need for the STTR Program.
The rationale for the program, which points to certain weaknesses in
SBIR and potential strengths in STTR, suggests some additional
questions that point more directly toward evaluating the need for
STTR.
AGENCY OFFICIALS NOTED NO
NEGATIVE EFFECTS OF STTR ON
SBIR OR OTHER FEDERAL R&D
---------------------------------------------------------- Chapter 4:1
The legislation establishing the STTR Program required us to assess
the effects of STTR on the SBIR Program and other agency R&D. Our
discussions with agency officials provided no evidence to suggest
that STTR was competing for quality proposals with SBIR or reducing
the quality of agency R&D in general. Instead, agency officials
expressed differing views regarding the effect of STTR on SBIR and
other agency R&D. A few officials noted some potentially beneficial
effects. Others said that sufficient data were not yet available to
determine the effect, if any, of STTR on SBIR or other agency R&D.
SBA officials contended that STTR was too small and too new a program
to have any real effect on SBIR or on the broader range of agency
research at the present time. The officials pointed out that the
program represented only 0.05 percent of each agency's external R&D
budget during its first year and that it was only 1 year old.
NIH's STTR Program manager told us that one of the main potential
effects of the program is that universities can have a greater role
than under SBIR. However, the manager noted that, in an SBIR survey
undertaken by NIH several years ago, collaboration between small
businesses and universities was already evident in well over half of
the SBIR projects.\3
In contrast to the view that STTR's effect was very limited, the
Army's STTR Program manager said that STTR was influencing SBIR in a
beneficial way. In his opinion, STTR is becoming known through
national conferences and other channels. As a result, small
businesses are realizing that they have more credibility and chance
of winning an award by collaborating with a university or other
research institution. The manager believes that the STTR Program has
already led to more collaboration in SBIR. In general, according to
the program manager, STTR is a promising program that may be as
successful as the SBIR Program. The manager also said that STTR will
influence agencies whose research has traditionally involved the
university community to a lesser extent. In DOD, for example, the
program manager believes that STTR's impact will be greater than in
certain other agencies (such as NIH) where the research program has
been closely tied to the universities.
DOE's STTR Program manager said that he has tried to make the topics
in the STTR solicitation somewhat different from SBIR. Regarding the
effect of STTR on DOE's research program, he said that STTR projects,
if successful, will help to meet the agency's R&D needs by
contributing to areas of particular interest to the agency. However,
the manager added that no conclusions can be drawn in the first year
of the program because the data are too limited.
--------------------
\3 During our meeting with DOD officials to discuss our draft report,
DOD's STTR program director noted that the STTR Program allows a
research institution to play a greater role in collaborating with a
small business than under the SBIR Program.
THREE QUESTIONS ARE RELEVANT IN
DETERMINING THE NEED FOR THE
STTR PROGRAM
---------------------------------------------------------- Chapter 4:2
The stated rationale for the STTR Program and the mandatory
collaboration suggest three additional questions that are relevant in
determining the need for the STTR Program: (1) Is the technology
originating primarily in the research institution as envisioned in
the rationale for the program or is it originating in the small
business? (2) Is the mandatory collaboration between the small
business and the research institution effective in transferring the
technology to the market place? (3) Can the SBIR Program accomplish
the same objective without the collaboration required by the STTR
Program?
IS THE TECHNOLOGY
ORIGINATING PRIMARILY IN THE
RESEARCH INSTITUTION?
-------------------------------------------------------- Chapter 4:2.1
The technology may originate with the research institution, the small
business, or a combination of the two. In the STTR Program, the
assumption is that the research institution will be the primary
originator of the new concept. However, data to determine the extent
to which research institutions are providing the core technologies
are not currently available. Neither SBA nor the agencies have
collected this information. DOE's SBIR/STTR Program manager said
that he would like to know whether the companies or the research
institutions were drafting the proposals, but such information is not
available.
The relative roles of the research institution and the small business
as the source of the technology bear directly on the need for the
STTR Program. If a high percentage of the ideas are originating with
small businesses rather than with research institutions, this finding
would raise questions about the need for the program. On the other
hand, if a high percentage of ideas are originating with research
institutions, this finding would suggest that the program was
achieving the first step in moving ideas from research institutions
to small businesses.
IS THE COLLABORATION
EFFECTIVE IN TRANSFERRING
IDEAS FROM THE RESEARCH
INSTITUTION TO THE MARKET
PLACE?
-------------------------------------------------------- Chapter 4:2.2
If the program is effective in moving ideas from research
institutions to small businesses, then the next logical question
concerns whether their collaboration is effective in moving them to
the market place. This question can be approached from two
directions: (1) Short-term views of how well the collaboration is
working in general and (2) long-term data on actual
commercialization.
Information on how well the collaboration is working can be obtained
in the near future. In particular, it would be useful to know how
the small businesses rated the contribution made by their research
partners to the research effort. Since most of the companies had not
completed even the first phase of their STTR award at the time of
this report, such information was not available, but it will be
obtainable in the next year or two.
Information on actual commercial outcomes will require a greater
amount of time before it can be obtained. Generally, 5 to 9 years
are needed to turn an initial concept into a marketable product.
Thus, it may be several years before the commercial effectiveness of
the program can be evaluated.
CAN THE SBIR PROGRAM
ACCOMPLISH THE SAME
OBJECTIVE WITHOUT STTR'S
MANDATORY COLLABORATION?
-------------------------------------------------------- Chapter 4:2.3
Because one important difference between the two programs is that
STTR makes a small business/research institution collaboration
mandatory, the question arises whether the SBIR Program could
accomplish the objective of transferring technology from research
institutions to the private sector without the mandatory
collaboration. The rationale for the STTR Program tends to assume
that such collaborations were relatively rare in the SBIR Program.
However, as noted above, NIH's Program manager told us that, in an
SBIR survey undertaken by NIH several years ago, collaboration
between small businesses and universities was already evident in well
over half of NIH's SBIR projects. By contrast, the Army's program
manager believed that STTR's impact will be greater in the Army than
in agencies such as NIH because the Army SBIR Program has had a
lesser degree of involvement with universities and other research
institutions in the past. Given the apparent variation from one
agency to another and the lack of current data, no definite
conclusion can be drawn at present concerning the need for STTR in
forging new collaborations.
Certain proposals may suggest a need for the STTR program. For
example, one NIH proposal involved a relationship between the
developer of a new type of microscope and a company with the
experience and capability to commercialize the instrument. The
principal investigator with the company told us that STTR led to a
partnership between his company and the research institution that
would not have existed otherwise. According to the investigator,
without STTR, his company would have been reluctant to devote a lot
of time to a technology that it did not have rights to. In such a
situation, according to the investigator, the company would have
provided specific components of the microscope to the research
institution, let the institution develop the product, and then
pursued a license to market it. Instead, under STTR, the company
developed an agreement with the research institution which led to the
current partnership.
However, other cases may suggest the opposite. For example, the
president of a company that has participated in both the SBIR and
STTR Programs said that, in his experience, the SBIR Program gave
companies a greater advantage in dealing with research institutions
as potential partners. Under STTR, according to this official, the
research institution has "veto power," but under SBIR, a company is
in a better negotiating position if it wants research to be done. In
addition, according to the official, the STTR Program will not be
able to alter the research-oriented outlook of the universities in a
more commercial direction. In general, the official said that almost
all STTR research could be accomplished through SBIR.
MAJOR CONTRIBUTORS TO THIS REPORT
=========================================================== Appendix I
RESOURCES, COMMUNITY, AND ECONOMIC
DEVELOPMENT DIVISION, WASHINGTON,
D.C.
Bernice Steinhardt, Associate Director
Robin Nazzaro, Assistant Director
Dennis Carroll, Evaluator
*** End of document. ***