Federal Research: Evaluation of Small Business Innovation Research Can Be
Strengthened (Testimony, 06/17/1999, GAO/T-RCED-99-198).

The United States depends heavily on innovation through research and
development. The Small Business Innovation Development Act of 1982,
which authorized the Small Business Innovation Research (SBIR) Program,
emphasizes the benefits of technological innovation and the ability of
small businesses to transform the results of research into new products.
As the program has matured in the 1990s, congressional concern has
focused on the companies' ability to commercialize the results of their
research and on the concentration of awards in certain states and
companies--commonly known as "frequent winners." This report discusses
the (1) distribution of awards by company and geographic area, (2)
extent to which federal agencies are considering commercial potential
and the program's other goals in making their awards, and (3) previous
evaluations of the SBIR program to identify opportunities to improve
measurements of the program's outcomes. This testimony summarizes GAO's
June 1999 report, GAO/RCED-99-114.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  T-RCED-99-198
     TITLE:  Federal Research: Evaluation of Small Business Innovation
	     Research Can Be Strengthened
      DATE:  06/17/1999
   SUBJECT:  Small business assistance
	     Research program management
	     Performance measures
	     Research and development contracts
	     Technology transfer
	     Authorization
	     Evaluation methods
	     Evaluation criteria
IDENTIFIER:  NSF Experimental Program to Stimulate Competitive Research
	     Small Business Innovation Research Program
	     SBA Tech-Net

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  This text was extracted from a PDF file.        **
** Delineations within the text indicating chapter titles,      **
** headings, and bullets have not been preserved, and in some   **
** cases heading text has been incorrectly merged into          **
** body text in the adjacent column.  Graphic images have       **
** not been reproduced, but figure captions are included.       **
** Tables are included, but column deliniations have not been   **
** preserved.                                                   **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                                            **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************

    United States General Accounting Office GAO
    Testimony Before the Subcommittee on Technology, Committee on
    Science House of Representatives For Release on Delivery
    FEDERAL RESEARCH Expected at 1 p.m. EDT Thursday June 17, 1999
    Evaluation of Small Business Innovation Research Can Be
    Strengthened Statement of Susan D. Kladiva, Associate Director,
    Energy, Resources, and Science Issues, Resources, Community, and
    Economic Development Division GAO/T-RCED-99-198 Madam Chair and
    Members of the Subcommittee: We are here today to present the
    results of our review1 of the Small Business Innovation Research
    (SBIR) program, which we performed for your subcommittee. As a
    nation competing in a global economy, the United States depends
    heavily on innovation through research and development. The Small
    Business Innovation Development Act of 1982, which authorized the
    SBIR program, directs agencies with sizeable external research and
    development budgets to set aside a portion of these budgets for
    projects by eligible small businesses. It emphasizes the benefits
    of technological innovation and the ability of small businesses to
    transform the results of research into new products. In
    reauthorizing the program in 1992, the Congress stated its
    intention to expand and improve the program, emphasize the
    program's goal of increasing the private sector's
    commercialization of technology developed through federal research
    and development, and improve the federal government's
    dissemination of information on the program. In its 16 years, the
    program has provided over 45,000 awards worth $8.4 billion in 1998
    dollars to thousands of small high-technology companies. In the
    1990s, congressional concern has focused on the companies' ability
    to commercialize the results of their research and on the
    concentration of awards in certain states and companies-commonly
    known as "frequent winners." Concern about frequent winners has
    arisen, in part, because studies conducted by us and the
    Department of Defense indicate that frequent winners generally
    achieve lower levels of commercialization than companies winning
    fewer awards. The report we are releasing today discusses * the
    distribution of awards by company and geographic area, with
    special emphasis on the share of awards received by the 25 most
    frequent winners; * the extent to which federal agencies are
    considering commercial potential and the program's other goals in
    making their awards; and * previous evaluations of the SBIR
    program to identify an opportunity to improve measurements of the
    program's outcomes. Our statement today highlights the message of
    our review. In summary, Madam Chair: 1Federal Research: Evaluation
    of Small Business Innovation Research Can Be Strengthened,
    (GAO/RCED-99-114, June 4, 1999). Page 1
    GAO/T-RCED-99-198 The 25 most frequent winners, which represent
    fewer than 1 percent of the companies in the program, received
    about 11 percent of the program's awards from fiscal year 1983
    through fiscal year 1997. These companies accumulated over $900
    million in total awards; the leading frequent winner received over
    $108 million. However, one-third of the companies receiving awards
    from fiscal year 1993 through fiscal year 1997 were first-time
    winners, indicating that the program is attracting an average of
    750 new companies annually. In our view, this level of
    participation by first-time winners is indicative of a substantial
    capacity to attract new participants each year. In response to the
    1992 reauthorization, agencies are considering commercial
    potential as a criterion when evaluating proposals and collecting
    data on commercialization by frequent winners. However, the
    reauthorization does not clarify how much weight should be given
    to the commercialization record as part of the goal of
    commercialization and how much weight should be given to the
    program's other goals, such as technological innovation or
    importance to an agency's mission. This lack of clarity has led to
    differences in agencies' evaluation approaches. For example, using
    an approach shared by none of the other agencies, the Department
    of Defense planned to give significantly lower scores to companies
    perceived as poor commercializers, but we found that this approach
    would penalize companies with relatively few awards and no sales
    but would not penalize frequent winners with limited sales.
    Defense has revised its approach to avoid these unintended
    consequences. Our report raises as a matter for congressional
    consideration how the commercialization record as part of the goal
    of commercialization should be balanced against the program's
    other goals in evaluations of proposals. Federal agencies and
    others have relied on various methods to evaluate the program's
    commercial outcomes. These methods have used "snapshots" of sales,
    data on additional developmental funding for projects, "success
    stories," and other indicators of success. However, they become
    quickly outdated and do not provide an ongoing, consistent, and
    programwide record. The use of a single method with uniform
    criteria for success focusing on commercial and other outcomes
    would help to satisfy the requirements of the Government
    Performance and Results Act.2 The Small Business Administration
    (SBA) is currently developing a new database called Tech-Net,
    which is scheduled for implementation in 1999. Tech-Net affords an
    opportunity to maintain current, consistent 2In December 1997, the
    Congress specified that information on the SBIR program must be
    included by each federal agency in the updates or revisions to its
    strategic plan required by the Results Act. 15 U.S.C. 638(t). Page
    2
    GAO/T-RCED-99-198 information on commercial outcomes and other
    indicators of success in response to the Results Act. Our report
    recommends that the SBA Administrator develop standard criteria
    for measuring the commercial and other outcomes of the SBIR
    program and incorporate these criteria into the new Tech-Net
    database. Let me be more specific about each of these issues.
    Frequent and             To analyze the distribution of SBIR
    awards by company, we divided the 3 Infrequent Winners
    winners of phase II awards  into three groups: the 25 companies
    with the most awards, the companies with between 1 and 4 awards,
    and a middle Are Major SBIR           group of companies with
    between 5 and 27 awards. Figure 1 presents the Players, but
    Frequent    distribution of phase II awards to these three groups
    from fiscal year 1984, when the first phase II awards were made,
    through fiscal year 1997, the Winners and Certain      latest year
    for which complete data are available. States Have Won a Large
    Share of the Program's Resources 3Phase II awards follow Phase I
    awards and are designed to further develop the scientific and
    technical merit and the feasibility of research ideas. Page 3
    GAO/T-RCED-99-198 Figure 1: Percentage of Phase II Awards Won by
    Various Groups of Participants, Fiscal Years 1984-97 7 Source:
    GAO's analysis of data from SBA's SBIR database. This figure bears
    out concerns about the concentration of awards, showing that
    nearly half of the program's awards have gone to just over a tenth
    of the participants represented by the top and middle groups.
    (Additional information about the 25 most frequent winners is
    provided in appendix I.) However, the figure also indicates wide
    participation, showing that thousands of other companies have
    received only one or just a few awards. Data for fiscal years 1993
    through 1997 further indicate that the program is attracting an
    average of over 750 new companies each year. Concern about the
    concentration of awards has also focused on their geographic
    distribution. SBA has found that the distribution of SBIR awards
    generally resembles the distribution of non-SBIR expenditures for
    research and development, venture capital investments, and
    academic research funds. Companies in a small number of states,
    especially California and Massachusetts, have won the majority of
    awards, largely because they have submitted the most proposals.
    Data for fiscal year 1998 show that proposals from companies in
    states with historically lesser amounts of Page 4
    GAO/T-RCED-99-198 federal research funding won awards at almost
    the same rate as proposals from companies in other states. To
    encourage greater participation by companies in states with fewer
    awards, the National Science Foundation has used a program it
    established about 20 years ago to support research in states with
    historically lesser amounts of federal research funding. The
    Foundation's Experimental Program to Stimulate Competitive
    Research (EPSCoR) began in 1981 and was funded at about $49
    million in fiscal year 1999. The Foundation has used this program
    to increase the number of SBIR awards to small businesses in
    states that have received lesser amounts of federal R&D funding.
    Eighteen states and the Commonwealth of Puerto Rico participate in
    the program.4 Since 1994, EPSCoR has awarded 82 phase I SBIR
    grants valued at over $7 million. Other agencies also have such
    programs but have not used them to assist their SBIR participants.
    Several agencies are considering such an initiative to increase
    their outreach efforts in the SBIR program. Agencies Are
    In reauthorizing the program in 1992, the Congress emphasized
    Considering                     commercialization, requiring
    agencies to consider commercial potential in making awards and to
    collect data on commercialization by companies Commercial
    Potential            that have received 15 or more phase II awards
    during the preceding 5 in Making Awards, but years. We found that,
    in response, agencies are weighing the commercial potential of all
    proposals and collecting data on commercialization by the Emphasis
    on                 companies, including frequent winners. However,
    neither the 1992 Commercialization               reauthorization
    nor a 1993 SBA policy directive on implementing the Raises
    Questions                legislation clarifies for the agencies
    what weight they are to assign to commercial potential relative to
    the program's other goals in evaluating About the Relative
    proposals or how they are to use the data they collect on
    companies' Importance of the               commercialization
    results. Without clarification, the agencies are developing
    different evaluation approaches, at least one of which would
    Program's Other                 have had unintended consequences.
    Goals 4The states are Alabama, Arkansas, Idaho, Kansas, Kentucky,
    Louisiana, Maine, Mississippi, Montana, Nebraska, Nevada, North
    Dakota, Oklahoma, South Carolina, South Dakota, Vermont, West
    Virginia, and Wyoming. Page 5
    GAO/T-RCED-99-198 Agencies Are Considering     As described in the
    1992 act, commercial potential has four indicators, one Commercial
    Potential to      of which is a company's commercialization
    record.5 Our review showed Varying Degrees in Making    that
    agencies are considering commercial potential, including a
    company's Awards                       commercialization record
    and the other indicators. Together, these four indicators can
    account for as much as one-third of a proposal's total score.
    However, the commercialization record alone has played a limited
    role because it is only one indicator of commercial potential
    that, in turn, is one among several criteria such as technical
    merit considered when evaluating proposals. At the Department of
    Defense, for example, the commercialization record currently
    accounts for about one-fourth of the commercial potential score
    and about one-twelfth of the total score for a proposal; at the
    Department of Energy, it accounts for about one-eighteenth of the
    total score. Thus, even if a company has a poor commercialization
    record, this factor has exercised only a limited influence on
    agencies' evaluations of proposals to date. According to SBA's
    Assistant Administrator for Technology, the 1992 reauthorization
    directs participating agencies to collect information on
    commercialization by companies with 15 or more phase II awards
    during the previous 5 years but does not clarify how they are
    supposed to use it. In response, the agencies have collected data
    on commercialization by companies, including frequent winners.
    Because the 1992 act and the SBA policy directive do not address
    how to use the information on commercialization records,
    differences among the agencies have emerged. The Department of
    Defense, for example, recently planned to implement a new
    evaluation approach designed to give greater prominence to the
    commercialization records of companies with 5 or more phase II
    awards. This approach would have led to significantly lower scores
    for companies with relatively few awards and no sales but would
    not have penalized frequent winners with only modest sales. The
    Department has since revised its approach to avoid these
    unintended consequences by taking into account the concept of
    statistical significance as it relates to companies with widely
    varying numbers of awards. 5Under the 1992 reauthorization,
    commercial potential was evidenced by "(i) the small business
    concern's record of successfully commercializing SBIR or other
    research; (ii) the existence of second phase funding commitments
    from private sector or non-SBIR funding sources; (iii) the
    existence of third phase, follow-on commitments for the subject of
    the research; and (iv) the presence of other indicators of the
    commercial potential of the idea." 15 U.S.C. 638(e)(4)(B). Page 6
    GAO/T-RCED-99-198 The Emphasis on             Despite the greater
    emphasis on commercialization since 1992, the Commercialization
    Raises    program's other goals, such as innovation and
    responsiveness to an Questions About the Role    agency's needs,
    remain important to the agencies when evaluating a of Other Goals
    in           company's accomplishments and subsequent proposals.
    According to Evaluating Companies'       some of the program
    managers, a relatively low level of commercialization may not
    signal failure because a company may have achieved other goals.
    Performance                 The difficulty for agencies of using
    any particular goal as a key criterion for selecting future
    proposals for funding stems from their not having (1) a clear
    definition of the program's goals, (2) information on the relative
    weight that should be given to these potential goals, and (3)
    criteria for judging whether the goals have been achieved. Finding
    practical ways to define and measure the SBIR program's goals in
    order to evaluate proposals has been difficult. For example,
    efforts to define and measure technological innovation have posed
    a challenge. Because technological innovation occurs in many
    different ways, no one indicator is an accurate measure of it. In
    addition, according to SBA's Assistant Administrator for
    Technology, the 1992 reauthorization lacks a clear definition of
    "commercialization," and he has sometimes differed with agencies
    on its meaning. This absence of a definition makes it more
    difficult, in his view, to determine when a frequent winner is
    "failing" to achieve a sufficient level of commercialization. The
    relative weight that should be given to the goals when evaluating
    proposals remains unclear. Innovation and responsiveness to an
    agency's needs, for example, may compete with commercialization.
    In the view of many program managers, innovation involves a
    willingness to undertake R&D with a higher element of risk and a
    greater chance that it may not lead to a commercial product;
    responsiveness to an agency's needs involves R&D that may be aimed
    at special niches with likewise limited commercial potential.
    According to the program managers, the challenge lies in striking
    the right balance between encouraging new, unproven technologies
    and achieving commercial sales. Agencies have also not agreed on
    criteria for "success" in meeting the program's goals. In general,
    it is difficult to determine the appropriate mix of higher-risk
    projects that lead less frequently to commercial outcomes and of
    lower-risk projects that lead more frequently to successful
    products. The difficulty caused by not having criteria is
    compounded by the fact that only a handful of projects in the
    program achieve substantial commercial success. Only about 1.5
    percent of the projects account for about half of the sales, and 4
    percent of the projects account for about Page 7
    GAO/T-RCED-99-198 75 percent of the sales. When the great majority
    of projects achieve no sales or only very limited sales,
    evaluating subsequent proposals from individual companies becomes
    more difficult if commercialization is considered the primary
    goal. SBA Has an                    Commercialization is only one
    of the program's objectives but has become Opportunity to
    the main outcome for measuring its effectiveness. Studies of
    commercialization have proliferated as agencies have tried to
    obtain data Standardize                   on commercial activity.
    Although these studies rely on different Evaluations of the
    approaches, they contain some common criteria for success, such as
    levels of sales and developmental funding. Expanding the SBIR
    database at SBA to Program's Outcomes            include
    information on commercial outcomes and other indicators of
    success, such as savings to agencies resulting from SBIR projects,
    would help to standardize evaluations of the program. As SBA
    develops a new database, called Tech-Net, which is scheduled for
    full implementation in 1999, it has an opportunity to include
    outcome-related measures that can be used to track
    commercialization and other indicators of success. Various Methods
    With          Our 1992 report on the SBIR program6 responded to a
    congressional Similar Criteria Have Been    mandate that we assess
    the commercial outcomes of the program. We Used in Attempting to
    surveyed companies that had won phase II awards from 1984 through
    Measure Outcomes              1987. Our survey asked, for example,
    "Has the technology associated with this project led to additional
    developmental funding and/or sales, and is further work on this
    technology under way?" This approach was also used in later
    surveys of the program, including those conducted by a support
    contractor for DOD in 1996 and for SBA in 1998. Although the
    Department of Energy used a different evaluation approach, it also
    focused on outcomes, seeking information on the results of SBIR
    technology rather than on individual awards. Another common
    evaluation approach, used by the National Science Foundation, DOD,
    the National Aeronautics and Space Administration, and other
    agencies, relies on success stories stemming from the agencies'
    awards. Although the agencies have used these stories to document
    the most significant results of their SBIR awards and to help
    companies market their technologies, they do not follow a
    consistent method and they ignore less successful projects,
    biasing the results of this approach. In our view, the approach is
    "open-ended," meaning that it can be used to develop a 6Federal
    Research: Small Business Innovation Research Shows Success but Can
    Be Strengthened (GAO/RCED-92-37, Mar. 30, 1992). Page 8
    GAO/T-RCED-99-198 detailed story for an individual company but
    does not lend itself to systematization. A Standard Approach
    The evaluation methods we identified do not provide consistent
    Involves the Use of         information across agencies on the
    program's results. The use of a single Uniform Criteria for
    method with uniform criteria focusing on outcomes would produce
    such Success and Improvements    information, enabling SBA and the
    agencies to satisfy the requirements of in SBA's New Database
    the Results Act. As the central administrative agency for the
    program, SBA has maintained a governmentwide database that brings
    together the data submitted by the individual agencies
    participating in the program. This database has had two major
    shortcomings. First, because it was developed long before the
    Results Act emphasized the measurement of outcomes, the database
    reflects the earlier attention given to inputs, such as the name
    of each company and the amount of funding it has received. In
    general, companies have not provided information on the actual
    results of their research. Second, the database has contained
    unreliable information because it has lacked a unique identifying
    code for each company. Slight variations in the spelling of a
    company's name have created difficulty because the database has
    counted each separate spelling as a separate company. In June
    1998, SBA announced the introduction of a new database called
    Tech-Net. SBA database managers and SBIR program managers are
    optimistic about their ability to expand Tech-Net to capture
    outcome-related data. Tech-Net will enable agencies to update
    their information on SBIR awards and companies to update key
    information about their activities. As a result, companies will be
    able to provide more information on the actual results of their
    SBIR awards. In addition, SBA plans to assign each company a
    unique identification number to eliminate the confusion about the
    identity of participants. Madam Chair, this concludes my
    statement. I would be happy to answer any questions you or the
    other Members of the Subcommittee may have. Page 9
    GAO/T-RCED-99-198 Appendix I An Overview of the Top 25 Frequent
    Winners, Fiscal Years 1983-97 Dollars in millions Percentage of
    annual revenue Phase II     Total    Dollar        from SBIR
    Company                awards      awards    value
    (1998) Foster Miller              147        573    108.2
    20 Physical Optics             96        377     71.2
    68 Creare                      87        281     61.4
    64 Physical Sciences           76        290     57.2
    42 Spire                       75        351     59.4
    26 Radiation Monitoring Devices                     59        187
    43.3                 38 Bend Research               58        166
    34.3                 23 EIC Laboratories            53        188
    38.1                 33 Mission Research            50        196
    39.8                  8 Science Research Laboratory
    49        147     33.4                 76 Advanced Technology
    Materials                   48        208     38.4
    10 Advanced Fuel Research                    42        154
    27.8                 52 Ultramet                    38        140
    28.4                 37 Aerodyne Research                    35
    134     27.5                 36 CFD Research                35
    107     24.7                 52 Sparta                      35
    162     28.3                   a TDA Research                35
    127     19.5                 70 Thermacore                  35
    102     25.8                   a American Research Corp. of
    Virginia                 34        102     19.3                 80
    Waterjet Technology                  34        102     21.5
    a Scientific Research Associates                  33        113
    24.0                   a Giner                       30        110
    22.1                 70 Schwartz Electro-optics              30
    104     20.1                  6 (continued) Page 10
    GAO/T-RCED-99-198 Appendix I An Overview of the Top 25 Frequent
    Winners, Fiscal Years 1983-97 Dollars in millions Percentage of
    annual revenue Phase II            Total    Dollar        from
    SBIR Company                             awards      awards
    value             (1998) Bio-Metric Systems
    29            89     18.5                  a Satcon Technology
    28           119     22.2                44 a Information was not
    available. Source: GAO's analysis of data from SBA's SBIR
    database. (141335)    Page 11
    GAO/T-RCED-99-198 Ordering Information The first copy of each GAO
    report and testimony is free. Additional copies are $2 each.
    Orders should be sent to the following address, accompanied by a
    check or money order made out to the Superintendent of Documents,
    when necessary. VISA and MasterCard credit cards are accepted,
    also. Orders for 100 or more copies to be mailed to a single
    address are discounted 25 percent. Orders by mail: U.S. General
    Accounting Office P.O. Box 37050 Washington, DC  20013 or visit:
    Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S.
    General Accounting Office Washington, DC Orders may also be placed
    by calling (202) 512-6000 or by using fax number (202) 512-6061,
    or TDD (202) 512-2537. Each day, GAO issues a list of newly
    available reports and testimony.  To receive facsimile copies of
    the daily list or any list from the past 30 days, please call
    (202) 512-6000 using a touchtone phone.  A recorded menu will
    provide information on how to obtain these lists. For information
    on how to access GAO reports on the INTERNET, send an e-mail
    message with "info" in the body to: [email protected] or visit
    GAO's World Wide Web Home Page at: http://www.gao.gov PRINTED ON
    RECYCLED PAPER United States General Accounting Office
    Bulk Rate Washington, D.C. 20548-0001     Postage & Fees Paid GAO
    Permit No. G100 Official Business Penalty for Private Use $300
    Address Correction Requested

*** End of document. ***