Small Business Innovation Research: Agencies Need to Strengthen
Efforts to Improve the Completeness, Consistency, and Accuracy of
Awards Data (19-OCT-06, GAO-07-38).
The Small Business Innovation Research (SBIR) program was created
to increase the use of small businesses to meet federal research
needs and commercialize the results of this research. To monitor
the program, the Small Business Administration (SBA) requires
participating agencies to provide, in a standard format, specific
data on all SBIR awards they make. SBA then compiles these data
into a database known as Tech-Net. Congress also required SBA to
create, by 2001, a restricted and more comprehensive database
that would provide information for government agencies to use in
evaluating the program. GAO was asked to identify the (1) types
of data that agencies report to SBA for inclusion in the Tech-Net
database, (2) extent to which these data are provided in a
standard format, (3) extent to which SBA has established the
government-use database, and (4) extent to which SBIR agencies
have developed and implemented techniques to track
commercialization of SBIR projects. GAO reviewed 8 of the 11
agencies participating in SBIR.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-07-38
ACCNO: A62346
TITLE: Small Business Innovation Research: Agencies Need to
Strengthen Efforts to Improve the Completeness, Consistency, and
Accuracy of Awards Data
DATE: 10/19/2006
SUBJECT: Awards
Data collection
Data integrity
Databases
Federal agencies
Interagency relations
Monitoring
Program evaluation
Program management
Reporting requirements
Schedule slippages
Small business
Policies and procedures
SBA Tech-Net
Small Business Innovation Research
Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-07-38
* [1]Results in Brief
* [2]Background
* [3]Agencies Provide Most of the Required SBIR-Award Data Elemen
* [4]Agencies Generally Comply with SBA's Formatting Guidance, bu
* [5]SBA Is Five Years Behind Schedule in Meeting its Obligation
* [6]Tracking Data on the Commercialization Success of SBIR Proje
* [7]Agencies' Efforts to Track SBIR Commercialization Success Va
* [8]Tracking the Commercialization Success of the Overall SBIR P
* [9]Conclusions
* [10]Recommendation for Executive Action
* [11]Agency Comments and Our Evaluation
* [12]Appendix I: Objectives, Scope and Methods
* [13]Appendix II: Comments from the Small Business Administration
* [14]GAO Comments
* [15]Appendix III: Comments from the Department of Defense
* [16]Appendix IV: Comments from the National Aeronautics and Spac
* [17]Appendix V: Comments from the National Institutes of Health
* [18]Appendix VI: GAO Contact and Staff Acknowledgments
* [19]GAO Contact
* [20]Acknowledgments
* [21]Order by Mail or Phone
Report to Congressional Committees
United States Government Accountability Office
GAO
October 2006
SMALL BUSINESS INNOVATION RESEARCH
Agencies Need to Strengthen Efforts to Improve the Completeness,
Consistency, and Accuracy of Awards Data
GAO-07-38
Contents
Letter 1
Results in Brief 4
Background 7
Agencies Provide Most of the Required SBIR-Award Data Elements to SBA, but
Some Data Submitted are Incomplete 9
Agencies Generally Comply with SBA's Formatting Guidance, but Key Data on
SBIR Award Recipients May Be Inconsistent or Inaccurate 10
SBA Is Five Years Behind Schedule in Meeting its Obligation to Implement a
Government-Use SBIR Database 13
Tracking Data on the Commercialization Success of SBIR Projects Varies
Among Agencies and Remains a Challenge 15
Conclusions 18
Recommendation for Executive Action 19
Agency Comments and Our Evaluation 19
Appendix I Objectives, Scope and Methods 21
Appendix II Comments from the Small Business Administration 24
GAO Comments 26
Appendix III Comments from the Department of Defense 27
Appendix IV Comments from the National Aeronautics and Space
Administration 29
Appendix V Comments from the National Institutes of Health 30
Appendix VI GAO Contact and Staff Acknowledgments 31
Abbreviations
USDA Department of Agriculture
DOD Department of Defense
DOE Department of Energy
EPA Environmental Protection Agency
FDA Food and Drug Administration
NASA National Aeronautics and Space Administration
NIST National Institute of Standards and Technology
NIH National Institutes of Health
NSF National Science Foundation
R&D research and development
SBA Small Business Administration
SBIR Small Business Innovation Research
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
United States Government Accountability Office
Washington, DC 20548
October 19, 2006
The Honorable Sherwood Boehlert Chairman The Honorable Bart Gordon Ranking
Minority Member Committee on Science House of Representatives
The Honorable Vernon J. Ehlers Chairman The Honorable David Wu Ranking
Minority Member Subcommittee on Environment, Technology and Standards
Committee on Science House of Representatives
Recognizing the potential of small businesses to be a source of
significant innovation, the Congress established the Small Business
Innovation Research (SBIR) program in 1982. From fiscal year 1983 through
fiscal year 2004, federal agencies that participated in the SBIR program
awarded over $17 billion in grants, contracts, or cooperative agreements
to over 82,000 projects. The primary goals of the SBIR program are to
stimulate technological innovation, meet federal research and development
(R&D) needs, foster participation by minority and disadvantaged persons in
technological innovation, and increase the commercial success
(commercialization) of innovation that is derived from federally funded
R&D.^1 Because the Congress did not define what constitutes commercial
success of federally funded R&D or how best to measure it, agencies have
used different commercialization outcomes for the SBIR program, such as
the sale of the resulting SBIR-funded product or process, the extent to
which SBIR firms have received non-SBIR funding, or the creation of new
jobs or products. The SBIR program is currently scheduled to expire on
September 30, 2008.
^1Small Business Innovation Development Act of 1982, Pub. L. No. 97-219
(1982).
Every federal agency with an R&D budget of $100 million or more is
required to establish and operate an SBIR program funded by 2.5 percent of
the agency's budget for research conducted by others, called extramural
research. Currently, 11 federal agencies participate in the SBIR
program.^2 Each agency manages its own program, including targeting
research areas, reviewing proposed projects, and making research awards
through grants, contracts, or cooperative agreements. The Small Business
Administration (SBA) plays a central administrative role by, for example,
issuing policy directives to the participating federal agencies,
collecting data from participating agencies on awards and recipients, and
reporting program results annually to the Congress.
Over the last 24 years, the SBIR program has been reauthorized and
modified by the Congress at various times. For example, the Small Business
Research and Development Enhancement Act of 1992 directed SBA and
participating agencies to, among other things, emphasize the goal of
increasing commercialization of research results and to improve the
government's dissemination of program-related data.^3 As a result,
agencies were required to include commercialization potential as a
criterion for selecting award recipients. During this same period, SBA
began to develop a publicly available database, known as Tech-Net, that
contained information on all awards made through the SBIR program.^4 The
Tech-Net database is intended to be, among other things, an electronic
gateway of technology information and resources for researchers,
scientists, and government officials about federally-funded leading edge
technology research. The Small Business Innovation Research Program
Reauthorization Act of 2000 formalized this database by requiring SBA to
develop, maintain, and make available to the public a searchable,
up-to-date, electronic database that contained SBIR award information.^5
The 2000 reauthorization act also required SBA to develop and maintain
another restricted government database that would contain additional
information on commercialization not contained in the public Tech-Net
database, thereby allowing better evaluations of the SBIR program on an
ongoing basis. This database was to be established by mid-2001 and made
available only to government agencies and certain other authorized users.
^2The eleven SBIR participating agencies are the departments of
Agriculture (USDA), Commerce, Defense (DOD), Education, Energy (DOE),
Health and Human Services, Homeland Security, and Transportation; the
Environmental Protection Agency (EPA); the National Aeronautics and Space
Administration (NASA); and the National Science Foundation (NSF).
^3Pub. L. No. 102-564 (1992).
^4The Tech-Net database also contains award information on SBA's Small
Business Technology Transfer (STTR) program. However, the focus of this
report is on the SBIR program and not the STTR program. As such, any
discussion of the Tech-Net database only refers to the information in the
database related to the SBIR awards.
^5Pub. L. No. 106-554 (2000).
As the program has evolved over time, congressional direction has focused,
among other things, on the ability of SBIR award recipients to
commercialize the results of their research, as evidenced by the increased
emphasis on commercialization in the 1992 and 2000 reauthorizations of the
program. At various points in the life of the program, we have reported
that the SBIR program has been successful in increasing commercialization
of research results and that agencies have used various methods to measure
the commercial success of the projects they fund.^6 Largely these methods
have consisted of surveys of award recipients to obtain data on indicators
of commercial success and soliciting "success stories" voluntarily
provided by SBIR award recipients. However, we also reported that these
methods provided only "snapshots" of commercial success and did not allow
for a systematic demonstration of changes in program commercialization
rates over time. We reported that the lack of clarity on how much emphasis
agencies should place on commercialization versus other SBIR program goals
had also created challenges for assessing the program's results.
In the context of efforts to monitor and evaluate the success of the SBIR
program, you requested that we identify the (1) types of data that
participating SBIR agencies are reporting to SBA for inclusion in the
Tech-Net database; (2) extent to which agencies provide data for the
Tech-Net database in a standard format to enable program evaluation; (3)
extent to which SBA has met the mandate to establish, by mid-2001, a
government-use database that can be used for program evaluation; and (4)
extent to which participating SBIR agencies have developed and implemented
techniques to track the commercialization success of SBIR projects.
^6See GAO, Federal Research: Observations on the Small Business Innovation
Research Program, [22]GAO-05-861T , June 28, 2005; Federal Research:
Observations on the Small Business Innovation Program,
[23]GAO/T-RCED-98-218 , June 4, 1998; Federal Research: Small Business
Innovation Research Shows Success but Can be Strengthened,
[24]GAO/RCED-92-37 , Mar. 30, 1992; and Federal Research: Assessment of
Small Business Innovation Research Programs, [25]GAO/RCED-89-39 , Jan. 23,
1989.
In conducting this study, we reviewed the SBIR-related activities at 8 of
the 11 SBIR participating agencies, which account for over 98 percent of
the total dollars awarded by the program in fiscal year 2005.^7 To
determine the types of data these participating agencies are reporting to
SBA and the extent to which SBA has complied with the requirement to
establish a government-use database that can be used for program
evaluation purposes, we compared the provisions in the Small Business
Innovation Research Program Reauthorization Act of 2000, SBA's amended
Policy Directive implementing the Act issued in September 2002, and other
guidance with agency SBIR Tech-Net database reports. To determine the
extent to which data for the Tech-Net database are provided in a standard
format to enable program evaluation, we compared the data from
participating agencies with data in SBA's Tech-Net database for fiscal
years 2004 and 2005, the 2 most recent years for which data were
available. Our data reliability review focused on SBA's data system and
internal controls, rather than on the systems and internal controls
agencies use to create the data provided to SBA. To assess the reliability
of the data in SBA's Tech-Net database, we interviewed SBA officials about
the database and reviewed related documentation. We determined that the
data are sufficiently reliable for our purpose, which is to report on
SBA's efforts to ensure consistency and completeness of the data it
receives. We used GAO's data reliability guidance to identify key
attributes of data quality that can facilitate program evaluation. We also
interviewed SBA and agency officials to determine the extent to which the
government-use database requirements have been implemented. To determine
the extent to which participating agencies and SBA have developed and
implemented techniques to evaluate commercialization success of SBIR
projects, we reviewed agency documentation and interviewed SBA and agency
officials. Appendix I contains a detailed discussion of the scope and
methodology of our review. We conducted our review from April 2006 through
September 2006 in accordance with generally accepted government auditing
standards.
Results in Brief
Federal agencies participating in the SBIR program annually submit to SBA
a wide range of descriptive information about each award they make. Each
year SBA requires participating agencies to provide over 40 data elements
for each SBIR award made. These data include information about the award
itself, such as the award amount, a descriptive abstract of the project,
and a unique tracking number; information about the award recipient such
as the name, gender, and socio-economic status; and information about the
type of firm that received the award, such as number of employees and
geographic location. However, we determined that although participating
agencies are providing most of the information requested by SBA, they are
not consistently providing the full range of required information. For
example, two of the eight agencies we reviewed had not consistently
provided SBA data on the gender and socio-economic status of SBIR award
recipients, and five did not provide data on the number of employees
working at the firms that received SBIR awards. As a result, certain
sections of the database needed for comprehensive program evaluation are
incomplete. Agencies cited a variety of reasons for not providing the data
requested by SBA, including frequent changes in SBA's data requirements
and differences in the types of data agencies collect versus the types of
data that SBA wants to collect.
^7The SBIR participating agencies included in this review are DOD, DOE,
EPA, NASA, National Institutes of Health (NIH) within the Department of
Health and Human Services, National Institute of Standards and Technology
(NIST) within the Department of Commerce, NSF, and USDA.
Some participating agencies are not submitting SBIR award data in the
standard format required by SBA, and while some of these problems are
corrected by SBA's quality assurance processes, others are not. Since
2000, SBA officials have worked with participating agencies to ensure
greater consistency in how the data required for the Tech-Net database are
formatted and have developed a reporting template that includes the
required data fields and instructions for appropriate data entry. However,
almost a quarter of the data provided by five participating agencies in
2004 and 2005 still did not comply with SBA's formatting guidance. As a
result, SBA officials said that they have to spend a considerable amount
of time and resources correcting these data, and are unable to correct all
of them. SBA's current quality assurance efforts therefore focus on
obtaining complete and accurate data only for those fields essential to
tracking specific awards, such as the tracking numbers and award amounts,
and not correcting data in fields that contain demographic information
about award recipients. The primary reason for the inconsistently
formatted data is that the template SBA has provided for reporting data
can be edited, and some agencies have done so and introduced different
formats. According to SBA, by fiscal year 2007, this issue should be
resolved because agencies will have to submit their data through an
Internet interface that will no longer allow the agencies to change the
format of titles and data. In light of the problems we identified with the
Tech-Net database and the implications for these errors to limit
evaluations of the SBIR program, we are recommending that SBA work with
participating agencies to strengthen efforts to improve the quality of the
data.
SBA has not met the congressional mandate to develop and implement, by
June 2001, a government-use database that can be used for evaluating the
SBIR program. To meet the congressional requirement to develop and
implement a government-use database, SBA planned to expand the existing
Tech-Net database and include a restricted government-use section that
would be accessible only to government agencies and other authorized
users. This government-use section of the Tech-Net database would rely on
information already gathered for the Tech-Net database and supplemented by
information on the commercialization outcomes for SBIR awards. However,
SBA officials told us that they have been unable to meet the requirement
to implement a government-use database by 2001 because of management
changes that have occurred at the agency and because of budgetary
constraints. To date, with the help of two contractors, SBA has developed
the framework for (1) importing data into the government-use section of
the database, and (2) an Internet-based interface that would allow
agencies and award recipients to access and enter commercialization
information. According to SBA, the government-use section of the database
will not be fully operational until the agency resolves certain
outstanding issues, such as making the commercialization information
secure because this information is considered proprietary and
confidential. SBA officials expect that the government-use section of the
database will be operational by October 1, 2006; however, they also
recognize that additional enhancements to improve the efficiency of the
database may still be needed after the system is operational.
Seven of the eight agencies participating in the SBIR program that we
reviewed are gathering data on the commercialization success of
SBIR-funded projects; however, the methodological rigor of their methods
varies significantly. Under the program's authorizing legislation and
implementation guidance, agencies have been given considerable flexibility
in tracking the commercial success of their SBIR-funded projects.
Specifically, five of the eight agencies systematically and periodically
survey SBIR award recipients to gather a range of information on program
participation, including commercialization experience. For example, DOD
annually surveys SBIR award recipients to gather data on (1) sales of
SBIR-funded research results, (2) commitments SBIR recipients have
received for additional development funding, and (3) whether SBIR-funded
research results have been used by other federal programs. Two of the
eight agencies we reviewed use less systematic data gathering efforts and
instead focus on gathering success stories and conducting periodic
follow-up with a small sample of SBIR award recipients. For example, USDA
periodically contacts a small group of award recipients to update and
obtain information on their commercialization experiences. The remaining
agency--NIST--has developed a survey that it plans to send to all of its
SBIR award recipients in 2006. Regardless of the methods used to gather
commercialization information, SBA and agency officials believe that
several factors complicate their efforts to obtain this information in a
standardized and complete manner. For example, these officials told us
that many years may elapse from the time an SBIR award is granted to the
time a product or process achieves commercial success and maintaining
contact with award recipients during this time period is often difficult.
Moreover, during this time, firms may change their names, be purchased by
other firms, or start new businesses to pursue the project's commercial
potential, making it even more difficult for the agencies to track these
firms and link them to their original SBIR awards.
Background
The Small Business Innovation Development Act of 1982 established the SBIR
program and identified four program goals: technological innovation,
commercialization of the research results, the use of small businesses to
meet agencies' R&D needs, and participation in federal R&D by minorities
and disadvantaged persons. The legislation provided for a competitive
three-phased program: phase I to determine the technical and scientific
merit and feasibility of a proposed research idea; phase II to further
develop the idea, taking into account its commercial potential; and phase
III to commercialize the resulting product or process with private or
federal investment but no additional SBIR funding.
Under the SBIR program provisions, federal agencies that have external R&D
budgets of $100 million or more are required to use 2.5 percent of these
budgets to establish and operate an SBIR program. SBA oversees and
coordinates the efforts of the eleven agencies currently participating in
the program. In this capacity, SBA coordinates the participating agencies'
schedules to announce opportunities for firms to apply for awards, called
a solicitation, and provides access to these solicitations through its Web
site. As part of its oversight effort, SBA collects SBIR data from the
participating agencies, aggregates the data, and uses the data to, among
other things, monitor the program and report annually to the Congress.
In reauthorizing the SBIR program in 1992, the Congress stated its
intention to expand and improve the program by emphasizing its goal of
increasing private sector commercialization, increasing participation in
federal R&D by small businesses, and improving the government's
dissemination of program-related information. One of the new provisions
under the 1992 legislation requires agencies, when evaluating phase II
proposals, to consider their commercial potential, including the
recipients' experiences commercializing the results from previous SBIR
awards, commitments accompanying the proposals for developmental funding
from sources other than the SBIR program, and other factors.
The SBIR program was again reauthorized in 2000 by the Small Business
Innovation Research Program Reauthorization Act of 2000. The 2000
legislation directed SBA and participating agencies to, among other
things, expand the scope of publicly available information on specific
awards and to annually report data on their SBIR programs to SBA. The act
required that SBIR phase II award recipients be requested to voluntarily
provide information to the agencies describing the outputs and outcomes of
their SBIR award. The act also required SBA to establish, by mid-2001, a
searchable and up-to-date electronic database available for public use,
and a restricted government-use database. To accomplish this mandate, SBA
envisioned expanding the electronic database, known as Tech-Net, that it
had developed in the late 1990s, into two sections: a public-use section
and a restricted government-use section. The public-use section of the
Tech-Net database would provide access to nearly all of the
statutorily-required award information for SBIR awards gathered by the
agencies. The public-use section was intended to be an electronic gateway
of technology information and resources for researchers, scientists, and
government officials who are seeking information on potential small
business partners, contractors, or leading edge technology research. The
government-use section would be solely used for program evaluation
purposes accessible only to government agencies and other authorized
users, and would contain commercialization data voluntarily supplied by
SBIR recipients upon completion of their phase II SBIR funding agreement,
such as revenue from the sale of new products or services resulting from
the research undertaken with the award. In addition, applicants for phase
II awards would be required to update information on the commercialization
success of any prior SBIR awards they had received. Currently, SBA has
created and is maintaining the public-use section of the Tech-Net
database, which is available on the Internet to the general public.
Agencies Provide Most of the Required SBIR-Award Data Elements to SBA, but Some
Data Submitted are Incomplete
While federal agencies participating in the SBIR program submit a wide
range of descriptive award information to SBA annually, these agencies are
not consistently providing all of the required information. As outlined in
SBA's guidance, each year SBIR participating agencies are required to
collect and maintain information from recipients and provide it to SBA so
that it can be included in the Tech-Net database. Specifically,
participating agencies are required to provide over 40 data elements for
each SBIR award they make. These data include award-specific information,
such as the date and amount of the award, an abstract of the project
funded by the award, and a unique tracking number for each award.
Participating agencies are also required to provide data about the award
recipient, such as gender and socio-economic status, and information about
the type of firms that received the awards, such as number of employees
and geographic location. Much of the data collected by participating
agencies are provided by the SBIR applicants at the time they apply for an
award. Agencies provide additional information, such as the grant/contract
number and the dollar amount of the award, after the award is made.
For the most part, all of the eight agencies we reviewed provided the
majority of the SBIR award data requested by SBA, including the program
identification number; company name, address, and contact information;
award year and amount; a unique tracking number that will stay with the
award through both phase I and phase II; and the title and abstract of the
project funded. However, we also determined that some of the agencies are
not providing the full range of information required by SBA. For example,
two of the eight agencies we reviewed had not provided SBA data on the
gender or socio-economic status of SBIR award recipients in 2004 and 2005.
Similarly, in 2005, five of the eight agencies failed to provide data on
the number of employees working at the firms that received the awards. As
a result, SBA does not have information on the number of employees at SBIR
awardee firms for about one-third of all the awards reported by these
agencies in 2004 and 2005.
SBA officials acknowledged that agencies do not routinely provide all of
the information requested because either they do not capture the
information in their agency databases or they are not requesting the
information from SBIR applicants. Officials at the participating agencies
also cited other reasons for the incomplete data they provided to SBA. For
example, NIH officials stated that for the past several years, the SBA
Tech-Net annual reporting requirements have changed each year. At the end
of calendar year 2003, SBA changed the field description from "minority"
to "socially and economically disadvantaged small business." According to
NIH officials, because the SBIR information is captured by the agency at
the time the application is submitted/received, there is a lag in time
between when application data is input into the NIH database and when the
agency receives new SBA data field requirements. According to these
officials, responding to SBA's changes in field names presents significant
challenges to NIH for collecting the data needed to complete the Tech-Net
reports, especially if the requirements change annually. As a result,
these officials stated that complying with the changing requirements
necessitates significant NIH resources and efforts to research and
identify the information needed for the new data fields, not all of which
can be provided. In commenting on a draft of this report, SBA stated that
it has only requested minor clarifications of data requirements and has
not made frequent changes as stated by the agencies. SBA believes that
this may have been caused by lack of clear communication to the agencies.
The agencies also noted that data for certain Tech-Net fields will be
absent from their reports to SBA if the data fields do not exist in the
NIH application or in its awards database. Similarly, USDA officials
stated that although they try to keep their records as up-to-date as
possible, problems occur when company or contact information changes and
the SBIR recipient fails to provide updated information to the agency,
such as the e-mail address for the central contact person. Additionally,
like NIH, USDA officials stated that certain information requested by SBA
is not collected by their agency. For example, as long as SBIR applicants
have certified that they meet the criterion of being a small business
(under 500 employees), they do not ask for nor do they record information
on the number of employees in the firm.
Agencies Generally Comply with SBA's Formatting Guidance, but Key Data on SBIR
Award Recipients May Be Inconsistent or Inaccurate
Participating agencies are providing some data that do not comply with
SBA's formatting guidance, and while some of these inconsistencies are
corrected by SBA's quality assurance processes, others are not. As a
result, some data elements in the Tech-Net database may be inconsistent or
inaccurate thereby, compromising the value of these data for program
evaluation. SBA's quality assurance efforts focus on obtaining complete
and accurate data for those fields essential to tracking specific awards,
such as the tracking number and award amount, rather than on those fields
that contain demographic information about the award recipient. Because
the data contained in the public-use section of Tech-Net will be
incorporated into the government-use section of the database, inaccurate
data in one section of the database will be replicated in the other, and
these inaccurate data will limit evaluations of the SBIR program.
Both SBA and agency officials acknowledge that SBA has worked with
participating agencies since 2000 to help ensure greater consistency in
the formatting of the SBIR award data reported by agencies each year. To
assist agencies in formatting the data and to minimize the number of
inconsistencies in the data reported to SBA, SBA has taken a number of
steps to improve the data formatting process. Specifically, SBA provided
all of the participating agencies specific guidance on its requirements
for the data, including its preferred program application for submitting
the data, the length of each data field, and whether data should be
entered as numbers, characters, or a mix of both. SBA has also included
discussions of the Tech-Net database as an agenda item at its quarterly
meetings with SBIR program managers from each of the participating
agencies. Additionally, SBA developed a reporting template for agencies to
use that includes the required data fields and instructions for
appropriate data entry.
Under the current process, participating agencies aggregate all of the
data provided by SBIR recipients in their award applications with
additional information on the award amounts and submit the combined data
to SBA by March 15 of each year. SBA then electronically checks the data
to locate and reformat inconsistencies, and adds the data to the Tech-Net
database. SBA officials told us that when they detect inconsistencies in
data fields essential to tracking a specific award, such as the award
tracking number, contact information for the recipient or principal
investigator, or awarding agency, they contact the agency to obtain the
correct information. However, SBA does not currently take steps to ensure
that agency-provided data are accurate and complete. For example, SBA does
not require agencies to submit a random sample of applications so that it
can compare the data submitted by agencies with the original applicant
information to ensure that the submitted data contain all the relevant
application information. Instead, SBA relies on the agencies to fully
report all the required application information on the awards they make.
SBA officials told us that they believe that over the past two years the
quality and consistency of the data received from participating agencies
had greatly improved.
In reviewing the SBIR Tech-Net data that the eight agencies reported to
SBA in 2004 and 2005, we determined that almost a quarter of the data
provided by five agencies was incorrectly formatted for one or more
fields. For example, phone number and award amount fields contained both
characters and numbers and first and last names of principal investigators
were combined into a single word that was used as both the first and last
names. Moreover, we found that agency-provided data on gender and
socio-economic status for over half of the awards reported in 2004 were
incorrectly formatted. SBA officials said that detecting and correcting
some of these formatting errors required a considerable amount of time and
effort, and that their electronic check does not detect all of the errors
that we identified.
The formatting inconsistencies that we identified arise primarily because
the SBA reporting template used by agencies to submit required data can be
edited. Consequently, agencies can and do edit the template. Agencies can
change the names of various data fields in SBA's template, delete fields
altogether, and enter data as numbers, characters, or both, regardless of
what SBA has specified. One of the data format issues identified by SBA
was the deletion of fields that agencies do not consider relevant or
necessary. For example, DOD is the only agency that uses the field called
"branches" to specify which component of DOD, such as the Army, made the
award; other agencies have deleted this field because it does not relate
to the structure of their agency. Similarly, we found instances where
agencies had entered data on the gender or socio-economic status of the
recipient and award amount in a format that differed from the numerical
format specified by SBA. For example, one agency entered "Y" and "N"
rather than "0" and "1." According to SBA officials, by fiscal year 2007,
this issue should be resolved because agencies will have to submit their
data through an Internet interface that will contain edit checks and
should eliminate many of these problems.
We also determined that inconsistencies or inaccuracies can arise in
certain data fields because SBA interprets the absence of certain data
elements as a negative entry without confirming the accuracy of such an
interpretation with the agency. In other words, if an agency did not
provide information on whether the recipient is a woman or a member of a
socio-economically disadvantaged group, SBA has entered a "no" into the
database. SBA stated that they generally do not contact agencies to obtain
correct information on data elements that are not used to track specific
awards, such as gender or socio-economic status of the recipient. However,
this inaccurate data on the award recipients could limit efforts to use
these fields for comprehensive program evaluation.
Information in the Tech-Net database will be used to populate the
government-use section of the database that SBA is developing (as
discussed below) for the purpose of supporting SBIR program evaluations.
However, SBA has no plans to correct any of the errors or inconsistencies
in the database that relate to the historical data already collected. As a
result, the errors in the existing database will migrate to the
government- use section of the database and we believe will compromise the
usefulness of the government-use database for program evaluation purposes.
SBA Is Five Years Behind Schedule in Meeting its Obligation to Implement a
Government-Use SBIR Database
SBA has not met its obligation to implement a restricted government-use
database that would allow SBIR program evaluation as directed by the 2000
SBIR reauthorization act. As outlined in the legislation, SBA, in
consultation with federal agencies participating in the SBIR program, was
to develop a secure database by June 2001 and maintain it for program
evaluation purposes by the federal government. SBA planned to meet this
requirement by expanding the existing Tech-Net database to include a
restricted government-use section that would be accessible only to
government agencies and other authorized users. In constructing the
government-use section of the database, SBA planned to supplement existing
data already gathered for the public-use section of the Tech-Net database
with information from SBIR recipients and from participating agencies on
commercialization outcomes for phase II SBIR awards. However, according to
SBA officials, the agency has been unable to meet this requirement,
primarily because of increased security and other information technology
project requirements, agency management changes, and budgetary
constraints. SBA's current goal for having the government-use section of
the Tech-Net database operational is October 1, 2006. In commenting on a
draft of this report, SBA modified this date, stating that they anticipate
having the government-use section of the Tech-Net database operational
early in fiscal year 2007.
To date, with the help of two contractors, SBA has developed the framework
for importing data into the government-use section of the Tech-Net
database and for an Internet-based interface that would allow agencies and
award recipients to access the database and enter commercialization
information. According to SBA officials, as currently envisioned, the
government-use section of the Tech-Net database will include the records
of all applicants, including those that did not receive SBIR awards.
Participating SBIR agencies will be asked to provide a unique business
identification number, called the Data Universal Numbering System or DUNS
number, for each award recipient, information about SBIR applicants that
were not funded, and any historical data they have obtained about the
commercialization of SBIR funded technologies. SBA has developed a
standardized electronic commercialization questionnaire to gather data for
the government-use section of the database from applicants and award
recipients. Information that will be captured in the questionnaire will
include the number of SBIR awards the company has received, the number of
patents or copyrights that have resulted from the award, sales revenue
realized as a result of the SBIR award, and sources of additional
investment funding. SBA officials told us that the commercialization
questionnaire will become an integral part of the SBIR application process
in the future, and any company applying for an SBIR award will be required
to complete and update the relevant information on phase II SBIR awards
previously received by the company at the time the application is
submitted. According to SBA officials, applications will not be accepted
until this information is completed, and failure to submit the information
may affect an applicant's ability to receive an award. In addition, SBIR
award recipients will be requested to voluntarily update the
commercialization information in the government-use section of the
Tech-Net database annually for a minimum period of five years following
the completion of their SBIR-funded project.
Although SBA has developed the majority of the functions needed to
populate the government-use section of the Tech-Net database with the data
currently gathered for the public-use section of the database, it can not
be made operational until certain security issues are resolved. For
example, because the government-use section of the database will contain
information from recipients that is considered proprietary and
confidential, SBA needs to ensure that adequate security measures are in
place to prevent unauthorized access to the data. This entails the
successful completion of a series of security and development checks to
ensure that the database system is operating as designed. While SBA
officials expect the government-use section of the database to be
operational by October 1, 2006, they also recognize that additional
enhancements, such as improving the user-friendliness of the interface for
online submission of SBIR data by participating agencies and recipients,
will be needed after the system is made operational. According to SBA
officials, the agency's priority is to get the government-use section of
the database up and running before considering further improvements to the
database.
Tracking Data on the Commercialization Success of SBIR Projects Varies Among
Agencies and Remains a Challenge
Seven of the eight participating agencies we reviewed have implemented
techniques to track the commercialization success of their SBIR-funded
projects, and the eighth is planning to do so, although the methodological
rigor of these techniques varies significantly. Under the program's
authorizing legislation and SBA's implementation guidance, agencies have
been given considerable flexibility to design, monitor, and evaluate the
extent to which their SBIR programs have achieved commercialization
success. For example, while some agencies use more systematic approaches
to gathering data, such as periodically surveying SBIR award recipients,
other agencies are less methodical, choosing instead to follow up
periodically with a relatively small sample of SBIR award recipients.
Regardless of how they track commercialization success, both SBA and
agency officials generally agree that several factors complicate their
efforts to obtain this information.
Agencies' Efforts to Track SBIR Commercialization Success Vary
Of the eight agencies we reviewed, five systematically and periodically
survey SBIR recipients to gather a variety of data on program
participation, including the recipients' commercialization experiences.
Specifically,
o Since 2000, DOD has systematically gathered information
electronically on the commercialization of phase II awards from
all phase I and phase II applicants and award recipients and
maintains the information in a commercialization database.
Commercialization outcomes that DOD monitors include such measures
as (1) sales revenue from new products and non-R&D services
resulting from the phase II technology; (2) additional investment
from sources other than the federal SBIR program in activities
that further the development, commercialization, or both of the
phase II technology; (3) whether the phase II technology has been
used in a DOD system or acquisition program, and if so, which
system or program; (4) the number of patents resulting from the
contractor's participation in the SBIR program; (5) growth in the
number of employees at the firm; and (6) whether the firm has
completed an initial public offering of stock resulting, in part,
from the phase II project. DOD uses the accumulated data to assign
a commercialization score to applicants that have received four or
more prior SBIR awards based on a comparison of their
commercialization experience with the average experience of other
comparable applicants and uses the score to help select proposed
projects for funding. In addition, recipients of phase II awards
are required to update the information one year after the start of
the project, at the completion of the project, and subsequently
when the recipient submits a new SBIR application to DOD. Firms
that do not submit a new SBIR application are asked to voluntarily
provide updates on an annual basis after the completion of their
phase II project. According to DOD officials, 66 percent of phase
II award recipients updated their commercialization information
when they submitted a new application, 11 percent provided the
information without submitting a new application, and 23 percent
did not update their information.
o For over 23 years, DOE has conducted an annual survey of SBIR
phase II recipients, both active and inactive, to track the
commercialization success of its SBIR-funded projects. The survey
requests recipients to (1) list all products and services derived
from their SBIR projects; (2) report on sales, phase III
investment related to these products and service, or both; and (3)
identify which phase II projects contributed to the development of
the products and services. According to DOE, approximately 90
percent of its phase II recipients respond to these annual
surveys.
o NASA has systematically gathered information on the
commercialization of SBIR awards through annual surveys of phase
II award recipients from 1997 to 2002. In these surveys, NASA
obtained data on various commercial outcomes, such as sales to
nongovernment markets of the SBIR-funded research results,
procurement of the research results by NASA or other federal
agencies, cumulative private capital funding, royalty and
licensing revenue from nongovernment sources based on the
SBIR-funded research results, creation of new business ventures
based on the SBIR-funded research results, and number of patents
and patent applications resulting from these awards. According to
NASA, approximately 91 percent of its phase II award recipients
responded to these annual surveys.
o NIH also surveys SBIR award recipients to gather
commercialization information. Specifically, in 2002, NIH
conducted a comprehensive evaluation of its SBIR program. As part
of this evaluation, NIH surveyed recipients of phase II awards
between 1992 and 2001 to obtain data relating to the range of SBIR
program goals, including commercialization of research results.
According to NIH officials, the 2002 survey results formed a
baseline that NIH staff could use to systematically monitor and
evaluate the program. In 2004 and 2005, NIH again contacted award
recipients to update the information obtained in the 2002 survey.
Commercialization outcomes tracked by NIH include (1) sales
realized for a product or service that resulted from the
SBIR-funded research; (2) status of the Food and Drug
Administration (FDA) approval process for the SBIR-funded research
results; (3) receipt of FDA approval for SBIR-funded research
results; and (4) receipt of additional non-SBIR funding or
capital.
o Since about 1998, NSF has collected historical commercialization
information from all phase II award recipients at the time an SBIR
application is submitted. This information is used in the proposal
review process to help select proposed projects for funding. In
2005, NSF developed an annual survey of phase II award recipients
that will be used to gather information three, five, and eight
years following their awards. Specific outcomes on which NSF will
gather data include sales revenue based on the SBIR-funded
research results, growth of overall company sales and employment,
receipt of additional non-SBIR funding, and patents related to the
SBIR funded research.
In contrast, two of the remaining three agencies we reviewed have
focused their efforts on gathering anecdotal success stories and
conducting periodic follow-up with a relatively smaller sample of
SBIR award recipients. For example, over the past 7 years, EPA has
contacted all phase II award recipients after their projects end
to learn about their commercial successes. Based on these
contacts, officials estimate that approximately 25 percent of the
phase II projects funded by EPA have been commercialized. EPA
defines "success" as the receipt of more than $300,000 in revenue
from sales of the SBIR-funded project, an amount greater than the
SBIR funds awarded by EPA. Similarly, USDA has periodically
contacted a sample of about 20 to 25 percent of award recipients
to obtain information about sales of their SBIR-funded research
results. USDA last surveyed its phase II award recipients in 1997.
USDA publishes the success stories on its Web site and in an
agency newsletter. About 2,500 people receive the newsletter and
USDA makes copies of the success stories available at SBIR
conferences. According to USDA officials, in the future they hope
to gather data more systematically and conduct site visits to the
SBIR firms. The eighth agency we reviewed, NIST, has recently
developed a Web-based survey that it plans to send in 2006 to all
of its SBIR-award recipients.
Tracking the Commercialization Success of the Overall SBIR Program
Remains a Challenge
Although each of the eight agencies we reviewed has implemented or
plans to implement a method for gathering commercialization data,
agency officials identified several factors that complicate their
efforts. First, agency officials stated that it is difficult to
track commercialization because it can take years before companies
achieve commercial success. For example, USDA officials stated
that, even over the short term, the effort to contact past award
recipients consumes considerable effort. During this time,
companies may move, change names, start a new business, or be
purchased by other firms, all of which make it difficult for the
agencies to track and link companies to the original SBIR awards.
Second, because the authorizing legislation lacks a clear
definition of what constitutes "commercialization" success,
agencies not only differed on the types of commercialization
outcomes they captured, but also in how they tracked commercial
success. SBA officials acknowledged that its guidance has provided
considerable latitude to agencies on this issue in light of the
wide range of industries represented by the participating
agencies. Commercialization outcomes captured by the participating
agencies included sales revenue based on the SBIR-funded research
results, receipt of additional non-SBIR funding to further develop
the research results, marketing activities ongoing or completed,
and public offering of company stock. However, not all agencies
are tracking all of these outcomes; therefore, assessing overall
commercial success of the SBIR program across the various agencies
remains a challenge.
Third, agency officials stated that SBIR award recipients may be
reluctant to provide information related to their trade and
business operations, which they consider proprietary and
sensitive. Companies are often not willing to provide
comprehensive data on their sales and particularly the investments
they receive due to competitive concerns. Finally, agency
officials told us that past recipients have no incentive to
voluntarily complete commercialization surveys and update the
information on their commercial experience unless they are
applying for a new SBIR award. As a result, they do not expect
that a large percentage of recipients will complete the
information needed for the government-use section of the Tech-Net
database. Agency officials believe that despite their best
efforts, the data needed to conduct evaluations of the SBIR
program are likely to be incomplete.
Conclusions
In the last 5 years, SBA has been unable to meet the congressional
directive to develop a government-use database that would provide
better information on the SBIR program and allow for program
evaluation. Although it has established a public-use Tech-Net
database and has worked with participating agencies to achieve
greater consistency in the data submitted for the database, the
quality of the data remains a concern. The steps on which SBA
relies to ensure that data are complete and accurate are
inadequate and it has no plans to correct errors or supply missing
data associated with the historical data already in the database.
We believe that unless necessary controls are established and
implemented to ensure the completeness, consistency, and accuracy
of the SBIR data reported to SBA by participating agencies, the
government-use section of the Tech-Net database, which depends on
the public-use Tech-Net database, will be of limited use for
program evaluation purposes when it becomes operational.
Recommendation for Executive Action
We recommend that the Administrator, SBA, and the SBIR
participating agencies work together to strengthen efforts to
ensure that the data collected for SBA's Tech-Net database are
complete, consistent, and accurate.
Agency Comments and Our Evaluation
We provided SBA and the eight SBIR participating agencies included
in this review a draft of this report for their review and
comment. SBA and the eight agencies generally agreed with the
report's findings and SBA and five of the eight agencies also
stated their concurrence with the recommendation. Three
agencies--EPA, NIST, and NSF--did not indicate whether they agreed
or disagreed with the recommendation.
In addition, SBA stated that it was concerned that our conclusions
did not reflect the fact that it plans to have the government-use
section of the Tech-Net database operational by early fiscal year
2007. We have not modified our conclusions because the fact
remains that SBA has not met the congressional directive to
establish a government-use database during the last five years.
Moreover, throughout this review, SBA officials told us that the
database would be operational by October 1, 2006. However, in its
official comments, the agency has again modified this date to some
time early in fiscal year 2007. SBA also provided us with
technical comments that we have incorporated as appropriate. SBA's
letter is included in appendix II and the letters that we received
from DOD, NASA, and NIH are included in appendix III through V.
As agreed with your office, unless you publicly announce the
contents of this report earlier, we plan no further distribution
until 30 days from the report date. At that time, we will send
copies of this report to interested congressional committees; the
Administrators of the Environmental Protection Agency, National
Aeronautics and Space Administration, and Small Business
Administration; the Directors of the National Institutes of
Health, National Institute of Standards and Technology, and
National Science Foundation; the Secretaries of Agriculture,
Defense, and Energy; and other interested parties. We will also
make copies available to others upon request. In addition, the
report will be available at no charge on the GAO Web site at
[26]http://www.gao.gov .
If you or your staff have any questions about this report, please
contact me at (202) 512-3841 or [27]mittala@gao.gov . Contact
points for our Offices of Congressional Relations and Public
Affairs may be found on the last page of this report. GAO staff
who made contributions to this report are listed in appendix VI.
Anu K. Mittal
Director, Natural Resources and Environment
Appendix I: Objectives, Scope and Methods
Our objectives for this review were to identify (1) the types of
data that participating Small Business Innovation Research (SBIR)
program agencies are reporting to the Small Business
Administration (SBA) for inclusion in the Tech-Net database, (2)
the extent to which agencies provide data for the Tech-Net
database in a standard format to enable program evaluation, (3)
the extent to which SBA has met the mandate to establish by early
2001 a government-use database that can be used for program
evaluation, and (4) the extent to which participating SBIR
agencies have developed and implemented techniques to track the
commercialization success of SBIR projects.
In conducting this study, we reviewed SBA and the SBIR-related
activities of 8 of the 11 SBIR participating agencies--Department
of Defense (DOD), Department of Energy (DOE), Environmental
Protection Agency (EPA), National Aeronautics and Space
Administration (NASA), National Institutes of Health (NIH),
National Institute of Standards and Technology (NIST), National
Science Foundation (NSF), and Department of Agriculture (USDA).
These eight agencies account for over 98 percent of the total
dollars awarded by the program in fiscal year 2005. To determine
the types of data these participating agencies are reporting to
SBA and the extent to which SBA has complied with the requirement
to establish a government-use database that can be used for
program evaluation purposes, we compared the provisions in the
Small Business Innovation Research Program Reauthorization Act of
2000, relevant legislative histories, SBA's Policy Directive
implementing the act issued in September 2002, and other guidance
with agency SBIR Tech-Net database reports. We identified and
interviewed SBIR program officials at each agency and officials
responsible for submitting program data to SBA. For these
interviews, we used a protocol guide to obtain information on
program operations, data reporting, data quality, and the Tech-Net
database, including development of the government-use section of
the database. We also reviewed documentation from each of the
eight agencies and SBA regarding data elements and formatting
instructions.
To determine the extent to which data for the Tech-Net database
are provided in a standard format, enabling program evaluation, we
compared data provided to SBA by the eight participating agencies
with data in SBA's Tech-Net database for fiscal years 2004 and
2005, the 2 most recent years for which data were available. The
agency data included information about the award itself, such as
the award amount, a descriptive abstract of the project, and a
unique tracking number; information about the award recipient,
such as gender and socio-economic status; and information about
the type of firm that received the award, such as number of
employees and geographic location. To assess the reliability of
the data provided by the agencies, we reviewed SBA's data system,
internal controls, and related guidance, rather than the systems
and internal controls participating agencies use to create the
data provided to SBA. To assess the reliability of the data in
SBA's Tech-Net database, we interviewed SBA officials about the
database and reviewed related documentation. We also conducted
tests of the data themselves. We used analytic software to compare
the data provided by participating agencies with the data
maintained by SBA. We focused our review on data fields considered
critical by SBA officials, such as the awarding agency, the date
of award, the award recipient, the amount of the award, and the
purpose of the award. We also reviewed data fields related to SBIR
program goals, such as gender and socio-economic status of the
recipient, and data on the number of employees at the recipient
firm. We used GAO's data reliability guidance to identify key
attributes of data quality that facilitate program evaluation.
These attributes include completeness, accuracy, and consistency
in format. Finally, we reviewed internal quality control
procedures. We determined that the data are sufficiently reliable
for our purpose, which is to report on SBA's efforts to ensure
consistency and completeness of the data it receives.
To determine the extent to which the government-use database
requirements have been implemented, we interviewed SBA officials,
and reviewed related documentation, such as minutes from meetings
of SBA and SBIR program directors, and overviews of the existing
and planned data systems. In addition, we attended a demonstration
of the proposed Internet interface for the government-use section
and interviewed the current contractor assisting SBA about
implementation progress. We also reviewed a contractor-prepared
analysis of the functional and data requirements for the
integration of the public- and government-use sections of the
Tech-Net database. At each of the eight participating agencies, we
interviewed SBIR officials regarding the extent to which SBA had
consulted them in the development of the government-use database.
To determine the extent to which participating agencies have
developed and implemented techniques to evaluate commercialization
success of SBIR projects, we reviewed agencies' documentation
about their commercialization assistance and monitoring efforts.
Specifically, we reviewed surveys that agencies had administered
to award recipients, resulting reports on survey results, and
anecdotal descriptions of commercialization success. We also
reviewed provisions in SBIR legislation, relevant legislative
histories, and SBA's Policy Directive regarding commercialization
of SBIR-funded projects, as well as past GAO reports. In addition,
we interviewed officials at each of the eight participating
agencies to obtain information on the specific commercialization
outcomes they monitor, the history of each agency's data
collection efforts, and the agencies' experience in obtaining such
information from current and past award recipients.
We conducted our work from April 2006 through September 2006 in
accordance with generally accepted government auditing standards.
Appendix II: Comments from the Small Business Administration
Note: GAO comments supplementing those in the report text appear at the
end of this appendix.
Page numbers in draft report may differ from those in this report.
See comment 4.
See comment 3.
See comment 2.
See comment 1.
See comment 5.
GAO Comments
The following are GAO's comments on the Small Business
Administration's (SBA) letter dated October 6, 2006.
1. We revised footnote 2 to more clearly reflect the
11 SBIR participating agencies.
2. We revised the text to clarify this statement.
3. We deleted reference to SBA in footnote 7.
4. We added SBA's position to the report.
5. We added SBA's position to the report. However,
because SBA has not met the congressional deadline to
develop a government-use database, we did not modify
our conclusions.
Appendix III: Comments from the Department of Defense
Appendix IV: Comments from the National Aeronautics and Space
Administration
Appendix V: Comments from the National Institutes of Health
Appendix VI: GAO Contact and Staff Acknowledgments
GAO Contact
Anu K. Mittal, (202) 512-3841
Acknowledgments
In addition to the contact named above, Cheryl Williams, Assistant
Director; Bernice H. Dawson; Vondalee R. Hunt; and Marcus L.
Oliver made key contributions to this report. Also contributing to
this report were Nancy Crothers, Grant Mallie, Gregory Marchand,
and Rebecca Shea.
GAO�s Mission
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in
meeting its constitutional responsibilities and to help improve
the performance and accountability of the federal government for
the American people. GAO examines the use of public funds;
evaluates federal programs and policies; and provides analyses,
recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony
The fastest and easiest way to obtain copies of GAO documents at
no cost is through GAO's Web site ( [28]www.gao.gov ). Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of
newly posted products every afternoon, go to [29]www.gao.gov and
select "Subscribe to Updates."
Order by Mail or Phone
The first copy of each printed report is free. Additional copies
are $2 each. A check or money order should be made out to the
Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are
discounted 25 percent. Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM
Washington, D.C. 20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax:
(202) 512-6061
To Report Fraud, Waste, and Abuse in Federal Programs
Contact:
Web site: [30]www.gao.gov/fraudnet/fraudnet.htm E-mail:
[31]fraudnet@gao.gov Automated answering system: (800) 424-5454 or
(202) 512-7470
Congressional Relations
Gloria Jarmon, Managing Director, [32]JarmonG@gao.gov (202)
512-4400 U.S. Government Accountability Office, 441 G Street NW,
Room 7125 Washington, D.C. 20548
Public Affairs
Paul Anderson, Managing Director, [33]AndersonP1@gao.gov (202)
512-4800 U.S. Government Accountability Office, 441 G Street NW,
Room 7149 Washington, D.C. 20548
(360677)
www.gao.gov/cgi-bin/getrpt?GAO-07-38 .
To view the full product, including the scope
and methodology, click on the link above.
For more information, contact Anu K. Mittal at (202) 512-3841 or
mittala@gao.gov.
Highlights of [35]GAO-07-38 , a report to congressional committees
October 2006
SMALL BUSINESS INNOVATION RESEARCH
Agencies Need to Strengthen Efforts to Improve the Completeness,
Consistency, and Accuracy of Awards Data
The Small Business Innovation Research (SBIR) program was created to
increase the use of small businesses to meet federal research needs and
commercialize the results of this research. To monitor the program, the
Small Business Administration (SBA) requires participating agencies to
provide, in a standard format, specific data on all SBIR awards they make.
SBA then compiles these data into a database known as Tech-Net. Congress
also required SBA to create, by 2001, a restricted and more comprehensive
database that would provide information for government agencies to use in
evaluating the program. GAO was asked to identify the (1) types of data
that agencies report to SBA for inclusion in the Tech-Net database, (2)
extent to which these data are provided in a standard format, (3) extent
to which SBA has established the government-use database, and (4) extent
to which SBIR agencies have developed and implemented techniques to track
commercialization of SBIR projects. GAO reviewed 8 of the 11 agencies
participating in SBIR.
[36]What GAO Recommends
GAO recommends that SBA and SBIR participating agencies work together to
improve the quality of the data in SBA's Tech-Net database. SBA and the
SBIR participating agencies included in this report generally agreed with
our recommendation.
Federal agencies participating in the SBIR program annually submit over 40
data elements to SBA for each award they make. These data include
information on the award, such as value and a descriptive abstract;
information on the recipient, such as name and gender; and information
about the firm receiving the award, such as number of employees and
location. Participating agencies submit most of the information required
by SBA, but they are not consistently providing all required data
elements, including the number of employees in the firm, and the gender
and socio-economic status of the award recipient, resulting in incomplete
sections of the database. Agencies stated that this happens because they
do not collect all of the information that SBA wants and because SBA's
requirements change regularly.
Some participating agencies are not submitting SBIR award data in the
standard format required by SBA, and although SBA's quality assurance
processes correct most of these problems, they do not correct all of them.
In 2004 and 2005, about 25 percent of the data provided by five
participating agencies did not comply with SBA's format. Formatting
inconsistencies occur because the template SBA has provided agencies for
reporting data can be edited. According to SBA, identifying and correcting
inconsistently formatted data involves considerable resources, therefore
the agency has focused its quality assurance efforts only on key data
elements needed to track awards; other fields, such as those containing
demographic data, are generally not corrected. As a result, comprehensive
program evaluations may be limited by the quality of the data in these
fields. SBA officials expect this problem to be resolved by fiscal year
2007, when all data will be submitted via an Internet interface that will
not allow changes in the format.
SBA is 5 years behind schedule in meeting the congressional mandate to
implement a restricted government-use database for the SBIR program. SBA
had planned to meet this requirement by expanding its Tech-Net database to
include a restricted government-use section. SBA officials attributed the
delay in meeting the 2001 deadline primarily to increased security
requirements needed for the database, agency management changes, and
budgetary constraints. SBA officials expect the government-use section of
Tech-Net to be operational by October 1, 2006, when safeguards to protect
the proprietary commercialization information in the database are in
place.
Most agencies GAO reviewed systematically gather data on the
commercialization success of SBIR-funded projects. Five of these eight
agencies regularly survey all awardees to gather information on program
participation, including commercial success, and one agency is about to
start a similar survey. In contrast, two agencies only gather anecdotal
success stories from a small sample of SBIR awardees. SBA and agency
officials generally agree that despite their best efforts, obtaining
commercialization information from awardees remains a major challenge.
References
Visible links
22. http://www.gao.gov/cgi-bin/getrpt?GAO-05-861T
23. http://www.gao.gov/cgi-bin/getrpt?GAO/T-RCED-98-218
24. http://www.gao.gov/cgi-bin/getrpt?GAO/RCED-92-37
25. http://www.gao.gov/cgi-bin/getrpt?GAO/RCED-89-39
26. http://www.gao.gov/
28. http://www.gao.gov/
29. http://www.gao.gov/
30. http://www.gao.gov/fraudnet/fraudnet.htm
34. http://www.gao.gov/cgi-bin/getrpt?GAO-07-38
35. http://www.gao.gov/cgi-bin/getrpt?GAO-07-38
*** End of document. ***