Measuring Performance: Challenges in Evaluating Research and Development
(Testimony, 04/10/97, GAO/T-RCED-97-130).
GAO discussed its work related to the National Institute of Standards
and Technology's Advanced Technology Program (ATP), Manufacturing
Extension Partnership (MEP) Program, and research and development (R&D)
performance measures.
GAO noted: (1) ATP has funded research projects that would have been
funded by the private sector as well as those that would not; (2) the
award recipients were nearly evenly divided when asked if they would
have pursued their projects if they had not received such funding; (3)
GAO also found that in most cases, the participants in its survey did
not look for funding from other sources, private or public, before
trying to obtain funding from ATP; (4) about half of the 45 applicants
that tried to find funding elsewhere before turning to ATP were told by
prospective funders that their projects were either too risky or
"precompetitive", characteristics that fulfill the aims of ATP; (5)
manufacturers viewed the manufacturing extension programs' services
positively, as was demonstrated in GAO's national survey of
manufacturers who had received substantive services from the programs in
1993; (6) most manufacturers responding to GAO's questionnaire, about 73
percent, reported that they believed that the type of assistance they
had received from these programs had positively affected their overall
business performance; (7) about 15 percent of the respondents reported
that they believed the programs' assistance had not affected their
overall business performance; (8) the amount of money spent on R&D, the
primary indicator of research investment, is useful as an input measure
of how much research is being performed; (9) however, the level of
spending is not a reliable indicator of research results; (10) GAO found
that there is no primary indicator of R&D results; (11) the companies
that GAO spoke with collect data on various output indicators, such as
return on investment and patents granted, but in general make limited
use of them in their investment decisions; (12) instead, they emphasized
that R&D contributes directly to the bottom line; (13) because companies
are profit oriented, many of the indicators tracked by the private
sector cannot be directly applied to the federal government; (14)
determining the specific outcomes resulting from federal R&D is a
challenge that will not be easily resolved; and (15) however, in
response to recent legislation requiring agencies to report on program
results, some progress is being made in measuring the impacts of
research.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: T-RCED-97-130
TITLE: Measuring Performance: Challenges in Evaluating Research
and Development
DATE: 04/10/97
SUBJECT: Research and development
Profits
Research program management
Federal funds
Research programs
Business assistance
Manufacturing industry
Technical assistance
Surveys
IDENTIFIER: NIST Advanced Technology Program
NIST Manufacturing Extension Partnership
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Before the Subcommittee on Technology, Committee on Science,
House of Representatives
For Release
on Delivery
Expected at
10 a.m. EDT
Thursday
April 10, 1997
MEASURING PERFORMANCE - CHALLENGES
IN EVALUATING RESEARCH AND
DEVELOPMENT
Statement by Allen Li,
Associate Director, Energy, Resources, and Science Issues,
Resources, Community, and Economic
Development Division
GAO/T-RCED-97-130
GAO/RCED-97-130T
(141046)
Abbreviations
=============================================================== ABBREV
NIST -
ATP -
MEP -
RCED -
GPRA -
============================================================ Chapter 0
Madame Chair and Members of the Subcommittee:
We are pleased to be here today to discuss our work related to two
programs at the National Institute of Standards and Technology
(NIST): the Advanced Technology Program (ATP) and the Manufacturing
Extension Partnership (MEP) Program. In addition, we will be
discussing our work performed in response to the Subcommittee's
request regarding research and development (R&D) performance
measures. This work resulted in our report entitled Performance
Measurement: Strengths and Limitations of Research Indicators
(GAO/RCED-97-91), which is being released today in conjunction with
this hearing.
Our report on measuring R&D performance highlights the difficulty in
measuring the impact of technology programs such as ATP and MEP. Our
work on these programs was performed in response to specific
congressional concerns. In the case of ATP, we were asked to assess
the impact of the program. In the case of MEP, we obtained
manufacturers' views on (1) the impact of the services provided by
the program, as well as other manufacturing extension programs, on
their business performance and (2) the factors affecting the impact
of these services. In the case of our report on performance
measurement, this Subcommittee asked us to evaluate various
indicators that are used to measure the results of R&D.
In summary, our work showed that:
-- The Advanced Technology Program has funded research projects
that would have been funded by the private sector as well as
those that would not.\1 The award recipients were nearly evenly
divided when asked if they would have pursued their projects if
they had not received such funding. We also found that in most
cases, the participants in our survey did not look for funding
from other sources, private or public, before trying to obtain
funding from the Advanced Technology Program. About half of the
45 applicants that tried to find funding elsewhere before
turning to the Advanced Technology Program were told by
prospective funders that their projects were either too risky or
"precompetitive"\2 --characteristics that fulfill the aims of
the Advanced Technology Program.
-- Manufacturers viewed the manufacturing extension programs'
services positively, as was demonstrated in our national survey
of manufacturers who had received substantive services from the
programs in 1993.\3 Most manufacturers responding to our
questionnaire--about 73 percent--reported that they believed
that the type of assistance they had received from these
programs had positively affected their overall business
performance. About 15 percent of the respondents reported that
they believed the programs' assistance had not affected their
overall business performance.\4
-- The amount of money spent on research and development, the
primary indicator of research investment, is useful as an input
measure of how much research is being performed. However, the
level of spending is not a reliable indicator of research
results. We found that there is no primary indicator of
research and development results. The companies that we spoke
with collect data on various output indicators, such as return
on investment and patents granted, but in general make limited
use of them in their investment decisions. Instead, they
emphasized that research and development contributes directly to
the bottom line. Because companies are profit oriented, many of
the indicators tracked by the private sector cannot be directly
applied to the federal government. Determining the specific
outcomes resulting from federal research and development is a
challenge that will not be easily resolved. However, in
response to recent legislation requiring agencies to report on
program results, some progress is being made in measuring the
impacts of research.
--------------------
\1 Measuring Performance: The Advanced Technology Program and
Private Sector Funding (GAO/RCED-96-47, Jan. 11, 1996).
\2 Precompetitive refers to the stage during R&D at which a
preliminary assessment of a technology's commercial potential can be
made but before commercial prototypes are developed.
\3 Manufacturing Extension Programs: Manufacturers' Views About
Delivery and Impact of Services (GAO/GGD-96-75, Mar. 14, 1996) and
Manufacturing Extension Programs: Manufacturers' Views of Services
(GAO/GGD-95-216BR, Aug. 7, 1995).
\4 Of the remaining 12 percent, 8 percent of the respondents said
that it was too early to determine if there was an effect, while 4
percent said they had no basis to estimate an effect.
BACKGROUND
---------------------------------------------------------- Chapter 0:1
ATP's mission is to stimulate economic growth in the United States
through technology development. The program seeks to accomplish that
mission by sharing the cost of R&D projects with private industry.
The projects selected by ATP for funding are characterized by the
program as having "a potential broad-based economic impact but a
relatively high technical risk and a long time horizon."
ATP's program guidance has stated that if the technical risk
associated with a project is very low, federal funding should not be
necessary. In addition, when submitting a research proposal,
applicants must sign a form stating that "this proposal is not
requesting funding for existing or planned research programs that
would be conducted in the same time period in the absence of
financial assistance under the ATP." This wording suggests that ATP
should not fund projects that other sources would have funded or,
when ATP does fund such projects, that ATP funds should enable
applicants to complete their projects in a shorter time.
Manufacturing extension programs offer manufacturers assistance in
modernizing or upgrading their operations, often with state and
federal funding. NIST manages federal funding of this type of
program through its Manufacturing Extension Partnership Program, or
MEP. In our prior reports, we used MEP to collectively refer to all
state, federal, and university manufacturing extension programs.\5
The primary mission of manufacturing extension programs is to give
"hands-on" technical assistance to small- and medium-sized
manufacturers trying to improve their operations through the use of
appropriate technologies.\6 These programs engage in a variety of
activities to assist small- and medium-sized manufacturers, often in
partnership with other business assistance providers, such as Small
Business Development Centers, community colleges, and federal
laboratories. The programs offer a wide range of business services,
including helping companies (1) solve individual manufacturing
problems, (2) obtain training for their workers, (3) create marketing
plans, and (4) upgrade their equipment and computers. The assistance
focuses on small- and medium-sized manufacturers because research by
the National Research Council and others has indicated that these
companies lack the resources necessary to improve their manufacturing
performance.
--------------------
\5 Manufacturing Extension Programs (GAO/GGD-96-75, Mar. 14, 1996)
and Manufacturing Extension Programs (GAO/GGD-95-216BR, Aug. 7,
1995).
\6 The Small Business Administration generally defines a small
business as having fewer than 500 employees. Some experts have
further divided small manufacturers into small firms with fewer than
100 employees and medium-sized firms with from 100 to 499 employees.
THE ADVANCED TECHNOLOGY PROGRAM
---------------------------------------------------------- Chapter 0:2
In our work on ATP, our objective was to examine, as one way to
assess the program's impact, whether research projects would have
been funded by the private sector if they had not received funds from
ATP.\7 We also examined ATP's impact in terms of other goals of the
program, such as aiding the formation of joint ventures.
We focused on two groups of ATP applicants, which we called "winners"
and "near winners." Both groups submitted proposals that were rated
highest during ATP's review, but the near winners did not ultimately
receive ATP funding. We surveyed all applicants that qualified as
winners or near winners during ATP's first 4 years (1990-93). We
achieved a 100-percent response rate from the 123 respondents that we
included in our analysis (89 winners and 34 near winners).
We found that ATP had funded research projects that would have been
funded by the private sector as well as those that would not. The
winners were nearly evenly divided when asked if they would have
pursued their projects even if they had not received ATP funding.
Half of the near winners continued their projects without relying on
ATP funding, while the other half discontinued their projects for
various reasons. Almost all the near winners that continued their
projects did so on a modified schedule, meeting the projects'
milestones later than they had scheduled in their proposals to ATP.
Of the 123 applicants we surveyed, 77,\8 or 63 percent, did not look
for funding from other sources before requesting it from ATP. Those
applicants that did look for funding looked for a long time and made
many attempts to find funding, on average. Seven applicants turned
down offers from private sources because they could not reach an
acceptable funding arrangement.
We also found that ATP had other effects. More than three-fourths of
the joint-venture applicants indicated that they had come together
solely to pursue an ATP project, thus satisfying ATP's goal of
serving as a catalyst for the formation of joint ventures.
Furthermore, of the 45 applicants that tried to find funding
elsewhere before turning to ATP, about half were told by prospective
funders that their projects were either too risky or
precompetitive--characteristics that fulfill the aims of ATP funding.
--------------------
\7 Measuring Performance (GAO/RCED-96-47, Jan. 11, 1996).
\8 One applicant did not know.
MANUFACTURING EXTENSION PROGRAM
SERVICES
---------------------------------------------------------- Chapter 0:3
We surveyed 766 U.S. manufacturers that had completed at least 40
hours of manufacturing extension program assistance, including NIST's
MEP, and received 551 responses.\9 We obtained respondents' views on
the impact of these services on their business performance and on the
factors affecting the impact of these services. We did not verify
either the positive or negative impacts reported by manufacturers,
nor did we evaluate the operations or management of specific federal
or state programs. We also obtained the views of other manufacturers
that had little or no experience with these programs to determine why
they made little or no use of them.
Most manufacturers responding to our questionnaire--about 73
percent--reported that they believed the programs' assistance had
positively affected their overall business performance. About 15
percent of the respondents reported that they believed the assistance
had not affected their overall business performance. Approximately 8
percent said that it was too early to determine the effect, and
another 4 percent said they had had no basis to estimate the effect.
In addition, most respondents reported that the assistance had
positively affected their use of technology in the workplace (about
63 percent), the quality of their product (about 61 percent), and the
productivity of their workers (about 56 percent). Between about 44
percent and 63 percent of the respondents reported that the programs'
assistance had positively affected certain specific indicators of
their business performance, such as customer satisfaction, profits,
and the ability to meet production schedules. Of those respondents
not reporting a positive impact on specific indicators of their
business performance, most said the programs' assistance had not had
any impact. Two percent or fewer of the respondents reported a
negative impact on each specific performance indicator.
Among the factors that manufacturers said had affected the impact of
MEP services was their own companies' input. The companies that had
committed their own financial resources to implement the programs'
recommendations reported greater benefits from the assistance
relative to other survey respondents. Of those 322 respondents who
had made a financial investment, 86 percent said that the programs'
assistance had positively affected their business performance.
However, 54 percent of those who had not made a financial investment
also reported an overall positive impact. Other factors, according
to the respondents, that influenced the effectiveness of the
programs' services were the expertise and experience of the programs'
staff and the affordability of the assistance.
In our related telephone survey of 200 additional manufacturers who
were not extensive users of the programs' services, about 82 percent
reported that they had not used the services because they were
unaware of these programs. About 10 percent said that although they
knew about these programs, they had not used them because they
believed the assistance would not be necessary. The companies we
interviewed said that other sources of modernization assistance
besides these programs were their customers, vendors and/or
suppliers, industry associations, and consultants.
--------------------
\9 Thirteen of these manufacturing extension programs received NIST
funding in fiscal year 1994. These 13 programs accounted for 36
percent of the 551 total respondents to our survey. See
Manufacturing Extension Programs (GAO/GGD-96-75, Mar. 14, 1996) and
Manufacturing Extension Programs (GAO/GGD-95-216BR, Aug. 7, 1995).
MEASURING THE IMPACTS OF
FEDERAL RESEARCH PROGRAMS
---------------------------------------------------------- Chapter 0:4
The report we are releasing today on performance measurement shows
that there is no single indicator or evaluation method that
adequately captures the results of R&D. However, indicators do
provide helpful information for making decisions about R&D. Whether
the focus is on basic research, applied research, or development, the
amount of money spent in that area is taken as an indication of how
much research is being performed. The major advantages of using
expenditure data as an indicator are that they are easily
understandable, readily available, and have been, in general,
consistently gathered over time. In addition, spending on different
projects in different research areas can be measured according to the
same unit, dollars, making comparisons between projects
straightforward.
The amount of funding, however, does not provide a good indication of
research results. Companies told us that they are switching their
spending to more short-term R&D projects rather than long-term
projects. However, the impacts of that change are unclear. The
reduced funding levels for long-term projects may not reflect the
fact that the R&D efforts can be performed with greater efficiency.
For example, one way in which the federal government and the private
sector have tried to use R&D resources more efficiently and
effectively is through consortia with universities or other
companies. By combining their research activities, companies attempt
to avoid expensive duplication and learn from each other.
We also found that because of the difficulties in identifying the
impacts of research, quantitative and qualitative indicators have
been developed as proxies to assess R&D results. The strengths and
limitations are evident in both types of indicators. Quantitative
indicators focus mainly on return on investment, patenting rates, and
bibliometrics--the study of publication-based data. While implying a
degree of precision, these indicators were not originally intended to
measure long-term R&D results. Qualitative assessment such as peer
review provides detailed information, but it relies on the judgments
of experts and may be expensive.
Because of these difficulties, the companies we interviewed stressed
marketplace results rather than R&D output indicators. While varying
in the types of indicators they collect, they emphasized the
difficulties in measuring R&D's specific contribution to a company's
overall performance. For example, one company stated that because so
many people have been involved in a product's evolution, it is
difficult to separate the contribution of the research unit from that
of other units. All of the companies interviewed have increased
their expectation that R&D contribute directly to their
profitability. However, instead of increasing their efforts at
measuring R&D results, they have shifted the responsibility for
making R&D decisions to the business units. For example, if the
business units believe that a particular R&D project would increase
their profits, the firm would budget for that R&D.
Many of the R&D output measures tracked by the private sector do not
apply directly to the federal government. In particular, while
facing the same increasing cost pressures as the private sector, the
federal government cannot rely on the profit motive to guide its
decisions.
This discussion of performance measures for R&D is particularly
relevant because of the current emphasis on the Government
Performance and Results Act (GPRA). In response to questions about
the value and effectiveness of federal programs, GPRA seeks to shift
federal agencies' focus away from such traditional concerns as
staffing, activity levels, and tasks completed toward a focus on
program outcomes. GPRA incorporates performance measurement as one
of its most important features. Under this act, executive branch
agencies are required to develop annual performance plans that use
performance measurement to reinforce the connection between the
long-term strategic goals outlined in their strategic plans and the
day-to-day activities of their managers and staff. However, the very
nature of the innovative process makes measuring the performance of
science-related projects difficult. For example, a wide range of
factors determines if and when a particular R&D project will result
in commercial or other benefits. It can also take many years for a
research project to achieve results.
Experiences from recent GPRA pilot efforts reinforce the fact that
output measures are highly specific to the management and mission of
each federal agency and that no single indicator exists to measure
the results of research. The Army Research Laboratory, which was
designated as a pilot project for performance measurement under the
act, has developed a multifaceted approach using quantitative
indicators, peer review, and customer feedback to evaluate the
results of R&D. Although this is not the only approach that can be
taken, this response to the challenges in measuring the impacts of
research shows that some progress is being made in response to GPRA.
Madame Chair, this concludes my prepared remarks. I would be happy
to respond to any questions you may have.
*** End of document. ***