Measuring Performance: Strengths and Limitations of Research Indicators
(Chapter Report, 03/21/97, GAO/RCED-97-91).
American taxpayers invested more than $60 billion in military and
civilian research and development projects in 1996. The private sector
spent more than $110 billion that same year. The technological
advancements stemming from these efforts have helped improve the
productivity of American workers and, correspondingly, the nation's
standard of living. However, although the contributions of research and
development to technological advancement are generally recognized, there
is no widely accepted method of measuring the outcome of that research.
This report evaluates the various indicators that are used to measure
the results of research and development. Specifically, GAO discusses the
strengths and limitations of the input and output indicators used by the
federal and private sectors to measure the results of research and
development. GAO also provides a historical perspective on spending for
research. GAO summarized this report in testimony before Congress; see:
Measuring Performance: Challenges in Evaluating Research and
Development, by Allen Li, Associate Director for Energy, Resources, and
Science Issues, before the Subcommittee on Technology, House Committee
on Science (GAO/T-RCED-97-130, Apr. 10).
--------------------------- Indexing Terms -----------------------------
REPORTNUM: RCED-97-91
TITLE: Measuring Performance: Strengths and Limitations of
Research Indicators
DATE: 03/21/97
SUBJECT: Research and development
Research and development costs
Investments
Non-government enterprises
Profits
Patents
Publications
Evaluation methods
Economic analysis
IDENTIFIER: NSF Science and Engineering Indicators Document
NSF Industrial Research and Development Survey
**************************************************************************
* This file contains an ASCII representation of the text of a GAO *
* report. Delineations within the text indicating chapter titles, *
* headings, and bullets are preserved. Major divisions and subdivisions *
* of the text, such as Chapters, Sections, and Appendixes, are *
* identified by double and single lines. The numbers on the right end *
* of these lines indicate the position of each of the subsections in the *
* document outline. These numbers do NOT correspond with the page *
* numbers of the printed product. *
* *
* No attempt has been made to display graphic images, although figure *
* captions are reproduced. Tables are included, but may not resemble *
* those in the printed version. *
* *
* A printed copy of this report may be obtained from the GAO Document *
* Distribution Facility by calling (202) 512-6000, by faxing your *
* request to (301) 258-4066, or by writing to P.O. Box 6015, *
* Gaithersburg, MD 20884-6015. We are unable to accept electronic orders *
* for printed documents at this time. *
**************************************************************************
Cover
================================================================ COVER
Report to Congressional Requesters
March 1997
MEASURING PERFORMANCE - STRENGTHS
AND LIMITATIONS OF RESEARCH
INDICATORS
GAO/RCED-97-91
Measuring Performance
(307738)
Abbreviations
=============================================================== ABBREV
AT&T -
DOD - Department of Defense
GAO - General Accounting Office
GPRA - Government Performance and Results Act
IBM -
IRI - Industrial Research Institute
NIH - National Institutes of Health
NSF - National Science Foundation
NASA - National Aeronautics and Space
Administration
OECD - Office of Economic Cooperation
and Development
OTA - Office of Technology Assessment
R&D - research and development
Letter
=============================================================== LETTER
(rced)
B-275241
March 21, 1997
The Honorable Constance A. Morella
Chairwoman
The Honorable Bart Gordon
Ranking Minority Member
Subcommittee on Technology
Committee on Science
House of Representatives
The Honorable John S. Tanner
House of Representatives
This report responds to your request for information on the
indicators used to evaluate the results of research and development
(R&D). The report discusses the relative strengths and limitations
of the input and output indicators used by the federal and private
sectors to measure the results of R&D. The report also provides a
historical perspective on research spending.
As agreed with your offices, we plan no further distribution of this
report until 30 days from its date of issue, unless you publicly
announce its contents earlier. We will then send copies to
interested parties, and we will also make copies available to others
upon request.
If you have any questions, I can be reached at (202) 512-3600. Major
contributors to this report are listed in appendix II.
Allen Li
Associate Director, Energy,
Resources, and Science Issues
EXECUTIVE SUMMARY
============================================================ Chapter 0
PURPOSE
---------------------------------------------------------- Chapter 0:1
American taxpayers invested more than $60 billion in federal funds in
military and civilian research and development (R&D) efforts in 1996.
The private sector invested more than $110 billion that same year.
The technological advancements resulting from these efforts are a
critical factor in improving the productivity of American workers
and, correspondingly, the nation's standard of living. However,
while the contribution of R&D to technological advancement is widely
recognized, there is no widely accepted method of measuring the
results of that research.
To facilitate discussions of the adequacy of the funding and of the
results of the R&D, the Subcommittee on Technology, House Committee
on Science, asked GAO to evaluate the various indicators that are
used to measure the results of R&D. Specifically, this report
discusses the strengths and limitations of the input and output
indicators used by the federal and private sectors to measure the
results of R&D. This report also provides a historical perspective
on spending for research.
BACKGROUND
---------------------------------------------------------- Chapter 0:2
The commitment to reduce the federal deficit is forcing the Congress
to reexamine the value of programs across the federal government.
Although scientific research is often considered to be intrinsically
valuable to society, there is pressure on all federal agencies,
including science agencies, to demonstrate that they are making
effective use of the taxpayers' dollars. This greater emphasis on
results is evident in the passage of the Government Performance and
Results Act of 1993 (GPRA). The act fundamentally seeks to shift the
focus of federal management and accountability from a preoccupation
with staffing, activity levels, and tasks completed to a focus on
results--that is, the real difference that federal programs make in
people's lives.
The experts in research measurement have tried for years to develop
indicators that would provide a measure of the results of R&D.
However, the very nature of the innovative process makes measuring
the performance of science-related projects difficult. For example,
a wide range of factors determine if and when a particular R&D
project will result in commercial or other benefits. It can also
take many years for a research project to achieve results.
RESULTS IN BRIEF
---------------------------------------------------------- Chapter 0:3
The amount of money spent on research and development, the primary
indicator of the investment in research, is useful as a measure of
how much research is being performed. Having been refined over many
years, these data are generally available for the research efforts in
both the public and private sectors. However, the level of spending
is not a reliable indicator of the level of results achieved by
research.
Unlike the situation with the input measures of research and
development, there is no primary indicator of the outputs. Output
indicators include quantitative analyses of return on investment,
patents granted, and other outputs as well as qualitative assessments
based on peer review. The companies that GAO spoke with collect data
on various output indicators but, in general, make limited use of
them in their investment decisions. Instead, the companies
emphasized that research and development contribute directly to their
"bottom line." Because companies are profit- oriented, many of the
indicators tracked by the private sector cannot be directly applied
to the federal government. Experiences from pilot efforts made under
the Government Performance and Results Act have reinforced the
finding that output measures are highly specific to the management
and mission of each federal agency and that no single indicator
exists to measure the results of research.
PRINCIPAL FINDINGS
---------------------------------------------------------- Chapter 0:4
FUNDING INDICATES RESEARCH
ACTIVITY BUT DOES NOT
MEASURE THE RESULTS OF
RESEARCH
-------------------------------------------------------- Chapter 0:4.1
Funding has been used as the primary input indicator for decades.
Whether a policymaker is interested in basic research, applied
research, or development, the amount of money spent in that area is
taken as an indication of how much research is being performed. The
major advantages of using expenditure data as an indicator are that
they are easily understandable, readily available, and have been, in
general, consistently gathered over time. In addition, spending on
different projects in different research areas can be measured
according to the same unit, dollars, making comparisons between
projects straightforward.
The amount of funding, however, does not provide a good indication of
the results of research. Companies told GAO that they are focusing
more of their spending on short-term R&D projects than on long-term
projects. However, the impacts of that change in focus are unclear.
The reduced funding levels may not reflect the fact that the R&D
efforts are being performed with greater efficiency. For example,
one way in which the federal government and the private sector have
tried to use R&D resources more efficiently and effectively is
through consortia with universities or other companies. By combining
their research activities, the companies attempt to avoid expensive
duplication and learn from each other.
R&D OUTPUT INDICATORS CAN
PROVIDE LIMITED INFORMATION
ABOUT THE RESULTS OF R&D
-------------------------------------------------------- Chapter 0:4.2
Because of the difficulties in identifying the impacts of research,
decisionmakers have developed quantitative and qualitative indicators
as proxies to assess the results of R&D activity. The strengths and
limitations are evident in both types of indicators. The current
quantitative indicators focus mainly on return on investment,
patenting rates, and bibliometrics--the study of published data.
While implying a degree of precision, these indicators were not
originally intended to measure the long-term results of R&D.
Qualitative assessment provides detailed information, but it relies
on the judgments of experts and may be expensive.
Because of these difficulties, the companies interviewed by GAO
stressed marketplace results rather than R&D output indicators.
While varying in the types of indicators they collect, they
emphasized the difficulties in measuring R&D's specific contribution
to a company's overall performance. For example, one company stated
that because so many people have been involved in a product's
evolution, it is difficult to separate the contribution of the
research unit from that of other units. All of the companies
interviewed have increased their expectation that R&D contribute
directly to their profitability, but instead of increasing their
efforts at measuring R&D results, they have shifted the
responsibility for R&D decisions to the business units. However,
many of the R&D output measures tracked by the private sector do not
apply directly to the federal government. In particular, while
facing the same increasing cost pressures as the private sector, the
federal government cannot rely on the profit motive to guide its
decisions.
The GPRA requires the executive agencies to develop their annual R&D
plans with suitable performance measures. The Research Roundtable, a
group of federal researchers and managers representing a
cross-section of R&D departments and agencies, warned about the
difficulties of quantifying the results of R&D and the potential for
incorrect application with subsequent harm to scientific endeavors.
The Army Research Laboratory, which was designated a pilot project
for performance measurement under the act, has developed a
multifaceted approach using quantitative indicators, peer review, and
customer feedback to evaluate the results of R&D.
RECOMMENDATIONS
---------------------------------------------------------- Chapter 0:5
This report contains no recommendations.
AGENCY COMMENTS
---------------------------------------------------------- Chapter 0:6
Because this report focuses broadly on the R&D of both the federal
and private sectors, and not on the effort of individual agencies,
GAO did not submit a draft of this report to federal agencies for
their review and comment.
INTRODUCTION
============================================================ Chapter 1
Over $180 billion was spent on research and development (R&D) in the
United States in 1996.\1 Most of that amount was spent by industry
and the federal government--$113 billion and $62 billion,
respectively; the balance was spent by universities and other
nonprofit organizations. The leading experts in the study of
research indicators agree that R&D has a significant, positive effect
on economic growth and the overall standard of living. However,
because of the complexity of linking the results of R&D to its
economic impacts, there is no widely accepted method of measuring the
results of R&D spending in either industry or the federal government.
The commitment to reduce the federal budget deficit is forcing the
Congress and the executive branch to undertake a basic reexamination
of the value of programs across the federal government. It is also
placing pressure on all federal agencies, including the civilian
science agencies, to clearly demonstrate that they are making
effective use of the taxpayers' dollars. The Government Performance
and Results Act of 1993 (GPRA) provides a legislative vehicle for the
agencies to use as they seek to demonstrate and improve their
effectiveness. Equally important, if successfully implemented, GPRA
should help the Congress make the difficult funding, policy, and
program decisions that the current budget environment demands.
--------------------
\1 These figures are based on preliminary 1996 statistics reported by
R&D performers to the National Science Foundation.
THE PROCESS OF INNOVATION AND
THE USE OF R&D INDICATORS
---------------------------------------------------------- Chapter 1:1
Researchers have developed their own terminology for describing the
process of transforming R&D into economic results. In its simplest
form, the theory underlying both public and private decision-making
has been that innovative activity positively affects economic
performance. Innovation can be thought of as the development and
application of a new product, process, or service. It can include
the use of an existing product in a new application or the
development of a new device for an existing application. Innovation
encompasses many activities, including scientific, technical, and
market research; product, process, or service development; and
manufacturing and marketing to the extent that they support the
dissemination and application of the invention. Innovation is a
combination of invention and commercialization. Invention describes
the initial conception of a new product, process, or service, but not
the act of putting it to use. Commercialization refers to the
attempt to profit from innovation through the sale or use of new
products, processes, and services.
This description as well as the traditional views of innovation has
been strongly influenced by the linear model of innovation, which
says that innovation proceeds sequentially through the stages of
basic research, applied research, development, manufacturing, and
marketing.\2 This model assumes that basic research serves as the
source of innovation and that new scientific knowledge initiates a
chain of events culminating in the development and sale of a new
product, process, or service. In this model, basic research is the
most uncertain part of the process; once basic research is conducted,
innovation and commercialization can proceed. The model suggests
that the firms with the best technology will likely be the first to
market and win the lion's share of profits.
The simplicity of this model makes it particularly useful in policy
discussions. Other models may be more accurate, but they provide a
more complex explanation of the relationship between science and the
commercialization of new technology. These models, such as the
"chain-linked model," include feedback loops that allow for
interaction among the different stages of the linear model. These
models also reflect the fact that the ideas for new inventions or
changes to existing products often arise from the recognition of new
market opportunities, advances in manufacturing capabilities, or
advances in technology independent of progress in the underlying
science.
Real-world examples show that technological breakthroughs can precede
as well as follow basic research. In many cases, science is not the
source of innovation. The Wright brothers, for example, developed
the first airplane without an understanding of aerodynamic theory,
and Chester Carlson developed the first xerographic copier without a
thorough understanding of photoconductive materials. These
inventions have resulted in considerable research into aerodynamic
theory and materials science, respectively, as scientists and
engineers attempted to improve the original invention.
In studying complex processes or concepts such as innovation, it is
not always possible to measure them directly. As a result,
researchers turn to the use of "indicators." Indicators point to or
illustrate the process or concept in question but do not directly
measure it. For example, in order to determine an object's
temperature, one can use a thermometer to measure it directly.
However, when trying to measure something as complex as the health of
a country's economy, one relies on different indicators, such as
unemployment rates, stock market averages, or trade balances that do
not provide a direct measurement of economic health but do give an
indication of its status. In the case of innovation, various "input"
and "output" indicators are used that are based on the linear model
of innovation. The following chapters of this report are broken down
according to these two sets of indicators. Chapter 2 is concerned
with R&D spending, or expenditure data, which is the most widely used
input indicator of innovation, and chapter 3 focuses on some widely
used output indicators.
--------------------
\2 The National Science Foundation uses the following definitions in
its resource surveys: Basic research has as its objective to gain a
more comprehensive knowledge or understanding of the subject under
study, without specific applications in mind. Applied research is
aimed at gaining knowledge or understanding to determine the means by
which a specific, recognized need may be met. Development is the
systematic use of the knowledge or understanding gained from research
directed toward the production of useful materials, devices, systems,
or methods. The Foundation recognizes the limitations of this
classification scheme but continues to use these categories to
maintain historical consistency, among other reasons.
PAST EFFORTS TO EVALUATE R&D
IMPACTS
---------------------------------------------------------- Chapter 1:2
For almost two decades, numerous reports have documented the
difficulties of quantifying the results of R&D. As noted above, the
identification of the economic and social effects of research
involves complex issues of measurement, analysis, and interpretation.
The following efforts may help to give perspective to the present
concerns about measuring R&D results.
In 1979, we reported on a wide range of factors that make the
measurement of R&D results difficult.\3 The report noted that
R&D expenditures are undertaken for a variety of reasons. Some
attempt to develop new knowledge; others are directed at meeting
needs, such as national defense, for which there is no
commercial market; and still others are directed at lowering the
cost of products. Furthermore, some projects produce
revolutions; others produce nothing. Most important, R&D is
only one input into a complex process. Thus, we concluded that
there is no possibility of eliminating the role that judgment
plays in the allocation of federal R&D resources.
In 1986, the Office of Technology Assessment (OTA) issued a
detailed report that questioned the utility of the effort to
quantify R&D returns.\4 According to OTA, the fundamental
stumbling block to placing an economic value on federal R&D is
that improving productivity or producing an economic return is
not the primary justification for most federal R&D programs.
The report added that the attempts to measure the economic
return to federal R&D are flawed because many of the research
outputs, such as national defense, cannot be assigned an
economic value. The report also noted that in industry, where
one might expect quantitative techniques to prevail because of
the existence of a well-defined economic objective, OTA found a
reliance on subjective judgment and good communications between
R&D, management, and marketing staffs.
In 1996, responding to concerns about the implementation of GPRA,
the National Science and Technology Council issued a report that
stressed the limited role of quantification in measuring the
results of R&D.\5 It stated that the insufficiency of
quantitative measures per se is one reason why other sources of
evidence, such as merit review of past performance, narrative
discussion, and descriptions of outstanding accomplishments and
more typical levels of achievement, should be included in annual
performance reports. The report concluded that the cornerstone
of world-class science will continue to be merit review with
peer evaluation, while the science community works to develop
the measurement tools and other methods needed to assess the
contributions of fundamental science.
At the international level, the Organization of Economic
Cooperation and Development (OECD) is also grappling with these
questions. Beginning with the first edition of the Standard
Practice for Surveys of R&D (the "Frascati Manual") in the
1960s, OECD has been developing international frameworks for the
measurement of R&D inputs. In connection with its more recent
effort to develop new output indicators (the "Oslo Manual"),
OECD stated that the new indicators and the underlying
statistics usually take two or more decades to reach general
acceptance and regular collection and publication. The
participants at a 1996 OECD conference extensively discussed the
organization's efforts to improve the quality of innovation
indicators. At this stage, these indicators appear to be most
useful in helping researchers study and describe the process of
innovation.
--------------------
\3 Assessing the "Output" of Federal Commercially Directed R&D
(GAO/PAD-79-69, Aug. 1979).
\4 Research Funding as an Investment: Can We Measure the Returns?
Office of Technology Assessment, 72 pp. (Apr. 1986).
\5 Assessing Fundamental Science, National Science and Technology
Council (July 1996).
THE GOVERNMENT PERFORMANCE AND
RESULTS ACT
---------------------------------------------------------- Chapter 1:3
In response to questions about the value and effectiveness of federal
programs, GPRA seeks to shift federal agencies' focus away from such
traditional concerns as staffing, activity levels, and tasks
completed toward a focus on program outcomes--that is, the real
difference the federal program makes in people's lives. In the GPRA
context, an "outcome measure" assesses the results of a program
activity compared to its intended purpose. An "output measure,"
according to GPRA, tabulates, calculates, or records the level of
activity or effort and can be expressed in a quantitative or
qualitative manner. The output indicators discussed in our report
could be considered either outcome measures or output measures,
depending on the context. For example, return-on-investment
calculations could be outcome measures in a business context since
gaining the maximum return on investment is the intended purpose of a
business. Patenting rates would be output measures in some
businesses because patents could serve as one measure of the level of
activity of a research unit.
The act recognizes how difficult it is to state the goals and measure
the results of some programs. While the law encourages the use of
objective measures of performance, it authorizes agencies--with the
approval of the Office of Management and Budget--to use alternative,
subjective measures of performance. Also, instead of having GPRA
take effect immediately after its passage in 1993, the Congress
allowed for a period of time for the government to learn how to
implement the act. As part of the learning process, the act called
for the use of pilot projects in performance measurement. Among the
approximately 70 agencies or parts of agencies that participated in
pilot projects, one addresses scientific research. Upon the full
implementation of the act, the executive branch agencies are required
to devise plans that are outcome-oriented. The act calls for the
agencies to develop three documents: a strategic plan, an annual
performance plan, and an annual performance report.
OBJECTIVES, SCOPE, AND
METHODOLOGY
---------------------------------------------------------- Chapter 1:4
In response to the Subcommittee on Technology's request, our
objective was to review various indicators that are used to measure
the results of R&D. Specifically, this report discusses the relative
strengths and limitations of the input and output indicators used by
the federal and private sectors to measure the results of R&D as well
as the claim that industry focuses on short-term profitability rather
than long-term R&D needs. This report also provides a historical
perspective on research spending. (See app. I.)
The impacts of innovation are widely studied in the public and
private sectors as well as in academia. Our work relied on a limited
number of experts in each of these sectors to provide us with the
prevailing understanding of and latest developments in the area. As
a result, our review does not provide an exhaustive examination of
R&D measures nor does it answer the question of how R&D should be
measured. It does, however, as agreed with your offices, consist of
information on the strengths and limitations in the use of these
indicators as well as anecdotal information based on interviews with
leading R&D companies. We also reviewed the relevant literature.
We interviewed a number of experts, including a former IBM Vice
President for Research and National Science Foundation (NSF)
Director, a former Executive Director of the Manufacturing Forum, a
former Associate Director of the White House Office of Science and
Technology Policy, the Director of the Special Projects Office for
the Army Research Laboratory, and the former Chief Financial Officer
for Apple Computer. We also conducted six teleconferences with
company officials who were typically at the director level. We
interviewed representatives of General Electric, Lucent Technologies
(formerly Bell Labs), Dow Chemical, Eastman-Kodak, IBM, and Microsoft
Corporation. These companies spent from $5 billion to $800 million
on R&D in 1995. We chose companies having among the largest total
R&D budgets, in terms of dollars spent, because we believed that
their experiences would be the most relevant to the needs of the
federal government.
Our work also covered the available data on R&D spending and output
indicators provided by the leading data-gathering organizations in
this field--the OECD, NSF, and the Industrial Research Institute
(IRI). Our collaboration with the OECD included participation in an
OECD conference on science and technology indicators. Throughout the
course of this work, we also interviewed the NSF staff responsible
for publishing the Science and Engineering Indicators.
We performed our audit work from June 1996 through February 1997 in
accordance with generally accepted government auditing standards.
R&D SPENDING DATA PROVIDE SOME
INFORMATION ABOUT INNOVATIVE
ACTIVITY BUT NOT ABOUT R&D RESULTS
============================================================ Chapter 2
R&D spending data provide an indication of how much research is being
performed but do not provide a measure of the impacts of that
spending. Spending data have been used for budgeting purposes;
however, they have also been used as an indicator of the level of
innovative activity within a nation or company. The use of R&D
spending data as an indicator has a number of advantages. For
example, it reduces the innovation process to a single figure for the
purposes of discussion. In addition, the data-gathering methods have
been refined over many years and are generally reliable over time.
However, the level of spending is not a reliable indicator of the
level of research results. For example, companies told us that they
are focusing more of their spending on short-term R&D projects than
on long-term projects, but the impacts of that change in emphasis are
unclear. The use of spending data is more appropriate for
discussions of R&D spending priorities than of the effectiveness and
impacts of R&D spending levels.
R&D SPENDING GIVES AN
INDICATION OF INNOVATIVE EFFORT
---------------------------------------------------------- Chapter 2:1
Traditionally, R&D expenditures have been taken to indicate the
"amount" of innovative activity that is occurring within a country or
a firm. One of the advantages of using expenditure data in this way
is that it simplifies the discussion of the complex process of
innovation to a single unit of measurement. Another advantage is
that the use of dollars as the unit of measurement enables direct
comparisons to be made. In addition, the gathering of spending data
has been refined over many years, increasing the data's reliability
and relevance to policy-making.
This straightforward rationale--the more R&D spending, the more
innovative activity--is the primary advantage to using expenditure
data in policy discussions. Its simplicity and close ties to the
linear model of innovation allow it to be readily understood by those
with little specialized knowledge, making it appealing in policy
discussions. These same simplifying characteristics may have led to
its use in other areas. In some contexts, countries and companies
are categorized according to their technological sophistication on
the basis of their R&D spending levels; little attention is given to
other factors.
Another advantage arises from the common use of "dollars" in the R&D
spending data. This usage enables the spending in different research
areas to be compared according to the same units. These
straightforward comparisons are useful in demonstrating the
priorities of the nation at large. For example, recent U.S. R&D
expenditure data show that a reduction in defense-related R&D was
somewhat counterbalanced by an increase in federal support for
civilian R&D programs, including those aimed at improving the
diagnosis and treatment of disease, cleaning up the environment, and
enhancing technological competitiveness and economic prosperity. In
addition, converting different foreign currencies to dollars allows
for international comparisons of research. After allowing for the
variation of inflation over time, funding data depict historical
patterns of real expenditures.
The accuracy and policy relevance of spending data are also important
advantages to their use as indicators. The typical source of these
data is the widely cited Science and Engineering Indicators document
published periodically by the National Science Board. The Science
Resources Studies Division of NSF has been gathering data to use in
Indicators for many years, and many improvements have been made in
the accuracy of the data.\6 For example, NSF has sponsored the Survey
of Industrial R&D since 1953 as a source of data for Indicators.
Recent improvements to this survey include selecting samples annually
(rather than less frequently) and increasing the sample size from
approximately 14,000 to nearly 24,000 firms. NSF took these steps to
account more accurately for the establishment of R&D-performing
entities in the survey universe and to survey more fully and
accurately the R&D performed by nonmanufacturing firms. In addition,
NSF is constantly searching for ways to make Indicators more
responsive to policymakers' needs. NSF began to make adjustments in
its surveys when it recognized that there was a need to supply more
information on the service sector to policymakers because the survey
historically had focused on the manufacturing industries in which R&D
performance had been heavily concentrated in the past.
--------------------
\6 Some problems with the data may still exist, however. For
example, NSF itself notes that the data on trends in the private
sector's basic research contain anomalous spending spikes in 1986 and
1991. The inconsistencies in the data appear to derive from changes
made in the NSF survey in those years. For example, in 1991 the
survey was expanded to cover a broader array of nonmanufacturing
firms.
R&D SPENDING IS NOT A GOOD
INDICATOR OF R&D RESULTS
---------------------------------------------------------- Chapter 2:2
The use of spending data is limited in its relevance to the impacts
of R&D. There is some correlation between the level of R&D spending
and innovative success. For example, if fewer research projects are
performed, then companies and countries forgo the potential benefits
of the research. However, spending alone does not guarantee
innovative success because many additional factors figure into the
innovation process and have important effects on the resulting
outputs. The reality of the process of innovation is much more
complex than expenditure data alone can reveal.
The usefulness of R&D spending data as an indicator of results is
limited because the data measure the amount of resources a firm or a
nation dedicates to innovation, but not its ability to convert that
effort into successful products, processes, and services. Accounts
in the press have noted that companies are proud of high R&D spending
levels because well-focused R&D generally pays off in long-term
revenue growth and profits. However, more is not always better, as
shown by the companies that ranked among the leaders for R&D spending
and disappeared shortly thereafter. For example, two companies
called Xonics and Transitron Electronic ranked as the top companies
in R&D spending per employee for 1984 and 1986, respectively; in the
same years that they achieved their top ranking, one company was
forced to file for Chapter 11 protection and the other company
dissolved.
There is great uncertainty in research and development investments.
The processes leading to commercially viable and socially useful
technologies are complex and involve substantial non-R&D factors.
One company official told us that it was impossible to determine
which business function was most important to the success of a new
product--research, marketing, or sales. While there is some
correlation between the level of R&D spending and innovative success,
spending alone does not guarantee success. One official from a
high-technology company told us that his company is constantly
evaluating research projects with respect to their targeted markets
and estimating the expected return on investment. However, in one
case the company lost approximately $100 million when the market
would not support a newly developed product at the price the company
expected.
The usefulness of R&D spending indicators is also limited because the
way in which innovative activities are structured and managed can be
as significant as the amounts of resources devoted to them in
determining their outcomes and effects on performance. Those nations
or firms with extremely efficient innovation systems can outperform
those that use greater R&D resources inefficiently. As a former
director of NSF pointed out to us, streamlining research efforts
could reduce both bureaucratic and financial overhead costs and make
up for spending reductions.
In addition, falling spending levels may hide the greater
efficiencies that might be realized by leveraging research. One
company official told us that although his company was cutting back
on long-term research, it was relying more on its relationships with
federal and university laboratories for similar work. Collaboration
between firms--through joint ventures, consortia, and
contracting--has recently been on the rise as firms attempt to
efficiently distribute risk, pool their resources, and tap into
external expertise. The magnitude of alliance formation is difficult
to gauge, as are the implications for the innovation and
commercialization of new technologies in the United States. However,
one academic told us that if the rhetoric about the efficiencies
coming from cooperative R&D is true, then no one should be too
disappointed with recent drops in R&D spending.
SPENDING PATTERNS SHOW A
GREATER EMPHASIS ON SHORT-TERM
RESEARCH
---------------------------------------------------------- Chapter 2:3
Federal and private R&D spending patterns reflect the changing
conditions in the country and the world at large. The share of basic
research supported by private sources peaked in the early post-World
War II period, driven by thriving domestic and international markets.
However, recent surveys and anecdotal evidence suggest that increased
international competitive pressures have forced companies to
emphasize short-term development over long-term research. Company
officials echoed this shift and suggested that the focus on
short-term research was part of a push toward more relevancy in their
R&D departments. They pointed out that their business units are
determining their research needs to a greater extent today than they
have in the past.
Some companies told us that this strategy could be viewed as
short-sighted. Others defended this strategy. For example, one
company official told us that his company emphasizes short-term
research because "competition is short-term." He stated that in
today's competitive environment, once a market is lost, it is gone
forever.
R&D OUTPUT INDICATORS CAN PROVIDE
LIMITED INFORMATION ABOUT R&D
RESULTS
============================================================ Chapter 3
Quantitative and qualitative indicators have been developed to
evaluate R&D activities and their results, but both types of
indicators have strengths and limitations. Our interviews with a
number of companies showed that the private sector stresses
marketplace results rather than relying on output indicators.
Because of the companies' profit orientation, many of the indicators
tracked by the private sector cannot be directly applied to the
federal government. In response to the GPRA, the federal science
agencies are exploring new ways to quantify the impacts of research.
However, it is too early to tell whether new performance measures can
be developed and whether they will meet the needs of the Congress.
CURRENT INDICATORS HAVE
STRENGTHS AND LIMITATIONS
---------------------------------------------------------- Chapter 3:1
Because of the difficulties in identifying the impacts of research,
decisionmakers in the public and private sectors typically have
chosen to measure outcomes using a variety of proxies. These
quantitative and qualitative indicators have strengths and
limitations. To illustrate these strengths and limitations, we
looked at three of the most frequently cited quantitative indicators:
return on investment, patents issued, and bibliometrics. While these
indicators imply a degree of precision, they were generally not
designed to measure the long-term results of R&D programs, nor are
they easily adaptable to such a purpose. Qualitative assessment
provides detailed, descriptive information, but it depends on
subjective judgments and may be costly.
RETURN ON INVESTMENT
-------------------------------------------------------- Chapter 3:1.1
This indicator aims at measuring the sales and profits resulting from
investments in R&D; as such, it addresses one of the fundamental
concerns about the value of such investments. However, a variety of
factors, such as the complexity of the innovation process and its
inherently long time frames, pose serious obstacles to the
calculation of these returns. The literature dealing with return on
investment is replete with words of caution against quantifying R&D
results. NSF's Science and Engineering Indicators 1996 pointed out
that not only is much of this information unobtainable or ambiguous,
but many of the gains from research are simply monetarily intangible.
Experts on the R&D process have stated frequently that the long time
periods and multiple inputs involved make the task of calculating the
return on basic research especially difficult. Productivity growth
may lag 20 years behind the first appearance of research in the
scientific community, and the lag for interindustry effects may be 30
years. A more serious impediment, however, is the fact that outcomes
are often not directly traceable to specific inputs or may result
from a combination of such inputs. The National Aeronautics and
Space Administration (NASA) and the National Bureau of Standards (now
the National Institute for Standards and Technology) attempted to
measure the economic impacts and benefits of certain of their
technologies in the 1970s and early 1980s. The studies at the Bureau
were discontinued, according to staff, because of serious theoretical
and methodological problems. As with the Bureau's studies, NASA's
studies evoked serious criticisms and were likewise discontinued.
Despite the difficulties in calculating return on investment, the
leading researchers in this field agree that R&D offers high private
and social returns in terms of high productivity. One recent survey
of 63 studies found that R&D activity achieves, on average, a 20- to
30-percent annual return on private (industrial) investments.\7
--------------------
\7 Nadiri, M.I. Innovations and Technological Spillovers (1993).
PATENTS
-------------------------------------------------------- Chapter 3:1.2
Patents show certain strengths as useful indicators in measuring
technical change and inventive input and output over time. For
example, they can reveal a variety of trends involving ownership and
levels of activity in technical areas. According to NSF, the data
concerning ownership show that the federal share of patents averaged
3.5 percent of the total number of U.S. patents during 1963 through
1980 but declined thereafter.
In addition, the data concerning a country's distribution of patents
by technical area have proved to be a reliable indicator of a
nation's technological strengths as well as its direction in product
development. For example, the three most emphasized U.S. patent
categories for inventors show specific contrasts between U.S. and
foreign patents. U.S. inventors obtained most of their patents in
wells, mineral oils, and certain areas of surgery. Japanese
inventors focused their efforts on certain areas of communications,
organic compounds, and refrigeration. Patent activity can be used to
pinpoint potentially important changes. In 1980 through 1987, U.S.
inventors led all other foreign inventors in radio and television
patents, but in 1987 the United States lost its front position to
Japanese inventors in this area.
Despite their usefulness as indicators of broad national and
international trends in various industries and areas of research,
patents also have several intrinsic drawbacks. Inconsistency across
industries in the number of patents granted is a major limitation
that results from the wide variations in the propensity to patent
inventions. Consequently, according to NSF, it is not advisable to
compare patenting rates between different technologies or industries.
Inconsistency in quality is a second drawback. The aggregated patent
statistics do not distinguish between those that led to major
innovations and those that led to minor improvements. Incompleteness
is a further limitation. Many inventions are not patented at all,
and trade secrecy may become a preferred alternative to patenting.
Another limitation is that patents do not lend themselves to the
evaluation of the most significant results achieved by an R&D
program. They can provide intermediate measures of progress, but
they are not usually the purpose for which the research was
undertaken.
In addition, the use of patents as a measure of federal R&D
effectiveness may be hampered by their limited relevance. The 1996
report entitled Assessing Fundamental Science by the National Science
and Technology Council noted that any use of patent counts should be
undertaken only with a full awareness of their limitations. A recent
academic study of patents and the evaluation of R&D also commented on
the extraordinarily limited applicability of patent evaluation to
government-performed R&D. Both studies pointed to the relatively low
level of federal patenting activity. The academic study noted that
most government laboratories are granted only one or two patents per
year, and only a few of them patent extensively. It concluded that
for government laboratories, one may question the overall wisdom of
evaluating public R&D with private techniques.\8
--------------------
\8 In its October 11, 1996 issue, Science magazine noted the French
government's announcement that patent records will form a part of the
evaluation of publicly funded researchers. The new proposal,
according to the magazine, is controversial because it might upset
the proper balance between basic and applied research by favoring
those who work in fields that are immediately applicable at the
expense of people doing basic research.
BIBLIOMETRICS
-------------------------------------------------------- Chapter 3:1.3
A third area of effort in developing quantitative measurements
involves bibliometrics, or the study of published data.
Bibliometrics counts citations in an attempt to address questions of
productivity, influence, and the transfer of knowledge. Its most
appropriate use is in quantifying the work of researchers whose
results are published. Thus, it may be especially applicable in
areas such as basic research where the results are more often
published than protected by firms. However, its usefulness as a
measure of research results remains somewhat controversial.
We believe that the use of bibliometrics as a source of information
on the quality of the publications or the citations being counted
needs to be approached with caution. Although bibliometric
indicators can be weighted by publication or other quality measure,
the frequency of citation, for example, provides no indication of the
level of research innovation. Another limitation is the problems
that arise in interdisciplinary comparisons of results. Some critics
have gone so far as to say that bibliometric findings should not be
used in science policy work until the problems with the analysis of
citations are addressed.
In addition, the relevance of bibliometric analysis to
decision-making by the federal government appears very limited. One
expert noted that a recent comprehensive review of bibliometrics
shows the sparsity of bibliometric studies for evaluations of the
impact of research reported by the federal government. Another
pointed out that few federal agencies use bibliometric analysis as an
evaluative tool. One of the few is the National Institutes of Health
(NIH), which uses this method to evaluate the effectiveness of its
different institutes, the comparative productivity of NIH-sponsored
research and similar international programs, and the responsiveness
of NIH's research programs to their congressional mandate.
PEER REVIEW
-------------------------------------------------------- Chapter 3:1.4
Recognizing the limitations of quantitative indicators, the National
Science and Technology Council concluded that it makes sense to track
relevant measures but that they cannot supplant the essential element
of expert judgment. Peer review, the most important form of
qualitative assessment, uses technical experts to judge R&D results
on the basis of the expert's evaluation of the quality of research.
However, peer review has serious shortcomings; it generally depends
on criteria that are inherently difficult to measure and on
subjective judgment that is vulnerable to bias.
Peer review has been used extensively in the selection of proposed
research projects. To a lesser extent, it has also been used to
evaluate R&D impacts. Peer review has come to be viewed by some
observers as the best assurance that quality criteria will prevail
over social, economic, and political considerations, while others
view it as an element of elitism in science that tends to discount
such concerns as economic significance.
Its major strength is its ability to bring together the leading
experts in the area of concern. Most peer review procedures require
a minimum of three reviewers; if the review involves a more ambitious
scope of coverage (such as an entire agency), dozens of reviewers may
be involved. The process of selecting the peer reviewers varies.
One of the chief responsibilities of the professional staff in
science agencies such as NSF and NIH is to stay in touch with a
specialized community of scientists who are qualified to judge the
agency's activities. However, others rely on in-house managers who
are not active researchers.
The major limitations of peer review are twofold. First, the
perception of quality depends largely on the expertise of the panel
members. It is based on the judgment of experts about a proposal or
a set of research-related results. Generally, a final judgment will
depend on the collective weight of the different opinions.
Frequently, a numerical rating scale--such as 1 for poor through 5
for excellent--is used. Despite the appearance of precision
conferred by a specific number, the numbers represent the best of
sometimes widely differing judgments. Consequently, although peer
review has been a mainstay in judging science for over three
centuries, questions remain about ways of improving it. For
instance, according to one academic, to improve "validity and
reliability," research needs to be done on the optimal numbers of
reviewers and on the advisability of training people to perform peer
reviews.
Second, peer review evaluation of completed or ongoing R&D projects
is a more thorough and expensive process than peer review for the
purpose of selecting proposals for funding. According to one study,
the cost of a 2-day, 10-person, face-to-face NSF merit review panel
is in the neighborhood of $20,000. Another study concluded that if
this method were applied annually to all federal research programs,
the cost in reviewer time alone would be enormous. For example, the
Army Research Laboratory has contracted for a peer review of its
activities; the contract calls for a 3-year review directed by the
National Research Council at approximately $650,000 per year.
THE PRIVATE SECTOR'S EMPHASIS
ON MARKETPLACE RESULTS LIMITS
LESSONS FOR THE FEDERAL
GOVERNMENT
---------------------------------------------------------- Chapter 3:2
The private-sector companies we interviewed varied in terms of the
types of quantitative and qualitative R&D indicators that they
collect, but in general they made limited use of these indicators in
their decisions. Many companies stressed the difficulties involved
in measuring the contribution of R&D to the firm's overall
performance using return on investment, patents issued, and other R&D
output measures. All of the firms mentioned that they were
increasing R&D that contributes directly to the bottom line of the
firm. Thus, they shifted the responsibility for R&D decisions to the
business-unit level so that the R&D would be tied more directly to
the profits of those units. The private sector's experience offers
general lessons to the federal government in terms of ensuring that
the R&D contributes directly to the mission of the organization,
although the specific output measures do not apply directly to
federal R&D efforts.
COMPANIES CITED DIFFICULTIES
IN MEASURING THE RESULTS OF
R&D
-------------------------------------------------------- Chapter 3:2.1
Companies told us that measuring the return on R&D investment is very
difficult. Companies stated that one factor making measurement more
difficult is the long time lag between the research and any revenue
that might be earned. Companies also stated that because so many
people have been involved in a product's evolution, it is difficult
to separate the contribution of the research unit from that of other
units.
Companies also mentioned difficulties with some of the other
indicators that they track. For example, one indicator was
alternatively labeled a "vitality index" or "product innovation
metric," which reflected the share of the firm's products that could
be considered new products. This measure provided an indication of
how rapidly the company was incorporating new ideas and research into
its products. Several company officials mentioned that this measure
had to be applied carefully because of the problem of defining a
"new" product; some products are completely different from their
predecessors, while others might incorporate cosmetic changes.
FIRMS INCREASED THEIR
ATTENTION TO R&D'S
CONTRIBUTION TO THE BOTTOM
LINE
-------------------------------------------------------- Chapter 3:2.2
One issue that was mentioned by all of the firms in our discussions
was the increasing emphasis on the relevance of R&D to business needs
and the "bottom line." These comments came up in a number of
contexts. For example, some firms mentioned that they look for
research with a shorter-term payoff in order to have a greater impact
on getting new products into the marketplace. Other firms cited the
increased emphasis on applied R&D, or the small amount of research
that they perform that could be called basic research. A common
element was that these firms were attempting to reduce what might be
called benevolent research: projects that benefit the industry or
the nation but do not have a payoff to the firm.
However, a number of the firms noted that this emphasis on the bottom
line does not necessarily create the opportunities for breakthrough
products. All of the firms either reserved a certain fraction of
their research funding for these types of projects or developed
processes that made it possible to continue some level of research in
areas that might not be directly aligned with any particular product
line. In this context, one firm mentioned the importance of
cooperation with the federal government and universities in the
pursuit of fundamental research.
FIRMS HAVE REDUCED EMPHASIS
ON R&D OUTPUT MEASURES
-------------------------------------------------------- Chapter 3:2.3
While this increased emphasis on getting the most out of R&D might be
expected to lead to greater efforts to measure the results of
research, most of the firms that we spoke to responded by changing
the organization so that measuring R&D outputs was no longer so
important. By shifting the responsibility for research decisions to
the business units in the firm that make use of the research outputs,
the companies have sidestepped the need for centralized indicators of
the quality of research. If the business units believe that a
particular R&D project would increase their profits, the firm would
budget for that R&D. If the business units do not perform up to
expectations in terms of their profitability, the entire unit would
be responsible.
For example, one firm shifted from a policy of centrally directed
research to a policy in which the individual business units make the
decisions on the appropriate research projects. Under the previous
arrangement, the various business units were assessed to support the
central R&D efforts; under the new arrangement, the units pay only
for those projects that they think are valuable. This shift was
designed to make the research more responsive to the needs of the
business units, in that these units do not pay for the research
unless they find it useful. This shift also greatly reduced the
emphasis on developing R&D output indicators in the central
laboratory because each of the business units would be reviewed on
the basis of its profitability. The reasoning is that if these units
were making poor decisions on R&D projects, the unit's overall
profitability would decrease.
PRIVATE SECTOR'S MEASURES DO
NOT DIRECTLY APPLY TO THE
FEDERAL GOVERNMENT
-------------------------------------------------------- Chapter 3:2.4
Our interviews with private-sector firms suggest that many of the R&D
output measures tracked by the private sector do not apply directly
to the federal government. Many of these measures are directly
related to the contribution of the R&D to the bottom line
profitability of the firm. However, federal agencies do not operate
in order to make a profit, but to accomplish a variety of other
missions and goals. For example, agencies conduct R&D to support a
variety of missions, such as maintaining national security or
improving citizens' health. R&D in these areas can contribute
greatly to the quality of life in the United States, even if it has a
negative return on investment.\9 Given these very different missions,
there is also no reason to believe that any single measure is
appropriate for different public-sector agencies.
Despite the lack of specific measures that can be translated from the
private to the public sector, there are general lessons to be learned
from the private sector's experience. Possibly the most important is
the recognition that as the pressures on costs increased at many of
the firms that we interviewed, the firms made significant efforts to
ensure that R&D contributed directly to the bottom line. The federal
government likewise faces increasing pressure on costs such as R&D
expenditures but, unlike the private sector, cannot rely on the
marketplace to ensure that the R&D contributes to the agency's goals.
Federal R&D is undertaken to support a variety of agency missions,
and producing an economic return is not the primary justification for
most federal R&D programs. In fact, the purpose of federal R&D is to
promote research that is socially beneficial but may not be
profitable for the private sector to pursue. Without this
competitive marketplace to ensure the relevance of R&D, the federal
agencies will continue to be challenged to develop better measures of
the outputs of their R&D.
The literature confirms this general finding of our discussions. For
example, in 1995 the National Research Council reported on the
results of a workshop on what the federal government might learn from
the corporate experience in research restructuring and assessment.\10
The Council invited senior corporate research managers from IBM,
AT&T, Ford, and Xerox to discuss their experiences in this area. The
report concluded that developing useful metrics and using them
appropriately is a difficult problem, but it is not impossible. In
addition, the participants were not successful in translating their
private-sector experience into specific lessons about what can be
measured and what makes sense to measure for the federal government.
--------------------
\9 The return-on-investment measure that is appropriate for the
federal government would compare the total "social benefits" to the
nation with the costs of the initial investment. Computing this
measure has many of the difficulties of computing the return on
investment for private firms, with the added complications associated
with placing a value on national security, improved quality of life,
and other intangible qualities.
\10 Research Restructuring and Assessment: Can We Apply the
Corporate Experience to Government Agencies?, 72 pp., National
Research Council (1995).
FEDERAL SCIENCE AGENCIES ARE
STILL EXPLORING WAYS TO MEASURE
THE IMPACTS OF R&D
---------------------------------------------------------- Chapter 3:3
Our July 1996 testimony on the implementation of GPRA noted that the
Congress recognized that successful implementation will not come
quickly or easily for many agencies.\11 To help address the
challenges of "measuring" the results of R&D programs, the Research
Roundtable, a group of federal researchers and managers representing
a cross-section of departments and agencies, has met periodically to
share ideas and approaches for implementing GPRA. The Army Research
Laboratory has also begun to address this issue in a pilot project
for performance measurement under GPRA.
The Research Roundtable has considered the extent to which R&D
agencies can and should adopt a common approach to measuring
performance. In 1995, it issued a paper based on 6 months of
discussions on the development of performance measures for research
programs. Although the Roundtable stated that the results of a
research program's performance can be measured, it cautioned that at
the same time, it is important to recognize the complexity of the
cause-effect relationship between R&D and its results. It added that
this complexity makes it difficult to establish quantifiable measures
that consistently measure program performance. It also noted that
such measures create a potential for incorrect application, which
could lead subsequently to a detrimental effect on scientific
endeavors. It warned that quantitative empirical demonstrations of
such cause-effect relationships should not be required and are often
not even possible.
The Army Research Laboratory was designated a pilot project for
performance measurement under GPRA. Of the more than 70 such pilot
projects governmentwide, the laboratory was the only pilot project
that addresses scientific research. As such, it attempted to break
new ground in both the planning and the evaluation of basic and
applied research. The Chief of the Army Research Laboratory's
Special Projects Office, who is mainly responsible for designing the
laboratory's approach to implementing GPRA, submitted a case study to
the Office of Management and Budget in 1996.\12 The case study
outlines an approach that makes use of "three pillars:" metrics, peer
review, and customer feedback.
In the case study, the laboratory identified about 60 metrics, most
of which measure input using fiscal, facilities, and personnel data.
Some of the metrics, such as tasks completed, patents awarded, and
articles published, measure output; none measure outcome. The
laboratory views the measures as useful tools for understanding the
functional health of the organization and the management of the
laboratory but cautions that the information will not enable it to
determine the real quality and impact of its R&D. The laboratory is
relying more heavily on peer review of its research and on customer
surveys for information about quality.
According to the case study, the laboratory has contracted with the
National Research Council of the National Academies of Science and
Engineering to conduct a retrospective peer review of research over a
2- or 3-year period. The Council is to assemble a Technical
Assessment Board that consists of six panels with 8 to 10 people,
each of whom is of high repute within the technical community. These
panels will appraise the quality of the laboratory's technical and
scientific efforts and, to a limited extent, productivity.
The case study also described the use of customer feedback. The
laboratory has identified a number of internal and external
customers. For those customers to whom it delivers specific items,
it uses a series of feedback questionnaires to determine their degree
of satisfaction on a 1 to 5 scale in terms of the quality,
timeliness, and utility of the deliverable. For those customers who
do not receive a specific, identifiable product, the laboratory is
developing a Stakeholders' Advisory Board of senior leadership and
user representatives to provide first-hand guidance and feedback.
--------------------
\11 Managing for Results: Key Steps and Challenges in Implementing
GPRA in Science Agencies (GAO/T-GGD/RCED-96-214, July 10, 1996).
\12 Brown, Edward A. "Applying the Principles of the Government
Performance and Results Act to the Research and Development Function"
(1996).
HISTORICAL PERSPECTIVE ON RESEARCH
SPENDING
=========================================================== Appendix I
Since World War II, U.S. public and private research and development
(R&D) spending patterns have reflected changing priorities as well as
reactions to the changing national and international economies. Some
of the more prominent events that have shaped spending in both
sectors have been the Cold War and the recent international
competitive pressures. Declines in spending in both sectors have
been less frequent than increases.
FEDERAL SPENDING
--------------------------------------------------------- Appendix I:1
In his July 1945 report, Science�The Endless Frontier, Vannevar Bush,
who headed the U.S. R&D effort during World War II, provided the
rationale for federal support of both basic research and research
related to national security, industry, and human health and welfare.
His plan contributed to the legislation adopted in 1950 that
established the National Science Foundation (NSF). By that time,
however, the National Institutes of Health (NIH) already had control
over most health-related research; the Office of Naval Research had
taken on a major role in supporting academic research in the physical
sciences; and the new Atomic Energy Commission had been assigned
control of the R&D on nuclear weapons and nuclear power. NSF's
mission focused on supporting fundamental research and related
educational activities. Its annual budget was less than $10 million
until the late 1950s. In contrast, NIH's annual budget, which had
been less than $3 million at the end of the war, grew to more than
$50 million by 1950.
The scope of federal R&D support grew in the decade after World War
II. Anxiety over the Cold War and the loss in 1949 of the U.S.
monopoly in nuclear weapons led to expanded R&D programs in the Army
and in the newly established Air Force and to a continuing buildup in
support for nuclear weapons R&D in the Atomic Energy Commission. On
the civilian side, R&D programs were established or expanded in
fields with direct practical importance, such as aeronautics
technology, water desalinization, and atmospheric disturbances and
weather. The launch of Sputnik by the Soviet Union in 1957 led to
immediate efforts to expand U.S. R&D, science and engineering
education, and technology deployment. Within months of the Sputnik
launch, the National Aeronautics and Space Administration (NASA) and
the Advanced Research Projects Agency (ARPA) in the Department of
Defense (DOD) were established. NASA's core included the aeronautics
programs of the National Advisory Committee on Aeronautics and some
of the space activities of DOD; ARPA's purpose was to enable DOD to
conduct advanced R&D to meet military needs.
Federal appropriations for R&D and for mathematics and science
education in NSF and other government agencies rose rapidly over the
next decade, often at double-digit rates in real terms. (See fig.
I.1.) During the early 1950s, the growth in federal funding for
health research slowed considerably from its fast pace in the
immediate postwar years. However, in the late 1950s federal support
for health research accelerated rapidly.
From 1966 to 1975, federal support for nondefense R&D dropped nearly
22 percent in real terms. The costs of the Vietnam War put downward
pressure on nondefense R&D along with other nondefense discretionary
spending. The conclusion of NASA's Apollo program contributed to the
decline in federal R&D funding during that period, also. Until
recently, this was the only period in which federal funds for R&D
were reduced substantially.
Figure I.1: U.S. R&D
Spending, 1953-96
(See figure in printed
edition.)
Source: National Science Foundation, Science Resources Studies
Division.
In the 1970s, new R&D-intensive agencies addressed environmental and
energy issues. Both the environmental movement and the energy crisis
of the 1970s, according to some analysts, led to increased public and
private spending on environmental and energy R&D. The Environmental
Protection Agency was established in 1970. In 1977, the Energy
Research and Development Administration and other federal
energy-related activities were combined to form the Department of
Energy, which was given major new responsibilities to fund
energy-related R&D.
In the 1980s, the federal role in R&D expanded to enable the United
States to compete in a global market. Out of this atmosphere,
several programs were initiated to provide financial and other
incentives for industrial R&D and for industrially related R&D
conducted at universities or federal laboratories.
Two major factors have influenced federal support for R&D funding in
the 1990s. These factors have been the efforts to reduce the budget
deficit and the defense drawdown. The Department of Defense and the
Department of Energy, two of the four largest sources of federal R&D
support, have had constant-dollar reductions in R&D obligations
during the 1990s. In fiscal year 1995, Defense accounted for roughly
half of all federal R&D obligations, down from nearly two-thirds of
the total in 1986 at the height of the defense build-up that occurred
during the Reagan administration.
At the same time, military-related R&D spending was being curtailed,
while federal investment in selected civilian R&D activities
increased, including support for research aimed at improving health
and the environment and for technology advancement. The Department
of Health and Human Services, which is a distant second to Defense in
terms of total R&D support, had the largest absolute increase--$3
billion--in federal R&D obligations during the 1990s. The proportion
of all U.S. R&D devoted to health-related projects has been
increasing continuously for nearly a decade. The Commerce Department
has registered the largest percentage increase in federal R&D
obligations during the 1990s so far. In addition, the federal
government, which supplies about three-fifths of all funds used to
perform R&D on college and university campuses, has been increasing
its support of academic research continuously since 1982, even after
adjusting for inflation.\13
--------------------
\13 Much of this section is based on Allocating Federal Funds for
Science and Technology by the Committee on Criteria for Federal
Support of Research and Development of the National Academy of
Sciences, National Academy Press (1995).
PRIVATE-SECTOR FUNDING
--------------------------------------------------------- Appendix I:2
During the early post-World War II period, thriving domestic and
international markets supported large profits and the rapid expansion
of R&D in both the central laboratories and the divisional
laboratories of large companies. Central R&D facilities focused
increasingly on fundamental research in many of these large firms,
leaving the development and application of new technologies, as well
as the improvement of established products and processes, to the
divisional laboratories. The data on basic research for 1953 through
1960 are less reliable than those for later years but suggest that
the share of total U.S. basic research financed by industry during
the postwar period may well have been at its peak during the 1950s
and early 1960s.
Severe competitive pressures from foreign firms, increases in the
real cost of capital, and a slowdown in the growth rate of the
domestic economy in the 1970s have been cited as causes for the
apparent decline in the returns to R&D investment during the
mid-1970s, and the rate of growth in real industry expenditures on
R&D declined. Industry's funding of basic research shrank, and many
of the central research facilities of the giant corporations entered
a period of budgetary austerity or cutbacks. After a resurgence in
the early 1980s, the rate of growth in industry-funded R&D declined
in the late 1980s.
Industry's funding of R&D was generally flat between 1991 and 1994;
several reasons have been cited for the lack of growth in some
companies' R&D programs during this time. These include corporate
downsizing, decentralization (i.e., a shift of R&D activity from
corporate laboratories to individual business units), and increasing
collaboration among industrial firms and with partners in academia,
in government, in the nonprofit sector, and in other countries. The
preliminary data for 1995 and 1996 indicate a slight upswing in
industrial R&D funding.
According to NSF, the most striking recent trend in industrial R&D
performance has been the increase in the proportion of total R&D
funded by the companies classified as nonmanufacturing industries.
Prior to 1983, nonmanufacturing industries accounted for less than 5
percent of the industry's total R&D. That share grew steadily during
the next decade so that in 1993, nonmanufacturing firms represented
more than 25 percent of all industrial R&D performed in the United
States. Part of this reported growth, however, reflects improvements
in the NSF's survey efforts to measure R&D in nonmanufacturing
industries, in addition to actual growth in service sector R&D
spending.
Between 1984 and 1994, some significant changes occurred among the
100 largest publicly held R&D funding companies, although the four
leading firms were the same in both 1984 and 1994. During the
decade, the number of pharmaceutical and computer hardware and
software companies among the largest R&D funders rose. In contrast,
the number of large defense contractors and chemical and petroleum
companies among the largest R&D funders fell.
U.S. firms have begun to alter their R&D patterns in response to
increasing competitive pressures. Firms have shifted a greater
portion of their R&D resources away from long-term investments and
toward shorter-term projects. U.S. companies now allocate 22
percent of their R&D spending to long-term projects when compared
with their Japanese counterparts, who devote 50 percent.
Increasingly, firms are emphasizing short-term R&D for immediate
problem-solving or near-term development over basic research; and
basic research is being directed toward the needs of product
development and manufacturing teams. Many central research
laboratories at large companies--such as AT&T, IBM, General Electric,
Kodak, and Xerox--have been downsized and work more closely with
product development divisions. They now receive a larger share of
their operating funds from individual business units rather than
general corporate funds.
MAJOR CONTRIBUTORS TO THIS REPORT
========================================================== Appendix II
RESOURCES, COMMUNITY, AND ECONOMIC
DEVELOPMENT DIVISION, WASHINGTON,
D.C.
Victor S. Rezendes, Director
Robin M. Nazzaro, Assistant Director
Andrew J. Vogelsang, Evaluator-in-Charge
Dennis Carroll, Senior Evaluator
OFFICE OF THE CHIEF ECONOMIST
Loren Yager, Assistant Director
*** End of document. ***