[House Hearing, 112 Congress]
[From the U.S. Government Publishing Office]
THE MERIT REVIEW PROCESS:
ENSURING LIMITED FEDERAL RESOURCES ARE INVESTED IN THE BEST SCIENCE
=======================================================================
HEARING
BEFORE THE
SUBCOMMITTEE ON RESEARCH AND SCIENCE EDUCATION
COMMITTEE ON SCIENCE, SPACE, AND TECHNOLOGY
HOUSE OF REPRESENTATIVES
ONE HUNDRED TWELFTH CONGRESS
FIRST SESSION
__________
TUESDAY, JULY 26, 2011
__________
Serial No. 112-31
__________
Printed for the use of the Committee on Science, Space, and Technology
Available via the World Wide Web: http://science.house.gov
_____
U.S. GOVERNMENT PRINTING OFFICE
67-522 PDF WASHINGTON : 2011
-----------------------------------------------------------------------
For sale by the Superintendent of Documents, U.S. Government Printing
Office Internet: bookstore.gpo.gov Phone: toll free (866) 512-1800; DC
area (202) 512-1800 Fax: (202) 512-2104 Mail: Stop IDCC, Washington, DC
20402-0001
COMMITTEE ON SCIENCE, SPACE, AND TECHNOLOGY
HON. RALPH M. HALL, Texas, Chair
F. JAMES SENSENBRENNER, JR., EDDIE BERNICE JOHNSON, Texas
Wisconsin JERRY F. COSTELLO, Illinois
LAMAR S. SMITH, Texas LYNN C. WOOLSEY, California
DANA ROHRABACHER, California ZOE LOFGREN, California
ROSCOE G. BARTLETT, Maryland DAVID WU, Oregon
FRANK D. LUCAS, Oklahoma BRAD MILLER, North Carolina
JUDY BIGGERT, Illinois DANIEL LIPINSKI, Illinois
W. TODD AKIN, Missouri GABRIELLE GIFFORDS, Arizona
RANDY NEUGEBAUER, Texas DONNA F. EDWARDS, Maryland
MICHAEL T. McCAUL, Texas MARCIA L. FUDGE, Ohio
PAUL C. BROUN, Georgia BEN R. LUJAN, New Mexico
SANDY ADAMS, Florida PAUL D. TONKO, New York
BENJAMIN QUAYLE, Arizona JERRY McNERNEY, California
CHARLES J. ``CHUCK'' FLEISCHMANN, JOHN P. SARBANES, Maryland
Tennessee TERRI A. SEWELL, Alabama
E. SCOTT RIGELL, Virginia FREDERICA S. WILSON, Florida
STEVEN M. PALAZZO, Mississippi HANSEN CLARKE, Michigan
MO BROOKS, Alabama
ANDY HARRIS, Maryland
RANDY HULTGREN, Illinois
CHIP CRAVAACK, Minnesota
LARRY BUCSHON, Indiana
DAN BENISHEK, Michigan
VACANCY
------
Subcommittee on Research and Science Education
HON. MO BROOKS, Alabama, Chair
ROSCOE G. BARTLETT, Maryland DANIEL LIPINSKI, Illinois
BENJAMIN QUAYLE, Arizona HANSEN CLARKE, Michigan
STEVEN M. PALAZZO, Mississippi PAUL D. TONKO, New York
ANDY HARRIS, Maryland JOHN P. SARBANES, Maryland
RANDY HULTGREN, Illinois TERRI A. SEWELL, Alabama
LARRY BUCSHON, Indiana
DAN BENISHEK, Michigan
RALPH M. HALL, Texas EDDIE BERNICE JOHNSON, Texas
C O N T E N T S
Tuesday, July 26, 2011
Page
Witness List..................................................... 2
Hearing Charter.................................................. 3
Opening Statements
Statement by Representative Mo Brooks, Chairman, Subcommittee on
Research and Science Education, Committee on Science, Space,
and Technology, U.S. House of Representatives.................. 10
Written Statement............................................ 10
Statement by Representative Daniel Lipinski, Ranking Minority
Member, Subcommittee on Research and Science Education,
Committee on Science, Space, and Technology, U.S. House of
Representatives................................................ 11
Written Statement............................................ 12
Witnesses:
Dr. Cora Marrett, Deputy Director, National Science Foundation
Oral Statement............................................... 14
Written Statement............................................ 15
Dr. Keith R. Yamamoto, Vice Chancellor for Research, University
of California San Francisco
Oral Statement............................................... 21
Written Statement............................................ 23
Dr. Nancy B. Jackson, President, American Chemical Society
Oral Statement............................................... 25
Written Statement............................................ 27
Dr. Jorge Jose, Vice President for Research, Indiana University
Oral Statement............................................... 30
Written Statement............................................ 32
Discussion....................................................... 36
Appendix: Answers to Post-Hearing Questions
Dr. Cora Marrett, Deputy Director, National Science Foundation... 58
Dr. Keith R. Yamamoto, Vice Chancellor for Research, University
of California San Francisco.................................... 64
Dr. Nancy B. Jackson, President, American Chemical Society....... 65
Dr. Jorge Jose, Vice President for Research, Indiana University.. 67
THE MERIT REVIEW PROCESS:
ENSURING LIMITED FEDERAL RESOURCES ARE INVESTED IN THE BEST SCIENCE
----------
TUESDAY, JULY 26, 2011
House of Representatives,
Subcommittee on Research and Science Education,
Committee on Science, Space, and Technology,
Washington, DC.
The Subcommittee met, pursuant to call, at 10:01 a.m., in
Room 2318 of the Rayburn House Office Building, Hon. Mo Brooks
[Chairman of the Subcommittee] presiding.
hearing charter
COMMITTEE ON SCIENCE, SPACE, AND TECHNOLOGY
SUBCOMMITTEE ON RESEARCH AND SCIENCE EDUCATION
U.S. HOUSE OF REPRESENTATIVES
The Merit Review Process:
Ensuring Limited Federal Resources Are Invested in the Best Science
tuesday, july 26, 2011
10:00 a.m.-12:00 p.m.
2318 rayburn house office building
Purpose
On Tuesday, July 26, 2011, the Subcommittee on Research and Science
Education will hold a hearing to examine the merit review grant award
process and its effect on federally funded scientific research, in an
effort to understand the strengths and potential weaknesses of the
process.
Witnesses
Dr. Cora Marrett, Deputy Director, National Science
Foundation
Dr. Keith Yamamoto, Vice Chancellor for Research,
University of California San Francisco
Dr. Nancy Jackson, President, American Chemical Society
Dr. Jorge Jose, Vice President for Research, Indiana
University
Overview
A number of federal agencies, from the Department of
Energy to the National Institutes of Health and the National Science
Foundation, use various types of peer or merit review to evaluate
proposals and make recommendations to award federal funds.
The National Science Foundation (NSF) has three funding
mechanisms: grants, cooperative agreements, and contracts. NSF makes
merit-based grant awards to researchers, educators, and students. In
Fiscal Year 2010 (FY 10), NSF received 55,542 proposals and awarded
12,996 grants, a 23 percent funding rate. Fifty percent of its budget
was devoted to new and continuing grants. \1\
---------------------------------------------------------------------------
\1\ Report to the National Science Board on the National Science
Foundation's Merit Review Process Fiscal Year 2010, p. 7.
Approximately 96 percent of NSF grant proposals are
evaluated through an external review process, commonly known as the NSF
merit review process. The process utilizes subject matter experts to
review proposals through the mail, in-person at a panel review, or
through a combination of both (early-concept grants, rapid response
grants, and small conferences and workshops are evaluated through an
internal merit review process).
The NSF merit review process evaluates proposals based on
two criteria, intellectual merit and broader impacts. A National
Science Board Task Force is currently examining the two criteria and a
report is forthcoming. Since 2007, NSF has also been promoting
potentially transformative concepts through additional language added
to the intellectual merit criteria.
Background
Federal funding is disbursed in a number of ways, including through
contracts, cooperative agreements and grants. The process by which many
federal agencies evaluate potential grant awards is often termed merit
or peer review. This process can take several different forms or
utilize different processes; however, in general, it requires that the
grant proposals be reviewed and evaluated by subject matter experts not
associated with the proposal.
The National Science Foundation (NSF) uses grants for the majority
of its funding disbursements. There are two basic grants. A standard
award has a duration of typically one 09five years, but is fully funded
in the first fiscal year. A continuing grant, also for a multi-year
project, is provided in annual increments. The first year of funding
for a continuing grant comes with a statement of intent to continue the
funding with continuing grant increments (CGIs) through completion of
the project, but the continuation is contingent on whether NSF deems
satisfactory progress, availability of funds, and the receipt and
approval of annual reports.
Cooperative agreements are used when the project requires
substantial agency involvement like research centers and multi-user
facilities.
NSF utilizes an internal merit review process for a fraction of its
grant awards, including the Early-concept Grants for Exploratory
Research (EAGER) and Grants for Rapid Response Research (RAPID). \2\
However, the bulk of NSF funded grants are evaluated through an
external grant review process, known as the NSF merit review process.
---------------------------------------------------------------------------
\2\ Early-concept Grants for Exploratory Research (EAGER) are two-
year awards for up to $300,000. These awards support quick-response
research, or research having a severe urgency with regard to
availability of data, facilities, or equipment.
---------------------------------------------------------------------------
NSF Merit Review Process
Grant proposals are required to be submitted electronically. NSF
program officers ensure each proposal has been assigned to the correct
office for review and determine the appropriate level of review
(internal or external). Proposals are returned without being reviewed
if they do not comply with NSF regulations, including separately
addressing the intellectual merit and broader impacts criteria.
Program officers choose proposal reviewers and panel members from
an NSF database of over 300,000 reviewers. They can also recruit
reviewers based on literature searches, professional activities, and
other reviewer recommendations. In addition, they also screen all
reviewers for potential conflicts of interest and provide guidance and
instructions.
Reviewers provide comments by mail or through the meeting of a
panel session (these are most often in person but panelists may also
meet virtually). Once reviewers return comments, or a panel convenes,
program officers are responsible for synthesizing comments and
recommending the award or decline of each proposal. Reviewers provide
narratives and categorical ratings which the program officer takes into
account. An ``Excellent'' rating does not guarantee the award of
funding. In FY 2010, 3,743 proposals that received an average review of
``Excellent'' were funded and 1,312 were not, and 4,560 proposals that
received an average review of ``Very Good to Excellent'' were funded
while 6,318 were not. \3\
---------------------------------------------------------------------------
\3\ Report to the National Science Board on the National Science
Foundation's Merit Review Process Fiscal Year 2010, p. 32.
The Division Director reviews the program officer's recommendations
and then passes them on to the Division of Grants and Awards, who vets
the eligibility of the awardee, negotiates any necessary changes, and
disburses the award. The Director's Review Board reviews any award in
excess of 2.5 percent of the awarding Division's budget. The National
Science Board (NSB) must approve any award in excess of $3 million
dollars, or one percent or more of the awarding Directorate's prior
year current plan, whichever is greater.
All those who submitted proposals receive notification as to
whether or not an award will be made. Those to whom funding was
declined receive copies of the reviews as well as information on the
number of grants awarded and the number of proposals in each category.
If a proposal is declined, the proposer may ask the program officer for
further clarification. If he is still unsatisfied, he may make a
reconsideration request to the relevant Assistant Director and a second
request to the Deputy Director. (See Appendix A for the NSF Flow
Chart.)
Types of Review
Proposals submitted to the merit review system are reviewed in
three ways. Through ``mail-only'' reviews, proposals are sent to
reviewers who are asked to submit written comments to NSF. Through
``panel-only'' reviews, reviewers serve on in-person (or virtual)
panels to discuss reviews and provide advice to the program officer.
Additionally, some proposals receive a combination of mail and panel
review, which can take place in a number of ways.
There is value in each type of review. Mail review allows for
better matching between the expertise of reviewers and proposals. Panel
review allows for interplay between reviewers in the evaluation of
proposals and the integration of different perspectives in the review
of proposals. According to NSF, ``Using panels in the review process
tends to reduce proposal processing time (time-to-decision), compared
to mail-only reviews. For example, in FY 10, 78 percent of all
proposals reviewed by panel-only were processed within six months,
compared to 72 percent for mail plus panel and 55 percent for mail-
only.'' \4\ While in-person panels are most common, ``virtual panels''
are being convened more often. Virtual panels allow reviewers to
participate from remote locations using interactive technology. The
combination of mail and panel reviews ``is used frequently because it
combines the in-depth expertise of mail review with the comparative
analysis of panel review.'' \5\
---------------------------------------------------------------------------
\4\ Report to the National Science Board on the Nationl Science
Foundation's Merit Review Process Fiscal Year 2010, page 29.
\5\ Ibid.
---------------------------------------------------------------------------
The Program Officer
NSF program officers are made up of permanent (54 percent) and non-
permanent (46 percent) employees; \6\ all are subject matter experts in
the areas they manage with advanced degrees or credentials. ``Some non-
permanent program officers are `on loan' as `Visiting Scientists,
Engineers, and Educators' (VSEEs) for up to three years from their host
institutions. Others are supported through grants to the home
institutions under the terms of the Intergovernmental Personnel Act
(IPA).'' \7\ These ``rotators'' ensure that new and fresh scientific
ideas and specialties come through the Foundation and help to prevent
institutional or innovative stagnation. One drawback, however, is the
loss of institutional knowledge when a rotator leaves and the challenge
of frequently bringing new rotators up to speed on NSF policies and
processes.
---------------------------------------------------------------------------
\6\ Report to the National Science Board on the Nationl Science
Foundation's Merit Review Process Fiscal Year 2010, page 34.
\7\ Ibid.
---------------------------------------------------------------------------
The Foundation expects program officers to administer balanced
portfolios and requires them to utilize the advice and expertise of the
proposal reviewers while assessing proposals in terms of each
portfolio. In order to create a balanced portfolio, program officers
are expected to additionally evaluate proposals for a number of
criteria, including, but not limited to: geographic distribution; novel
approaches to significant research questions; capacity building in a
new and promising research area; potential impact on the development of
human resources and infrastructure; and NSF core strategies, including
integration of research and education, broadening participation, and
promoting partnerships. \8\
---------------------------------------------------------------------------
\8\ Report to the National Science Board on the Nationl Science
Foundation's Merit Review Process Fiscal Year 2010, p. 26.
---------------------------------------------------------------------------
Merit Review Criteria
Since initial approval in 1997, every NSF grant proposal has been
reviewed based on two merit review criteria, intellectual merit and
broader impacts. While additional consideration may be given for a
number of reasons including special requirements of the program,
intellectual merit and broader impacts remains the cornerstone of the
NSF merit review process.
Intellectual Merit. What is the intellectual merit of the
proposed activity? How important is the proposed activity to advancing
knowledge and understanding within its own field or across different
fields? How well qualified is the proposer (individual or team) to
conduct the project? (If appropriate, the reviewer will comment on the
quality of prior work.) To what extent does the proposed activity
suggest and explore creative, original, or potentially transformative
concepts? How well conceived and organized is the proposed activity? Is
there sufficient access to resources?
Broader Impacts. What are the broader impacts of the
proposed activity? How well does the activity advance discovery and
understanding while promoting teaching, training, and learning? How
well does the proposed activity broaden the participation of
underrepresented groups (e.g., gender, ethnicity, disability,
geographic, etc.)? To what extent will it enhance the infrastructure
for research and education, such as facilities, instrumentation,
networks and partnerships? Will the results be disseminated broadly to
enhance scientific and technological understanding? What may be the
benefits of the proposed activity to society? \9\
---------------------------------------------------------------------------
\9\ Report to the National Science Board on the Nationl Science
Foundation's Merit Review Process Fiscal Year 2010, pp. 21 0922.
The America COMPETES Reauthorization Act of 2010 expands the
broader impacts criteria to include activities to achieve the following
goals: (1) increase the economic competitiveness of the United States;
(2) development a globally competitive STEM workforce; (3) increase
participation of women and underrepresented minorities in STEM; (4)
increase partnerships between academia and industry; (5) improve pre-K
0912 STEM education and teacher development; (6) improve undergraduate
STEM education; (7) increase public scientific literacy; and (8)
increase national security.
In February 2010, the NSB reconstituted the Task Force on Merit
Review. The Board charged the Task Force with ``examining the two Merit
Review Criteria and their effectiveness in achieving the goals for NSF
support for science and engineering research and education.'' \10\
---------------------------------------------------------------------------
\10\ NSB Task Force on Merit Review, http://www.nsf.gov/nsb/
committees/tskforce-mr^charge.jsp.
---------------------------------------------------------------------------
In June 2011, after a year of review, NSB and NSF put out a call
for public comment (closed July 14, 2011) on proposed revisions to the
merit review criteria. The proposed changes maintain the themes of
intellectual merit and broader impacts while establishing key
principles of the merit review criteria. The proposed changes include
the identification of national goals which every NSF project should
seek to advance, including but not limited to: the increased economic
competitiveness of the United States; the increased participation of
women, persons with disabilities, and underrepresented minorities in
STEM; the increased public scientific literacy and public engagement
with science and technology; and increased national security. (See
Appendix B for the complete proposal.)
Potential Challenges
While the NSF merit review process is widely considered the most
effective of its type for the awarding of federal funding, there are
existing challenges to be considered in an effort to strengthen the
process. Questions remain about the way in which scientific priorities
are established and whether the process is truly supporting innovative
research and researchers. Below are some additional challenges:
Transformative Research Research that questions existing
science often faces additional hurdles when facing review by scientific
researchers in that field, especially during lean economic times as
experts favor more conservative funding approaches. Since 2007, NSF has
been working to ensure transformative research is considered
appropriately and such proposals are provided an opportunity to compete
through the merit review process, including adding explicit language in
the intellectual merit criteria for transformative concepts. EAGER
grants are intended to be used, in part, to fund potentially
transformative ideas for which there is little to no preliminary data
and, as such, would fare poorly in the standard merit review process.
NSF has also incorporated efforts to encourage transformative research
in its training of program officers and reviewers. It is also
experimenting with modifications in the review process to help identify
transformative proposals. Are these efforts working? Is there more to
be done, within the process, to encourage transformative science?
Ensuring a Pipeline for U.S. Students by Encouraging New
Principal Investigators New Principal Investigators (PIs) often do not
have the same level of experience or access to resources that
established PIs have, both considerations included as part of the
intellectual merit criteria. In FY 10, new PIs submitted 21,545
proposals and received 3,620 awards, a funding rate of 17 percent;
prior PIs were funded at a rate of 28 percent. \11\ A strong system
properly encourages new investigators to participate in the scientific
arena in order to ensure a pipeline for U.S. student participation in
scientific endeavors. The Faculty Early Career Development (CAREER)
Program offers specific funding opportunities for new PIs to help in
this endeavor, but is this enough? Does the merit review process
encourage the participation of new PIs?
---------------------------------------------------------------------------
\11\ Report to the National Science Board on the National Science
Foundations's Merit Review Process Fiscal Year 2010, p. 8.
Parity for Institutions Institutions that are not regular
grant recipients do not always have the same resources or proficiencies
as those institutions that consistently receive federal funding. ``For
FY 10, the average funding rate was 26 percent for the top 100 Ph.D.-
granting institutions (classified according to the amount of FY 10
funding received). In comparison, the rate was 17 percent for Ph.D.-
granting institutions that are not in the top 100 NSF-funded
category.'' \12\ Are those institutions, not regularly in receipt of
federal funding, encouraged to submit grant proposals and participate
in the merit review process?
---------------------------------------------------------------------------
\12\ Report to the National Science Board on the National Science
Foundations's Merit Review Process Fiscal Year 2010, pp. 10 0911.
Multidisciplinary Review As NSF seeks to grow its
multidisciplinary projects, the merit review process must consider the
management of reviews that incorporate a combination of scientific
disciplines in order to fund the strongest multidisciplinary proposals.
Is the current process able to effectively encourage and evaluate
---------------------------------------------------------------------------
multidisciplinary projects?
In all, the merit review process must continue to balance these
challenges with the inherent need to fund the strongest science.
Chairman Brooks. The Subcommittee on Research and Science
Education will come to order.
Good morning. Welcome to today's hearing entitled ``The
Merit Review Process: Ensuring Limited Federal Resources Are
Invested in the Best Science.'' I am going to give an opening
statement.
Our hearing today presents us with an opportunity to
examine the merit review process for awarding federal grant
funds. It is our goal to highlight the benefits of the process,
while acknowledging that no process involving human decision
making is flawless. The focus of today's hearing will primarily
be on the merit review process at the National Science
Foundation.
The National Science Foundation Act of 1950 directs NSF to
initiate and support basic scientific research and programs to
strengthen scientific research potential and science education
programs at all levels. NSF works to accomplish this
Congressional directive through the issuance of merit-based
awards to researchers, educators and students at approximately
1,900 U.S. colleges, universities and other institutions. In
1994, the National Academies touted it as among the best
procedures known for insuring the technical excellence of
research projects that receive public support, but the process
has changed since then, and we need to make sure that is still
the case.
As we know, a large number of potentially fundable
proposals are declined each year. The Foundation received over
55,000 proposals for funding in fiscal year 2010 and funded
nearly 13,000, or 23 percent, of them. Many of the proposals
received were not worthy of federal funding, but it is also
true that many were not funded because federal funds are
limited. So given that those limited dollars should go to the
very best scientific research, NSF must maintain a robust and
transparent merit review process.
Today, our witnesses will share their thoughts on how the
process works and its strengths and weaknesses. We want to know
if the current process spurs or stifles innovation, how award
decisions are actually made after receiving peer review, and if
there are flaws in the system that may be providing precious
federal funds to lower rated proposals over more highly rated
proposals.
In exercising its oversight role, this Subcommittee must
ensure that federal dollars are being spent on the best
science. This examination of the merit review process will help
us to understand how programmatic funding decisions are made
and how, in turn, those decisions interact with Foundation-wide
priorities.
I look forward to hearing from each of our witnesses on
this important topic. I thank you for joining us.
[The prepared statement of Mr. Brooks follows:]
Prepared Statement of Chairman Mo Brooks
Good morning, and welcome to each of our witnesses. Our hearing
today presents us with an opportunity to examine the merit review
process for awarding federal grant funds. It is our goal to highlight
the benefits of the process, while acknowledging that no process
involving human decision making is flawless. The focus of today's
hearing will primarily be on the merit review process at the National
Science Foundation (NSF).
The National Science Foundation Act of 1950 directs NSF to ``to
initiate and support basic scientific research and programs to
strengthen scientific research potential and science education programs
at all levels.'' NSF works to accomplish this Congressional directive
through the issuance of merit-based awards to researchers, educators
and students at approximately 1,900 U.S. colleges, universities, and
other institutions. In 1994, the National Academies touted it as being
among ``the best procedures known for insuring the technical excellence
of research projects that receive public support,'' but the process has
changed since then, and we need to make sure that is still the case.
As we know, a large number of potentially fundable proposals are
declined each year. The Foundation received over 55,000 proposals for
funding in Fiscal Year 2010 and funded nearly 13,000, or 23 percent, of
them. Many of the proposals received were not worthy of federal
funding, but it is also true that many were not funded because federal
funds are limited. So, given that those limited dollars should go to
the very best scientific research, NSF must maintain a robust and
transparent merit review process.
Today, our witnesses will share their thoughts on how the process
works and its strengths and weaknesses. We want to know if the current
process spurs or stifles innovation, how award decisions are actually
made after receiving peer review, and if there are flaws in the system
that may be providing precious federal funds to lower rated proposals
over more highly rated proposals.
In exercising its oversight role, this Subcommittee must ensure
that federal dollars are being spent on the best science. This
examination of the merit review process will help us to understand how
programmatic funding decisions are made and how, in turn, those
decisions interact with Foundation-wide priorities.
I look forward to hearing from each of our witnesses on this
important topic; thank you for joining us.
Chairman Brooks. At this point, I recognize Mr. Lipinski,
who is the Ranking Member of this Subcommittee.
Mr. Lipinski. Thank you, Chairman Brooks, and thank you for
holding the hearing this morning.
It will not surprise you when I say that I have had a great
interest in this topic since I first submitted a grant proposal
to the National Science Foundation when I was a graduate
student. That interest continued throughout my academic career,
and when I was elected to Congress I joined what was then the
Science and Technology Committee to a large extent--actually it
was the Science Committee at the time when I joined--to a large
extent because of my strong belief in the NSF and its mission.
I agree with the statements of all of the witnesses here
today that NSF's merit review system remains the gold standard
for the world. And I don't say that just because that NSF grant
application I submitted when I was in grad school was
successful.
At the same time, I recognize that there are challenges in
any system for allocating limited research dollars. I agree
with Chairman Brooks that it is our job on this Subcommittee to
hold hearings such as this one to discuss these challenges and
collectively imagine how we might continue to make NSF, and the
merit-review system that it manages, even stronger.
Particularly in this tight budget environment, it is incumbent
upon us all to make sure that the system for funding excellent
science is as efficient and effective as possible.
In 2009, when I was Chair of this Subcommittee, we looked
at a few slices of the broader topic being addressed today when
we held a hearing on high-risk, high-reward, or what we also
call transformative research. Dr. Yamamoto was not on the panel
for that hearing, but a report that he helped author, the ARISE
report, played a central role and remains relevant today. I
look forward to learning from his deep expertise on this topic.
But there are many issues that we have not examined in
detail, including the extent to which faculty from lesser-
resourced institutions face an uneven playing field. I am also
interested in the extent to which the institutional structures
underlying NSF's peer-review system influence decisions and the
benefits and drawbacks of different approaches to peer review.
For example, I am intrigued by proposals to conduct
committee review in virtual environments such as Second Life.
While I am perhaps a little skeptical, I recognize that virtual
review has the potential to save taxpayers a lot of money in
travel expenses, as well as broaden the pool of reviewers. So I
am glad that the NSF is looking into innovative approaches. But
we need to be confident that the group dynamics in a virtual
environment, while certain to be different, do not in any way
undermine the quality of NSF merit review.
I am particularly interested in hearing the panel's
recommendations about some of the alternatives to standard
merit review, not as a replacement of, but rather as a
complementary approach. For example, in last year's COMPETES
Act, I authored a provision that authorizes prize programs at
all of the science agencies. While NSF, as a basic research
agency, would need to design and implement a prize program that
looks very different from those run by DARPA or NASA, I
continue to believe that the NSF should experiment with some
pilot projects to award research prizes. I look forward to
hearing an update from Dr. Marrett on the Foundation's thoughts
on this subject.
So I see this hearing as an opportunity to examine the real
challenges that do exist in a very strong review system and to
discuss current and novel approaches to overcoming those
challenges.
Finally, Mr. Chairman, I think it is important to point out
that the National Science Board is in the middle of a process
to review and revise the existing merit review criteria. While
I believe this hearing is important, it is critical that in the
fall when the Board has finished its work and produced a new
set of review criteria for the scientific community, this
Committee should examine what the Board has come out with.
Especially in light of the provision in last year's COMPETES
bill directing the agency to clarify the purpose and
implementation of the broader impacts review criterion, it will
be helpful at some point later this year or early next year to
revisit this topic, and I certainly hope that we can do that.
With that, Mr. Chairman, I thank all of the witnesses for
being here today and I yield back.
[The prepared statement of Mr. Lipinski follows:]
Prepared Statement of Ranking Member Daniel Lipinski
Thank you, Chairman Brooks, and thank you for holding this hearing
this morning. It will not surprise anyone when I say that I have had a
great interest in this topic since I first submitted a grant proposal
to the National Science Foundation as a graduate student. That interest
continued throughout my academic career and when I was elected to
Congress I joined what was then the Science and Technology committee to
a large extent because of my strong belief in the NSF and its mission.
I agree with the statements of all of the witnesses here today that
NSF's merit-review system remains the gold standard for the world. And
I don't say that just because that NSF grant application I submitted
when I was in grad school was successful.
At the same time, I recognize that there are challenges in any
system for allocating limited research dollars. I agree with Chairman
Brooks that it is our job, on this Subcommittee, to hold hearings such
as this one to discuss these challenges and collectively imagine how we
might continue to make NSF, and the merit-review system that it
manages, even stronger. Particularly in this tight budget environment
it is incumbent upon us all to make sure that the system for funding
excellent science is as efficient and effective as possible.
In 2009, when I was chair of this Subcommittee, we looked at a few
slices ofthe broader topic being addressed today when we held a hearing
on high-risk, high-reward, or what we also call transformative
research. While Dr. Yamamoto was not on the panel for that hearing, a
report that he helped author, the ARISE report, played a central role
and remains relevant today. I look forward to learning from his deep
expertise on this topic.
But there are many issues that we have not examined in detail,
including the extent to which faculty from lesser resourced
institutions face an uneven playing field. I am also interested in the
extent to which the institutional structures underlying NSF's peer-
review system influence decisions and the benefits and drawbacks of
different approaches to peer review.
For example, I am intrigued by proposals to conduct committee
review in virtual environments such as Second Life. While I am perhaps
a little skeptical, I recognize that virtual review has the potential
to save taxpayers a lot of money in travel expenses, as well as to
broaden the pool of reviewers. So I'm glad that the NSF is looking into
innovative approaches. But we need to be confident that the group
dynamics in a virtual enviromnent, while certain to be different, do
not in any way undermine the quality of NSF merit review.
I'm particularly interested in hearing the panel's recommendations
about some of the alternatives to standard merit review, not as a
replacement of, but rather as a complementary approach. For example, in
last year's COMPETES Act, I authored a provision that authorizes prize
programs at all ofthe science agencies. While NSF, as a basic research
agency, would need to design and implement a prize program that looks
very different from those run by DARPA or NASA, I continue to believe
that the NSF should experiment with some pilot projects to award
research prizes. I look forward to hearing an update from Dr. Marrett
on the Foundation's thoughts on this subject.
So I see this hearing as an opportunity to examine the real
challenges that do exist in a very strong review system and to discuss
current and novel approaches to overcoming those challenges.
Finally, Mr. Chairman (as you already mentioned), I think it's
important to point out that the National Science Board is in the middle
of a process to review and revise the existing merit review criteria.
While I believe this hearing is critically important, it is unfortunate
that we couldn't wait until this fall when the Board has finished its
work and produced a new set of review criteria for the scientific
community and this Committee to examine. Especially in light of the
provision in last year's COMPETES bill directing the agency to clarify
the purpose and implementation of the Broader Impacts Review Criterion,
it will he helpful at some point later this year or next year to
revisit this topic.
With that, Mr .Chairman, I thank all of the witnesses for being
here today and I yield back.
Chairman Brooks. Thank you, Mr. Lipinski.
At this time I would like to introduce our witnesses for
today's hearing. First, we have Dr. Cora B. Marrett, the Deputy
Director of the National Science Foundation. Since January
2009, she has served as NSF's Acting Deputy Director and Senior
Advisor until her confirmation as Deputy Director in May 2011.
Dr. Marrett holds a bachelor of arts degree from Virginia Union
University, a master of arts and a doctorate from the
University of Wisconsin-Madison, all in sociology.
Dr. Keith Yamamoto is Vice Chancellor for Research at the
University of California at San Francisco. Dr. Yamamoto served
on grant review panels for the NSF Biology Directorate, and as
an ad hoc member of the National Science Board Taskforce on
Transformative Research.
Dr. Nancy Jackson serves as President of the American
Chemical Society. Dr. Jackson is employed at Sandia National
Laboratories and earned her Ph.D. in chemical engineering from
the University of Texas at Austin.
And Dr. Jorge Jose is the Vice President for Research at
Indiana University. A theoretical physicist and neurobiologist,
Dr. Jose received his undergraduate, master's and Ph.D. in the
Department of Physics at the National Autonomous University of
Mexico.
As our witnesses should know, spoken testimony is limited
to five minutes each, after which the members of the committee
will have five minutes each to ask questions.
I now recognize our first witness, Dr. Cora Marrett, for
five minutes.
STATEMENT OF DR. CORA MARRETT,
DEPUTY DIRECTOR, NATIONAL SCIENCE FOUNDATION
Dr. Marrett. Thank you, Chairman Brooks, Ranking Member,
Lipinski, and the distinguished Members of the Subcommittee
staff. I am pleased to have this opportunity to discuss the
National Science Foundation's merit review process.
For over 60 years, NSF has been a steward of the Nation's
science and engineering enterprise with a proven track record
for producing results. Despite its relatively small size, NSF
has an important impact on scientific and engineering knowledge
and academic capacity. NSF's investments in discovery, learning
and innovation have been important to increasing America's
competitiveness, our economic strength, national security and
overall quality of life.
NSF relies on a merit-based competitive process that is
critical to fostering the highest standards of excellence and
accountability. The process lies at the heart of the agency's
strategy for accomplishing its overall mission.
Now, of the 256 American Nobel Prize recipients in science
since NSF first began to award research grants, of those 256,
approximately 75 percent have received NSF funding at some
point in their careers. That includes 19 individuals in the
last five years.
The agency has a strong record of funding the most
insightful ideas and visionary investigators. Our model of
high-quality merit review has been emulated and replicated by
other nations. Our merit review process helps assure that
awards made by NSF are of the highest quality, are relevant to
our goals and objectives and have an appropriate balance for
the resulting portfolio. This assurance is critical as nearly
90 percent of NSF's funding is allocated through merit review,
allocated either as grants or cooperative agreements.
As you have already noted, in the fiscal year 2010, NSF
evaluated over 55,000 proposals through this process and made
approximately 13,000 new awards. This entailed conducting about
287,000 proposal reviews and engaging nearly 46,000 members of
the science and engineering community as reviewers. Underlying
these statistics is a strategy that helps ensure that each
proposal submitted to NSF is reviewed in a fair, competitive,
transparent, and in-depth manner.
First, our program officers are experts in the scientific
and engineering programs they manage. They hold strong
credentials, usually a Ph.D. or equivalent in a STEM field, and
their funding recommendations are subject to several additional
layers of review. Each proposal submitted to NSF is reviewed by
these in-house experts, and they in turn rely extensively on
external experts to keep NSF-funded research at the frontier.
As we emphasize, this is a unique aspect of NSF, the knowledge
of our program officers, who are then empowered to make
recommendations on funding.
Each proposal must meet the highest standards in terms of
two merit review criteria: intellectual merit and broader
impacts. Intellectual merit encompasses the potential of the
research to advance knowledge, the originality and creativity
of the proposed activity and the qualifications of the
researchers. Broader impacts include technological innovation,
societal benefits and opportunities to include a diversity of
participants.
Finally, the merit review process itself is constantly
assessed through evaluations by committees of visitors,
advisory committees and other stakeholders. NSF continuously
strives to maintain and improve the quality and transparency of
the process.
So in summary, the National Science Foundation is dedicated
to ensuring that the merit review process remains robust,
rigorous, and beyond reproach. This process enables us to carry
out our mission. I appreciate, then, this opportunity to appear
before the Subcommittee on this important topic and would be
pleased to answer any questions you might have.
[The prepared statement of Dr. Marrett follows:]
Prepared Statement of Dr. Cora Marrett,
Deputy Director, National Science Foundation
Chairman Brooks, Ranking Member Lipinski, and distinguished Members
of the Subcommittee, thank you for inviting me to participate in this
hearing on ``The Merit Review Process.''
I am delighted to discuss the National Science Foundation's (NSF)
Merit Review Process with you. As you well know, NSF is the primary
Federal agency supporting research at the frontiers of knowledge,
across all fields of science and engineering (S&E) and all levels of
S&E education. Its mission, vision and goals are designed to maintain
and strengthen the vitality of the U.S. science and engineering
enterprise. As part of the overall national R&D enterprise, the basic
research and education activities supported by NSF are vital to the
economic advancement of the U.S. and provide the know-how that allows
the U.S. to respond rapidly and effectively to a range of unexpected
challenges. The NSF merit review process lies at the heart of the
agency's strategy for accomplishing its overall mission. As such, NSF
is continuously striving to maintain and improve the quality and
transparency of the process.
Before I begin my discussion of the unique elements of the NSF
merit review system, let me first describe the essential features of
merit review writ large. In general, merit review refers to an
independent assessment of a plan's worthiness. The Code of Federal
Regulations (Section 600.13 of title 10) defines Merit Review as a
``thorough, consistent and objective examination of applications based
on pre-established criteria by persons who are independent of those
individuals submitting the applications and who are knowledgeable in
the field of endeavor for which support is requested.''
I would also like to note here that although the terms ``merit
review'' and ``peer review'' are often used interchangeably, they are
not equivalent terms. NSF made this distinction clear back in 1986,
based on a report from an external Advisory Committee on Merit Review,
established by then-director Erich Bloch at the request of the National
Science Board. As is described by Marc Rothenberg, the NSF historian,
in his 2010 article ``Making Judgments about Grant Proposals: A Brief
History of the Merit Review Criteria at the National Science
Foundation:''
According to the committee, the term ``peer review'' was
properly a restrictive term referring to the evaluation of the
technical aspect of the proposal. However, for more and more federally
funded research, ``technical excellence'' was, in the words of the
committee, ``a necessary but not fully sufficient criterion for
research funding.'' Acknowledging that the NSF (as well as other
federal agencies) was using a wide range of nontechnical criteria as
part of the decision-making process, the committee suggested that the
term ``merit review'' more accurately described the NSF selection
process.
The committee's recommendation was accepted by Director
Bloch, and since then NSF has used the term ``merit review'' to
describe our process.
Since its founding, NSF has relied on the merit review process to
allocate the vast majority of its funding. As in other agencies, this
has involved the use of proposals from prospective researchers that are
judged on their merits by knowledgeable persons. But there are several
elements that give merit review at the NSF its distinct features. For
one, right from the beginning, NSF utilized the project grant mechanism
(as opposed to a contract mechanism) for providing funds. This was a
rather radical concept back in 1951, when most government operations
used contracts. Since that time, the use of the grant mechanism has
been adopted by many federal extramural research funding organizations.
NSF's process for deciding which proposals to fund differs from the
approach of a number of other funding agencies and organizations (such
as philanthropic foundations) nationally and internationally. Perhaps
the most distinctive differences are our reliance on expertise from
both outside and within the Foundation, and the discretionary authority
vested in the NSF program officer to make funding recommendations.
Unlike many philanthropic foundations (and even some federal research
funding programs), NSF policy requires that the program officers seek
external expert advice before making most of their funding
recommendations. However, in contrast to a number of other funding
bodies, the external reviewers do not make binding recommendations that
the program officer is obliged to follow, although program officers
always pay close attention to all external reviews. Because of the
responsibility we give our program officers, NSF sets a high standard
for excellence in that position. Our program officers are subject
matter experts in the scientific areas that they manage, and bring
strong credentials with them, including advanced educational training
(e.g., a Ph.D. or equivalent credentials) in science or engineering,
and deep experience in research, education, and/or administration.
NSF has chosen to give the program officer the responsibility for
making funding recommendations to enable a more strategic and long-term
approach for building the award portfolio. As important as the input of
the external scientific experts is, they have only a snapshot view of
the current set of proposals they are evaluating. The NSF program
officer is responsible for putting that snapshot view into the larger
context of the entire award portfolio they are managing, which can lead
to a more diverse and robust portfolio overall. Together with the
division directors, who have the authority to review and act on the
program officers' recommendations, program officer teams are poised to
identify promising research that responds to national priorities
identified by Congress and the Administration. In addition, program
officers can incorporate agency or programmatic priorities, which are
articulated in the annual agency budget, special solicitations, and
standing program descriptions, all of which are available to the
community via the NSF Web site.
The NSF merit review process is described in full detail on the NSF
Web site (http://www.nsf.gov/bfa/dias/policy/meritreview/). There is
also a summary of the major steps in the merit review process in the
annual Report to the National Science Board on the Merit Review Process
(the most recent report covering activities in FY 2010 can be found at
http://www.nsf.gov/nsb/publications/2011/nsb1141.pdf). It is worth
noting here that the key features of the NSF process have remained
remarkably stable over time. Any changes that have been incorporated
have sought primarily to clarify the process and make it more
transparent. For example, initially only excerpts of the external
reviews were shared with the proposal authors. Over time, NSF provided
the verbatim reviews (but not the identities of the reviewers) to the
applicant. Similarly, over time there have been modifications to the
number and clarity of the review criteria. In the America COMPETES
Reauthorization Act, the broader impacts criterion is specifically
mentioned, and the National Science Board is in the process of
analyzing the many comments received on this topic.
A flowchart that graphically depicts the major steps in the merit
review process and a timeline is attached to this testimony as Appendix
I. These steps include:
Assignment to the appropriate program for review.
Principal investigators initiate this process by selecting the program
or programs to which they wish to submit their proposal. Once
submitted, the cognizant program officers for those programs confirm
that the assignment is appropriate. On occasion, a proposal may be
reassigned to another program where there is a better fit. During this
initial assignment process, it is not uncommon for proposals to be
assigned to multiple programs for review, if the subject is
interdisciplinary in nature, or if the question is of interest and
relevance to more than one program.
Administrative review of all proposals for compliance with
NSF regulations. These regulations, which are intended to ensure
fairness in the review process, are described in the Grant Proposal
Guide, which is widely available to the NSF community on the NSF Web
site (http://www.nsf.gov/pubs/policydocs/pappguide/nsf11001/
nsf11-1.pdf). Proposals that do not comply with these
regulations may be returned without review.
Merit review of all proposals that pass the administrative
review. As noted above, a critical feature of NSF's process is the use
of both external review by experts in the field and internal review by
NSF's corps of program officers. The program officers are responsible
for administering the merit review process from beginning to end,
starting with identifying and recruiting appropriate peer reviewers
from the external community to serve either as individual reviewers for
a particular proposal (referred to as ``ad hoc'' reviewers) or as
members of a panel of reviewers who evaluate a larger set of proposals.
To ensure that they receive substantive reviews from a variety of
perspectives, the program officers reach out to a broad range of
experts for input--in fiscal year 2010, over 46,000 external peer
reviewers from academia, government, and occasionally industry provided
authoritative advice to the Foundation. Selection of expert peer
reviewers may be based on the program officer's knowledge, references
listed in the proposal, individuals cited in recent publications or
relevant journals, presentations at professional meetings, reviewer
recommendations, bibliographic and citation databases, or suggestions
from the proposal author (subject to the program officer's discretion).
In making these selections, program officers pay very careful attention
to avoiding conflicts of interest, both real and perceived.
NSF takes seriously its responsibility to ensure that the merit
review process is fair and equitable. One of the ways in which we
address this responsibility is through the briefings that are given to
each review panel before it begins its work. In these briefings,
panelists are instructed on NSF's review criteria (Intellectual Merit
and Broader Impacts), and on maintaining confidentiality and avoiding
conflicts of interest. In addition, review panel briefings typically
include alerting the reviewers to the phenomenon of implicit bias,
which may adversely impact new investigators, smaller institutions, and
underrepresented groups. By guarding against the effects of implicit
bias in the review process, NSF is working to ensure that there are
equitable opportunities for all investigators.
I should note here that while the vast majority of the proposals
received at NSF (96%) are subject to both external and internal merit
review, for some proposals the external review requirement is waived.
This waiver provides necessary flexibility for handling proposals for
which most of the external community would be conflicted (such as
proposals for small conferences, workshops, or symposia), those for
which there is a severe urgency (submitted through the Grants for Rapid
Response Research, or RAPID, mechanism used, for example, on rapid-
response research to the Deepwater Horizon oil spill), and those that
request support for high-risk, potentially transformative exploratory
work (submitted through the Early Grants for Exploratory Research, or
EAGER, mechanism). These proposals are usually only reviewed internally
by program officers with appropriate expertise.
Development of funding recommendations. A central tenet of
the NSF merit review process is that the reviewer input is advisory in
nature. Funding recommendations are developed by the program officer,
who is responsible for synthesizing the advice of the reviewers along
with several other factors, with the goal of allocating funding to a
diverse portfolio of projects that addresses a variety of
considerations and objectives. In addition to their scientific
expertise noted above, NSF program officers bring their own unique
perspective born from their experience of working with hundreds,
thousands, or--in some cases--tens of thousands of proposals. In
developing recommendations within the larger context of their overall
portfolio, program officers consider carefully the individual merits of
each proposal with respect to both its intellectual merit and the
potential broader impacts of the project, and how each proposal might
help advance a variety of portfolio goals such as:
Achieving special program objectives and initiatives;
Fostering novel approaches to significant research and
education questions;
Building capacity in a new and promising research area;
Supporting high-risk proposals with potential for
transformative advances;
Supporting NSF's core strategies of integration of
research and education and integrating diversity into NSF's programs;
Potential impact on human resources and infrastructure;
Other available funding sources; and
Geographic distribution.
NSF has set a goal for completing this process within six months,
from the time the proposal is submitted to the point at which the
proposal is either declined or recommended for funding and forwarded to
the Division of Grants and Agreements for the final stages of review
and processing. The proposal assignment and administrative review stage
is typically complete within a few weeks. The bulk of the time is spent
in the merit review stage, which can take three to four months to
complete. Despite the volume of proposals that NSF receives annually
(in FY 2010, over 55,000 proposals were submitted, an increase of 23%
over the previous year), NSF routinely processes the majority of these
proposals (+75%) in fewer than six months.
To ensure the integrity of the process, all program officer
recommendations are reviewed by the division director (or other
appropriate NSF official), who examines whether the process used to
arrive at the decision has been executed in accordance with NSF's
policies and that the decision has been based on a thorough analysis of
the merits of the proposal. Large awards may receive additional review,
either by the Director's Review Board (DRB) or additionally by the
National Science Board (NSB). The DRB examines award recommendations
with an average annual award amount of 2.5 percent or more of the
awarding division's prior year current plan. The NSB reviews
recommended awards with an annual award amount of one percent or more
of the awarding Directorate's or Office's prior year current plan, or
less than one percent or more of the prior year total NSF budget at the
enacted level. Once the funding recommendation is approved (at whatever
level is appropriate), the Division of Grants and Agreements ensures
that the award recommendation meets all of NSF's requirements before
officially issuing the award.
In addition to having multiple layers of review of individual award
recommendations, NSF requires that all programs undergo an external
review by Committees of Visitors (COVs) every three years. COV reviews
provide NSF with external expert assessments of the quality and
integrity of program operations and program-level technical and
managerial matters pertaining to the merit review and final proposal
decisions. Finally, retrospective analysis of the process is
periodically performed on a Foundation-wide basis, including the
statistical reports submitted to the NSB every year and the Impact of
Proposal and Award Management Mechanisms (IPAMM) report of 2007 (http:/
/www.nsf.gov/pubs/2007/nsf0745/nsf0745.pdf).
At the request of Congress, in 2005 the NSB undertook an
examination of NSF's Merit Review Process (http://www.nsf.gov/nsb/
documents/2005/nsb05119.pdf). The report concludes that:
``The Board fully supports the current NSF system of merit review,
which utilizes the peer review process as the principal driver in
funding decisions. The Board also strongly endorses the role of NSF
program officers' discretionary authority, in concurrence with division
directors, for ensuring the implementation and goals of both Merit
Review Criteria, along with achieving a balanced portfolio of research
and education awards, both within directorates and across the suite of
NSF programs. Unlike a system based solely on peer reviews' scores,
NSF's merit review process incorporates peer review in a system that
also considers those attributes of a proposal (risk, multidisciplinary
nature, novelty) that are not readily accommodated by a numerical
score, but essential to identifying the most innovative proposals.''
The National Academy of Sciences, in the 1994 report ``Major Award
Decisionmaking at the National Science Foundation,'' stated that, ``The
United States has built the most successful research system in the
world. The use of peer review to identify the best ideas for support
has been a major ingredient in this success. Peer review-based
procedures such as those in use at NSF, the National Institutes of
Health, and other federal research agencies remain the best procedures
known for ensuring the technical excellence of research projects that
receive public support.'' In November 2009, the Executive Director of
the Transportation Research Board at the National Research Council,
provided testimony before Congress on how to facilitate the
implementation of research at the Department of Transportation. In that
testimony, the Director endorsed strongly the fact that NSF's merit
review process is well suited to the mission of the agency. His
observation: ``The more applied mitigation and adaptation research
topics should be steered by the concerns and needs of policy makers and
practitioners, while the fundamental research topics should be
organized along the NSF model in which scholars and experts are guiding
the decisions about which projects are likely to be most promising.''
NSF's merit review process has served the agency, the scientific
community, and indeed the country well for many years. Many Nobel
Laureates, National Medal of Science and Technology winners, and
MacArthur Foundation Fellows (popularly known as recipients of Genius
Grants) have been supported by NSF at various stages in their careers.
Through separate programs and in the course of funding specific
scientific progress, over the past 25 years NSF has also supported the
training of hundreds of thousands of graduate and post-graduate
scholars in STEM fields. Discoveries stemming from NSF-funded projects
have led to advances across all areas of science, engineering and
education, with far-reaching impacts in the fields of nanotechnology,
information technology, environmental science, genomics, STEM
education, and many others.
The high quality of NSF's merit review process is recognized
globally, as evidenced by the fact that it has been used as a model by
countries around the world that are newly establishing their own
funding agencies. The merit review system for L'Agence Nationale de la
Recherche (ANR), the French counterpart to NSF, is explicitly modeled
after NSF, as is that of the Foundation for Polish Science. NSF helped
the European Research Council establish its merit review system some
five years ago, and was instrumental in helping Ireland establish
Science Foundation Ireland. Back in 1986, a Chinese official came to
NSF for six months to learn about our merit review and decision making
processes, and subsequently incorporated what he had learned in
establishing the National Natural Science Foundation of China (NSF
09C). These are just a few examples of international agencies where NSF
has had an explicit role in helping develop their merit review systems,
but there are literally dozens of others that have borrowed our
approach over the years.
As the nature of research and the scientific enterprise continues
to change--becoming more interdisciplinary, technological,
international and collaborative--NSF continues to explore ideas and
strategies that could strengthen the merit review process by enlarging
the range of tools that can be used in proposal evaluation. These ideas
have come from a variety of sources--internally, from the research
community, from the practices of other funding agencies, and from the
scientific literature on merit review. One idea that we are actively
exploring is a greater use of technology-mediated virtual panels when
and where it makes sense, with the hope that decreasing the travel
burden will expand the potential pool of reviewers. Among the benefits
that NSF would derive from an expanded pool of reviewers are the
inclusion of more and varied perspectives, increased opportunities for
participation by underrepresented groups, decreased review burden per
individual reviewer, and decreased travel costs for the agency. We have
established an internal working group to identify other viable
candidates for pilot activities, and to develop plans for running and
evaluating those pilot activities. We will be discussing these with an
advisory committee over the next few months to get their help in
refining the processes.
For over 60 years, NSF has been forward looking in terms of how the
agency manages its research and education portfolio. Merit review
fosters the ``process of discovery,'' the means by which researchers
can identify emerging scientific challenges and innovative approaches
for addressing them. NSF is dedicated to ensuring that the merit review
process remains robust, rigorous, and beyond reproach, in support of
our mission and enabling us to pursue our goal of funding the world's
best research in science, engineering and education.
I appreciate the opportunity to appear before the Subcommittee to
speak to you on this important topic. I would be pleased to answer any
questions that you may have.
Appendix I: NSF Proposal and Award Process and Timeline
Chairman Brooks. Thank you, Dr. Marrett.
I now recognize our second witness, Dr. Keith Yamamoto, for
five minutes.
STATEMENT OF DR. KEITH R. YAMAMOTO,
VICE CHANCELLOR FOR RESEARCH,
UNIVERSITY OF CALIFORNIA SAN FRANCISCO
Dr. Yamamoto. Good morning, Chairman Brooks, Ranking Member
Lipinski, distinguished Members of the Subcommittee. Thank you
for the invitation to present a statement before you today. I
am Keith Yamamoto, a molecular biologist, professor, and
administrator at the University of California San Francisco. My
lab's research has been funded continuously for 35 years by
grants from the NIH and NSF.
I have also been active in matters of science and public
policy, leading or serving on dozens of committees focused on a
broad range of these topics. One of these topics has been the
federal merit review system for evaluation of biomedical and
biological research grant applications, especially those
overseen by NIH and to a lesser extent the NSF.
Today, I shall describe the key conceptual and operational
features of the merit review process, assess how well that
process works from my point of view, consider the impact of a
specific proposal to change the NSF process, and mention two
modifications that might make merit review even stronger.
Every application for NIH and NSF life sciences research
support, and at NIH, that means 80,000 applications per year,
we just heard the 55,000 at NSF, every application is
rigorously reviewed and prioritized for scientific merit by
special communities of experts. The details of the review
process differ among the different federal agencies but there
are two critical features that to my knowledge are common
across agencies.
The first is that the merit review is carried out by other
scientists because only scientific peers have the knowledge and
perspective required to assess the scientific significance,
innovation, and impact of proposed research projects, and the
second is that the review committees judge only scientific
merit. It is left to others to assess relevance of the
applications to the goals or portfolio of the funding agencies
and to make the funding decisions themselves. Thus, peer review
with a solitary focus on scientific merit.
How does the process work and how well does it work? The
review committees are federally chartered and populated by
volunteer expert scientists who set aside time from their own
research at institutions across the country to carefully
evaluate written proposals for scientific investigation. The
reviewers exercise their individual and collective scientific
judgments, prioritizing the scientific merit of the
applications. To help avoid nonscientific biases, both the
applications and the reviews are standardized. For example, in
the NIH process, every application is reviewed and scored
according to the same set of criteria, significance, approach,
investigator, environment and overall impact. By any measure,
peer-driven merit review has been spectacularly successful at
identifying and prioritizing the most interesting, innovative
and significant scientific research, thus enhancing profoundly
the strength of federally funded research. In contrast, merit
review is not intended to influence the breadth or type of
research that is funded. Rather, breadth and type are affected
by funding mechanisms and by the stated missions of the funding
agencies.
How might the merit review process be changed to further
strengthen it? Let me mention first a change currently under
consideration that would, in my view, not serve the process and
with two examples of changes that might enhance it.
The National Science Board Taskforce on Merit Review has
proposed a revision of the broader impacts criteria for NSF
merit review, and I am quoting, ``to ensure consideration of
how the proposed project advances national goals.'' This
criterion departs, in my view, from the singular focus on
scientific merit that is essential to the merit review process
and calls upon reviewers to judge grant applications by metrics
outside of their expertise.
Moreover, NSF is the Nation's basic research agency as
mandated by the NSF Act of 1950, which created the Foundation.
The impact of individual basic research projects on broad
national goals may be impossible to judge at the time of
proposal but may emerge with time in unexpected and profound
ways.
Broad national goals of course are vital and need to be
advanced collectively by federal science research agencies.
They cannot, however, be mandated for individual NSF grant
applications. Indeed, national goals can be addressed by
development of funding mechanisms and by defining agency
priorities, not by the merit review process.
Now, while the current merit review process has great
strengths, there are potential changes that might improve it
further. I cite several in my written testimony. Let me mention
just two here. Consider recognizing and validating that a grant
application is not a contract for sure, in fact, not even a
roadmap of experiments but rather an exercise in which the
applicant demonstrates his or her capacity to pose a scientific
problem and devise a research plan that would impact and
advance a field. In reality, scientists pursue the implications
of each day's results rather than adhering to the course of
experiments imagined in their grants. Thus, grant application
formats and merit review criteria would closely assess the
merits of a proposed idea and of the investigator while
reducing the current focus on experimental detail and
feasibility.
Second, consider formally denoting two separate classes of
research, one I will call innovative and one we have been
referring to as transformative, both essential but each
requiring distinct merit review mechanisms. Innovative research
would advance and deepen our understanding of current
paradigms. Transformative research would disrupt or destroy
prevailing paradigms and force creation of new ones.
In closing, the merit review process for federal support of
research is indisputably the best system for identifying
highest quality of science. Its primary features, peer-driven
review and singular focus on merit, have been critical in
identifying grant applications that describe the best science
by the best scientists. A healthy and robust merit review
process is critical to maintaining and extending U.S.
leadership in scientific research and innovation, which in turn
is essential to reaching broad national goals.
This concludes my testimony. I will be pleased to answer
questions and address your comments. Thank you again for the
opportunity to discuss this important matter with you.
[The prepared statement of Dr. Yamamoto follows:]
Prepared Statement of Dr. Keith R. Yamamoto,
Vice Chancellor for Research,
University of California San Francisco
Good morning, Chairman Brooks, Ranking Member Lipinski, and Members
of the Subcommittee. Thank you for the invitation to present a
statement before you today. I am Keith R. Yamamoto, Vice Chancellor for
Research, Executive Vice Dean of the School of Medicine and Professor
of Cellular and Molecular Pharmacology at the University of California,
San Francisco. I received a Bachelor of Science from Iowa State
University and a Ph.D. from Princeton University before migrating to
San Francisco, where I have been on the faculty for 35 years. My
molecular biology lab has been studying the detailed mechanisms by
which small molecules made in our bodies, hormones, control important
physiological processes such as metabolism, stress responses and
immunity; in the course of that work, I have had primary responsibility
for training approximately 100 graduate students and postdoctoral
scholars. Our research has been funded throughout by grants from the
NIH and NSF.
For the past 30 years, I have also been active in matters of
science and public policy, leading or serving on dozens of committees
focused on a broad range of issues, challenges and opportunities. One
of the major areas of emphasis for those activities has been federal
merit review system for evaluation of biomedical and biological
research grant applications, especially those overseen by the National
Institutes of Health (NIH) and the National Science Foundation (NSF).
For example, I co-chaired the NIH effort to assess and enhance its peer
review process, and served on the National Science Board/NSF task force
on transformative research. These extensive experiences have provided
me with a rather deep perspective on the merit review process and its
relationship to the U.S. life science research enterprise.
In my testimony today, I shall: (1) describe the key operational
and organizational features of the merit review process for biomedical
and biological research; (2) assess how well that process works; (3)
consider the impact of a particular change to the process that is
currently being considered; and (4) enumerate some potential
modifications that might further improve the process.
Every application for NIH and NSF life sciences research support
(NIH, for example, receives some 80,000 applications per year)
undergoes rigorous review and prioritization of scientific merit by
committees of peers, typically by scientists who themselves hold grants
from the same agency, and whose research is in the same area of
research as the proposed study. The details of the review process
differ among the different federal agencies, but evaluation by peers is
the crucial common feature.
To populate the federally chartered merit review committees,
agencies enlist volunteer service from expert scientists in each area
of research, who agree to set aside time from their own scientific
research at academic and research institutions around the country,
typically several times a year, to carefully evaluate written proposals
for scientific investigation. The reviewers exercise their individual
and collective scientific judgments free of other biases, prioritizing
the scientific merit of the applications.
To help prevent nonscientific biases, the formats of both the
applications and the reviews are tightly delineated, whereas the
applicants have broad flexibility in choosing the scientific subject
matter of the applications. In the NIH process, for example, a
standardized set of five criteria (significance, approach, innovation,
investigator, and environment) is mandated as the basis for rating
every application. Assessment of the investigator is focused solely on
past performance and qualification for carrying out the proposed study.
Importantly, merit review committees are charged to judge only
scientific merit. In particular, they do not assess relevance of the
applications to the portfolio of the funding agency, nor do they not
make funding decisions. This singular focus on merit (together with
peer-driven review) is the second key feature of the merit review
process.
Thus, it is important that decisions of scientific merit are
insulated and separated from decisions of funding. In addition, of
course, the merit review process has no control over the level of
funding allocated to support meritorious applications. When funding
levels fall far below the capacity to support some of the very best
applications (as is presently the case), the merit review process
appears to fail, i.e., outstanding science goes unfunded. This apparent
failure instead reflects a misalignment between the number of highly
meritorious applications and the number of dollars available to fund
those applications.
Any merit review process that depends upon peers to carry out
evaluations must acknowledge and address at least two intrinsic and
related conflicts of interest: reviewers might unfairly support
applications from their friends to create an ``old boys'' network, or
they might unfairly disadvantage applications from competitors or from
those outside of some perceived ``inner circle.'' In general, these
intrinsic conflicts have been addressed successfully by well-crafted
regulations, and more importantly, by a universal ``culture of
respect'' from the participating scientists who serve as reviewers.
While peer-driven merit review plays a crucial role in identifying
excellence among proposed ideas and research plans, it is also the case
that an element of conservatism is intrinsic to peer review, which
complicates recognition and prioritization of ``transformative'' ideas
and approaches. This is because the majority of scientists, and
therefore the majority of peer reviewers, embrace and extend prevailing
scientific paradigms, whereas transformative research disrupts or
destroys accepted paradigms and creates new ones. Because both types of
research are essential, approaches to address intrinsic conservatism
are important. One strategy is to adopt special funding mechanisms, and
perhaps some special elements of the review process, designed to
identify ideas.
The current merit review processes, which have been in place in the
U.S. for over 65 years, recognize that only scientific peers have the
knowledge and perspective required to assess the relevance, innovation
and impact of proposed research projects. Indeed, the current system of
peer-driven merit review is widely held to be not the ``best good
system'' for evaluation and prioritization of merit, but ``the only
good system.''
By any measure (e.g., quality of scientific publications resulting
from support of meritorious applications, honors and prizes given in
recognition of the highly meritorious research, products and services
that are developed from the results of supported research, creation of
an outstanding scientific workforce resulting from training of students
and fellows supported), peer-driven merit review has been spectacularly
successful at identifying and prioritizing the most interesting,
innovative and significant scientific research projects. Hence, the
merit review process enhances profoundly the strength of the research
funded by the federal government. In contrast, the merit review process
is not intended to influence the breadth or type of research that is
funded. Rather, breadth and type are strongly influenced by funding
mechanisms, and by the range and diversity of disciplinary foci that
are chosen by the funding agencies.
Are there specific changes that might further strengthen the merit
review process to ensure support of the best science? Let me mention
first a change currently under consideration that would in my view
damage the process, and then end with a brief enumeration of some
potential changes that might enhance it.
The National Science Board (NSB) Task Force on Merit Review is
currently reviewing the NSF's merit review criteria, and has proposed a
revision of both the ``Intellectual Merit'' and the ``Broader Impacts''
requirements, with the goal of clarifying their intent and the ways
that they would be used in the merit review process. The purpose of the
Broader Impacts criterion ``is to ensure the consideration of how the
proposed project advances a national goal(s).'' This criterion, as
stated, would in my view adversely affect the merit review process
because it departs from the singular focus on scientific merit that is
essential to the process, and because it obligates peer reviewers to
judge grant applications by metrics outside of their expertise.
This being said, I concur fully that broader national goals are
essential, and as stated in the NSB Merit Review Principles,
``collectively, NSF projects should help to advance a broad set of
national goals.'' It is important, however, to remain mindful of the
language of the NSF Act of 1950, which directs the Foundation ``to
initiate and support basic scientific research and programs to
strengthen scientific research potential and science education at all
levels.'' Indeed, NSF itself originated from ``Science the Endless
Frontier,'' the redoubtable 1945 policy initiative of Vannevar Bush,
which called out untargeted basic research as ``the pacemaker of
technological progress'' in which ``new products and new processes are
founded on new principles and new conceptions, which in turn are
painstakingly developed by research in the purest realms of science.''
This implies that broad national goals should be advanced by the
composite federally funded scientific research endeavor, and in
particular should not be mandated for individual NSF research grant
applications. In general, such goals should be addressed by development
of funding mechanisms and by defining agency priorities, and not as a
part of the merit review process. Moreover, it seems that broad
national goals might specifically be mandates for mission-driven
agencies that seek to support research relevant to health, environment,
energy, food and agriculture, or national security, rather than for the
National Science Foundation.
While the current merit review process has great strength, there
are conceptual and operational aspects that might improve it further.
Some examples for consideration across a wide spectrum:
Reconfigure the grant application conceptually to be
viewed less as a ``contract,'' and more as demonstrations of an
investigator's capacity to identify an important scientific problem and
devise tests that could advance knowledge and understanding, and
thereby impact the field. Thus, the grant application is not intended
as a ``roadmap of experiments'' projected three to five years into the
future. Grant application formats and merit review criteria should
place greater focus on the merits of the proposed idea and of the
investigator, whole reducing the current focus on proposed experimental
details and feasibility.
Motivate top scientists to maintain active participation
in the merit review process, in part by developing mechanisms that more
effectively encourage applicants to submit bold scientific ideas.
Establish and formalize two separate investigator-
initiated funding mechanisms, innovative and transformative, which
invite, identify and support research that, respectively, advances and
deepens our understanding of current paradigms, or disrupts and
destroys prevailing paradigms, and forces creation of new ones.
Consider unique aspects of merit review process for transformative
applications.
The topical/disciplinary focus of review committees has
been eroded by the demands of reviewing a rapidly increasing proportion
of grant applications that include a remarkable diversity of
experimental approaches. To recover the conceptual focus of merit
review committees, institute a ``focused external review'' process in
which ad hoc reviewers are requested to contribute electronically brief
comments that address solely those technologies or approaches for which
specific expertise is lacking on the chartered committee. Such external
reviewers would not be asked to assess the overall scientific merit of
the application; that responsibility would reside solely with the
chartered committee.
The merit review process used for review of federal grant
applications for support of biological and biomedical research is
indisputably the best system for managing this important
responsibility. Its primary features of peer-driven review and singular
focus on merit have been critical in identifying grant applications
that describe the best science by the best scientists. Once unique in
the world, the process is being widely emulated.
This concludes my testimony. I would be pleased to answer your
questions or address your comments. Thank you again for the opportunity
to discuss this important matter with you.
Chairman Brooks. Thank you, Dr. Yamamoto.
I recognize our next witness, Dr. Nancy Jackson, for her
five minutes.
STATEMENT OF DR. NANCY B. JACKSON,
PRESIDENT, AMERICAN CHEMICAL SOCIETY
Dr. Jackson. Good morning, Chairman Brooks and
distinguished Members of the Subcommittee.
As President of the American Chemical Society (ACS) during
2011, the International Year of Chemistry, it is my great
pleasure to address the Subcommittee this morning. Founded in
1876, ACS is the world's largest scientific society with more
than 163,000 members. A nonprofit organization, ACS was
chartered by the Congress in 1937 to advance chemistry in all
its branches, promote scientific research and inquiry, and
foster public welfare and education. ACS members work in
industry, academia and government. ACS is a long-time supporter
and strong supporter of NSF and its merit review process, which
is recognized globally as the gold standard for identifying the
best research to fund. My testimony today will concentrate on
chemistry and NSF's impact on our science.
The Federal Government is an important source of support,
particularly for basic research conducted within the chemical
enterprise that also trains tomorrow's researchers. By
stimulating innovation, the Federal Government empowers a
competitive U.S. chemical enterprise, which contributes to U.S.
economic growth.
Mr. Chairman, I would like to focus on three areas from my
written testimony that I believe are of particular value in
these times of constrained resources and economic challenges.
Those three areas are: one, balancing the NSF research
portfolio; two, streamlining the merit review process; and
three, measuring return on NSF investment.
My first point has to do with portfolio management. As
anyone with a retirement fund has been told, managing a
portfolio is critical to its long-term strength, finding the
right balance between proven performing stocks and riskier,
higher-return investments. Managing a research portfolio is
similar, striving to find the right balance between more
incremental research that builds on previous successes and
those ideas that are riskier and could lead to game-changing
developments. In times of tight budgets and restricted funding,
NSF mustn't become too conservative. The agency must ensure
adequate attention to providing opportunities for young
researchers as well as for out-of-the-box research that may
create economic renewal, produce jobs and train the U.S.
scientific workforce of the future.
Secondly, NSF should be able to triage the least
competitive submission from the review process. At the present
time, all proposals submitted to NSF must go through an
extensive peer review, even if they are intellectually weak
with no hope of ultimately getting NSF funding. For every 100
grant proposals a program officer reviews, a small number,
perhaps around 10, will be of such high quality that it is
obvious to everyone that it should be funded. Another 50
proposals easily will be recognized as not competitive.
However, in the current system, they all take considerable time
to review and process. The real agonizing choices must be made
concerning the remaining 40 proposals, all very good proposals.
Under current funding, only 13 of those 40 proposals would be
funded. The merit review process should be focused on these
choices.
Over the last 10 years, the number of grant proposals
submitted to the NSF chemistry division has doubled while the
size of the NSF chemistry division staff has remained the same,
thereby presenting a significant challenge to NSF to continue
to perform its job of identifying the best science.
ACS is responsible for a research fund, albeit small
compared to NSF. It is the ACS Petroleum Research Fund (PRF).
It has seen the number of grant proposals skyrocket in recent
years as well. To help manage this increase, a new policy was
recently implemented that empowers PRF research managers to
reject proposals at the outset that they deem to be poor in
quality. The practical result has been that more time is freed
up for PRF staff and the volunteer peer reviewers to focus on
selecting the right balance of research from the strong
proposals. Empowering NSF research managers to likewise would
be a practical step to focus a steadily increasing NSF workload
and maintain excellence in its merit review process.
And that brings me to my third point. We need new tools and
methods to evaluate the success of NSF investment. While ACS
strongly supports NSF's broader impacts criteria, we need
better tools to determine their overall effectiveness.
Measuring how many fellowships or grants are funded is easy;
measuring increased innovation, improved national security or
broadening participation is complex and must take place over a
longer time scale.
It is difficult to single out how one individual effort has
impacted a complex collection of national priorities. I
understand that finding the right metrics for measuring these
issues is very difficult but it is worthwhile. Empowering NSF
to more easily gather data is necessary to ensure that research
is successful in achieving the broader impacts and that the
selection process is meeting its many goals.
Chairman Brooks, I thank you for the opportunity to testify
today and I would be happy to answer any questions you or your
Subcommittee have.
[The prepared statement of Dr. Jackson follows:]
Prepared Statement of Dr. Nancy Jackson,
President, American Chemical Society
As president of the American Chemical Society, or ACS, it is my
great pleasure to address the Subcommittee this morning on the topic of
the merit review process of the National Science Foundation (NSF).
Founded in 1876, ACS has grown to be the world's largest scientific
society with more than 163,000 members and one of the world's leading
sources of authoritative scientific information. A nonprofit
organization, ACS was chartered by the U.S. Congress in 1937 to advance
chemistry in all its branches, promote scientific research and inquiry,
and foster public welfare and education. ACS members work in industry,
universities and colleges, and at national laboratories.
ACS is at the forefront of the evolving worldwide chemical
enterprise. It is the premier professional home for chemists, chemical
engineers and related professionals around the world as well as a
global leader in chemical information. ACS publishes 41 world-class
scientific journals and operates the Chemical Abstracts Service, which
provides the most comprehensive databases of disclosed research in
chemistry and related sciences.
Every year, ACS gives more than $11 million in grants for basic
research in petroleum and related fields through the Petroleum Research
Fund (PRF). Twenty-five researchers, who were recipients of these
grants, later went on to become Nobel Laureates.
The Society also plays a leadership role in educating and
communicating with public policy makers and the general public about
the importance of chemistry in our lives. This includes identifying new
solutions to global challenges, improving public health, protecting the
environment, and contributing to the economy.
ACS has been a strong, long-time supporter of the National Science
Foundation, which is of particular importance at this critical time in
our nation's history. My testimony will concentrate on chemistry and
NSF's impact on our science. While I think my observations and
recommendations are broadly applicable, chemistry is my area of
expertise.
Chemistry is the fundamental science that is at the heart of
processes and products that meet our most fundamental needs for food,
shelter, and health, as well as developments and materials that are
vital to advances in biotechnology, computing, and telecommunications.
It is a keystone of U.S. manufacturing and is essential to a range of
industries.
America's $720 billion chemical industry is one of our nation's top
exporters, with $171 billion in annual exports, which accounts for more
than 10 cents of every dollar in total U.S. merchandise exports. Within
the United States, the chemical industry employs 784,000 people and is
a driver of innovation. The industry invests $55 billion in research
and development annually, and one in five U.S. patents is chemistry
related. In addition, the industry contributes to human and
environmental health. Drug innovations, made possible through
chemistry, have helped increase life expectancy in the United States by
30 years over the past century.
I mention this today because the success of the chemical enterprise
is due largely to scientific and technological breakthroughs and
advances made in industrial, academic, and government laboratories.
Although much of the nation's chemical research is carried out by
scientists, engineers, and technicians employed in industry and
academia, the Federal Government is an important source of support,
particularly for basic research conducted by our nation's universities
and government laboratories. By stimulating the roots of innovation,
the Federal Government plays a fundamental role in ensuring the ability
of the U.S. chemical industry to stay competitive in the long term. And
because so many other industries depend on chemicals, the federal
investment enhances the ability of the United States to compete
globally by enabling a high-tech, competitive chemical industry to
supply new products at prices that give our nation's producers an edge.
The NSF plays a unique role in the U.S. scientific enterprise.
While other federal agencies have missions directed at advancing
specific science and technology in health or energy, for example, the
core mission of NSF is to foster a healthy scientific enterprise here
in America. Supporting the best ideas and exploring new frontiers
across research disciplines have been the hallmark of NSF and the
backbone of the American research system.
NSF has played a pivotal role in paving the way for scientific
discovery, in large part, by awarding grants to members of the
scientific research community that have demonstrated outstanding merit.
The Foundation accomplishes its mission by supporting fundamental
research and education in science and engineering. From aircraft
design, pioneering medical tools and robotics, to discovering how
children can learn chemistry better, NSF has played a key role in
funding discoveries that have driven the nation's economy, improved our
quality of life, and enhanced national security. It also supports high-
risk research and novel collaborations that could deliver exceptionally
high rewards.
NSF is not just about research. It's also about developing and
training tomorrow's scientific workforce. There is a symbiotic
relationship between research and education. When a graduate student or
a post-doctoral student works with a researcher funded by NSF, the
student is honing skills and adding new scientific knowledge. In this
way, the torch is passed from one generation of researchers to the
next. To put it another way, this is how we keep pushing the edge of
the envelope. If the United States is to continue to be a leader in
science and technology, then we need to have the trained workforce
working in that space.
NSF provides more than 20 percent of the federal support for basic
research at academic institutions and supports roughly 10,000 new
awards per year through the merit reviews of over 40,000 proposals
received. Every year, an estimated 200,000 people, from undergraduates
to senior faculty, participate directly in NSF research and education
programs.
The NSF merit review process is the gold standard worldwide, and is
one of the reasons why U.S. science has been as successful as it is.
When other countries seek to set up their own national research
efforts, they often look to the U.S. NSF as the role model to emulate.
At NSF, all proposals are evaluated for intellectual merit and
broader impacts. NSF receives far more meritorious proposals than it
could ever fund. While a proposal with weak intellectual merit has no
hope of getting NSF funding, many proposals are rated ``excellent''
with strong intellectual merit and still do not get funded because of
the stiff competition. The broader impacts criteria take into
consideration which research is the most urgent or has the greatest
relevance to improving the quality of life. This merit review process
enables NSF to ensure that precious R&D money goes only to the most
pressing R&D needs.
As anyone with a retirement fund has been told, managing a
portfolio is critical to its long-term strength. Financial advisors
stress that it's important to find the right balance between solid
performing stocks and riskier investments that may provide higher
returns. Managing a research portfolio is similar: the research
manager, whether working in industry or at NSF, strives to find the
right balance between science that will deliver steady advances and
ideas that are out of the box, but could result in game-changing
developments. This point is especially important in times of restricted
funding. It's human nature to make more conservative choices and be
risk-aversive when times are tough. However, now more than ever,
America needs pioneering research that will create economic renewal,
produce jobs, and train the scientific workforce of the future. Extra
efforts and attention must be paid to cultivating young researchers and
game-changing ideas.
One of the reasons why the merit review process is so successful is
because it draws from the collective wisdom of the scientific
community. Many NSF personnel come directly from the scientific
community and will return to their research institutions at the end of
their two- or-three-year rotations. Relying on rotating directors means
the managers are up to date on the most recent scientific developments.
The panels that perform the peer review of proposals are fellow
researchers in the field, and as such, are also up to speed on the
latest developments. This scientific community service, whether
performed by grant proposal reviewers or NSF program officers, is an
integral part of scientific culture. Many scientists dedicate their
time in this way because it provides an opportunity to remain in touch
with and influence the cutting edge, as well as because they understand
that the system only works if everyone volunteers to play their part.
In a way, it is the science community's way of 11paying it forward.''
The merit review process requires significant efforts by both NSF
employees and scientist volunteers. To better understand how the
process plays out, consider this example from the NSF chemistry
division. The division receives about 1,800 proposals annually. Each
program officer in the division manages about 100 proposals a year.
These managers are responsible for picking peer reviewers, and they
must do so with an eye for diversity across a large number of factors
such as ensuring that the reviewers reflect a balanced group based on
type of institution (e.g., small undergraduate colleges vs. large
research universities), geography, and racial and gender
characteristics.
Peer reviewers must also be experts within the proposal's subfield
of chemistry. Generally, a program officer approaches three reviewers
to find one who will accept the call to serve. Since each proposal
requires three to five reviewers, this means the officers approach six
to 10 reviewers for each proposal. Therefore, on the average, the NSF
chemistry division approaches between 10,000 to 18,000 researchers to
serve as peer reviewers for the proposals submitted.
For every 100 grant proposals a program officer reviews, a small
number (perhaps 10) will be of such high quality that it is obvious
they should be funded. Another 50 proposals will be recognized as
clearly not competitive; however, they still must be considered through
the process. An agonizing choice must then be made over the 40
proposals in the middle. These include proposals that may be considered
excellent or very good. In fiscal year 2010, the NSF award rate was 23
percent. In our analogy of 100 proposals, this would mean that 13 out
of the remaining 40 would be funded.
The broader impacts criteria include considerations about whether
the research proposal would broaden underrepresented minorities'
participation in science, strengthen U.S. infrastructure, improve
national security, or foster innovation. Some of these impacts are the
result of language in the America COMPETES bill enacted last year. The
broader impacts criteria take into consideration which research is the
most urgent or has the greatest relevance to improving the quality of
life. The broader impacts criteria enables NSF to choose between
meritorious and even more meritorious proposals, and is a way to ensure
that precious R&D money goes to the most pressing R&D needs.
It should be added that the number of grant proposals submitted to
the NSF chemistry division steadily increases each year, more than
doubling from levels 10 years ago. And while the number of proposals
has doubled, the size of the NSF chemistry division staff has remained
the same. NSF is challenged to continue to perform its job of
supporting the best science, even as the sheer number of proposals
competing for funding has ballooned.
I mentioned in my introduction that ACS is responsible for the
management and administration of the ACS Petroleum Research Fund (PRF),
which was established in 1944 by seven oil companies as a perpetual
trust to advance science education and fundamental research in the
petroleum field. In 2010, the Fund provided $11.4 million for research
grants.
Although PRF is a small research fund, like the NSF, it has seen
the number of submitted grant proposals skyrocket in recent years.
Perhaps some insights gleaned from PRF would be useful in considering
how to strengthen the NSF merit review process.
Several years ago, to relieve the growing peer review burden on the
science community and to lighten the administrative load on the PRF
staff, a policy was implemented to withdraw proposals from
consideration that were deemed to be ``poor'' from the get-go: This
includes those that are poorly written, use bad science, or do not
address the specific scientific areas that were eligible for funding.
In practice, this means that the managers now triage approximately 20
percent of the grants that come their way. These managers err on the
side of caution: If there is any doubt that a proposal may have some
merit, it is forwarded to the peer review panels for consideration and
ranking. The result has been that, while some ``poor'' proposals are
removed from the evaluation process, more time and energy is freed up
for PRF staff and the volunteer peer reviewers to focus on selecting
the right balance of research from the strongest proposals.
Currently, NSF does not have the freedom to remove any proposals
from the very bottom of the pile from consideration. Empowering NSF
research managers to do so--provided that specific criteria are taken
into consideration--would be a simple step to help NSF maintain
excellence in its merit review process. If the average acceptance rate
for an NSF proposal is about 23 percent, this means that 77 percent of
funding proposals will be turned away. Enabling managers to remove the
lowest 20 percent of those that would normally be rejected from
consideration is highly unlikely to result in a potentially great
proposal not getting its due consideration. Instead, this approach may
be a practical step to balancing a steadily increasing NSF workload.
I have mentioned the important role the broader impacts criteria
play in the NSR merit review process. NSF promotes broadening
participation of underrepresented minorities and women, and persons
with disabilities. This also includes increasing diversity in the NSF
portfolio with respect to types of institutions supported and the
geographic regions represented. Broadening participation is one way to
address the broader impacts criteria; however, other activities are
also appropriate.
The importance of NSF efforts to broaden participation of
underrepresented minorities in science and engineering is well
understood and supported by the scientific community. We know that if
the scientific work force doesn't reflect the demographics of our
country, we risk missing out on bringing the best minds and talents
from every community to work on the scientific challenges that will
impact all of our lives.
In spite of NSF efforts in the broader impact criteria areas, we
could use better tools to measure how effective these NSF efforts have
been. One difficulty of measuring the long-term impact of the broader
impact criteria is that it's easier to measure the inputs than the
outputs. Measuring how many fellowships or grants are funded is easy.
Measuring increased national innovation, improved national security, or
broadening participating is complex and must take place over a longer
time scale. These differences make it difficult to single out how one
individual effort has impacted a complex collection of national
priorities.
As a scientist, I want these efforts to be successful. As a senior
administrator, I recognize that it's nearly impossible to measure
success if you can't measure long-term outputs. In industry, we
understand that finding the right metrics is very difficult, but it's
worthwhile to try. I do not know how to resolve this issue, but I do
believe that empowering NSF to more easily gather the data needed to
measure the success of the broader impacts criteria would be a
necessary step to ensuring those efforts achieve their desired effect
to the maximum extent possible.
Chairman Brooks, I thank you for the opportunity to testify today
and to share these thoughts with you. ACS believes the NSF is the
cornerstone of the U.S. scientific enterprise, and we stand ready to
assist you efforts to strengthen the agency for the benefit of the
scientific community and the entire nation.
Chairman Brooks. Thank you, Dr. Jackson.
Next, we recognize our final witness, Dr. Jose.
STATEMENT OF DR. JORGE JOSE,
VICE PRESIDENT FOR RESEARCH, INDIANA UNIVERSITY
Dr. Jose. Good morning, Chairman Brooks, Ranking Member
Lipinski and distinguished Members of the Subcommittee, and
especially Representative Bucshon from Indiana. Thank you for
allowing me the opportunity to speak with you today.
My remarks arise from my experience as a primary
investigator funded for many years by the National Science
Foundation, as a member of NSF review panels, and an advisor
for strategic planning at both the NSF and the National
Institute of Health (NIH). I have been the Vice President for
Research at two major research universities, for five years at
the State University of New York in Buffalo, and since last
August at Indiana University. Both institutions are members of
the American Association for Universities and the Association
for Public Land-Grant Universities.
There are three points I would like to emphasize this
morning. First, merit review is the most effective process we
have for ensuring that federal funds are used to support the
most important and far-reaching scientific research. Second,
merit review is the best way to ensure impartiality in awarding
funding to research. And finally, federal funding for
scientific research awarded through rigorous process of merit
review is a necessary and important component of continued
American preeminence in the world and is the foundation of our
Nation's long-term economic and national security.
A very large number of scientific problems are interesting
in principle but only a small number of those problems is
important and deserves the investment necessary to search for
and find solutions. I support what Dr. Jackson said about not
all of the problems that people want to study are interesting
or relevant for study. My broad experience in the process by
which the Federal Government funds scientific research at
universities leaves me to conclude while no review system is
perfect, merit review is the best way to identify those
problems that are important to support.
One indication of the strength of the merit review system
is the number of major scientific breakthroughs resulting from
research that was awarded federal funding as a result of peer
review by experts, people uniquely able to recognize the
potential of proposed research. For instance, one can imagine
how the average person in the street would respond if asked
whether or not we should spend taxpayer dollars to study C.
elegans, which is a tiny, transparent worm, a millimeter in
size. Yet federally funding both at the NSF and the NIH on this
worm has enabled scientists to identify key genes regulating
organ development and programmed cell death and has shown that
corresponding genes exist in high-end species including humans,
of course. Nine recipients of Nobel Prizes since 2002 focused
their work on C. elegans. Research on C. elegans has led to
clinical trials on the treatment of macular degeneration,
asthma, diabetes and brain diseases. These advances occurred
because experts in the field recognized that this tiny worm
would make a good model system to study and could help us learn
important things about humans.
I would like to comment on the broader impact criteria that
are part of the merit review. In my experience, the broader
impacts criteria are used only to decide between proposals of
equal scientific merit, and this is entirely appropriate.
Having said that, I would like to stress the importance of
expanding the participation of underrepresented groups in the
sciences, a goal which over the past 10 years has been the
primary broader impact criteria for NSF. This is not a matter
of establishing quotas or prioritizing this participation over
the scientific importance of a proposal but the NSF emphasis on
this is the demonstrated way of increasing the pool of talented
and trained scientists which in turn enhances American economic
competitiveness and leadership in scientific research,
discovery and innovation.
In these challenging times, it is more important than ever
that federal funds are spent wisely. Federal research funding
has for 70 years been the cornerstone of American economic
security, scientific and educational preeminence, maintaining
our competitiveness globally, continuing American leadership in
scientific research and innovation and ensuring that our
children and grandchildren enjoy a future in which American
higher education and industry remain the envy of the world. All
these rely on continued robust federal funding for scientific
research and discovery, funding awarded through the merit
review system as my colleagues have already stated.
Thank you very much for inviting me here this morning and I
look forward to answering any questions you may have. Thank
you.
[The prepared statement of Mr. Jose follows:]
Prepared Statement of Dr. Jorge Jose,
Vice President for Research, Indiana University
Chairman Brooks, Ranking Member Lipinski and distinguished Members
of the Subcommittee, it is an honor to be here this morning to speak
with you about the important topic of the merit review process in
federal funding for scientific research, and in particular at the
National Science Foundation. My remarks today arise from my experience
as a primary investigator funded for many years by the NSF, as a member
of NSF review panels, and as an advisor for strategic planning at both
the National Science Foundation and the National Institutes of Health.
I also speak from my experience as Vice President for Research at two
major research universities, for five years at the State University of
New York at Buffalo, and since last August, at Indiana University. Both
institutions are members of the American Association of Universities
and the Association of Public and Land-Grant Universities. This morning
I would like to speak with you about the importance of the merit review
process, specifically, its integral place in establishing American
preeminence in hIgher education, scientific investigation, and economic
innovation.
My broad experience in different aspects of the process by which
the federal government funds scientific research at universities leads
me to conclude that while no system of review is perfect, nor
guaranteed to fund only the best scientific research, the merit review
system is the most effective process we have for ensuring that federal
funds are used most effectively in support of scientific research, in
particular at this time of limited resources when we need to prioritize
how the taxpayer dollars are best invested.
In some respects, the challenges facing federal funding agencies
such as the NSF and the National Institutes of Health are very much
like the challenges I face as Vice President for Research at Indiana
University. As you may know, research universities often invest some of
their limited resources to catalyze programs of research into issues
that are of fundamental importance to our State, our Nation, and the
world. My goal is to help our researchers identify and address the most
important scientific, social and economic problems ofthe 21st century,
such as energy security, health care, national security and our global
competitiveness. Addressing these problems is not only valuable but is
also a necessity because the problems will not solve themselves.
Seemingly intractable problems can be solved only when the best minds
with the appropriate expertise are brought together, and America's
current and future well-being depends in an essential way upon the
results of research into these problems.
As important as this work is, research is just one among many
important areas of the University's work and available resources--at IU
as within the Federal Government--are limited. Given limited resources
and given the importance of tbe problems, it is crucial that IU directs
available resources to the projects with the strongest likelihood of
being transformative and successful. I rely heavily on the ability of
experts to assess each proposal, the work plan, and the potential of
specific people to carry out a project successfully. This guidance
helps to ensure that funding decisions are made on the basis of
scientific merit rather than personal or political considerations. In
short, merit review must be the foundation of funding decisions we make
at IU because merit review enhances the likelihood that we will
properly invest our limited available funds into research projects with
the strongest potential for innovation, for transforming a field, or
addressing successfully an important problem.
We are, of course, following the path that was set up by Vannevar
Bush in his ``Science--the endless frontier'' developed right after
World War II, that led to the formation of the National Science
Foundation. NSF introduced the merit review process as an essential
component to assess and determine how tax dollars should be best
allocated to scientific research. The merit review process has for the
last 60 years led to many NSF notable successes. It is very important
to recognize that a very large number of scientific problems are
interesting in principle but a much smaller subset of those problems is
important and deserves the investment necessary to search and find
solutions.
Merit review is the best way to identify the important problems. It
is the best way to ensure that federal funds are invested in a healthy
array of important problems, covering a breadth of areas and approaches
within a particular field. And merit review is the best way to ensure
impartiality, so that the best science and the best scientists are
funded.
In his planning for the NSF, Vannevar Bush drew on the experience
of wartime scientific research, organized through the National Defense
Research Council (NDRC). Wartime federal investment in scientific
research resulted in the development of penicillin, the radar, and most
famously, the atomic bomb. NDRC research brought together the very best
scientific minds from Europe and the U.S. to work on the most important
problems of the time--and resulted in discoveries that helped America
become the pre-eminent economic, military, and scientific power of the
20th century. Since the end of the Second World War, federal investment
in research at American universities has been central to the
development of universities that are the envy of the world, and merit
review has been central to funding research that enables the United
States to remain the leader in scientific inquiry, the development of
new technologies, and the translation of fundamental research into
applications that shape our lives every day. While there are many ways
to assess and demonstrate the value of the merit review system, allow
me to focus on one. The strength of relying upon merit review to
determine what small percentage of proposals will be funded is
demonstrated by its adoption by other countries across the world.
Funding agencies in Europe, South America, and Asia, which are trying
to emulate the research breakthroughs the U.S. has had in the last 60
years, all rely on a merit review system in which experts assess
proposed research much as we do in the United States.
I can give you a long list of items we use every day which were
often developed as a result of the research that was funded by the
federal investments in scientific research. One which the large
majority of Americans use is the cell phone; another is the Global
Positioning System, which was developed by the Department of Defense
but which was based on a trail of research discoveries that started
with the work by Einstein on his mathematical theory of gravity. GPS
use is so common that it is hard to imagine how we found our way
anywhere before it became commercially viable! We also hear the weather
report, in particular during these hot days; the food we eat has been
produced using scientific breeding techniques. We often don't think
about where all these things came from, but we do know that Americans
as a whole are very proud of the long tradition of scientific research,
inventiveness and innovation, a tradition which has made the U.S. the
advanced technological society that it is today.
It might be helpful to reflect for a moment on a couple of specific
examples of the impact of federal funding for scientific research,
because these examples point to the success of merit review. They also
demonstrate that it is not always possible to anticipate what kind of
impact research may have. For example, Caenorhabditis elegans (c.
elegans), a transparent worm that most of us can go our whole lives
without thinking too much about, but which has been the subject of
significant research since the early 1960s. Only one millimeter in
length, it was the C. elegans which first had its full genome
sequenced, prior to the big achievement of decoding the human genome.
For many reasons, C. elegans is a useful model organism, enabling
researchers to learn about genetics, cell biology, and the pathogenesis
that relate to many human diseases. Three times since 2002, Nobel
Prizes have been awarded to researchers working with C. elegans. The
2002 Nobel Prize in Physiology or Medicine was shared by three
researchers, Sidney Brenner (Berkeley), John Sulston (Cambridge), and
Robert Horvitz (MIT), whose work was funded largely by the National
Institutes of Health through its processes of merit review. The 2002
Laureates identified key genes regulating organ development and
programmed cell death and showed that corresponding genes exist in
higher species, including man. The 2006 Nobel Prize in Physiology or
Medicine was awarded to longtime NIH grantees Andrew Fire (Stanford)
and Craig C. Mello (U. Massachusetts) for their discovery of RNA
interference in C. elegans--work which has led to clinical trials in
the treatment of macular degeneration, asthma, diabetes, and brain
diseases. Further, the 2008 Nobel Prize in Chemistry was awarded to
Martin Chalfie (Columbia) for his work on green fluorescent protein in
C. elegans (along with Roger Tsien and Osamu Shimomura, who studied
these proteins in other contexts). I still read many C. elegans papers
in the current scientific literature that keeps unraveling important
new discoveries with very likely applications including useful drugs to
treat diseases.
You can imagine the response if you were to ask a regular person in
the street if they felt that investing tax dollars would be justified
to study a simple, almost insignificant worm. As the results show, this
research has been a good investment--and it's an investment that relied
on the experts who recognized that this organism was indeed an animal
model system which had many properties in common with higher organisms
like humans and that it was worth studying. It is important to note
that a significant percentage of the Nobel Prize winners for the last
50 years have been American or working in the U.S. This can be
connected with the existence of our merit review system, which tries to
fund and identify only the best and more promising ideas for funding.
In the social and behavioral sciences, federal funding awarded
through merit review has been at the foundation of substantial and
important research. Elinor Ostrom (Indiana University) \1\ and Oliver
Williamson (Berkeley) won the Nobel Prize for their work on ``Economic
Governance.'' Ostrom explored how communities often govern shared
property and common resources more effectively than institutions do.
Her research not only challenges the logical assumption about the
inefficiency of informal groups, but also demonstrates that economic
analysis can help understand myriad forms of social organization.
Williamson provided a theory of why some economic transactions take
place within firms and other similar transactions take place between
firms in the marketplace. Their work informs us about how to handle one
ofthe most basic choices in human organization: When should decision
power be controlled inside an organization, and when should decisions
be left to the market or governments themselves. Ostrom is the only
woman who has won the Nohel Prize in Economics, and she benefited from
long-term funding from the Social and Behavioral Sciences section ofthe
National Science Foundation. Those funds were awarded through merit
review.
---------------------------------------------------------------------------
\1\ Distinguished Professor of Political Science, and Senior
Research Director, Workshop in Political Theory and Policy Analysis,
Indiana University; Research Professor and Founding Director, Center
for the Study of Institutional Diversity, Arizona State University.
Why is the merit review process for awarding federal funds
considered a strong or beneficial process? What is the impact of the
merit review process on the breadth, type, and strength of research
---------------------------------------------------------------------------
funded by the Federal Government?
At the center of the merit review process is peer review, in which
experts review each proposal for its importance, soundness, and
possible transformational and broader impacts when considering if it
should be funded or not. The experts who serve on review panels not
only have expertise in specific research; they also have an
understanding of the broader context of the discipline. They are
therefore uniquely qualified to assess whether an idea is important as
well as interesting; the extent to which the research methods proposed
and the qualification of the researchers is appropriate for the problem
under consideration; the degree to which a proposed research project
has transformative potential. The strength of the merit review process
rests largely on the service of subject experts, their willingness to
read proposals carefully, often for free, so as to identify those most
deserving of funding. It is important to recognize that the expected
results from the proposed projects for funding are not known in
advance; otherwise they would not be called research. It is for this
reason that expert reviewers are able to best assess the probability of
success of a project based on the previous track record of the
investigators submitting the proposal and the track record of the
reviewers on the subject matter under consideration.
Of course, even the most well-intentioned group of subject experts
is not immune from the biases, limitations, and agendas that are part
and parcel of being human. However, in my experience as a reviewer for
many years and after having submitted many proposals to the funding
agencies, it is clear that the agencies try to reduce as much as they
can all possible types of bias. By and large, most of the reviews by
the experts are more than likely to arrive at the correct overall
decision about whether or not it is appropriate to fund a proposal.
There can be outliers to the general process, but reviewers for the
most part are very strict about basing their decisions on scientific
merit, which is why the percentage of proposals which are strongly
recommended for funding is so low.
At its best, the merit review process is a system of checks and
balances not unlike our system of government, in that program officers
can also offer a counterbalance to the limitations of the peer
reviewers. Program officers bring to the decision-making process an
awareness of agency priorities and funding trends that individual
reviewers or review panels will not have. In selecting reviewers,
framing the review process, and interpreting the panel's
recommendations, program officers place funding decisions in a broader
context than just subject expertise. Together, program officers and
reviewers have been remarkably successful at identifying the best
proposals across a wide variety of fields, basing their funding
decisions first and foremost on the intellectual merit as assessed by
the experts, the transformative potential, and intrinsic importance
ofthe research proposed.
With limited federal resources, what role does the merit review
process play in ensuring that the best scientific and potentially
transformative ideas receive funding? How do the broader impacts
criteria requirements, in addition to intellectual merit, affect these
funding decisions?
The merit review process is the best way to ensure that limited
resources are directed to the best ideas. As I stated above, the
collaboration of subject experts and program officers enhances the
likelihood that funded projects will be the ones addressing important
questions, in ways that are methodologically appropriate and that can
lead to transformative changes in the subject matter at hand. The merit
review process is, in my view, the best way to minimize the potential
for politicizing scientific research, and to ensure that limited funds
are allocated as well as possible, in particular when the federal
financial situation is as precarious as it is at the moment.
The broader impacts considerations offer important additional
criteria for funding. Allow me to stress that in my experience, the
broader impacts criteria are used to decide between proposals of equal
scientific merit--the contribution of a proposed project to achieving
the extra-scientific goals included within these criteria does not
outweigh scientific considerations. That is, assuming two proposals
offer programs that are of equal importance and potential, then the
broader impacts criteria should be used to distinguish them. In my
view, funding decisions made in this way find the appropriate balance
between intellectual merit and broader impacts of research.
As members of the Subcommittee are likely aware, in response to the
America COMPETES Act of 2010, the National Science Board recently
proposed an expanded list of broader impact criteria for NSF proposals,
a list that does give some cause for concern. I do not believe that
this expanded list might alter the appropriate balance between
intellectual merit and broader impacts in funding decisions. Rather, my
concern is that the expanded list will diminish the National Science
Foundation's admirable and necessary leadership in promoting the
participation of underrepresented groups (women, racial, and ethnic
minorities) in the sciences. Expanding the participation of
underrepresented groups is an economic and intellectual necessity.
Science and scientific innovation are increasingly important to a
strong economy, and so American economic security as well as American
pre-eminence in scientific inquiry and higher education depend upon
expanding scientific education, at both the K 0912 and higher education
levels. To the extent that the expanded list of broader impacts
criteria diverts attention from this priority, it risks damaging our
economic and national security. \2\
---------------------------------------------------------------------------
\2\ See attachment A.
---------------------------------------------------------------------------
Expanding the participation of members of underrepresented groups
in the sciences is not a matter of establishing quotas or prioritizing
this participation over the scientific importance of a proposal. The
NSF emphasis upon this is a demonstrated way of increasing the pool of
talented and trained scientists. This in turn enhances overall American
competitiveness, and over the long run, it will ensure that the United
States continues to be the world leader in scientific research,
discovery, and innovation.
How does the merit review process work to ensure review
impartiality for all applicants while maintaining high standards of
excellence?
Among the strengths of the merit review process is a degree of
flexibility and breadth that other possible review systems would be
unable to match. By this I refer to the ways in which our proposal
review processes recognize that merit has many facets. For instance, an
investigator at an early stage in her/his career cannot be expected to
have the track record that a more senior investigator has--and yet, the
early career proposal may be as meritorious and promising as the ones
submitted by the senior investigator earlier in their career. The NSF
and other agencies have worked to develop programs that attend to
differences in seniority, to allow the junior investigator to develop
and flourish since they represent our future. CAREER Awards, available
through the NSF, are targeted specifically at researchers early in
their academic careers. But outside of these programs, the merit review
system contributes to impartiality of reviews because it brings a group
of scientific experts together to review proposals. Barring
personalities or idiosyncratic agendas that subvert the process, merit
review increases the likelihood of fair and open review of all
proposals over any alternative review process I am aware of.
Please discuss any potentially novel ideas that should be
considered in order to strengthen the process.
I am, as you have likely gathered, a strong proponent of the merit
review system. I believe it is the process more likely to ensure that
limited federal funds for research are awarded to the most significant
scientific research, with the highest likelihood of long-term impact.
Nonetheless, the process is not perfect. I have alluded to the
possibility that review panels are not free from bias that can
influence their funding recommendations. One weakness of the peer
review process is that scientists can be somewhat conservative. By
this, I mean that it is easier to identify and fund proposals with the
likelihood of incremental scientific advances than it is to identify
and fund proposals with the potential for transformative breakthroughs.
To the extent that bias toward the incremental means that we fail to
fund innovative, paradigm-shifting research, we may be missing
opportunities for precisely the kind of scientific advances that
characterized federally funded research in World War II.
This conservatism is recognized by funding agencies, which have
occasionally responded by altering the charge given to review panels.
``For example, when NIH concluded that it was not awarding enough high
risk/high payoff grants, it changed its charge to panels accordingly.''
\3\ Adapting guidelines given to review panels is one way to ensure
flexibility and strength within merit review processes. ARPA 09E, the
Department of Energy program that focuses on funding breakthrough
research, offers another model that might strengthen the merit review
process. By using a multi-part proposal and review process, ARPA 09E
attempts to give reviewers a better understand of extraordinary
proposals, and thus a better chance of recommending the best and most
feasible proposals for funding. \4\ Similar multipart proposal and
review processes might effectively be adopted by other funding
agencies, in specific programs if not throughout all funding programs.
This might strengthen the merit review process by making funding of
extraordinary projects with transformative potential increasingly
likely. Of course, scientific research advances slowly and having
transformative discoveries is not as common as one would wish. Thus I
am not advocating that funding be directed solely to risky research.
Rather, I am suggesting that the overall research funding apparatus
could be improved by a measured emphasis on potentially disruptive
discoveries and having mechanisms that allow them to be funded.
---------------------------------------------------------------------------
\3\ James Turner, ``Best Practices in Merit Review: A Report to
the U.S. Department of Energy,'' APLU, December 2010, p. 7.
www.aplu.org/document.doc?id=2948.
\4\ Ibid., p. 10.
---------------------------------------------------------------------------
In these challenging economic times, it is more important than ever
that federal funds are spent wisely. Continued federal funding for
scientific research remains an important priority. Federal research
funding--via the National Defense Research Council during the Second
World War, and via the NSF and other government agencies since then--
has for 70 years been a cornerstone of American economic security,
scientific and educational preeminence. Maintaining our competitiveness
globally, continuing American leadership in scientific research and
innovation, and ensuring that our children and grandchildren enjoy a
future in which American higher education and industry remain the envy
of the world--all of these rely on continued, robust federal funding
for scientific research and discovery. Yet each dollar we spend must be
spent wisely--and a strong, flexible and rigorous merit review process
is the best possible guarantee that American scientific research will
continue to lead the way forward and that every taxpayer dollar is
spent with the highest possible return on investment.
I thank the Subcommittee on Research and Science Education for
allowing me to express my opinions about the merit review systems used
by the federal agencies in general but in particular by the National
Science Foundation, which funded a significant portion of my research
for close to 25 years. I will gladly respond to any questions you might
have.
Chairman Brooks. Thank you, Dr. Jose, and thank you, panel,
for the information that you shared with us. I am reminding the
Members that Committee rules limit questioning to five minutes.
However, if time permits and there are Members who wish to
engage in a second round of questions, we may do that. The
Chair will at this point open the round of questions and as
such I recognize myself.
The first question is for Dr. Marrett. Based on fiscal year
2010 data, please explain why over 2,700 proposals rated from
poor to good to very good received federal funding over the
8,000-plus proposals that received very good to excellent or
excellent ratings and were not funded. More particularly, in
looking at the graph that we have been provided, there were
some proposals, one that had a poor to fair rating but was
funded, we had 98 proposals that received a fair to good rating
that were funded, while on the other hand we had 1,312
proposals that received an excellent rating that were not
funded, and similarly, 6,318 proposals that received a very
good to excellent rating that were not funded. So generally
speaking, why this kind of variation?
Dr. Marrett. Yes. Now, I have to explain, to go back to one
of my key points, the role of program officers who have to
bring their expertise to bear. What you are reporting on are
the results of the reviews of external panels. The external
panels look at, as already indicated, very much the technical
merit of proposals. There are other things that have to be
taken into account, and they have been the founding of the
Foundation. Thus, a program officer has to ask about what else
might be known about the proposal. The program officer also
pays a lot of attention to what Dr. Jackson mentioned, a
portfolio. Thus, if we are to be concerned about how well we
are serving the entire United States, how well a portfolio
represents within a discipline the varying ways in which work
might be done in that discipline so that there isn't complete
overlap in what is being funded. So making sure that there this
is this kind of balanced portfolio is a responsibility that
leads then to the fact that our program officers who make their
recommendations to the next levels of review in the process,
program officers are not then bound by the kinds of assessments
that might be given since those assessments are on technical
merit. They are not bound to have to rely exclusively on those
recommendations from the external reviewers.
So let me close by saying, I will note again, those are
recommendations that are made and they are not the final
decisions or how the funding will be done.
Chairman Brooks. You mentioned other things taken into
account and then you gave as an example overlap. Are there any
other factors taken into account that you have not shared?
Dr. Marrett. Well, there can be a number of things. I think
some of the questions from the Committee that we received asked
about such questions as geographical balance. One of the
questions commented on the fact that we know that talent is not
limited to any specific part of the Nation and thus there is a
question often of how are we looking across, but this is not to
suggest that these are non-meritorious. You don't start with
the assumption that you will do away with the merit but the
meritoriousness of proposals. So there are numbers of other
things. The potential contributions to innovation can be
another criterion that a program officer looks at. We then do
have--and these don't depart significantly from what will
appear in all of the materials that we share with the
community, with what will appear in the broader impacts
criteria, but all of these can be brought to bear as
assessments are made about how to use the funding in the most
effective ways for the Nation.
Chairman Brooks. These other criteria that you mentioned,
are they required by statute, required by Code of Federal
Regulations or internal to the way in which the NSF makes its
decisions?
Dr. Marrett. They are internal to NSF. We do abide by the
code that affects all the federal agencies when it comes to
merit review so merit review in a broad sense. They are sets of
principle shared by all federal agencies because of the code.
But then there are the specific criteria approaches that
the Foundation has outlined and those are the ones that we try
to make sure--not try--we actually make sure they are
communicated, understood by the community, thus our Web sites,
our communications always are very explicit about what is going
to be expected of those who submit proposals and how the
process will work.
Chairman Brooks. And finally, the graph that I have talks
about these proposals in terms of numbers but not in terms of
dollars, so I know no score a number, poor to fair, a number of
proposals that fit that criteria, fair to good, good to very
good, very good to excellent, and excellent and so forth. Does
the NSF have this information broken down by dollar figures or
is it strictly on a number basis?
Dr. Marrett. We can provide you the information about
dollars if you would like to know.
Chairman Brooks. Will you please provide our Subcommittee
staff with that information?
Dr. Marrett. We will.
Chairman Brooks. Thank you.
Next I will recognize Mr. Lipinski for his questions.
Mr. Lipinski. Thank you, Mr. Chairman.
Dr. Marrett, as I mentioned in my opening statement, I
offered a provision in last year's COMPETES Act that authorized
prize programs at all science agencies. This language began
with a bill I introduced with Congressman Frank Wolf that would
have created an innovation inducement prize pilot program at
NSF. We introduced this legislation based on a 2007 National
Academies report which concluded that ``an ambitious program of
innovation inducement prize contests will be a sound investment
in strengthening the infrastructure for U.S. innovation.'' I
have also worked with Congressman Wolf to include report
language for the fiscal year 2012 Commerce, Justice, Science
appropriations bill that asks NSF to make use of its new
authority, especially the mechanism for funding high-risk,
high-reward research projects.
Dr. Marrett, can you tell me whether NSF has had any
discussions on this topic and whether it has any plans to
administer a prize program?
Dr. Marrett. Thank you, Mr. Lipinski. We have had extensive
discussions on the topic and appreciate the intent, or we
appreciate what has happened to give us the authority to try
out other kinds of tools in the toolkit. I appreciate your
earlier comment that we must make sure that what we do fits
into the kind of support for fundamental work that lies at the
heart of what the National Science Foundation does and so we
have been looking at what would be appropriate in that context,
and we will be launching a series of experimental efforts to
try to see how these fit within our context and so we are--we
haven't been ready to announce very broadly because these are
highly experimental but we are giving a lot of attention to the
use of prizes and awards as ways for advancing innovation and
advancing knowledge in the Nation. So this would not be
inconsistent with what our overall mission is but certainly we
have paid close attention to the strong interest being
expressed by this Subcommittee, members of the larger community
in the fact that prizes and awards could be a mechanism for
expanding our portfolio.
Mr. Lipinski. Thank you.
I want to turn now to Dr. Jose. One of the inherent
challenges of NSF's merit review process is overcoming
potential conflict that exists when an established researcher
reviews a transformative proposal that may question that
reviewer's work. Now, NSF proposes to address this challenge by
awarding EAGER grants, retraining program officers and
experimenting with alternative approaches to the merit review
process. In your testimony, you mentioned a similar challenging
experience by Indiana University and how crucial it is that IU
directs available resources to projects with the strongest
likelihood of being transformative and successful. Could you
expand on how you ensure that funding decisions of
transformative research are based on scientific merit rather
than the personal considerations of reviewers, specifically, if
such research conflicts with that of a reviewer? How does your
process compare with NSF's process, and if it differs, how do
you recommend incorporating Indiana University's process into
NSF's process?
Dr. Jose. Yes. Thank you very much for the question, Mr.
Lipinski. One of the characteristics of a scientific endeavor
is it moves slowly, usually moves slowly, and once in a while
there is a big discovery and it has a big jump and then it
continues. Scientists generally actually are conservative. When
they review proposals, they are conservative and they are less
willing often to risk the funds that they have to decide to
award to a researcher because they are not sure that it is
going to succeed. Now, doing research means that we don't know
what the answer is going to be. That is why we call it research
because we pose a question that we would like to find an
answer. Often it is not easy to see how to get the answer. We
have to have a plan of how we are going to do it.
Now, when--we had a similar problem, as you mentioned, at
Indiana University. We also have limited resources that we want
to invest in trying to catalyze the research that the faculty
may want to do that is risky and is transformative. Now, the
way of doing that is by requiring from the reviewers to
consider what will happen if this project will succeed, how
much is it going to change the nature of the field in which the
proposal is being submitted, is it going to be transformative.
What does it mean to be transformative? It is really not going
to be just small change in the way things have been done but is
really going to be significant in the way that it is being
done.
The NSF and the NIH have recognized that that is a problem.
In fact, I would say that the way they review proposals in the
NSF is different than the way they review proposals in the NIH.
NIH has review panels that are study sections that actually
review panels for an extended period of time. In the National
Science Foundation, the reviewers are usually confidential;
their names are not known. And what happens is that the
reviewers are often trying to make sure that the money is not
going to be wasted so they are not as willing to risk funding a
project that they are not so sure is going to work out. One
needs to have some policies, and the NSF and NIH have done
that.
They have introduced projects, like the ones you have
mentioned at the NSF, in which a certain amount of money is
going to be given for projects that are risky but the payoff is
going to be very high, if they succeed, it is going to be very
high, money to be given for a project that actually going to be
transformative. For example, the NIH has a program that is
called Pioneer Awards. Pioneer Awards are given to projects
that are very ambitious, very risky, that are really going to
be transformative in the field in which they are given. So the
agencies are trying to do that. It is not easy to do but it
needs to be done. Having said that, funding proposals where the
advance is slowly moving is also important because science
doesn't advance in jumps suddenly. Thank you.
Mr. Lipinski. Thank you, Dr. Jose. Thank you, Mr. Chairman.
Chairman Brooks. The Chair recognizes next Mr. Bucshon from
Indiana.
Mr. Bucshon. Thank you, and welcome, Dr. Jose.
Dr. Jose. Thank you.
Mr. Bucshon. A couple comments. First of all, I was a
cardiovascular surgeon in my previous career, and so I know a
little bit about research and I would just like to say that a
lot of discoveries, as everyone would recognize, are made by
students and people who are just getting their careers started
in science and so that is kind of the area that I am going to
focus on a little bit.
Some areas of research may only have a few experts in the
field, so to speak, and the review process, even though
technically external may actually be consistent with an in-the-
family-type review, and that happens a lot, as we know, in
science. So I guess, Dr. Marrett, I would like to ask, how do
we guard against the politics involved? Because everybody knows
there is politics involved in this process. In a highly
competitive academic environment, both at the institutional
level and at the individual level, and is there a review
process for the reviewers that shows their consistency and
their willingness to assess projects that appear to be
nonpartial, even in this setting?
Dr. Marrett. Let me note again first that what the
reviewers, the panels that we call in or the mail reviewers,
those are again advisory, and when you mention, we are very
concerned to make sure that the process is free of lots of
conflicts. There are a couple of ways in which those are
handled. One is by having all reviewers do have to sign the
conflict forms to indicate that they are not in conflict. Our
program officers do training for ethics and conflict training.
In fact, interestingly, we also have that now, ethics training,
for students, undergraduates, graduates, post-doctoral
students. There is also a requirement for institutions, that
institutions must assure that there isn't a conflict of
interest that would be there.
Even with all of that, we still have other sorts of
processes to oversee how final awards are made and so the
awards are not made exclusively on the basis of these
recommendations. They go up through other levels for review.
Then finally, if there still remain problems, those can be
undertaken through our Office of the Inspector General. If
there is evidence of waste, fraud or abuse, those we would
report to the Inspector General and the Inspector General gets
them through other processes. So we have had--we put in place
checks and balances to reduce the likelihood that there will be
the kind of personal considerations that I know would always be
of concern, and that is why we have what I say again are the
checks and balances to assist them that will not rely
exclusively on any single individual.
Mr. Bucshon. Thank you, because looking over the process,
as you know, there is a disparity in the percentage of grants
being made to new investigators versus established as well as
universities that traditionally get federal funding versus
those that may be new to the process, and I understand all the
reasons why those may occur. People honestly are more
experienced at submitting their proposals and so they may be
more likely to be successful. But the reality is, there
probably is a little bit of this internal family-type thing
that does come into play. Again, saying that, you know,
students and new researchers are ones that frequently make the
most discoveries. I am glad to hear your explanation that these
people, you know, they appear to have the same opportunity.
Dr. Jose, I will ask you a question. In your experience,
many years of experience, has the review--I mean, is there a
lot of political interplay in the review process, and you are
free to----
Dr. Jose. I wouldn't call it political. I think that often
people that have been working in the field for some time, they
think that they have the right answer, and when someone else
comes in and say well, maybe I have another way of looking at
the problem with a different answer, they may say oh, no, no,
no. There is some resistance at the beginning. But eventually
one of the nice things about science and the review process is
that other researchers will actually come up with results that
will say yes, the new way of looking at the problem is the way
to do it. But it is always, just as in human nature, I mean,
all of a sudden you have a way of understanding a problem, you
think you have the answer and someone else comes in with a more
innovative way of doing it and says oh, maybe not, maybe it is
not a good idea to support it but eventually I think the system
works itself out to actually recognize new ways of doing
things, and even when you have a group of people that are--what
do you call them, in the family--that look at problems in one
way, eventually they have to give credit to the new ways of
looking at a problem and solving the problem.
Now, let me say something about the young investigators.
The NSF has a program for--and NIH too--for recognizing junior
investigators that gives them some extra points, if you wish,
when they start the running because they can't compete with the
ones that have been in the field for 40 years. So there are
programs that are working very well. They are not as good as
they should be, but they have recognized those problems.
Mr. Bucshon. Great. Thank you. I yield back.
Chairman Brooks. Thank you, Mr. Bucshon.
The Chair next recognizes Mr. Clarke.
Mr. Clarke. Thank you, Mr. Chair. We are a few days from
possible government default, and to me, I am the new guy here,
relatively, like a few months here in Congress. Many years ago,
I was a staffer. So I am going to say that maybe I have a
perspective that could give us a clearer picture on things
because I haven't been part of the system for a long time. But
one underlying issue that I see that Members of Congress have
regarding taxpayer-funded basic research as we are dealing with
here is they want to make sure that we are not wasting money,
that it also goes to make a constructive difference in our
country, and the broader impacts criteria kind of lays out what
our objectives would be.
Dr. Jackson indicated, I think, a very important issue,
though, is that how do we best measure whether that research is
effectively achieving those broader impact goals, especially
when it is research that is totally innovative and totally
transformative. It is almost like, you know, asking the veteran
Members of Congress to quit blaming each other and think of a
different way of actually passing legislation by inspiring the
public with an enlightened point of view. I am not sure if that
would actually come from entrenched folks here in this system
but it could.
So in that sense, I am just posing this question to all of
you. What type of measures do you think we should look at? Dr.
Jackson indicated that maybe empowering the NSF to easily
gather more data to measure the success of the research could
be effective, but along those lines, we here in Congress want
to make sure that when we are spending taxpayers' dollars that
the taxpayers are getting the best benefit for those dollars.
How do we measure it?
Dr. Jose. If I may answer the question, I mean, there is a
program now, a federal program that is called NSTAR where they
are going to try to find out for every dollar that is spent by
the Federal Government, how many jobs it will create, how many
people are educated, what is the impact that those dollars have
in the economy, for example, and the number of jobs and the
competitiveness of the United States when comparing with other
countries. It is a very difficult thing to do. It is a multi-
variable type of problem. One answer is not that simple. We had
this question asked of us when the stimulus money was given to
universities, and we were asked, and I was interviewed a few
times, and people would ask me, well, you are going to get $20
million so how many jobs will you create, how many jobs were
created from that $20 million, and it was not an easy question
to answer, particularly because the money had just arrived a
few months before, and second, because in particular when it
has to do with research, it takes some time actually to see the
results.
But I will say that in my view, the money that the United
States has invested in funding research and particularly after
the second World War has maintained the preeminence and the
dominance of the United States economy. Furthermore, it has
been the seed money that was used to create the best university
system we have in the world and is the envy of the rest of the
world. I would call that evidence that everything has been
working out. Now, having said that, we need to review
constantly how we are doing things, how we are investing the
money, because times change and we should look at things in a
different way like today with the budget problem that we have,
the Federal Government budget problem that we need to be very
careful about how we invest money, federal research taxpayer
money.
Dr. Marrett. I would just elaborate a bit on that. I think
the points are very well taken. The National Science Foundation
pays attention constantly to issues of metrics because we take
seriously our responsibilities to be accountable to the larger
public. Those metrics are reflected in many ways in the kinds
of requirements for what must be done, broader impacts. There
are the assessments that have to take place as people are
reporting on the broader impacts. So we are constantly looking
at, reviewing, examining what would tend to be approaches that
are possible.
I think the other issue, a key issue here is, it is much
easier often to think about short-term returns for those things
that might have--we can look at, then the longer-term returns,
and it is the longer-term returns that are very important for
an agency such as the National Science Foundation. We have done
studies that have looked at--we take account of some of
important developments and track them back. Sometimes those
have been over an extended period of time. Thus, I think a part
of Dr. Jackson's comments had to do with ensuring that our
attention on returns would not focus us so heavily on immediate
returns that we are unable to think about and invest in what
will have to be the longer terms and the metrics appropriate
for the longer life of activities.
Dr. Jackson. And yes, I would say that is exactly true.
When you analyze and measure the impact of fundamental and
basic research, it takes a long time sometimes to see what
those impacts are, so it is a long-term process, not just a
short-term one.
Dr. Yamamoto. I agree with that. I think you raise a really
important question, and the way that I would look at it, as I
said in my remarks, is that this must be viewed as a continuum
that recognizes that application that really reaches a level of
being evident in addressing national goals, requires a
continuous input of fundamental information about complicated
processes and that for any one grant application, it could be
impossible actually to ascribe how they may be approaching
national goals.
Let me do one quick example. My colleague, Herb Boyer, in
the early 1970s was working on a process that was sort of an
esoteric process in which DNA that got into bacteria was broken
down for some reason and he didn't understand, and this was
called DNA restriction. The kind of DNA that would get into
bacteria would be restricted by breaking it down. And very few
people, very few scientists were interested in this process. I
would wager that if this were put up against the test of
addressing national goals, it might not do very well. That
discovery of what DNA restriction modification was turned into
the capacity to make recombinant DNA, which then turned into
the biotechnology industry worth billions and billions of
dollars and leading the world. So it was impossible to ascribe
with that particular grant what its impact would be but there
it was.
So I think that the goal would be in fact not to look at
these things individually, to ask collectively whether the
federally funded scientific research endeavor in this country
is actually moving things toward national goals, to recognize
that we understand so little about these processes that a
continued input of investigation at the fundamental level is
essential and that that continuum really does work.
The NSF is the basic research engine of the Federal
Government and as such really needs to be celebrated and
protected. There are mission-oriented agencies including the
NIH in fact, the DOE, the DOD, Department of Agriculture,
Commerce, elements of Commerce and so forth that have specific
missions and I think that those should be tested in the short
term against whether they are addressing national goals but
always to recognize that input of fundamental science from the
National Science Foundation and elsewhere is always going to be
essential.
Chairman Brooks. The Chair next recognizes Mr. Hultgren.
Mr. Hultgren. Thank you, Mr. Chairman, and thank you all
for being here.
I have a question for Dr. Yamamoto and Dr. Jackson and Dr.
Jose. Each of you has testified about broader impacts imposed
on the Foundation by Congress in the 2010 American COMPETES
Reauthorization Act and that it will adversely affect funding
decisions by requiring reviewers to consider metrics far
outside the scope of the intellectual merit. Dr. Jose suggests
that rather than enhancing competitiveness and national
security, they risk damaging our economic and national
security. I wondered, are reviewers even privy to the national
priorities prior to the meeting of a panel, and just asking if
you would comment, if each of you would comment on this.
Dr. Jose. I would like to say that all of us are aware of
our environment. All of us are aware about what is going on
with Congress today. For example, all of us read the news, all
of us know what is important. All of us are very aware about
what is happening in the growth of the economies in the rest of
the world and how the United States is having to compete very
strongly with China, which in particular is growing very fast
and we know or we feel that one of the ways in which the United
States is going to succeed is by innovation, by being
innovative, by being creative and introducing new knowledge and
new ideas, and I think that we should not limit the imagination
of investigators and researchers by having a large number of
restrictions about how the money is going to be awarded. I
mean, we put too many restrictions that constrain the freedom
that researchers have that often leads to the creation of an
industry like biotechnology, as Dr. Yamamoto just said. It is
the essence actually of the freedom of thinking freely that
creates new innovations that were totally unexpected.
Mr. Hultgren. Dr. Yamamoto or Dr. Jackson?
Mr. Yamamoto. As I said in my comments, I think that one of
the critical features of merit review is a singular focus on
scientific merit and that anything outside of that is valuable
and important and in fact essential as it may be such as
addressing broader national goals should reside outside of that
process and that asking merit reviewers to make an assessment
of how well the fit is of a given project to national goals I
think is inappropriate. I think the agency individually and
collectively, as I said, the federal scientific research
endeavor needs to be paying attention to whether national goals
are being addressed but that neither reviewers nor applicants,
in my opinion, should be mandated to make such a statement. If
there is a clear application, of course, it would be to the
advantage of the applicant to make that clear, but if not, and
it is a fundamental question, then you are really asking
reviewers to make guesses, and I think one of the great things
about the merit review process is that there is not a lot of
guesswork there and that we shouldn't be asking reviewers to
step outside of their expertise.
Dr. Jackson. As I mentioned in my testimony, there are a
lot of proposals that are sent to NSF that are very, very good,
excellent proposals that just cannot get funded because there
is not enough money so that when the program officer at NSF
looks at 40 proposals that are all great and could have a huge
impact on this country and they have to decide which of those
40 to choose, then the broader impact does come as an advantage
to choose between all these great proposals, and that is an
important role, I think, for the broader impacts criteria and
should be kept as such because there are clearly far more many
proposals that are excellent that can be funded.
Mr. Jose. May I say one more thing?
Mr. Hultgren. Certainly.
Mr. Jose. Just going along with what you said, I just came
to Indiana University last year, and we have limited resources
but we introduced a seed funding program for people that would
be willing in interdisciplinary research. We actually asked
people first to have new projects, new collaborations and so on
and so forth. We know of about 165 proposals and we could only
fund 10 percent. There were many that were recommended for
funding but we had to make a decision at the end. We decided
never to fund something that was not recommended on merit, but
then we have to use some criteria to decide which projects were
more likely to be good and producing new results or new
discoveries and so we had to reduce the total number of
projects that we were going to fund using other criteria. So it
is important to have a combination perhaps of the two.
Mr. Hultgren. My time is winding down. If I could just ask
Dr. Marrett just to follow up quickly, Mr. Chairman?
Chairman Brooks. We are going to do a second round, given
our time availability. Go ahead and----
Mr. Hultgren. Let me just quickly, Dr. Marrett, I know this
is something that Congress has directed you to do. It wasn't
necessarily something internal that you decided. I wonder if
you would have any comment on what your colleagues at the table
stated, what your feelings are about this.
Dr. Marrett. Well, it is not quite true that this is only,
that we only are talking about broader impacts because of
Congressional activity. No, from the outset, from the time that
NSF was founded and started making awards, it has always
considered both the technical side and how we address the
national goals. The idea, the terminology of broader impacts
was really brought in in 1998 when the Foundation said we are
now going to collapse several criteria down to the two
criteria, intellectual merit and broader impacts. The broader
impacts statement or the expectation is that the goals, these
do have to reflect some of the kinds of goals that are expected
by the Nation of the investments in science. So they would
include economic competitiveness, development of a STEM
workforce, increased participation by underrepresented groups,
partnerships between academia and industry. In other words, we
have seen these as very critical for the Nation and reasonable
as an agency thinks about how it will allocate resources. I
think the only difference between the NIH and NSF approach, NIH
does much of the same kind of a thing, it is at a different
level. So the advisory councils take up the questions of how
should the portfolio and how should these other national
considerations be taken into account. They happen to be done as
a part of our regular merit review process in NSF. But this is
not a new idea.
Mr. Hultgren. But wouldn't you say, though, it is an--this
expansion is mandated by COMPETES? Isn't that true and that
makes it unique?
Dr. Marrett. No, what COMPETES did was to say be much
clearer and there are other things with reference to broader
impacts, and so that is what the Science Board is looking at,
that is what the expectation of the Foundation, and that is
what we do believe we can be much clearer on what broader
impacts should mean and how it would come up with indications
appropriate metrics, but again, this was a part of an ongoing
process and we appreciate then the interest of COMPETES, the
interest of others in helping us continue to move forward in
explaining and being accountable, again, to the Nation.
Mr. Hultgren. Mr. Chairman, you have been very gracious. I
yield back. Thank you.
Chairman Brooks. Now for the second round of questions. The
Chair will start first.
Dr. Jose's testimony stated that the merit review process
is at best a system of checks and balances, and this question
is for all of you. Being a member of a government body built to
provide a system of checks and balances, I am interested in
this theory. In the context of the financial circumstances that
Congressman Clarke so notably described that we face here in
Washington, DC, do you or any of your colleagues sitting with
you today have any additional suggestions on how to strengthen
the system based on our need for a balanced approach to federal
funding, particularly to ensure the advancement of science as
our sole or primary funding criteria?
Dr. Jose. Actually, let me actually say something. I mean,
NSF has no specific mandate to do other than basic research and
try to uncover new laws of biology or physics. The NIH has a
very clear mandate to try to find the best way to improve
American citizens' health. The Department of Energy has very
special goals, the Department of Defense, each one of them has
a merit review system that works differently than the one at
the NSF. The NSF is to some extent unique.
Having said that, I think that when I said checks and
balances, what I mean is that the reviewing process of papers,
not just proposals for funding--I mean, we are supposed to
publish, many of us that work in science, our research, our
results. It is very hard to get results published in the best
journals if they are not of the highest quality. The same thing
happens in the merit review process. You are not going to get
funded unless you have the highest quality. So in a sense, we
have like a voting system. Our colleagues actually vote about
what we do, if they think it is good or not, as we get elected
or reelected in Congress or not, I mean, the voters have to
decide if we are doing a good job or not. As researchers, if we
are doing a good job, our colleagues will vote that yes, get
funded, yes, get published, yes, get evaluated for lectures and
so on and so forth. So that is what I mean in the sense that we
have checks and balances.
When a recent result is wrong, the community will find out
that it is wrong because when they try to reproduce that
discovery and they cannot reproduce it, like cold fusion, for
example, it just will die immediately. That is what I meant by
checks and balances. That is how science advances, not in a
straight line continuum.
Chairman Brooks. Thank you for the additional information
with respect to checks and balances. Do you have any specific
suggestions on how we can strengthen or improve the process or
system in order to help ensure that these scarce federal
dollars are being spent on basic science or research rather
than being affected by the criteria that are extraneous to
those primary functions?
Dr. Jose. Yes, of course. I think that, as I said, it is a
human endeavor when people try to review proposals. As
Representative Bucshon said, I mean, there is some element of
human nature that interferes and how do we review proposals,
who is reviewing the proposals, is the best investment that one
can make or not. But we recognize that is a problem we have. We
develop all kinds of checks and balances from the agencies,
from you. You all the time are telling us you have to convince
us that you are doing the right thing, otherwise we won't give
you money to do the research, and it is not perfect and we keep
changing it and correcting it as we go.
Chairman Brooks. Well, my question is, do you have any
specific suggestions on how to improve the process?
Dr. Jose. Yes. I think that--let me--yes. I think that--
actually I have thought a lot about that. There are blind types
of reviews because often the name of the scientist plays a role
in if it is going to get funded or not going to get funded.
Perhaps even the institution, perhaps the project and the area.
I think that--let me see. How do improve the system? I think
that we have to make sure that we choose the right reviewers.
We have to make sure that there is no conflict of interest with
the reviewers being part of the same group of people that are
funding each other. Other than that, I think I have many things
to say but it would take a while to describe.
Chairman Brooks. Well, thank you, Dr. Jose. You mentioned
blind reviews. What are those?
Dr. Jose. Well, for example, if you submit a proposal, you
don't know the name of the author of the proposal, the
institution of the proposal, as you don't know the name of the
referees, and then you will just blindly decide if this is
really good or not whereas the way we do it now, I mean, we
know the name of the reviewer. We can know if they are very
good or not very good and decide--that may influence it a
little bit.
Chairman Brooks. Are you recommending that we have blind
reviews?
Dr. Jose. No, no. We have tried to do that. In fact, there
is a journal trying to do that and they have mixed reviews,
mixed results. I mean, they tried to do that, not having the
name of the author, not having the name of the reviewers, and
they just review it blindly. I am not sure that the end result
was satisfactory as expected.
Chairman Brooks. Dr. Jackson or Dr. Yamamoto or Dr.
Marrett, do you all have any specific suggestions that you
think we should look at in order to try to improve the process?
Dr. Jackson. Well, I do think--I will go back to what I
mentioned in my testimony about giving NSF program officers the
ability to triage proposals. At this point in time, all these
proposals have to go through merit review. That taxes the
scientific community, that taxes the program officers, and we
need to make this--since we can't fund all the proposals, and
there are a lot of excellent proposals--there are more
excellent proposals than we can fund--it makes sense for us to
spend our time thinking about which of those excellent
proposals we should fund rather than spreading our time across
the whole gamut of proposals.
Chairman Brooks. I am sorry. I understand what you have
already testified to, and my question should have been better.
Do you have anything specific in addition to what you have
already informed us of?
Dr. Jose. I am sorry. I must say something about that,
about what Dr. Jackson has. I mean, at the NSF, there are
increasing number of proposals that have been submitted to the
NSF for review and they have decided to assign to the
university to have a limited submission-type thing. You can
only submit two or three grants when there is several million
dollars rather than having 10 or 15. That is a change that was
made and it is a change also that has been made in the NIH as
well.
Chairman Brooks. Thank you.
Dr. Jackson, I don't know if you have thought of anything
in addition to what you have previously shared with us?
Dr. Jackson. It is a messy human process and it would be
difficult to make it more efficient. It is always something you
are struggling to improve.
Chairman Brooks. Well, I have to admit that when I heard it
compared to the election process, that raised some concern.
Dr. Yamamoto or Dr. Marrett, do you all have any other
specific suggestions on what we can do to improve the process?
Dr. Yamamoto. Let me make two comments that I want to frame
as an expansion of what President Obama said when he addressed
the National Academy of Sciences in April 2009. He said that a
basic research project may not work for a year or a decade or
ever, and it is for that reason that public funds, government
funds, should support basic research because the private sector
will always underinvest in that side because when a project
works, it works for everyone. When it doesn't work, of course,
the private sector loses. Maintaining that basic research and
engine is really critical, and so let me make one suggestion,
which is an expansion of what I said about transformative
research, and I will try to frame briefly why I think it really
needs to be a distinct sector of research that is supported in
this country, and then a second comment that really approaches
this question of broader impacts.
So transformative research, I suggested in my comments, and
I will expand very briefly, should be viewed as a different
kind of research. It is destructive of current views rather
than extending them or deepening our understanding of current
paradigms, and to do that requires that we recognize that
special characteristic and have a review process that is very
distinctive. I proposed the transformative research track at
the NIH in 2006. It has been installed in a limited way. I
chaired the first two rounds of review of those applications
when they came in, and an essential feature of I think making
that program successful is actually carving out a different
mode of merit review, one that is driven by reviewers that are
not deep content experts in the areas that are being proposed
but rather are generalists who think deeply, who are able to
recognize really high-impact ideas when they see them and are
able to celebrate the fact that those ideas may actually bring
down their current ways of thinking about a process.
Luckily, such scientists exist in this country. They can be
found. They can be identified. They are willing to participate
in this way. They are the kinds of people that are finally
getting challenge grants from the Gates Foundation that are
proposing--making proposals to the transformative track and are
willing to review in that way. So it takes a different kind of
review that recognizes that you don't bring in content experts
who made the paradigms because they are going to be immediately
critical of someone who comes in and says I think the current
paradigm is wrong. I could expand on that further.
Let me move to the second point, something I didn't
mention, didn't talk about at all, and in a sense, you could
view this as being outside of the merit review process, but I
think not, and that is the idea of recognizing, especially with
this broader impacts mandate, that basic research has advanced
to a point where it can increasingly be applied, and since I
work in the biomedical sphere, I see this all of the time with
the opportunities of our faculty in my own laboratory to be
able to move fundamental research to application, but there is
the so-called valley of death that I am sure you have heard
about, that especially in the current economy has made it very
difficult to be able to do the few experiments that are needed
to make it evident to pharmaceutical companies and the
biomedical sphere that there is a worthy investment there. I
think that we have an opportunity to be able to extend the
continuum to be able to build new interfaces between academic
basic research funded by the NSF and other agencies and build
public-private partnerships that can more effectively move
ideas into application, and in that way be able to really
directly approach this challenge of addressing broader impacts
that the COMPETES Act really challenged all of us to be able to
do.
Chairman Brooks. Thank you, Dr. Yamamoto.
And Dr. Marrett, I don't know if you have any suggestions.
It occurred to me that if there some, you probably would have
already implemented them as Deputy Director and Acting Director
but nonetheless, if you have any other suggestions on how you
think we can improve the process, please share it.
Dr. Marrett. And I take your question to mean, what
especially might the subcommittee do, and I have two
recommendations there. One is to share broadly the
understanding of the process of merit review. I am not sure how
widely the process is understood by, I wouldn't just say
Congress, or general public, yet if we are drawing on public
resources, we need to make sure that that process is
understood. So this hearing represents a good approach for
broadening knowledge about the process.
The second recommendation I would make for the Subcommittee
is, I hope you would be open to our sharing with you the
experiments, the pilots that we are trying. We do say that
there are areas where there can be improvement. Let me take the
case of potentially transformative research. We are not all
that satisfied but we are experimenting with, do we advance the
potentially transformative research by identifying proposals
once they have come in, by soliciting new ideas, by thinking of
the sorts of experience from NIH. We are trying all of that. We
would welcome then coming back as we have learned from those
experiments to say more about what should be advanced. We are
not quite ready yet to say these are the things that we think
ought to be put in place. And finally there, I would say
whatever, we think that at the core, that core has to be
maintained. I mean by that core that commitment to merit and so
whatever the sorts of changes, the changes are around the
margins in many respects. They never should undermine the
importance of relying on the meritoriousness of the idea and
the approach that is being proposed. So thank you.
Chairman Brooks. Thank you, Dr. Marrett.
Mr. Lipinski, thank you for being patient.
Mr. Lipinski. Thank you, Mr. Chairman. I think that--I am
glad we went to a second round of questions because I think we
really got into some really good ideas here about how, you
know, to make improvements to the system, and I think actually
I would ask--the Chairman asked Dr. Marrett if we could
possibly do, maybe a tutorial, and get into how exactly this
process does work, because we are talking about it here but
even someone who has presented a proposal and has gotten a
grant, I would like to really know better how the whole system
actually works and maybe the opportunity for members to sit
down and go through a tutorial. I think that would be a great
idea for all of us.
Chairman Brooks. That is a good suggestion, and I welcome
it.
Mr. Lipinski. Because I think there is a lot to the
process, and we are talking about the process but I think there
is a lot of details to the process and things that probably
each of us doesn't completely understand and might be able to
get a better understanding of that.
Two things I want to get into. The first one, Dr. Jackson
had--one of the main proposals you had to sort of lessen the
work is for the program officers to be able to essentially
triage and put aside proposals that come in that the program
officer doesn't see there being merit to. I wanted to get a
better understanding of what you are saying how that would be
done, and then I will get Dr. Marrett's response on that. Are
you saying then that immediately set aside these proposals
without doing any sort of response to them? Because right now
all the proposals get a response from the reviewers. It would
seem the only way you could save time really is if the program
officer just put these aside without any comments and just send
it back and say rejected right off the top. In that way, you
would not be getting--you would not be giving the researchers
any feedback at all. I was just wondering how exactly you saw
that working.
Dr. Jackson. Right, and you are right. That is the downside
to triaging proposals that clearly do not--clearly aren't up to
snuff, so to speak, is that they miss the opportunity for a
certain amount of feedback, and there are several ways you can
handle that. Certainly for the petroleum research fund at ACS,
there are certain ways we deal with that. One is that you can
ask for a review from the program officer who in our case and
in NSF's case is an expert in the field so that program officer
does have to write up something about what is a problem with
the project and this sort of thing. Another thing that we do
with the Petroleum Research Fund is to provide workshops
particularly focused at younger researchers, but anybody can
come, that talk about what makes for a good proposal and so we
are training the new generation and new people to the field how
to write proposals, what we are looking for and what needs to
be done, so we are doing this to counteract maybe some of the
individual feedback that could be--that would be given if it
was sent out to review.
But you have to understand that a proposal goes out not
just to one reviewer but to a number of reviewers, so there are
possibilities too you could have; if you realized that this is
not going to be funded, you could ask one reviewer perhaps to
provide a review, to provide advice to the proposal writer, or
there are a number of other things you can do that would
decrease the amount of work for each proposal. But in the PRF,
we recognize that these are just not going to get funded and so
we spend a limited amount of time with them and try to do our
education separately.
Mr. Lipinski. Yeah, I understand the need to save time and
I think we all want to find ways, because there is--reviewers
are overburdened and it is hard to get people to serve in that
capacity. I just want to give--and I am not, you know, being
critical of that proposal. I want to see what Dr. Marrett would
have to say about that.
Dr. Jackson. Well, and we also--one more thing. We also
encourage those proposal writers to call our program managers
and talk to them about their proposal so there can be some kind
of exchange and learning going on.
Dr. Marrett. We are absolutely intrigued by anything that
could help reduce the burdens on program officers, on
reviewers. The number of proposals we are getting, the number
continues to climb and it is very difficult. On the other hand,
there is the kind of view, a very pervasive view that a part of
the whole process is to help give good feedback, especially
help in the cultivation of new investigators and thus our
program officers are hesitant simply to say let us try to move
this as quickly as possible. Because I think there is a
responsibility to help people understand where the problems
might have been, to help cultivate the ways in which they might
develop stronger proposals. It is a dilemma then of how to have
an effective, efficient strategy that is at the same time an
educational strategy that we are all seeking to accomplish.
So as I said, we welcome ideas but we are trying to examine
them in the context of the larger kinds of things that we seek
to do in the whole development of the science and engineering
community of the United States.
Mr. Lipinski. And I wanted to go on to another, if the
Chairman will indulge me.
Chairman Brooks. Feel free.
Mr. Lipinski. I wanted to ask about conducting the
committee reviews in virtual environments. It is something I
mentioned in my opening statement. I personally have had no
experience at all. I have no avatar. I have not gotten into
this so I can't really speak from any expertise here, but I
know this is something that has been looked at, and I was
wondering if--I will start with Dr. Marrett. I want to ask all
the other panelists about their opinions on using this as a way
to make it easier in some ways in doing this in a virtual
environment. You don't have to fly people into Washington to do
this, and other possible merits to using this system. I would
also think that the blind review process--and also having
people come together in a way that they don't know each other
that could impact, change the dynamics of the group. So, Dr.
Marrett, we will start with you.
Dr. Marrett. Yes, we are looking very much at virtual
environments. There are experiments that have been taking
place, and these environments, thinking of virtual panels
especially, we have been thinking of them for two reasons of
reducing some of the cost, the cost for reviewers, and for
potentially expanding the pool of reviewers. In other words,
there are some people who would find it difficult to do the
traveling since most of these are here in this area. We thought
this would be a way then to bring more people into the process.
What we are looking at, what are the sorts of things that
have to be taken into account. There are matters of security,
for example, that you have to make sure that you have been
abiding by. You raised the questions about the kind of
interpersonal dynamics. There, we are actually drawing some of
the research being sponsored by our Directorate for Social
Behavioral Economic Sciences where there are studies of human
dynamics, enough for us to understand how those dynamics might
come into play and actual studies specifically of virtual
environments. All of this then we think we need to take into
account and that is why we are experimenting but we are trying
to bring the knowledge to bear as much as we can to ask what
would be the consequences and where there might be some savings
but where there might be some losses if we are not careful
about how we try to institute the idea of virtual environments.
Mr. Lipinski. Any other panelists have any opinions here? I
will start with Dr. Jose.
Dr. Jose. Yeah, let me say that since I am quite old and I
was funded for many years, I can see the evolution of how
proposals are reviewed by the NSF. For a long time, there was
only by mail review, and they only waited until the reviews
were done and they analyzed what the kind of grade each one of
the reviewers gave. Then they went into having a combination of
having mail and panel reviews. And I can tell you that by
having done both, there is a significant change in the actual
assessment of the proposals that arises from meeting with
people live than just from reviewing it by yourself.
Okay. Now, with the virtual analysis they have to do, it is
very important to know what kind of human dynamics are going to
influence to some extent the reviewing process, and the reason
I am saying this is that one of the committees that I was a
member of, the mail reviewers gave much better grades to the
proposals than when we sat down and discussed the proposals and
interacted and listened to everybody else live, okay, so what
effect would it have to have it virtually and not face to face,
one has to wonder. But it is course a very useful and important
thing to explore, yes.
Mr. Lipinski. Who else? Dr. Jackson?
Dr. Jackson. Living in Albuquerque, New Mexico, which seems
to have no direct flights to anywhere, the idea of having
virtual meetings is very exciting to me and to my colleagues
because we are more likely to be able to be involved in these
sort of--you know, become more--participate more in these sort
of activities. But also because of that, I have done a lot more
of this sort of virtual meetings and this sort of thing and I
would say that yes, perhaps when you first start doing it, it
is a little unusual but it is remarkable how with time you can
become very accustomed to it, and I highly encourage NSF to
consider these sorts of possibilities because I think the kind
of expertise they could tap into will only become greater and
richer.
Mr. Lipinski. Yeah, obviously I brought up avatars. You
don't have to do it that way, but Dr. Yamamoto?
Dr. Yamamoto. I agree. I think there are two drivers for
this increased--rapid evolution toward electronic environment
for review. The first is that those of us who didn't grow up
using those media are disappearing and being replaced
fortunately by people who have screens in front of them their
entire lives and are very comfortable working in that
environment. And the second is the technology development is
marching along very quickly and there are electronic video
conferencing modes that you are looking across the room at a
screen and it looks exactly like we are looking across the room
at you. And so the opportunity for interactions, which I agree
with Dr. Jose, is essential in being able to come to this sort
of agreement about the scientific merit of an application that
can really occur increasingly seamlessly in this sort of
environment.
Let me make one further comment, because you asked,
Congressman Lipinski, about this idea of blind reviews that Dr.
Jose had mentioned, and just say there again here is the
spectrum of experiments ongoing in different agencies. You may
know that the Howard Hughes Medical Institute does actually the
opposite of a blind review. In fact, rather than choosing
scientific projects as the agencies that we all represent and
are working and do, they simply choose people, and so the
identification of the person is the whole ballgame for the
Howard Hughes Institute. The Gates Foundation in their initial
round of reviews for their Grand Challenges grants, which I
participated in several times, is blinded to the identity of
the applicants, and after they make it through the first round,
then that identity is revealed, and then NSF and NIH, the
federal agencies that I am familiar with, the identity is
always known.
There is a risk of conflict of interest or other kinds of
biases that can be introduced at that point, but let me say
that having been experienced in review for at least NSF and NIH
extensively that I don't actually see that actually occurring,
and instead knowing the identity of the investigator, in fact,
building it in as an explicit criterion for the merit review
process, turns out to be very important. There have been
studies done and arguments made that in fact the strongest
predictor of success of a given scientific proposal is in fact
the track record of the investigator who is making the
proposal. So in that sense, knowing that person, their past
capacity for doing innovative work--and let me just insert here
that this does not mean--because very often that statement that
I just made raises concern that this would bias against young
investigators. In fact, it does not, and I think the clearest
view of that is that I have been involved, and I am sure my
colleagues have as well, for years in making judgments about
graduate fellowships, postdoctoral fellowships, young trainees
who have very little in the way of conventional track records
but it is very easy, I think leaving those meetings, those of
us who are on those panels, feel very confident that we have
been able to make the right choices for identifying the best
candidates based on what they have done before, even though
they have been in a training mode and very different from being
an independent investigator, and data prove that out that those
people that win such fellowships and awards go on to be able to
do very good work.
And so my view is that having that identify early on at
least--I think the Gates Foundation strategy actually works
quite well--but being able to unmask those individuals early
on, to be able to see their track record and build it into the
merit review process is important.
Mr. Lipinski. Well, let me throw my own two cents in here.
I am not always certain from--I think it depends on the field
possibly. In my background as a political scientist, I was not
always convinced that it was helpful to the advancement of
knowledge to know who was proposing something because sometimes
I think that had an influence that wasn't good on review
processes, and I am talking more about journal reviews. But
that is my own two cents. It is very interesting to hear,
though, Dr. Yamamoto's and everyone else's experiences with
this, and I thank all of you very much and I especially thank
the Chairman for his indulgence here and for this hearing.
Chairman Brooks. My pleasure, but we are not finished yet.
I am going to follow up on something that Mr. Lipinski noted,
and that is, we have talked about the virtual panels but what
is the cost of in-person panels?
Dr. Marrett, do we have any kind of transportation costs or
hotel costs or is there a budget line item, or how much does it
cost per panel or how many panels are there per year? Can you
share some information with us about that?
Dr. Marrett. We will get that information to you. We
actually have done the calculations of what the costs are
associated with panels, and those do include the travel costs,
of course, and there are hotel issues there, and we had started
as we were looking at what would be the savings from virtual
panels, and I should mention that in the case of avatars, we
have had reviews done in Second Life. But in terms of the sorts
of costs associated with virtual panels, we are not quite sure
because there are other kinds of things. We just haven't
explored enough now. We can't do the comparative information
but we can get to you how many panels, what the cost is of an
average panel for the National Science Foundation.
Chairman Brooks. And if you can do a comparison format,
too, I would anticipate that the virtual panels are
significantly less expensive. But if we have some kind of
comparison model, that would be beneficial.
Dr. Marrett. We will try that, but why I say it is not
necessarily going to be much cheaper because there is the
question of the technology that we might have it on our side,
but it is also got to be at the other side, and one of the
things that has been talked about will be using regional panels
where there will be cost of travel to the regional panels as
well, the enhanced security costs that come into play, so we
are not quite sure at the outset that there would be lots of
savings that could happen over time in a number of ways. That
is why I said we won't have for you--it will be very rough
kinds of estimates when it comes to virtual panels but we can
tell you about the investments currently made in the in-person
panels.
Chairman Brooks. Thank you, Dr. Marrett.
As a short concluding statement, I don't know if you all
have access to this, perhaps Dr. Marrett already has it, but
some kind of compilation of success stories with NSF, the kind
of information that this Subcommittee can share with the public
or the Members can be familiar with so when asked, we can share
it with the public. I am sure you are all familiar with the
relatively famous shrimp on a treadmill commercial that has
been running over the last month or two, and I am afraid that
that may give the general public some dissatisfaction with the
way their tax dollars are being spent, so I think it is
important to also have information that we can share with the
public that shows success stories and the kinds of returns on
the investments that we are asking taxpayers to make.
Having said that, I want to thank the witnesses for their
valuable testimony and the Members for their questions. If the
Members of the Subcommittee have additional questions for the
witnesses, you are free to submit those, and we will ask you to
respond to those in writing. The record will remain open for
two weeks for additional comments from the Members.
The witnesses are excused and this hearing is adjourned.
[Whereupon, at 11:43 a.m., the Subcommittee was adjourned.]
----------
Answers to Post-Hearing Questions
Answers to Post-Hearing Questions
Responses by Dr. Cora Marrett,
Deputy Director, National Science Foundation
Questions submitted by Chairman Mo Brooks
Q1. As requested during the hearing, please provide the total amount of
federal funding awarded to those proposals rated from ``Poor'' to
``Good or Very Good'' for FY 10. While you touched on it briefly at the
hearing, please also expand on why those proposals received funding
over proposals rated ``Very Good to Excellent'' and ``Excellent.''
A1. All funded proposals are determined to be highly meritorious based
on a combination of reviews by individuals, panel deliberations and
program officer evaluation. On average, NSF proposals are reviewed by
four to six individuals, depending on the type of review. All reviewers
are chosen for their specific expertise related to the subject, and the
collection of persons brings different points of view to the decision-
making process. When the average reviewer score is in the ``good''
range, it often represents a split of ``excellent'' or ``very good''
reviews with some ``fair'' or ``poor'' review scores that lowered the
average. It is important to note that the proposal rating data included
in the annual NSB Merit Review Report reflects proposal ratings before
panel deliberations and, therefore, not the final panel evaluation. The
panel evaluation is based on a thorough discussion of the proposal's
strengths and weaknesses in the context of the full set of proposals
being reviewed; this discussion forms the basis for placing a proposal
in a particular category. These in-depth discussions can often clarify
perceived weaknesses and result in a proposal being recommended for
funding despite the initial average review score. Likewise, some
proposals with high average review scores are not recommended by panels
as a result of a detailed discussion that uncovers weaknesses that
might not have been reflected in the initial reviews.
The expertise of the NSF Program Officer making the final
recommendation is also an important voice in the process. Program
Officers take into consideration other factors that might not have been
considered by expert reviewers. For example, proposals for innovative
new ideas often use unproven methods or techniques that might be
considered risky by reviewers and panelists. Risky proposals often
result in transformative research that accelerates the pace of
discovery. Although Program Officers consider concerns about risk
expressed by panels, they also see the value of funding potentially
transformative research. Proposals that do not review well at panel
because the methods are unproven or risky can be given small awards to
allow enough work for a ``proof of concept.'' Program Officers will
also consider broader impacts that might not be obvious to reviewers,
such as an infrastructure need that will serve a large number of
people. There are also many dimensions of portfolio balance that
influence the final recommendation. In addition to maintaining a
diverse scientific portfolio, Program Officers strive to fund proposals
from diverse institution types across the U.S., and from both young and
experienced investigators.
As explained above, the reviewer rating data reported in the Merit
Review Report are only initial reviewer ratings, which is just the
beginning of the merit review process leading to a final dertermination
of whether any given proposal should be funded. Initial reviewer
ratings do not reflect panel deliberations or Program Officer input. In
FY 2010, NSF funded approximately $46K in proposals initially rated as
poor, $21M in proposals initially rated as fair, $818M in proposals
initially rated as good, and $1.6B in proposals initially rated as very
good. Following panel discussion and analysis, all of these proposals
were determined to be highly meritorious nothwithstanding their initial
rankings.
Q2. In your testimony, you described experimenting with innovative
approaches to identify potentially transformative research. Please
expand on the ``ideas factory sandpit'' approach and tell us what you
are learning from it and other novel approaches.
A2. NSF has experimented with an approach to identifying potentially
transformative high-risk research that it is now calling ``Ideas Lab.''
The Ideas Lab is closely modeled on the ``sandpit'' process developed
by the UK's Engineering and Physical Sciences Research Council (EPSRC).
The essential element of the Ideas Lab is an intensive interactive
residential workshop involving 20-30 participants, with the aim of
developing bold, often risky, new approaches to grand challenge
questions in areas that could benefit from creative ``out-of-the-box''
thinking. A fundamental aspect of the EPSRC sandpit that has been
incorporated into the Ideas Lab is the use of a highly
multidisciplinary mix of participants (including active researchers
from diverse fields and potential users of research outcomes) to
address specific research challenges. A description of the process used
follows. Slight variations should be anticipated as NSF gains
experience with the process and as it is adapted to different topics.
To identify potential participants, a solicitation is issued that
includes an open call for participants. Interested individuals submit
short preliminary proposals that include concise descriptions of their
pertinent experience and expertise as well as their communication
skills, collaborative activities, and creative abilities. A panel of
reviewers evaluates the applications and identifies a pool of potential
participants from a range of disciplines and backgrounds who have high
potential to work at the interface between disciplines and to develop
new and highly original research ideas. NSF Program Officers make the
final selection from the pool to ensure a diverse mix of participants.
Industrial psychologists provide advice that guides but does not decide
participant selection.
During the multi-day Ideas Lab workshop, participants interact in
unconventional new ways to develop innovative research project ideas on
the selected topic area. Professional facilitators, experienced in
sandpit-like activities, integrate creative problem-solving techniques,
iterative project-development activities, and real-time peer review by
both participants and a resident panel of experts (called the mentor
group) to advance the most innovative ideas. Outcomes at the end of the
workshop are research project concepts that vary in scale and scope in
addressing the grand challenge topic of the Ideas Lab. At the end of
the Ideas Lab, the panel of reviewers provides a consensus report
summarizing its evaluation of each project concept. Based on this
review, the Program Officers invite the submission of full proposals
for some, none, or all of the project concepts. The invited groups have
six to eight weeks to submit full proposals, which are then reviewed by
the same panel of mentors using NSF's two merit review criteria. Based
on that review, NSF then makes a decision whether to fund some or all
of the proposals. Taking part in the Ideas Lab does not mean that the
participant is guaranteed to be funded under an award resulting from
the Ideas Lab process.
Experimentation with the Ideas Lab is still at an early stage. A
total of four Ideas Labs have been conducted to date. The first three
resulted in 12 awards and the fourth is currently awaiting full
proposal submissions. Feedback from participants in the Ideas Labs has
been positive. However, the resulting funded projects are still in
their beginning phases. As they progress, NSF will look at the outcomes
of these projects to evaluate whether they resulted in transformative
research.
Q3. Researchers will send in proposals whenever they have an idea that
they would like to have funded. However, NSF also puts out
solicitations for specific areas of research. Please explain how
decisions are made on what types of research areas warrant a specific
solicitation from the Foundation? What happens if the Foundation does
not receive high quality proposals for a solicitation? Do you pick from
what you have or do you rework the solicitation?
A3. Solicitations are formal NSF publications that encourage the
submission of proposals in specific program areas of interest to NSF.
Solicitations are generally more focused than program announcements,
and normally apply for a limited period of time. Ideas for new
solicitations can be initiated within Divisions by Division Directors,
at the Directorate level by Assistant Directors, or by groups of
Assistant Directors who see the need for a new cross-cutting activity.
Program Directors also commonly suggest ideas for new initiatives. The
initial spark for a new initiative often comes from interaction with
the scientific community through scientific meetings or other
communications. When an idea for a new solicitation is suggested, a
working group is formed that includes Program Officers who are expert
in the research area. The working group collaborates on a detailed plan
for the new solicitation, which is then discussed and reviewed by
various levels of leadership before approval.
Specific factors that are considered when deciding whether to
develop a new solicitation include the following:
the intellectual reason for the Program, activity, or
initiative;
whether the new activity(ies) will generate sufficient
interest in the targeted community;
whether the Program, activity, or initiative is new, how
it supports the long-range goals of the Directorate and/or NSF;
whether the size of the effort justifies a separate
announcement and/or competition;
the total funding available for the proposal competition,
including estimated proposal receipts and anticipated number of awards
and funding levels;
cross-Directorate participation (and implications) in the
Program.
Program Solicitations often specify submission limits, award
conditions or reporting requirements, and provide supplemental proposal
preparation guidance in addition to what is in the Grant Proposal
Guide. Program solicitations also provide specific review criteria when
reviewing proposals. In cases where the Foundation does not receive
high quality proposals for a solicitation, we decline the proposals
that are not of high quality. The solicitation could be revised and
recompeted to attract high quality proposals. Because NSF funding
opportunities generally attract more high quality proposals that we can
fund, this would be a rare occurrence.
Q4. After reviewing the flow chart for the proposal and award process
and timeline, the Directorate Assistant Director seems conspicuously
absent from the process. Please describe what the role and
responsibilities of the Assistant Director are in the funding process,
both from a programmatic and overall agency funding priorities
perspective.
A4. While Assistant Directors (ADs) are not involved in the day-to-day
review and processing of proposals submitted to the Foundation, as
described in the NSF Grant Proposal Guide Exhibit III 091 (http://
www.nsf.gov/pubs/policydocs/pappguide/nsf1100/gpg-3ex1.pdf),
they fulfill a vital role in the overall funding process. (See the
attached referenced flowchart.)
ADs are knowledgeable about the award portfolios in their
directorates, but they are not involved in the decision-making process
itself, because their role is to set the vision and strategic goals and
objectives for the divisions/offices that report to them.
Assistant Directors also play an important role in the formal
reconsideration process. If a PI is dissatisfied with the explanation
they receive for why a proposal has been declined, he/she may request a
reconsideration of the decision. ADs/Office Heads are responsible for
responding to these requests, and review the proposal record to
determine whether NSF's review of a declined proposal was fair and
reasonable, substantively and procedurally. If they were involved in
the decision-making process, the would have a conflict of interest in
responding to any official reconsideration request.
Q5. How does the Foundation leadership ensure that Program Officers
``produce and manage a balanced portfolio of awards that address a
variety of considerations and objectives'' as the FY 10 NSB Report
states?
A5. Portfolio balance is reviewed at a variety of levels at different
times during the decision-making process. Program Officers consider
many dimensions of portfolio balance when they are making decisions
about what proposals should be recommended for awards. Some of the
factors that are considered include: balance across disciplines and
subdisciplines, award size and duration, awards to new investigators,
geographical distribution of awards, awards to different types of
institutions, innovative/potentially transformative projects, projects
with elements of risk, inter- and multidisciplinary projects, projects
that integrate research and education, and projects that are relevant
to agency mission or national priorities. Division Directors review the
recommendations by Program Officers for portfolio balance before they
concur with the award recommendations. Portfolio balance is also
reviewed by our Committees of Visitors who review programs at three-
year intervals. Some programs also contract for external evaluations of
their portfolio periodically to inform how they might make changes to
their programs. The results of both COV reports and external portfolio
analyses are reveiwed by Directorate Advisory Comittees.
Q6. According to the FY 10 Board Report, NSF awarded approximately five
percent of its annual budget to federal agencies and laboratories. What
kinds of awards were these and did they go through the formal merit
review process?
A6. The 2010 Merit Review Report to the National Science Board
reported that NSF funded $351.2M in awards to Federally Funded Research
and Development Centers (FFRDCs). The majority of this funding went to
two organizations that build and manage large astronomy facilities for
university consortia: Associated Universities Inc. (AUI) and the
Association of Universities for Research in Astronomy (AURA). AUI
received $111M in funding associated with the National Radio Astronomy
Observatory, the Atacama Large Milimeter Array (ALMA), and other
related projects. The Association for Universities for Research in
Astronomy (AURA) received $234.3M for a number of projects including
building the Advanced Technology Solar Telescope and the Advanced
Technology Solar Telescope and operations and management of the Gemini
Observatory, the National Optical Astronomy Observatory, and the
National Solar Observatory. In addition, $0.4M in funding went to fund
several much smaller projects through another FFRDC, Aerospace
Corporation. Proposals submitted to NSF by FFRDCs go through the same
merit review process as other proposals. The large awards for building
and operating large facilities go through a very lengthy and detailed
review process that includes site visits, cost reviews, design reviews,
and approval by the National Science Board.
The $351.2M reported in the Merit Review Report also includes $5M
in contracts to fund the Science and Technology Policy Institute (STP)
operated by the Institute for Defense Analyses. STPI provides rigorous
and objective analysis of science and technology (S&T) policy issues
for the White House Office of Science and Technology Policy (OSTP) and
other offices and councils within the Executive Branch of the U.S.
Government and federal agencies. IDA was selected to operate STPI in
2003 following a competition and undergoes reviews at five-year
intervals.
Note that the funding to FFRDCs described in the FY 2010 Merit
Review Report did not include contract funds to or from other federal
agencies through interagency agreements.
Q7. What kind of peer reviewers are coming from industry, non-profits,
and government? Do they all have Ph.D.s? What role does a panelist from
the government play? What qualifications do they have?
A7. Reviewers are chosen for their expertise in areas covered by the
proposals that they are asked to review. For research proposals,
reviewers are typically researchers in domains of science relevant to
the topic of the proposal. In the review of proposals for facilities,
in addition to the reviewers who can provide input on the research
impacts, technical feasibility and soundness of the faciltity's design,
Program Officers may also include reviewers with expertise in other
relevant fields such as project management, systems engineering,
complex acquisition processes, architectural design, etc.
In many scientific and engineering disciplines, some of the leading
researchers work in industry, non-profits, government laboratories and
FFRDCs. Examples include computer science and a number of others. Such
researchers are very much the peers of their academic counterparts and
are included in NSF's pool of peer reviewers. Some of these researchers
may have spent part of their research careers in academia and some in
industry or an FFRDC, allowing them to bring important perspectives on
the state of the art in the different environments and their potential
broader impacts. These reviewers tend to have the qualifications that
are typical for the research communities to which they belong. In many
disciplines, this is often a Ph.D., although occasionally it is simply
long experience doing cutting-edge research. Typically, what signals
the expertise of a researcher is his or her record of research
achievement, including significant publication in peer-reviewed
journals and conference proceedings.
Reviewers from government and industry are often more familiar with
project management and complex acquisition processes than some of their
academic counterparts, and so such individuals are sometimes asked to
bring this expertise to review teams looking at proposals for research
facilities. Such individuals may or may not have Ph.D.s.
In general, what NSF looks for in its choices of reviewers is
expertise in the topics under review.
Q8. How does the Foundation train reviewers to prevent the phenomenon
of implicit bias?
A8. The frontline of the merit review process are the approximately
520 NSF Program Officers (POs) who select experts who can provide the
information needed to make a recommendation in accordance with the
National Science Board (NSB) approved criteria for selection of
projects. Program Officers are trained on conflicts of interest, the
importance of getting a diversity of perspectives, and guarding against
the influence of subjective or biased input.
Proposals submitted to NSF receive rigorous and objective treatment
and POs ensure that this takes place. Proposals are evaluated by
independent reviewers consisting of scientists, engineers, and
educators who do not work at NSF or for the institution that employs
the proposing researchers. NSF selects the reviewers from among a pool
of experts in each field, and their evaluations are anonymous. On
average, about 50,000 experts give their time to serve on review panels
each year. POs ensure that there is diverse representation within the
review group. The goal is to achieve a balance among various
characteristics, including type of organization represented, reviewer
diversity, age distribution and geographic balance.
The reviewer's job is to provide advice to NSF on which projects
are the highest priorities. This competitive process ensures that many
voices are heard and that only the best projects make it to the funding
stage. When someone is asked to review a proposal (either as an ad hoc
or panel reviewer), they are provided with information on the
confidentiality of the process and the potential for conflicts of
interest. Panelists sign a ``Conflict-of-interests and Confidentiality
Statement'' whenever they participate in a panel. For ad hoc reviewers,
by submitting their review, they are acknowledging that they've been
informed of such policies. Again, NSF POs are responsible for assuring
that appropriate, qualified merit reviewers are selected and the entire
process is overseen by Section Heads and/or Division Directors who
supervise the Program Officers.
Q9. The 2010 reauthorization of the America COMPETES Act required the
Foundation to ``Apply a Broader Impacts Review Criterion to achieve''
various goals. Witnesses at the hearing raised some concerns with the
draft criteria that is currently being weighed by the Board. Have the
goals, now specified in statute, been considered in the past when
making funding decisions? Are the peer reviewers taking these goals
into consideration during their review or are the Program Officers
simply tasked with this responsibility? Based on the work being
conducted by NSB and NSF and your experience with the merit review
process, is the legislative requirement achiveable and is it necessary?
A9. NSF strives to invest in a robust and diverse portfolio of
projects that creates new knowledge and enables braekthroughs in
understanding across all areas of science and engineering research and
education. To identify which projects to support, NSF relies on a a
merit review process that incorporates consideration of both the
technical merits of a proposed project and its potential to contribute
more broadly to advancing NSF's mission ``to promote the progress of
science; to advance the national health, prosperity, and welfare; to
secure the national defense; and for other purposes.'' In 1997, these
considerations were put into action through the two primary merit
review criteria of Intellectual Merit and Broader Impacts. Each
reviewer must consider, and address, both merit review criteria for
each proposal.
As noted in your question, the importance of incorporating
consideration of potential broader impacts in deciding which projects
to fund was re-emphasized in the America COMPETES Reauthorization Act
of 2010. Having the reinforcement of Congressional support on the
fundamental nature of the Foundation's Organic Act is always an
important, and appreciated, development. However, there is a danger of
viewing the Broader Impacts criterion as a ``one size fits all''
checklist, which would be a mistake.
This COMPETES Reauthorization identified a number of societally
relevant outcomes that may result as a consequence of NSF-funded
research. Stated more broadly, these outcomes include (but are not
limited to) increased participation of women, persons with
disabilities, and underrepresented minorities in STEM; improved STEM
education at all levels; increased public scientific literacy and
public engagement with science and technology; improved well being of
individuals in society; development of a globally competitive STEM
workforce; increased partnerships between academia, industry, and
others; increased national security; increased economic competitiveness
of the United States; and enhanced infrastructure for research and
education. These represent examples of societally relevant outcomes.
The NSF will strive to clarify that these examples should not be
considered either comprehensive or prescriptive, and that investigators
may include appropriate outcomes not covered by these examples.
Responses by Dr. Keith Yamamoto,
Vice Chancellor for Research, University of California San Francisco
Questions submitted by Chairman Mo Brooks
Thank you for these insightful questions, Chairman Brooks.
Q1. In basic science research, when one cannot possibly know what the
outcome may be, how can one establish what the broader impacts will be?
A1. Basic science research, by definition, is untargeted, driven only
by the curiosity, intuition, background and expertise of the
investigator(s) who define a question and develop a plan to answer it.
For such investigations, it is impossible to ascribe broader impacts at
the time of merit review. Only after the work has been completed, often
long after its completion, can the wisdom of retrospection be applied
to define broader impacts. Thus, inclusion of the broader impacts
criterion in the merit review of grant applications is misguided and
damaging to the integrity of the merit review process.
Q2. Having sat on review panels, please tell me how heavily the broader
impacts of a proposal are weighed when rating the proposals you were
considering? Did the evaluation of broader impacts have any effect on
the evaluation of the intellectual merit of the proposals?
A2. My experience on NSF review panels predated the 1998 development
of the broader impacts criterion, so I cannot provide experience-based
perspective here. However, as indicated in my testimony, any departure
from singular focus on scientific merit, as mandated by this assessment
of broader impacts, erodes the evaluation of the intellectual merit of
the proposals.
In this regard, I wish to comment in particular on the point raised
by Ms. Jackson that the broader impact criterion provides a metric to
distinguish proposals that are equally meritorious on scientific
grounds. She viewed this metric as ``an important role for the broader
impacts criteria and should be kept as such because there are clearly
far more proposals that are excellent than can be funded.'' With due
respect, my opinion is sharply different. Application of this approach
would discriminate against true basic science research as defined in my
response to your first question above, and favor funding of work with a
clear application. To the extent that this evaluation metric would
become known in the scientific community, this approach would lead to a
decline in basic science proposals. Fundamental discovery remains
absolutely essential for scientific progress, and would suffer
significantly from negative discrimination applied at the merit review
stage.
In NIH peer review, as noted by Dr. Marrett, this conflict is
avoided by placing broader impact considerations in a separate review
process carried out by separate review committees, thus maintaining the
singular focus of merit review.
Responses by Dr. Nancy B. Jackson,
President, American Chemical Society
Questions submitted by Chairman Mo Brooks
Q1. In basic science research, when one cannot possibly know what the
outcome may be, how can one establish what the broader impacts will be?
A1. At the National Science Foundation (NSF), all proposals are
evaluated for intellectual merit and broader impacts. Two major
determinate factors in estimating the long-term broad impacts of
scientific proposals are the amount of resources available to fund
grants and the standards of the Request for Proposal (RFP).
NSF receives many more meritorious proposals than it could ever
fund. While a proposal with weak intellectual merit has no hope of
getting NSF funding, many proposals are rated ``excellent'' with strong
intellectual merit and still do not get funded because of the stiff
competition. The broader impacts criteria allow decision makers to
gauge which research is the most urgent or has the greatest relevance
to improving the quality of life. This merit review process enables NSF
to ensure that precious R&D money goes only to the most relevant R&D
needs.
As anyone with a retirement fund has been told, managing a
portfolio is critical to its long-term strength. Financial advisors
stress that it's important to find the right balance between solid
performing stocks and riskier investments that may provide higher
returns. Managing a research portfolio is similar.
The research manager, whether working in industry or at a
government agency, strives to find the right balance between science
that will deliver steady advances and ideas that are out of the box,
but could result in game-changing developments. This point is
especially important in times of restricted funding. It's human nature
to make more conservative choices and be risk-aversive when times are
tough. Expanding resources allows grant reviewers to ``take a chance''
on promising but less established scientific ideas that may result in a
major scientific leap when compared to a safer but more conservative
proposal.
For every 100 grant proposals a program officer reviews, a small
number (perhaps 10) will be of such high quality that it is obvious
they should be funded. Another 50 proposals will be recognized as
clearly not competitive; however, they still must be addressed through
the process. An agonizing choice must then be made over the 40
proposals in the middle. These include proposals that may be considered
excellent or very good. In fiscal year 2010, the NSF award rate was 23
percent. In our analogy of 100 proposals, this would mean that 13 out
of the remaining 40 would be funded.
Unfortunately, it has proven difficult to pinpoint from where the
next game-changing scientific moment will occur. In 1994, when NSF
funded the Stanford Integrated Digital Library Project, the Web was a
tiny portion of the Internet, which was dominated by file sharing and
gopher sites. This innovative grant to fund research into developing
methods for searching Internet databases eventually lead to the
creation of a $200 billion dollar company (Google, Inc.), which
revolutionized almost every aspect of the way humans interact with each
other and information. However, it would have been difficult to
pinpoint that particular project as the most promising grant NSF would
fund in 1994.
Similarly, Dr. Robert Grubbs won the 2005 Nobel Prize in chemistry
for his work in olefin metathesis. This complex organic reaction
essentially allows molecules to swap components and is broadly used
today in the pharmaceutical industry. The basic understanding of olefin
metathesis dates back to the 1950s; however, at that time it was deemed
expensive and dangerous for industrial use. For four decades corporate
and academic researchers labored to refine and improve the reaction,
which did not become industrially productive until the late 1990s. NSF
was a major contributor to Dr. Grubbs' work, which is revolutionizing
the way drug companies improve the environmental footprint of chemical
reactions. Without NSF's long-term dedication to the rigor and quality
of Dr. Grubbs' (along with many others) work, safe and economically
viable olefin metathesis may not have occurred.
These two examples (one based on a single grant that revolutionized
the information age, the other, the result of four decades of research
by multiple teams of researchers) show the need for sustained,
predictable research funding. Ensuring a steady stream of research
funds would allow NSF to fund not only the most rigorous and reliable
research, but also take risks in cutting-edge ideas based on promising
data. By doing so, the government would be encouraging the broadest
possible impacts of taxpayers' precious dollars.
One of the reasons why the merit review process is so successful is
because it draws from the collective wisdom of the scientific
community. Many NSF personnel come directly from the scientific
community and return to their research institutions at the end of their
two- or three-year rotations. Relying on rotating directors means the
managers are up to date on the most recent scientific developments. The
panels that perform the peer review of proposals are fellow researchers
in the field, and as such, are also up to speed on the latest
developments. This scientific community service, whether performed by
grant proposal reviewers or NSF program officers, is an integral part
of scientific culture. Many scientists dedicate their time in this way
because it provides an opportunity to remain in touch with and
influence the cutting edge, as well as because they understand that the
system only works if everyone volunteers to play their part. In a way,
it is the science community's way of ``paying it forward.''
The broader impacts criteria include considerations about whether
the research proposal would broaden underrepresented minorities'
participation in science, strengthen U.S. infrastructure, improve
national security, or foster innovation. Including some of these
impacts is the result of language in the America COMPETES bill enacted
last year. The broader impacts criteria take into consideration which
research is the most urgent or has the greatest relevance to improving
the quality of life. The broader impacts criteria enables NSF to choose
between meritorious and even more meritorious proposals and is a way to
ensure that precious R&D money goes to the most pressing R&D needs.
Responses by Dr. Jorge Jose,
Vice President for Research, Indiana University
Questions submitted by Chairman Mo Brooks
Q1. In basic science research, when one cannot possibly know what the
outcome may be, how can one establish what the broader impacts will be?
A1. Thank you, Chairman Brooks, for the question. As you correctly
note, in basic science research, it is not possible to know in advance
what the outcome of research will be. We pose a question for which we
would like to find an answer, and we lay out a research plan that in
principle should lead us to an answer. The research plan is based on
prior knowledge and an educated hypothesis of the expected results. A
researcher may not know for certain that a particular plan will yield
the results they hope for, but they do know aspects of a plan that is
likely to do so. Reviewers, in fact, will base their review on whether
the hypothesis and the research plan seem to be an appropriate way to
ascertain the ansers to the question being posed. In the course of an
investigation, other questions or an unexpected discovery may arise
that may lead the researcher to significantly change the direction of
the research.
My sense is that a similar awareness of cause and effect can shape
our expectations for what the broader impacts of any given research
project might be. Take, for example, the broader impact goal of
expanding the participation of underrepresented minorities in science,
technology, engineering, and mathematics (STEM). I cannot know for
certain that a research project will have that result, but I can know
some aspects of a research plan that are likely to do so. If my
research plan, for instance, includes some form of collaboration with
colleagues from an HBCU, then it is more likely to expand the
participation of underrepresented minorities than if it does not
include that collaboration. Of course the collaboration is not a
failsafe guarantee what the broader impact will be, but it increases
the likelihood--just as certain elements of the scientific research
plan will increase the likelihood of certain experimental outcomes.
Q2. Having sat on review panels, please tell me how heavily the broader
impacts of a proposal are weighed when rating the proposals you were
considering. Did the evaluation of broader impacts have any effect on
the evaluation of the intellectual merit of the proposals?
A2. Thank you, Chairman Brooks, for that question. As Dr. Marrett
noted, the NSF has many processes in place to guard against bias within
review committees--to make sure, in other words, that as far as is
possible, the process identifies the projects with the most significant
intellectual merit. Ideally, questions of broader impact are used only
to decide afterwards between proposals of equal intellectual merit.
Certainly at Indiana University, when we are allocating resources, we
make every effort to do the same thing--to remove conflicts of
interest, to focus on the intellectual merit of competing proposals,
and to leave ``broader impact'' and strategic considerations for later.
Realistically, of course, it is impossible to guard against all
variables that might prioritize some other aspect of a proposal over
its intellectual merit. Indeed, one might worry that the expansion of
broader impact goals as laid out in the America Competes Act increases
that difficulty. But my experience is that more often than not, the
intellectual merit of a proposal is the issue that is most
determinative in funding recommendations, since consideration of
broader impacts is generally a second-level concern.