[House Hearing, 112 Congress]
[From the U.S. Government Publishing Office]



                       THE MERIT REVIEW PROCESS:
  ENSURING LIMITED FEDERAL RESOURCES ARE INVESTED IN THE BEST SCIENCE

=======================================================================

                                HEARING

                               BEFORE THE

             SUBCOMMITTEE ON RESEARCH AND SCIENCE EDUCATION

              COMMITTEE ON SCIENCE, SPACE, AND TECHNOLOGY
                        HOUSE OF REPRESENTATIVES

                      ONE HUNDRED TWELFTH CONGRESS

                             FIRST SESSION

                               __________

                         TUESDAY, JULY 26, 2011

                               __________

                           Serial No. 112-31

                               __________

 Printed for the use of the Committee on Science, Space, and Technology








       Available via the World Wide Web: http://science.house.gov

                                _____

                  U.S. GOVERNMENT PRINTING OFFICE
67-522 PDF                WASHINGTON : 2011
-----------------------------------------------------------------------
For sale by the Superintendent of Documents, U.S. Government Printing 
Office Internet: bookstore.gpo.gov Phone: toll free (866) 512-1800; DC 
area (202) 512-1800 Fax: (202) 512-2104  Mail: Stop IDCC, Washington, DC 
20402-0001







              COMMITTEE ON SCIENCE, SPACE, AND TECHNOLOGY

                    HON. RALPH M. HALL, Texas, Chair
F. JAMES SENSENBRENNER, JR.,         EDDIE BERNICE JOHNSON, Texas
    Wisconsin                        JERRY F. COSTELLO, Illinois
LAMAR S. SMITH, Texas                LYNN C. WOOLSEY, California
DANA ROHRABACHER, California         ZOE LOFGREN, California
ROSCOE G. BARTLETT, Maryland         DAVID WU, Oregon
FRANK D. LUCAS, Oklahoma             BRAD MILLER, North Carolina
JUDY BIGGERT, Illinois               DANIEL LIPINSKI, Illinois
W. TODD AKIN, Missouri               GABRIELLE GIFFORDS, Arizona
RANDY NEUGEBAUER, Texas              DONNA F. EDWARDS, Maryland
MICHAEL T. McCAUL, Texas             MARCIA L. FUDGE, Ohio
PAUL C. BROUN, Georgia               BEN R. LUJAN, New Mexico
SANDY ADAMS, Florida                 PAUL D. TONKO, New York
BENJAMIN QUAYLE, Arizona             JERRY McNERNEY, California
CHARLES J. ``CHUCK'' FLEISCHMANN,    JOHN P. SARBANES, Maryland
    Tennessee                        TERRI A. SEWELL, Alabama
E. SCOTT RIGELL, Virginia            FREDERICA S. WILSON, Florida
STEVEN M. PALAZZO, Mississippi       HANSEN CLARKE, Michigan
MO BROOKS, Alabama
ANDY HARRIS, Maryland
RANDY HULTGREN, Illinois
CHIP CRAVAACK, Minnesota
LARRY BUCSHON, Indiana
DAN BENISHEK, Michigan
VACANCY
                                 ------                                

             Subcommittee on Research and Science Education

                     HON. MO BROOKS, Alabama, Chair
ROSCOE G. BARTLETT, Maryland         DANIEL LIPINSKI, Illinois
BENJAMIN QUAYLE, Arizona             HANSEN CLARKE, Michigan
STEVEN M. PALAZZO, Mississippi       PAUL D. TONKO, New York
ANDY HARRIS, Maryland                JOHN P. SARBANES, Maryland
RANDY HULTGREN, Illinois             TERRI A. SEWELL, Alabama
LARRY BUCSHON, Indiana                   
DAN BENISHEK, Michigan                   
RALPH M. HALL, Texas                 EDDIE BERNICE JOHNSON, Texas













                            C O N T E N T S

                         Tuesday, July 26, 2011

                                                                   Page
Witness List.....................................................     2

Hearing Charter..................................................     3

                           Opening Statements

Statement by Representative Mo Brooks, Chairman, Subcommittee on 
  Research and Science Education, Committee on Science, Space, 
  and Technology, U.S. House of Representatives..................    10
    Written Statement............................................    10

Statement by Representative Daniel Lipinski, Ranking Minority 
  Member, Subcommittee on Research and Science Education, 
  Committee on Science, Space, and Technology, U.S. House of 
  Representatives................................................    11
    Written Statement............................................    12

                               Witnesses:

Dr. Cora Marrett, Deputy Director, National Science Foundation
    Oral Statement...............................................    14
    Written Statement............................................    15

Dr. Keith R. Yamamoto, Vice Chancellor for Research, University 
  of California San Francisco
    Oral Statement...............................................    21
    Written Statement............................................    23

Dr. Nancy B. Jackson, President, American Chemical Society
    Oral Statement...............................................    25
    Written Statement............................................    27

Dr. Jorge Jose, Vice President for Research, Indiana University
    Oral Statement...............................................    30
    Written Statement............................................    32


Discussion.......................................................    36


              Appendix: Answers to Post-Hearing Questions

Dr. Cora Marrett, Deputy Director, National Science Foundation...    58

Dr. Keith R. Yamamoto, Vice Chancellor for Research, University 
  of California San Francisco....................................    64

Dr. Nancy B. Jackson, President, American Chemical Society.......    65

Dr. Jorge Jose, Vice President for Research, Indiana University..    67

 
                       THE MERIT REVIEW PROCESS:
  ENSURING LIMITED FEDERAL RESOURCES ARE INVESTED IN THE BEST SCIENCE

                              ----------                              


                         TUESDAY, JULY 26, 2011

                  House of Representatives,
    Subcommittee on Research and Science Education,
               Committee on Science, Space, and Technology,
                                                    Washington, DC.

    The Subcommittee met, pursuant to call, at 10:01 a.m., in 
Room 2318 of the Rayburn House Office Building, Hon. Mo Brooks 
[Chairman of the Subcommittee] presiding.



                            hearing charter

              COMMITTEE ON SCIENCE, SPACE, AND TECHNOLOGY

             SUBCOMMITTEE ON RESEARCH AND SCIENCE EDUCATION

                     U.S. HOUSE OF REPRESENTATIVES

                       The Merit Review Process:

  Ensuring Limited Federal Resources Are Invested in the Best Science

                         tuesday, july 26, 2011
                         10:00 a.m.-12:00 p.m.
                   2318 rayburn house office building

Purpose

    On Tuesday, July 26, 2011, the Subcommittee on Research and Science 
Education will hold a hearing to examine the merit review grant award 
process and its effect on federally funded scientific research, in an 
effort to understand the strengths and potential weaknesses of the 
process.

Witnesses

     Dr. Cora Marrett, Deputy Director, National Science 
Foundation

     Dr. Keith Yamamoto, Vice Chancellor for Research, 
University of California San Francisco

     Dr. Nancy Jackson, President, American Chemical Society

     Dr. Jorge Jose, Vice President for Research, Indiana 
University

Overview

     A number of federal agencies, from the Department of 
Energy to the National Institutes of Health and the National Science 
Foundation, use various types of peer or merit review to evaluate 
proposals and make recommendations to award federal funds.

     The National Science Foundation (NSF) has three funding 
mechanisms: grants, cooperative agreements, and contracts. NSF makes 
merit-based grant awards to researchers, educators, and students. In 
Fiscal Year 2010 (FY 10), NSF received 55,542 proposals and awarded 
12,996 grants, a 23 percent funding rate. Fifty percent of its budget 
was devoted to new and continuing grants. \1\
---------------------------------------------------------------------------
    \1\  Report to the National Science Board on the National Science 
Foundation's Merit Review Process Fiscal Year 2010, p. 7.

     Approximately 96 percent of NSF grant proposals are 
evaluated through an external review process, commonly known as the NSF 
merit review process. The process utilizes subject matter experts to 
review proposals through the mail, in-person at a panel review, or 
through a combination of both (early-concept grants, rapid response 
grants, and small conferences and workshops are evaluated through an 
internal merit review process).
     The NSF merit review process evaluates proposals based on 
two criteria, intellectual merit and broader impacts. A National 
Science Board Task Force is currently examining the two criteria and a 
report is forthcoming. Since 2007, NSF has also been promoting 
potentially transformative concepts through additional language added 
to the intellectual merit criteria.

Background

    Federal funding is disbursed in a number of ways, including through 
contracts, cooperative agreements and grants. The process by which many 
federal agencies evaluate potential grant awards is often termed merit 
or peer review. This process can take several different forms or 
utilize different processes; however, in general, it requires that the 
grant proposals be reviewed and evaluated by subject matter experts not 
associated with the proposal.
    The National Science Foundation (NSF) uses grants for the majority 
of its funding disbursements. There are two basic grants. A standard 
award has a duration of typically one 09five years, but is fully funded 
in the first fiscal year. A continuing grant, also for a multi-year 
project, is provided in annual increments. The first year of funding 
for a continuing grant comes with a statement of intent to continue the 
funding with continuing grant increments (CGIs) through completion of 
the project, but the continuation is contingent on whether NSF deems 
satisfactory progress, availability of funds, and the receipt and 
approval of annual reports.
    Cooperative agreements are used when the project requires 
substantial agency involvement like research centers and multi-user 
facilities.




    NSF utilizes an internal merit review process for a fraction of its 
grant awards, including the Early-concept Grants for Exploratory 
Research (EAGER) and Grants for Rapid Response Research (RAPID). \2\ 
However, the bulk of NSF funded grants are evaluated through an 
external grant review process, known as the NSF merit review process.
---------------------------------------------------------------------------
    \2\  Early-concept Grants for Exploratory Research (EAGER) are two-
year awards for up to $300,000. These awards support quick-response 
research, or research having a severe urgency with regard to 
availability of data, facilities, or equipment.

---------------------------------------------------------------------------
NSF Merit Review Process

    Grant proposals are required to be submitted electronically. NSF 
program officers ensure each proposal has been assigned to the correct 
office for review and determine the appropriate level of review 
(internal or external). Proposals are returned without being reviewed 
if they do not comply with NSF regulations, including separately 
addressing the intellectual merit and broader impacts criteria.
    Program officers choose proposal reviewers and panel members from 
an NSF database of over 300,000 reviewers. They can also recruit 
reviewers based on literature searches, professional activities, and 
other reviewer recommendations. In addition, they also screen all 
reviewers for potential conflicts of interest and provide guidance and 
instructions.
    Reviewers provide comments by mail or through the meeting of a 
panel session (these are most often in person but panelists may also 
meet virtually). Once reviewers return comments, or a panel convenes, 
program officers are responsible for synthesizing comments and 
recommending the award or decline of each proposal. Reviewers provide 
narratives and categorical ratings which the program officer takes into 
account. An ``Excellent'' rating does not guarantee the award of 
funding. In FY 2010, 3,743 proposals that received an average review of 
``Excellent'' were funded and 1,312 were not, and 4,560 proposals that 
received an average review of ``Very Good to Excellent'' were funded 
while 6,318 were not. \3\
---------------------------------------------------------------------------
    \3\  Report to the National Science Board on the National Science 
Foundation's Merit Review Process Fiscal Year 2010, p. 32.




    The Division Director reviews the program officer's recommendations 
and then passes them on to the Division of Grants and Awards, who vets 
the eligibility of the awardee, negotiates any necessary changes, and 
disburses the award. The Director's Review Board reviews any award in 
excess of 2.5 percent of the awarding Division's budget. The National 
Science Board (NSB) must approve any award in excess of $3 million 
dollars, or one percent or more of the awarding Directorate's prior 
year current plan, whichever is greater.
    All those who submitted proposals receive notification as to 
whether or not an award will be made. Those to whom funding was 
declined receive copies of the reviews as well as information on the 
number of grants awarded and the number of proposals in each category. 
If a proposal is declined, the proposer may ask the program officer for 
further clarification. If he is still unsatisfied, he may make a 
reconsideration request to the relevant Assistant Director and a second 
request to the Deputy Director. (See Appendix A for the NSF Flow 
Chart.)

Types of Review

    Proposals submitted to the merit review system are reviewed in 
three ways. Through ``mail-only'' reviews, proposals are sent to 
reviewers who are asked to submit written comments to NSF. Through 
``panel-only'' reviews, reviewers serve on in-person (or virtual) 
panels to discuss reviews and provide advice to the program officer. 
Additionally, some proposals receive a combination of mail and panel 
review, which can take place in a number of ways.
    There is value in each type of review. Mail review allows for 
better matching between the expertise of reviewers and proposals. Panel 
review allows for interplay between reviewers in the evaluation of 
proposals and the integration of different perspectives in the review 
of proposals. According to NSF, ``Using panels in the review process 
tends to reduce proposal processing time (time-to-decision), compared 
to mail-only reviews. For example, in FY 10, 78 percent of all 
proposals reviewed by panel-only were processed within six months, 
compared to 72 percent for mail plus panel and 55 percent for mail-
only.'' \4\ While in-person panels are most common, ``virtual panels'' 
are being convened more often. Virtual panels allow reviewers to 
participate from remote locations using interactive technology. The 
combination of mail and panel reviews ``is used frequently because it 
combines the in-depth expertise of mail review with the comparative 
analysis of panel review.'' \5\
---------------------------------------------------------------------------
    \4\  Report to the National Science Board on the Nationl Science 
Foundation's Merit Review Process Fiscal Year 2010, page 29.
    \5\  Ibid.

---------------------------------------------------------------------------
The Program Officer

    NSF program officers are made up of permanent (54 percent) and non-
permanent (46 percent) employees; \6\ all are subject matter experts in 
the areas they manage with advanced degrees or credentials. ``Some non-
permanent program officers are `on loan' as `Visiting Scientists, 
Engineers, and Educators' (VSEEs) for up to three years from their host 
institutions. Others are supported through grants to the home 
institutions under the terms of the Intergovernmental Personnel Act 
(IPA).'' \7\ These ``rotators'' ensure that new and fresh scientific 
ideas and specialties come through the Foundation and help to prevent 
institutional or innovative stagnation. One drawback, however, is the 
loss of institutional knowledge when a rotator leaves and the challenge 
of frequently bringing new rotators up to speed on NSF policies and 
processes.
---------------------------------------------------------------------------
    \6\  Report to the National Science Board on the Nationl Science 
Foundation's Merit Review Process Fiscal Year 2010, page 34.
    \7\  Ibid.
---------------------------------------------------------------------------
    The Foundation expects program officers to administer balanced 
portfolios and requires them to utilize the advice and expertise of the 
proposal reviewers while assessing proposals in terms of each 
portfolio. In order to create a balanced portfolio, program officers 
are expected to additionally evaluate proposals for a number of 
criteria, including, but not limited to: geographic distribution; novel 
approaches to significant research questions; capacity building in a 
new and promising research area; potential impact on the development of 
human resources and infrastructure; and NSF core strategies, including 
integration of research and education, broadening participation, and 
promoting partnerships. \8\
---------------------------------------------------------------------------
    \8\  Report to the National Science Board on the Nationl Science 
Foundation's Merit Review Process Fiscal Year 2010, p. 26.

---------------------------------------------------------------------------
Merit Review Criteria

    Since initial approval in 1997, every NSF grant proposal has been 
reviewed based on two merit review criteria, intellectual merit and 
broader impacts. While additional consideration may be given for a 
number of reasons including special requirements of the program, 
intellectual merit and broader impacts remains the cornerstone of the 
NSF merit review process.

     Intellectual Merit. What is the intellectual merit of the 
proposed activity? How important is the proposed activity to advancing 
knowledge and understanding within its own field or across different 
fields? How well qualified is the proposer (individual or team) to 
conduct the project? (If appropriate, the reviewer will comment on the 
quality of prior work.) To what extent does the proposed activity 
suggest and explore creative, original, or potentially transformative 
concepts? How well conceived and organized is the proposed activity? Is 
there sufficient access to resources?

     Broader Impacts. What are the broader impacts of the 
proposed activity? How well does the activity advance discovery and 
understanding while promoting teaching, training, and learning? How 
well does the proposed activity broaden the participation of 
underrepresented groups (e.g., gender, ethnicity, disability, 
geographic, etc.)? To what extent will it enhance the infrastructure 
for research and education, such as facilities, instrumentation, 
networks and partnerships? Will the results be disseminated broadly to 
enhance scientific and technological understanding? What may be the 
benefits of the proposed activity to society? \9\
---------------------------------------------------------------------------
    \9\  Report to the National Science Board on the Nationl Science 
Foundation's Merit Review Process Fiscal Year 2010, pp. 21 0922.

    The America COMPETES Reauthorization Act of 2010 expands the 
broader impacts criteria to include activities to achieve the following 
goals: (1) increase the economic competitiveness of the United States; 
(2) development a globally competitive STEM workforce; (3) increase 
participation of women and underrepresented minorities in STEM; (4) 
increase partnerships between academia and industry; (5) improve pre-K 
0912 STEM education and teacher development; (6) improve undergraduate 
STEM education; (7) increase public scientific literacy; and (8) 
increase national security.
    In February 2010, the NSB reconstituted the Task Force on Merit 
Review. The Board charged the Task Force with ``examining the two Merit 
Review Criteria and their effectiveness in achieving the goals for NSF 
support for science and engineering research and education.'' \10\
---------------------------------------------------------------------------
    \10\  NSB Task Force on Merit Review, http://www.nsf.gov/nsb/
committees/tskforce-mr^charge.jsp.
---------------------------------------------------------------------------
    In June 2011, after a year of review, NSB and NSF put out a call 
for public comment (closed July 14, 2011) on proposed revisions to the 
merit review criteria. The proposed changes maintain the themes of 
intellectual merit and broader impacts while establishing key 
principles of the merit review criteria. The proposed changes include 
the identification of national goals which every NSF project should 
seek to advance, including but not limited to: the increased economic 
competitiveness of the United States; the increased participation of 
women, persons with disabilities, and underrepresented minorities in 
STEM; the increased public scientific literacy and public engagement 
with science and technology; and increased national security. (See 
Appendix B for the complete proposal.)

Potential Challenges

    While the NSF merit review process is widely considered the most 
effective of its type for the awarding of federal funding, there are 
existing challenges to be considered in an effort to strengthen the 
process. Questions remain about the way in which scientific priorities 
are established and whether the process is truly supporting innovative 
research and researchers. Below are some additional challenges:

     Transformative Research Research that questions existing 
science often faces additional hurdles when facing review by scientific 
researchers in that field, especially during lean economic times as 
experts favor more conservative funding approaches. Since 2007, NSF has 
been working to ensure transformative research is considered 
appropriately and such proposals are provided an opportunity to compete 
through the merit review process, including adding explicit language in 
the intellectual merit criteria for transformative concepts. EAGER 
grants are intended to be used, in part, to fund potentially 
transformative ideas for which there is little to no preliminary data 
and, as such, would fare poorly in the standard merit review process. 
NSF has also incorporated efforts to encourage transformative research 
in its training of program officers and reviewers. It is also 
experimenting with modifications in the review process to help identify 
transformative proposals. Are these efforts working? Is there more to 
be done, within the process, to encourage transformative science?

     Ensuring a Pipeline for U.S. Students by Encouraging New 
Principal Investigators New Principal Investigators (PIs) often do not 
have the same level of experience or access to resources that 
established PIs have, both considerations included as part of the 
intellectual merit criteria. In FY 10, new PIs submitted 21,545 
proposals and received 3,620 awards, a funding rate of 17 percent; 
prior PIs were funded at a rate of 28 percent. \11\ A strong system 
properly encourages new investigators to participate in the scientific 
arena in order to ensure a pipeline for U.S. student participation in 
scientific endeavors. The Faculty Early Career Development (CAREER) 
Program offers specific funding opportunities for new PIs to help in 
this endeavor, but is this enough? Does the merit review process 
encourage the participation of new PIs?
---------------------------------------------------------------------------
    \11\  Report to the National Science Board on the National Science 
Foundations's Merit Review Process Fiscal Year 2010, p. 8.

     Parity for Institutions Institutions that are not regular 
grant recipients do not always have the same resources or proficiencies 
as those institutions that consistently receive federal funding. ``For 
FY 10, the average funding rate was 26 percent for the top 100 Ph.D.-
granting institutions (classified according to the amount of FY 10 
funding received). In comparison, the rate was 17 percent for Ph.D.-
granting institutions that are not in the top 100 NSF-funded 
category.'' \12\ Are those institutions, not regularly in receipt of 
federal funding, encouraged to submit grant proposals and participate 
in the merit review process?
---------------------------------------------------------------------------
    \12\  Report to the National Science Board on the National Science 
Foundations's Merit Review Process Fiscal Year 2010, pp. 10 0911.

     Multidisciplinary Review As NSF seeks to grow its 
multidisciplinary projects, the merit review process must consider the 
management of reviews that incorporate a combination of scientific 
disciplines in order to fund the strongest multidisciplinary proposals. 
Is the current process able to effectively encourage and evaluate 
---------------------------------------------------------------------------
multidisciplinary projects?

    In all, the merit review process must continue to balance these 
challenges with the inherent need to fund the strongest science.



    Chairman Brooks. The Subcommittee on Research and Science 
Education will come to order.
    Good morning. Welcome to today's hearing entitled ``The 
Merit Review Process: Ensuring Limited Federal Resources Are 
Invested in the Best Science.'' I am going to give an opening 
statement.
    Our hearing today presents us with an opportunity to 
examine the merit review process for awarding federal grant 
funds. It is our goal to highlight the benefits of the process, 
while acknowledging that no process involving human decision 
making is flawless. The focus of today's hearing will primarily 
be on the merit review process at the National Science 
Foundation.
    The National Science Foundation Act of 1950 directs NSF to 
initiate and support basic scientific research and programs to 
strengthen scientific research potential and science education 
programs at all levels. NSF works to accomplish this 
Congressional directive through the issuance of merit-based 
awards to researchers, educators and students at approximately 
1,900 U.S. colleges, universities and other institutions. In 
1994, the National Academies touted it as among the best 
procedures known for insuring the technical excellence of 
research projects that receive public support, but the process 
has changed since then, and we need to make sure that is still 
the case.
    As we know, a large number of potentially fundable 
proposals are declined each year. The Foundation received over 
55,000 proposals for funding in fiscal year 2010 and funded 
nearly 13,000, or 23 percent, of them. Many of the proposals 
received were not worthy of federal funding, but it is also 
true that many were not funded because federal funds are 
limited. So given that those limited dollars should go to the 
very best scientific research, NSF must maintain a robust and 
transparent merit review process.
    Today, our witnesses will share their thoughts on how the 
process works and its strengths and weaknesses. We want to know 
if the current process spurs or stifles innovation, how award 
decisions are actually made after receiving peer review, and if 
there are flaws in the system that may be providing precious 
federal funds to lower rated proposals over more highly rated 
proposals.
    In exercising its oversight role, this Subcommittee must 
ensure that federal dollars are being spent on the best 
science. This examination of the merit review process will help 
us to understand how programmatic funding decisions are made 
and how, in turn, those decisions interact with Foundation-wide 
priorities.
    I look forward to hearing from each of our witnesses on 
this important topic. I thank you for joining us.
    [The prepared statement of Mr. Brooks follows:]

                Prepared Statement of Chairman Mo Brooks
    Good morning, and welcome to each of our witnesses. Our hearing 
today presents us with an opportunity to examine the merit review 
process for awarding federal grant funds. It is our goal to highlight 
the benefits of the process, while acknowledging that no process 
involving human decision making is flawless. The focus of today's 
hearing will primarily be on the merit review process at the National 
Science Foundation (NSF).
    The National Science Foundation Act of 1950 directs NSF to ``to 
initiate and support basic scientific research and programs to 
strengthen scientific research potential and science education programs 
at all levels.'' NSF works to accomplish this Congressional directive 
through the issuance of merit-based awards to researchers, educators 
and students at approximately 1,900 U.S. colleges, universities, and 
other institutions. In 1994, the National Academies touted it as being 
among ``the best procedures known for insuring the technical excellence 
of research projects that receive public support,'' but the process has 
changed since then, and we need to make sure that is still the case.
    As we know, a large number of potentially fundable proposals are 
declined each year. The Foundation received over 55,000 proposals for 
funding in Fiscal Year 2010 and funded nearly 13,000, or 23 percent, of 
them. Many of the proposals received were not worthy of federal 
funding, but it is also true that many were not funded because federal 
funds are limited. So, given that those limited dollars should go to 
the very best scientific research, NSF must maintain a robust and 
transparent merit review process.
    Today, our witnesses will share their thoughts on how the process 
works and its strengths and weaknesses. We want to know if the current 
process spurs or stifles innovation, how award decisions are actually 
made after receiving peer review, and if there are flaws in the system 
that may be providing precious federal funds to lower rated proposals 
over more highly rated proposals.
    In exercising its oversight role, this Subcommittee must ensure 
that federal dollars are being spent on the best science. This 
examination of the merit review process will help us to understand how 
programmatic funding decisions are made and how, in turn, those 
decisions interact with Foundation-wide priorities.
    I look forward to hearing from each of our witnesses on this 
important topic; thank you for joining us.

    Chairman Brooks. At this point, I recognize Mr. Lipinski, 
who is the Ranking Member of this Subcommittee.
    Mr. Lipinski. Thank you, Chairman Brooks, and thank you for 
holding the hearing this morning.
    It will not surprise you when I say that I have had a great 
interest in this topic since I first submitted a grant proposal 
to the National Science Foundation when I was a graduate 
student. That interest continued throughout my academic career, 
and when I was elected to Congress I joined what was then the 
Science and Technology Committee to a large extent--actually it 
was the Science Committee at the time when I joined--to a large 
extent because of my strong belief in the NSF and its mission.
    I agree with the statements of all of the witnesses here 
today that NSF's merit review system remains the gold standard 
for the world. And I don't say that just because that NSF grant 
application I submitted when I was in grad school was 
successful.
    At the same time, I recognize that there are challenges in 
any system for allocating limited research dollars. I agree 
with Chairman Brooks that it is our job on this Subcommittee to 
hold hearings such as this one to discuss these challenges and 
collectively imagine how we might continue to make NSF, and the 
merit-review system that it manages, even stronger. 
Particularly in this tight budget environment, it is incumbent 
upon us all to make sure that the system for funding excellent 
science is as efficient and effective as possible.
    In 2009, when I was Chair of this Subcommittee, we looked 
at a few slices of the broader topic being addressed today when 
we held a hearing on high-risk, high-reward, or what we also 
call transformative research. Dr. Yamamoto was not on the panel 
for that hearing, but a report that he helped author, the ARISE 
report, played a central role and remains relevant today. I 
look forward to learning from his deep expertise on this topic.
    But there are many issues that we have not examined in 
detail, including the extent to which faculty from lesser-
resourced institutions face an uneven playing field. I am also 
interested in the extent to which the institutional structures 
underlying NSF's peer-review system influence decisions and the 
benefits and drawbacks of different approaches to peer review.
    For example, I am intrigued by proposals to conduct 
committee review in virtual environments such as Second Life. 
While I am perhaps a little skeptical, I recognize that virtual 
review has the potential to save taxpayers a lot of money in 
travel expenses, as well as broaden the pool of reviewers. So I 
am glad that the NSF is looking into innovative approaches. But 
we need to be confident that the group dynamics in a virtual 
environment, while certain to be different, do not in any way 
undermine the quality of NSF merit review.
    I am particularly interested in hearing the panel's 
recommendations about some of the alternatives to standard 
merit review, not as a replacement of, but rather as a 
complementary approach. For example, in last year's COMPETES 
Act, I authored a provision that authorizes prize programs at 
all of the science agencies. While NSF, as a basic research 
agency, would need to design and implement a prize program that 
looks very different from those run by DARPA or NASA, I 
continue to believe that the NSF should experiment with some 
pilot projects to award research prizes. I look forward to 
hearing an update from Dr. Marrett on the Foundation's thoughts 
on this subject.
    So I see this hearing as an opportunity to examine the real 
challenges that do exist in a very strong review system and to 
discuss current and novel approaches to overcoming those 
challenges.
    Finally, Mr. Chairman, I think it is important to point out 
that the National Science Board is in the middle of a process 
to review and revise the existing merit review criteria. While 
I believe this hearing is important, it is critical that in the 
fall when the Board has finished its work and produced a new 
set of review criteria for the scientific community, this 
Committee should examine what the Board has come out with. 
Especially in light of the provision in last year's COMPETES 
bill directing the agency to clarify the purpose and 
implementation of the broader impacts review criterion, it will 
be helpful at some point later this year or early next year to 
revisit this topic, and I certainly hope that we can do that.
    With that, Mr. Chairman, I thank all of the witnesses for 
being here today and I yield back.
    [The prepared statement of Mr. Lipinski follows:]

          Prepared Statement of Ranking Member Daniel Lipinski
    Thank you, Chairman Brooks, and thank you for holding this hearing 
this morning. It will not surprise anyone when I say that I have had a 
great interest in this topic since I first submitted a grant proposal 
to the National Science Foundation as a graduate student. That interest 
continued throughout my academic career and when I was elected to 
Congress I joined what was then the Science and Technology committee to 
a large extent because of my strong belief in the NSF and its mission. 
I agree with the statements of all of the witnesses here today that 
NSF's merit-review system remains the gold standard for the world. And 
I don't say that just because that NSF grant application I submitted 
when I was in grad school was successful.
    At the same time, I recognize that there are challenges in any 
system for allocating limited research dollars. I agree with Chairman 
Brooks that it is our job, on this Subcommittee, to hold hearings such 
as this one to discuss these challenges and collectively imagine how we 
might continue to make NSF, and the merit-review system that it 
manages, even stronger. Particularly in this tight budget environment 
it is incumbent upon us all to make sure that the system for funding 
excellent science is as efficient and effective as possible.
    In 2009, when I was chair of this Subcommittee, we looked at a few 
slices ofthe broader topic being addressed today when we held a hearing 
on high-risk, high-reward, or what we also call transformative 
research. While Dr. Yamamoto was not on the panel for that hearing, a 
report that he helped author, the ARISE report, played a central role 
and remains relevant today. I look forward to learning from his deep 
expertise on this topic.
    But there are many issues that we have not examined in detail, 
including the extent to which faculty from lesser resourced 
institutions face an uneven playing field. I am also interested in the 
extent to which the institutional structures underlying NSF's peer-
review system influence decisions and the benefits and drawbacks of 
different approaches to peer review.
     For example, I am intrigued by proposals to conduct committee 
review in virtual environments such as Second Life. While I am perhaps 
a little skeptical, I recognize that virtual review has the potential 
to save taxpayers a lot of money in travel expenses, as well as to 
broaden the pool of reviewers. So I'm glad that the NSF is looking into 
innovative approaches. But we need to be confident that the group 
dynamics in a virtual enviromnent, while certain to be different, do 
not in any way undermine the quality of NSF merit review.
    I'm particularly interested in hearing the panel's recommendations 
about some of the alternatives to standard merit review, not as a 
replacement of, but rather as a complementary approach. For example, in 
last year's COMPETES Act, I authored a provision that authorizes prize 
programs at all ofthe science agencies. While NSF, as a basic research 
agency, would need to design and implement a prize program that looks 
very different from those run by DARPA or NASA, I continue to believe 
that the NSF should experiment with some pilot projects to award 
research prizes. I look forward to hearing an update from Dr. Marrett 
on the Foundation's thoughts on this subject.
    So I see this hearing as an opportunity to examine the real 
challenges that do exist in a very strong review system and to discuss 
current and novel approaches to overcoming those challenges.
    Finally, Mr. Chairman (as you already mentioned), I think it's 
important to point out that the National Science Board is in the middle 
of a process to review and revise the existing merit review criteria. 
While I believe this hearing is critically important, it is unfortunate 
that we couldn't wait until this fall when the Board has finished its 
work and produced a new set of review criteria for the scientific 
community and this Committee to examine. Especially in light of the 
provision in last year's COMPETES bill directing the agency to clarify 
the purpose and implementation of the Broader Impacts Review Criterion, 
it will he helpful at some point later this year or next year to 
revisit this topic.
    With that, Mr .Chairman, I thank all of the witnesses for being 
here today and I yield back.

    Chairman Brooks. Thank you, Mr. Lipinski.
    At this time I would like to introduce our witnesses for 
today's hearing. First, we have Dr. Cora B. Marrett, the Deputy 
Director of the National Science Foundation. Since January 
2009, she has served as NSF's Acting Deputy Director and Senior 
Advisor until her confirmation as Deputy Director in May 2011. 
Dr. Marrett holds a bachelor of arts degree from Virginia Union 
University, a master of arts and a doctorate from the 
University of Wisconsin-Madison, all in sociology.
    Dr. Keith Yamamoto is Vice Chancellor for Research at the 
University of California at San Francisco. Dr. Yamamoto served 
on grant review panels for the NSF Biology Directorate, and as 
an ad hoc member of the National Science Board Taskforce on 
Transformative Research.
    Dr. Nancy Jackson serves as President of the American 
Chemical Society. Dr. Jackson is employed at Sandia National 
Laboratories and earned her Ph.D. in chemical engineering from 
the University of Texas at Austin.
    And Dr. Jorge Jose is the Vice President for Research at 
Indiana University. A theoretical physicist and neurobiologist, 
Dr. Jose received his undergraduate, master's and Ph.D. in the 
Department of Physics at the National Autonomous University of 
Mexico.
    As our witnesses should know, spoken testimony is limited 
to five minutes each, after which the members of the committee 
will have five minutes each to ask questions.
    I now recognize our first witness, Dr. Cora Marrett, for 
five minutes.

                 STATEMENT OF DR. CORA MARRETT,

          DEPUTY DIRECTOR, NATIONAL SCIENCE FOUNDATION

    Dr. Marrett. Thank you, Chairman Brooks, Ranking Member, 
Lipinski, and the distinguished Members of the Subcommittee 
staff. I am pleased to have this opportunity to discuss the 
National Science Foundation's merit review process.
    For over 60 years, NSF has been a steward of the Nation's 
science and engineering enterprise with a proven track record 
for producing results. Despite its relatively small size, NSF 
has an important impact on scientific and engineering knowledge 
and academic capacity. NSF's investments in discovery, learning 
and innovation have been important to increasing America's 
competitiveness, our economic strength, national security and 
overall quality of life.
    NSF relies on a merit-based competitive process that is 
critical to fostering the highest standards of excellence and 
accountability. The process lies at the heart of the agency's 
strategy for accomplishing its overall mission.
    Now, of the 256 American Nobel Prize recipients in science 
since NSF first began to award research grants, of those 256, 
approximately 75 percent have received NSF funding at some 
point in their careers. That includes 19 individuals in the 
last five years.
    The agency has a strong record of funding the most 
insightful ideas and visionary investigators. Our model of 
high-quality merit review has been emulated and replicated by 
other nations. Our merit review process helps assure that 
awards made by NSF are of the highest quality, are relevant to 
our goals and objectives and have an appropriate balance for 
the resulting portfolio. This assurance is critical as nearly 
90 percent of NSF's funding is allocated through merit review, 
allocated either as grants or cooperative agreements.
    As you have already noted, in the fiscal year 2010, NSF 
evaluated over 55,000 proposals through this process and made 
approximately 13,000 new awards. This entailed conducting about 
287,000 proposal reviews and engaging nearly 46,000 members of 
the science and engineering community as reviewers. Underlying 
these statistics is a strategy that helps ensure that each 
proposal submitted to NSF is reviewed in a fair, competitive, 
transparent, and in-depth manner.
    First, our program officers are experts in the scientific 
and engineering programs they manage. They hold strong 
credentials, usually a Ph.D. or equivalent in a STEM field, and 
their funding recommendations are subject to several additional 
layers of review. Each proposal submitted to NSF is reviewed by 
these in-house experts, and they in turn rely extensively on 
external experts to keep NSF-funded research at the frontier. 
As we emphasize, this is a unique aspect of NSF, the knowledge 
of our program officers, who are then empowered to make 
recommendations on funding.
    Each proposal must meet the highest standards in terms of 
two merit review criteria: intellectual merit and broader 
impacts. Intellectual merit encompasses the potential of the 
research to advance knowledge, the originality and creativity 
of the proposed activity and the qualifications of the 
researchers. Broader impacts include technological innovation, 
societal benefits and opportunities to include a diversity of 
participants.
    Finally, the merit review process itself is constantly 
assessed through evaluations by committees of visitors, 
advisory committees and other stakeholders. NSF continuously 
strives to maintain and improve the quality and transparency of 
the process.
    So in summary, the National Science Foundation is dedicated 
to ensuring that the merit review process remains robust, 
rigorous, and beyond reproach. This process enables us to carry 
out our mission. I appreciate, then, this opportunity to appear 
before the Subcommittee on this important topic and would be 
pleased to answer any questions you might have.
    [The prepared statement of Dr. Marrett follows:]

                Prepared Statement of Dr. Cora Marrett,
              Deputy Director, National Science Foundation
    Chairman Brooks, Ranking Member Lipinski, and distinguished Members 
of the Subcommittee, thank you for inviting me to participate in this 
hearing on ``The Merit Review Process.''
    I am delighted to discuss the National Science Foundation's (NSF) 
Merit Review Process with you. As you well know, NSF is the primary 
Federal agency supporting research at the frontiers of knowledge, 
across all fields of science and engineering (S&E) and all levels of 
S&E education. Its mission, vision and goals are designed to maintain 
and strengthen the vitality of the U.S. science and engineering 
enterprise. As part of the overall national R&D enterprise, the basic 
research and education activities supported by NSF are vital to the 
economic advancement of the U.S. and provide the know-how that allows 
the U.S. to respond rapidly and effectively to a range of unexpected 
challenges. The NSF merit review process lies at the heart of the 
agency's strategy for accomplishing its overall mission. As such, NSF 
is continuously striving to maintain and improve the quality and 
transparency of the process.
    Before I begin my discussion of the unique elements of the NSF 
merit review system, let me first describe the essential features of 
merit review writ large. In general, merit review refers to an 
independent assessment of a plan's worthiness. The Code of Federal 
Regulations (Section 600.13 of title 10) defines Merit Review as a 
``thorough, consistent and objective examination of applications based 
on pre-established criteria by persons who are independent of those 
individuals submitting the applications and who are knowledgeable in 
the field of endeavor for which support is requested.''
    I would also like to note here that although the terms ``merit 
review'' and ``peer review'' are often used interchangeably, they are 
not equivalent terms. NSF made this distinction clear back in 1986, 
based on a report from an external Advisory Committee on Merit Review, 
established by then-director Erich Bloch at the request of the National 
Science Board. As is described by Marc Rothenberg, the NSF historian, 
in his 2010 article ``Making Judgments about Grant Proposals: A Brief 
History of the Merit Review Criteria at the National Science 
Foundation:''

     According to the committee, the term ``peer review'' was 
properly a restrictive term referring to the evaluation of the 
technical aspect of the proposal. However, for more and more federally 
funded research, ``technical excellence'' was, in the words of the 
committee, ``a necessary but not fully sufficient criterion for 
research funding.'' Acknowledging that the NSF (as well as other 
federal agencies) was using a wide range of nontechnical criteria as 
part of the decision-making process, the committee suggested that the 
term ``merit review'' more accurately described the NSF selection 
process.

     The committee's recommendation was accepted by Director 
Bloch, and since then NSF has used the term ``merit review'' to 
describe our process.

    Since its founding, NSF has relied on the merit review process to 
allocate the vast majority of its funding. As in other agencies, this 
has involved the use of proposals from prospective researchers that are 
judged on their merits by knowledgeable persons. But there are several 
elements that give merit review at the NSF its distinct features. For 
one, right from the beginning, NSF utilized the project grant mechanism 
(as opposed to a contract mechanism) for providing funds. This was a 
rather radical concept back in 1951, when most government operations 
used contracts. Since that time, the use of the grant mechanism has 
been adopted by many federal extramural research funding organizations.
    NSF's process for deciding which proposals to fund differs from the 
approach of a number of other funding agencies and organizations (such 
as philanthropic foundations) nationally and internationally. Perhaps 
the most distinctive differences are our reliance on expertise from 
both outside and within the Foundation, and the discretionary authority 
vested in the NSF program officer to make funding recommendations. 
Unlike many philanthropic foundations (and even some federal research 
funding programs), NSF policy requires that the program officers seek 
external expert advice before making most of their funding 
recommendations. However, in contrast to a number of other funding 
bodies, the external reviewers do not make binding recommendations that 
the program officer is obliged to follow, although program officers 
always pay close attention to all external reviews. Because of the 
responsibility we give our program officers, NSF sets a high standard 
for excellence in that position. Our program officers are subject 
matter experts in the scientific areas that they manage, and bring 
strong credentials with them, including advanced educational training 
(e.g., a Ph.D. or equivalent credentials) in science or engineering, 
and deep experience in research, education, and/or administration.
    NSF has chosen to give the program officer the responsibility for 
making funding recommendations to enable a more strategic and long-term 
approach for building the award portfolio. As important as the input of 
the external scientific experts is, they have only a snapshot view of 
the current set of proposals they are evaluating. The NSF program 
officer is responsible for putting that snapshot view into the larger 
context of the entire award portfolio they are managing, which can lead 
to a more diverse and robust portfolio overall. Together with the 
division directors, who have the authority to review and act on the 
program officers' recommendations, program officer teams are poised to 
identify promising research that responds to national priorities 
identified by Congress and the Administration. In addition, program 
officers can incorporate agency or programmatic priorities, which are 
articulated in the annual agency budget, special solicitations, and 
standing program descriptions, all of which are available to the 
community via the NSF Web site.
    The NSF merit review process is described in full detail on the NSF 
Web site (http://www.nsf.gov/bfa/dias/policy/meritreview/). There is 
also a summary of the major steps in the merit review process in the 
annual Report to the National Science Board on the Merit Review Process 
(the most recent report covering activities in FY 2010 can be found at 
http://www.nsf.gov/nsb/publications/2011/nsb1141.pdf). It is worth 
noting here that the key features of the NSF process have remained 
remarkably stable over time. Any changes that have been incorporated 
have sought primarily to clarify the process and make it more 
transparent. For example, initially only excerpts of the external 
reviews were shared with the proposal authors. Over time, NSF provided 
the verbatim reviews (but not the identities of the reviewers) to the 
applicant. Similarly, over time there have been modifications to the 
number and clarity of the review criteria. In the America COMPETES 
Reauthorization Act, the broader impacts criterion is specifically 
mentioned, and the National Science Board is in the process of 
analyzing the many comments received on this topic.
    A flowchart that graphically depicts the major steps in the merit 
review process and a timeline is attached to this testimony as Appendix 
I. These steps include:

     Assignment to the appropriate program for review. 
Principal investigators initiate this process by selecting the program 
or programs to which they wish to submit their proposal. Once 
submitted, the cognizant program officers for those programs confirm 
that the assignment is appropriate. On occasion, a proposal may be 
reassigned to another program where there is a better fit. During this 
initial assignment process, it is not uncommon for proposals to be 
assigned to multiple programs for review, if the subject is 
interdisciplinary in nature, or if the question is of interest and 
relevance to more than one program.

     Administrative review of all proposals for compliance with 
NSF regulations. These regulations, which are intended to ensure 
fairness in the review process, are described in the Grant Proposal 
Guide, which is widely available to the NSF community on the NSF Web 
site (http://www.nsf.gov/pubs/policydocs/pappguide/nsf11001/
nsf11-1.pdf). Proposals that do not comply with these 
regulations may be returned without review.

     Merit review of all proposals that pass the administrative 
review. As noted above, a critical feature of NSF's process is the use 
of both external review by experts in the field and internal review by 
NSF's corps of program officers. The program officers are responsible 
for administering the merit review process from beginning to end, 
starting with identifying and recruiting appropriate peer reviewers 
from the external community to serve either as individual reviewers for 
a particular proposal (referred to as ``ad hoc'' reviewers) or as 
members of a panel of reviewers who evaluate a larger set of proposals. 
To ensure that they receive substantive reviews from a variety of 
perspectives, the program officers reach out to a broad range of 
experts for input--in fiscal year 2010, over 46,000 external peer 
reviewers from academia, government, and occasionally industry provided 
authoritative advice to the Foundation. Selection of expert peer 
reviewers may be based on the program officer's knowledge, references 
listed in the proposal, individuals cited in recent publications or 
relevant journals, presentations at professional meetings, reviewer 
recommendations, bibliographic and citation databases, or suggestions 
from the proposal author (subject to the program officer's discretion). 
In making these selections, program officers pay very careful attention 
to avoiding conflicts of interest, both real and perceived.
    NSF takes seriously its responsibility to ensure that the merit 
review process is fair and equitable. One of the ways in which we 
address this responsibility is through the briefings that are given to 
each review panel before it begins its work. In these briefings, 
panelists are instructed on NSF's review criteria (Intellectual Merit 
and Broader Impacts), and on maintaining confidentiality and avoiding 
conflicts of interest. In addition, review panel briefings typically 
include alerting the reviewers to the phenomenon of implicit bias, 
which may adversely impact new investigators, smaller institutions, and 
underrepresented groups. By guarding against the effects of implicit 
bias in the review process, NSF is working to ensure that there are 
equitable opportunities for all investigators.
    I should note here that while the vast majority of the proposals 
received at NSF (96%) are subject to both external and internal merit 
review, for some proposals the external review requirement is waived. 
This waiver provides necessary flexibility for handling proposals for 
which most of the external community would be conflicted (such as 
proposals for small conferences, workshops, or symposia), those for 
which there is a severe urgency (submitted through the Grants for Rapid 
Response Research, or RAPID, mechanism used, for example, on rapid-
response research to the Deepwater Horizon oil spill), and those that 
request support for high-risk, potentially transformative exploratory 
work (submitted through the Early Grants for Exploratory Research, or 
EAGER, mechanism). These proposals are usually only reviewed internally 
by program officers with appropriate expertise.

     Development of funding recommendations. A central tenet of 
the NSF merit review process is that the reviewer input is advisory in 
nature. Funding recommendations are developed by the program officer, 
who is responsible for synthesizing the advice of the reviewers along 
with several other factors, with the goal of allocating funding to a 
diverse portfolio of projects that addresses a variety of 
considerations and objectives. In addition to their scientific 
expertise noted above, NSF program officers bring their own unique 
perspective born from their experience of working with hundreds, 
thousands, or--in some cases--tens of thousands of proposals. In 
developing recommendations within the larger context of their overall 
portfolio, program officers consider carefully the individual merits of 
each proposal with respect to both its intellectual merit and the 
potential broader impacts of the project, and how each proposal might 
help advance a variety of portfolio goals such as:

        Achieving special program objectives and initiatives;

        Fostering novel approaches to significant research and 
education questions;

        Building capacity in a new and promising research area;

        Supporting high-risk proposals with potential for 
transformative advances;

        Supporting NSF's core strategies of integration of 
research and education and integrating diversity into NSF's programs;

        Potential impact on human resources and infrastructure;

        Other available funding sources; and

        Geographic distribution.

    NSF has set a goal for completing this process within six months, 
from the time the proposal is submitted to the point at which the 
proposal is either declined or recommended for funding and forwarded to 
the Division of Grants and Agreements for the final stages of review 
and processing. The proposal assignment and administrative review stage 
is typically complete within a few weeks. The bulk of the time is spent 
in the merit review stage, which can take three to four months to 
complete. Despite the volume of proposals that NSF receives annually 
(in FY 2010, over 55,000 proposals were submitted, an increase of 23% 
over the previous year), NSF routinely processes the majority of these 
proposals (+75%) in fewer than six months.
    To ensure the integrity of the process, all program officer 
recommendations are reviewed by the division director (or other 
appropriate NSF official), who examines whether the process used to 
arrive at the decision has been executed in accordance with NSF's 
policies and that the decision has been based on a thorough analysis of 
the merits of the proposal. Large awards may receive additional review, 
either by the Director's Review Board (DRB) or additionally by the 
National Science Board (NSB). The DRB examines award recommendations 
with an average annual award amount of 2.5 percent or more of the 
awarding division's prior year current plan. The NSB reviews 
recommended awards with an annual award amount of one percent or more 
of the awarding Directorate's or Office's prior year current plan, or 
less than one percent or more of the prior year total NSF budget at the 
enacted level. Once the funding recommendation is approved (at whatever 
level is appropriate), the Division of Grants and Agreements ensures 
that the award recommendation meets all of NSF's requirements before 
officially issuing the award.
    In addition to having multiple layers of review of individual award 
recommendations, NSF requires that all programs undergo an external 
review by Committees of Visitors (COVs) every three years. COV reviews 
provide NSF with external expert assessments of the quality and 
integrity of program operations and program-level technical and 
managerial matters pertaining to the merit review and final proposal 
decisions. Finally, retrospective analysis of the process is 
periodically performed on a Foundation-wide basis, including the 
statistical reports submitted to the NSB every year and the Impact of 
Proposal and Award Management Mechanisms (IPAMM) report of 2007 (http:/
/www.nsf.gov/pubs/2007/nsf0745/nsf0745.pdf).
    At the request of Congress, in 2005 the NSB undertook an 
examination of NSF's Merit Review Process (http://www.nsf.gov/nsb/
documents/2005/nsb05119.pdf). The report concludes that:

    ``The Board fully supports the current NSF system of merit review, 
which utilizes the peer review process as the principal driver in 
funding decisions. The Board also strongly endorses the role of NSF 
program officers' discretionary authority, in concurrence with division 
directors, for ensuring the implementation and goals of both Merit 
Review Criteria, along with achieving a balanced portfolio of research 
and education awards, both within directorates and across the suite of 
NSF programs. Unlike a system based solely on peer reviews' scores, 
NSF's merit review process incorporates peer review in a system that 
also considers those attributes of a proposal (risk, multidisciplinary 
nature, novelty) that are not readily accommodated by a numerical 
score, but essential to identifying the most innovative proposals.''

    The National Academy of Sciences, in the 1994 report ``Major Award 
Decisionmaking at the National Science Foundation,'' stated that, ``The 
United States has built the most successful research system in the 
world. The use of peer review to identify the best ideas for support 
has been a major ingredient in this success. Peer review-based 
procedures such as those in use at NSF, the National Institutes of 
Health, and other federal research agencies remain the best procedures 
known for ensuring the technical excellence of research projects that 
receive public support.'' In November 2009, the Executive Director of 
the Transportation Research Board at the National Research Council, 
provided testimony before Congress on how to facilitate the 
implementation of research at the Department of Transportation. In that 
testimony, the Director endorsed strongly the fact that NSF's merit 
review process is well suited to the mission of the agency. His 
observation: ``The more applied mitigation and adaptation research 
topics should be steered by the concerns and needs of policy makers and 
practitioners, while the fundamental research topics should be 
organized along the NSF model in which scholars and experts are guiding 
the decisions about which projects are likely to be most promising.''
    NSF's merit review process has served the agency, the scientific 
community, and indeed the country well for many years. Many Nobel 
Laureates, National Medal of Science and Technology winners, and 
MacArthur Foundation Fellows (popularly known as recipients of Genius 
Grants) have been supported by NSF at various stages in their careers. 
Through separate programs and in the course of funding specific 
scientific progress, over the past 25 years NSF has also supported the 
training of hundreds of thousands of graduate and post-graduate 
scholars in STEM fields. Discoveries stemming from NSF-funded projects 
have led to advances across all areas of science, engineering and 
education, with far-reaching impacts in the fields of nanotechnology, 
information technology, environmental science, genomics, STEM 
education, and many others.
    The high quality of NSF's merit review process is recognized 
globally, as evidenced by the fact that it has been used as a model by 
countries around the world that are newly establishing their own 
funding agencies. The merit review system for L'Agence Nationale de la 
Recherche (ANR), the French counterpart to NSF, is explicitly modeled 
after NSF, as is that of the Foundation for Polish Science. NSF helped 
the European Research Council establish its merit review system some 
five years ago, and was instrumental in helping Ireland establish 
Science Foundation Ireland. Back in 1986, a Chinese official came to 
NSF for six months to learn about our merit review and decision making 
processes, and subsequently incorporated what he had learned in 
establishing the National Natural Science Foundation of China (NSF 
09C). These are just a few examples of international agencies where NSF 
has had an explicit role in helping develop their merit review systems, 
but there are literally dozens of others that have borrowed our 
approach over the years.
    As the nature of research and the scientific enterprise continues 
to change--becoming more interdisciplinary, technological, 
international and collaborative--NSF continues to explore ideas and 
strategies that could strengthen the merit review process by enlarging 
the range of tools that can be used in proposal evaluation. These ideas 
have come from a variety of sources--internally, from the research 
community, from the practices of other funding agencies, and from the 
scientific literature on merit review. One idea that we are actively 
exploring is a greater use of technology-mediated virtual panels when 
and where it makes sense, with the hope that decreasing the travel 
burden will expand the potential pool of reviewers. Among the benefits 
that NSF would derive from an expanded pool of reviewers are the 
inclusion of more and varied perspectives, increased opportunities for 
participation by underrepresented groups, decreased review burden per 
individual reviewer, and decreased travel costs for the agency. We have 
established an internal working group to identify other viable 
candidates for pilot activities, and to develop plans for running and 
evaluating those pilot activities. We will be discussing these with an 
advisory committee over the next few months to get their help in 
refining the processes.
    For over 60 years, NSF has been forward looking in terms of how the 
agency manages its research and education portfolio. Merit review 
fosters the ``process of discovery,'' the means by which researchers 
can identify emerging scientific challenges and innovative approaches 
for addressing them. NSF is dedicated to ensuring that the merit review 
process remains robust, rigorous, and beyond reproach, in support of 
our mission and enabling us to pursue our goal of funding the world's 
best research in science, engineering and education.
    I appreciate the opportunity to appear before the Subcommittee to 
speak to you on this important topic. I would be pleased to answer any 
questions that you may have.

Appendix I: NSF Proposal and Award Process and Timeline



    Chairman Brooks. Thank you, Dr. Marrett.
    I now recognize our second witness, Dr. Keith Yamamoto, for 
five minutes.

              STATEMENT OF DR. KEITH R. YAMAMOTO,

                 VICE CHANCELLOR FOR RESEARCH,

             UNIVERSITY OF CALIFORNIA SAN FRANCISCO

    Dr. Yamamoto. Good morning, Chairman Brooks, Ranking Member 
Lipinski, distinguished Members of the Subcommittee. Thank you 
for the invitation to present a statement before you today. I 
am Keith Yamamoto, a molecular biologist, professor, and 
administrator at the University of California San Francisco. My 
lab's research has been funded continuously for 35 years by 
grants from the NIH and NSF.
    I have also been active in matters of science and public 
policy, leading or serving on dozens of committees focused on a 
broad range of these topics. One of these topics has been the 
federal merit review system for evaluation of biomedical and 
biological research grant applications, especially those 
overseen by NIH and to a lesser extent the NSF.
    Today, I shall describe the key conceptual and operational 
features of the merit review process, assess how well that 
process works from my point of view, consider the impact of a 
specific proposal to change the NSF process, and mention two 
modifications that might make merit review even stronger.
    Every application for NIH and NSF life sciences research 
support, and at NIH, that means 80,000 applications per year, 
we just heard the 55,000 at NSF, every application is 
rigorously reviewed and prioritized for scientific merit by 
special communities of experts. The details of the review 
process differ among the different federal agencies but there 
are two critical features that to my knowledge are common 
across agencies.
    The first is that the merit review is carried out by other 
scientists because only scientific peers have the knowledge and 
perspective required to assess the scientific significance, 
innovation, and impact of proposed research projects, and the 
second is that the review committees judge only scientific 
merit. It is left to others to assess relevance of the 
applications to the goals or portfolio of the funding agencies 
and to make the funding decisions themselves. Thus, peer review 
with a solitary focus on scientific merit.
    How does the process work and how well does it work? The 
review committees are federally chartered and populated by 
volunteer expert scientists who set aside time from their own 
research at institutions across the country to carefully 
evaluate written proposals for scientific investigation. The 
reviewers exercise their individual and collective scientific 
judgments, prioritizing the scientific merit of the 
applications. To help avoid nonscientific biases, both the 
applications and the reviews are standardized. For example, in 
the NIH process, every application is reviewed and scored 
according to the same set of criteria, significance, approach, 
investigator, environment and overall impact. By any measure, 
peer-driven merit review has been spectacularly successful at 
identifying and prioritizing the most interesting, innovative 
and significant scientific research, thus enhancing profoundly 
the strength of federally funded research. In contrast, merit 
review is not intended to influence the breadth or type of 
research that is funded. Rather, breadth and type are affected 
by funding mechanisms and by the stated missions of the funding 
agencies.
    How might the merit review process be changed to further 
strengthen it? Let me mention first a change currently under 
consideration that would, in my view, not serve the process and 
with two examples of changes that might enhance it.
    The National Science Board Taskforce on Merit Review has 
proposed a revision of the broader impacts criteria for NSF 
merit review, and I am quoting, ``to ensure consideration of 
how the proposed project advances national goals.'' This 
criterion departs, in my view, from the singular focus on 
scientific merit that is essential to the merit review process 
and calls upon reviewers to judge grant applications by metrics 
outside of their expertise.
    Moreover, NSF is the Nation's basic research agency as 
mandated by the NSF Act of 1950, which created the Foundation. 
The impact of individual basic research projects on broad 
national goals may be impossible to judge at the time of 
proposal but may emerge with time in unexpected and profound 
ways.
    Broad national goals of course are vital and need to be 
advanced collectively by federal science research agencies. 
They cannot, however, be mandated for individual NSF grant 
applications. Indeed, national goals can be addressed by 
development of funding mechanisms and by defining agency 
priorities, not by the merit review process.
    Now, while the current merit review process has great 
strengths, there are potential changes that might improve it 
further. I cite several in my written testimony. Let me mention 
just two here. Consider recognizing and validating that a grant 
application is not a contract for sure, in fact, not even a 
roadmap of experiments but rather an exercise in which the 
applicant demonstrates his or her capacity to pose a scientific 
problem and devise a research plan that would impact and 
advance a field. In reality, scientists pursue the implications 
of each day's results rather than adhering to the course of 
experiments imagined in their grants. Thus, grant application 
formats and merit review criteria would closely assess the 
merits of a proposed idea and of the investigator while 
reducing the current focus on experimental detail and 
feasibility.
    Second, consider formally denoting two separate classes of 
research, one I will call innovative and one we have been 
referring to as transformative, both essential but each 
requiring distinct merit review mechanisms. Innovative research 
would advance and deepen our understanding of current 
paradigms. Transformative research would disrupt or destroy 
prevailing paradigms and force creation of new ones.
    In closing, the merit review process for federal support of 
research is indisputably the best system for identifying 
highest quality of science. Its primary features, peer-driven 
review and singular focus on merit, have been critical in 
identifying grant applications that describe the best science 
by the best scientists. A healthy and robust merit review 
process is critical to maintaining and extending U.S. 
leadership in scientific research and innovation, which in turn 
is essential to reaching broad national goals.
    This concludes my testimony. I will be pleased to answer 
questions and address your comments. Thank you again for the 
opportunity to discuss this important matter with you.
    [The prepared statement of Dr. Yamamoto follows:]

              Prepared Statement of Dr. Keith R. Yamamoto,
                     Vice Chancellor for Research,
                 University of California San Francisco
    Good morning, Chairman Brooks, Ranking Member Lipinski, and Members 
of the Subcommittee. Thank you for the invitation to present a 
statement before you today. I am Keith R. Yamamoto, Vice Chancellor for 
Research, Executive Vice Dean of the School of Medicine and Professor 
of Cellular and Molecular Pharmacology at the University of California, 
San Francisco. I received a Bachelor of Science from Iowa State 
University and a Ph.D. from Princeton University before migrating to 
San Francisco, where I have been on the faculty for 35 years. My 
molecular biology lab has been studying the detailed mechanisms by 
which small molecules made in our bodies, hormones, control important 
physiological processes such as metabolism, stress responses and 
immunity; in the course of that work, I have had primary responsibility 
for training approximately 100 graduate students and postdoctoral 
scholars. Our research has been funded throughout by grants from the 
NIH and NSF.
    For the past 30 years, I have also been active in matters of 
science and public policy, leading or serving on dozens of committees 
focused on a broad range of issues, challenges and opportunities. One 
of the major areas of emphasis for those activities has been federal 
merit review system for evaluation of biomedical and biological 
research grant applications, especially those overseen by the National 
Institutes of Health (NIH) and the National Science Foundation (NSF). 
For example, I co-chaired the NIH effort to assess and enhance its peer 
review process, and served on the National Science Board/NSF task force 
on transformative research. These extensive experiences have provided 
me with a rather deep perspective on the merit review process and its 
relationship to the U.S. life science research enterprise.
    In my testimony today, I shall: (1) describe the key operational 
and organizational features of the merit review process for biomedical 
and biological research; (2) assess how well that process works; (3) 
consider the impact of a particular change to the process that is 
currently being considered; and (4) enumerate some potential 
modifications that might further improve the process.
    Every application for NIH and NSF life sciences research support 
(NIH, for example, receives some 80,000 applications per year) 
undergoes rigorous review and prioritization of scientific merit by 
committees of peers, typically by scientists who themselves hold grants 
from the same agency, and whose research is in the same area of 
research as the proposed study. The details of the review process 
differ among the different federal agencies, but evaluation by peers is 
the crucial common feature.
    To populate the federally chartered merit review committees, 
agencies enlist volunteer service from expert scientists in each area 
of research, who agree to set aside time from their own scientific 
research at academic and research institutions around the country, 
typically several times a year, to carefully evaluate written proposals 
for scientific investigation. The reviewers exercise their individual 
and collective scientific judgments free of other biases, prioritizing 
the scientific merit of the applications.
    To help prevent nonscientific biases, the formats of both the 
applications and the reviews are tightly delineated, whereas the 
applicants have broad flexibility in choosing the scientific subject 
matter of the applications. In the NIH process, for example, a 
standardized set of five criteria (significance, approach, innovation, 
investigator, and environment) is mandated as the basis for rating 
every application. Assessment of the investigator is focused solely on 
past performance and qualification for carrying out the proposed study. 
Importantly, merit review committees are charged to judge only 
scientific merit. In particular, they do not assess relevance of the 
applications to the portfolio of the funding agency, nor do they not 
make funding decisions. This singular focus on merit (together with 
peer-driven review) is the second key feature of the merit review 
process.
    Thus, it is important that decisions of scientific merit are 
insulated and separated from decisions of funding. In addition, of 
course, the merit review process has no control over the level of 
funding allocated to support meritorious applications. When funding 
levels fall far below the capacity to support some of the very best 
applications (as is presently the case), the merit review process 
appears to fail, i.e., outstanding science goes unfunded. This apparent 
failure instead reflects a misalignment between the number of highly 
meritorious applications and the number of dollars available to fund 
those applications.
    Any merit review process that depends upon peers to carry out 
evaluations must acknowledge and address at least two intrinsic and 
related conflicts of interest: reviewers might unfairly support 
applications from their friends to create an ``old boys'' network, or 
they might unfairly disadvantage applications from competitors or from 
those outside of some perceived ``inner circle.'' In general, these 
intrinsic conflicts have been addressed successfully by well-crafted 
regulations, and more importantly, by a universal ``culture of 
respect'' from the participating scientists who serve as reviewers.
    While peer-driven merit review plays a crucial role in identifying 
excellence among proposed ideas and research plans, it is also the case 
that an element of conservatism is intrinsic to peer review, which 
complicates recognition and prioritization of ``transformative'' ideas 
and approaches. This is because the majority of scientists, and 
therefore the majority of peer reviewers, embrace and extend prevailing 
scientific paradigms, whereas transformative research disrupts or 
destroys accepted paradigms and creates new ones. Because both types of 
research are essential, approaches to address intrinsic conservatism 
are important. One strategy is to adopt special funding mechanisms, and 
perhaps some special elements of the review process, designed to 
identify ideas.
    The current merit review processes, which have been in place in the 
U.S. for over 65 years, recognize that only scientific peers have the 
knowledge and perspective required to assess the relevance, innovation 
and impact of proposed research projects. Indeed, the current system of 
peer-driven merit review is widely held to be not the ``best good 
system'' for evaluation and prioritization of merit, but ``the only 
good system.''
    By any measure (e.g., quality of scientific publications resulting 
from support of meritorious applications, honors and prizes given in 
recognition of the highly meritorious research, products and services 
that are developed from the results of supported research, creation of 
an outstanding scientific workforce resulting from training of students 
and fellows supported), peer-driven merit review has been spectacularly 
successful at identifying and prioritizing the most interesting, 
innovative and significant scientific research projects. Hence, the 
merit review process enhances profoundly the strength of the research 
funded by the federal government. In contrast, the merit review process 
is not intended to influence the breadth or type of research that is 
funded. Rather, breadth and type are strongly influenced by funding 
mechanisms, and by the range and diversity of disciplinary foci that 
are chosen by the funding agencies.
    Are there specific changes that might further strengthen the merit 
review process to ensure support of the best science? Let me mention 
first a change currently under consideration that would in my view 
damage the process, and then end with a brief enumeration of some 
potential changes that might enhance it.
    The National Science Board (NSB) Task Force on Merit Review is 
currently reviewing the NSF's merit review criteria, and has proposed a 
revision of both the ``Intellectual Merit'' and the ``Broader Impacts'' 
requirements, with the goal of clarifying their intent and the ways 
that they would be used in the merit review process. The purpose of the 
Broader Impacts criterion ``is to ensure the consideration of how the 
proposed project advances a national goal(s).'' This criterion, as 
stated, would in my view adversely affect the merit review process 
because it departs from the singular focus on scientific merit that is 
essential to the process, and because it obligates peer reviewers to 
judge grant applications by metrics outside of their expertise.
    This being said, I concur fully that broader national goals are 
essential, and as stated in the NSB Merit Review Principles, 
``collectively, NSF projects should help to advance a broad set of 
national goals.'' It is important, however, to remain mindful of the 
language of the NSF Act of 1950, which directs the Foundation ``to 
initiate and support basic scientific research and programs to 
strengthen scientific research potential and science education at all 
levels.'' Indeed, NSF itself originated from ``Science the Endless 
Frontier,'' the redoubtable 1945 policy initiative of Vannevar Bush, 
which called out untargeted basic research as ``the pacemaker of 
technological progress'' in which ``new products and new processes are 
founded on new principles and new conceptions, which in turn are 
painstakingly developed by research in the purest realms of science.''
    This implies that broad national goals should be advanced by the 
composite federally funded scientific research endeavor, and in 
particular should not be mandated for individual NSF research grant 
applications. In general, such goals should be addressed by development 
of funding mechanisms and by defining agency priorities, and not as a 
part of the merit review process. Moreover, it seems that broad 
national goals might specifically be mandates for mission-driven 
agencies that seek to support research relevant to health, environment, 
energy, food and agriculture, or national security, rather than for the 
National Science Foundation.
    While the current merit review process has great strength, there 
are conceptual and operational aspects that might improve it further. 
Some examples for consideration across a wide spectrum:

     Reconfigure the grant application conceptually to be 
viewed less as a ``contract,'' and more as demonstrations of an 
investigator's capacity to identify an important scientific problem and 
devise tests that could advance knowledge and understanding, and 
thereby impact the field. Thus, the grant application is not intended 
as a ``roadmap of experiments'' projected three to five years into the 
future. Grant application formats and merit review criteria should 
place greater focus on the merits of the proposed idea and of the 
investigator, whole reducing the current focus on proposed experimental 
details and feasibility.

     Motivate top scientists to maintain active participation 
in the merit review process, in part by developing mechanisms that more 
effectively encourage applicants to submit bold scientific ideas.

     Establish and formalize two separate investigator-
initiated funding mechanisms, innovative and transformative, which 
invite, identify and support research that, respectively, advances and 
deepens our understanding of current paradigms, or disrupts and 
destroys prevailing paradigms, and forces creation of new ones. 
Consider unique aspects of merit review process for transformative 
applications.

     The topical/disciplinary focus of review committees has 
been eroded by the demands of reviewing a rapidly increasing proportion 
of grant applications that include a remarkable diversity of 
experimental approaches. To recover the conceptual focus of merit 
review committees, institute a ``focused external review'' process in 
which ad hoc reviewers are requested to contribute electronically brief 
comments that address solely those technologies or approaches for which 
specific expertise is lacking on the chartered committee. Such external 
reviewers would not be asked to assess the overall scientific merit of 
the application; that responsibility would reside solely with the 
chartered committee.

    The merit review process used for review of federal grant 
applications for support of biological and biomedical research is 
indisputably the best system for managing this important 
responsibility. Its primary features of peer-driven review and singular 
focus on merit have been critical in identifying grant applications 
that describe the best science by the best scientists. Once unique in 
the world, the process is being widely emulated.
    This concludes my testimony. I would be pleased to answer your 
questions or address your comments. Thank you again for the opportunity 
to discuss this important matter with you.

    Chairman Brooks. Thank you, Dr. Yamamoto.
    I recognize our next witness, Dr. Nancy Jackson, for her 
five minutes.

               STATEMENT OF DR. NANCY B. JACKSON,

              PRESIDENT, AMERICAN CHEMICAL SOCIETY

    Dr. Jackson. Good morning, Chairman Brooks and 
distinguished Members of the Subcommittee.
    As President of the American Chemical Society (ACS) during 
2011, the International Year of Chemistry, it is my great 
pleasure to address the Subcommittee this morning. Founded in 
1876, ACS is the world's largest scientific society with more 
than 163,000 members. A nonprofit organization, ACS was 
chartered by the Congress in 1937 to advance chemistry in all 
its branches, promote scientific research and inquiry, and 
foster public welfare and education. ACS members work in 
industry, academia and government. ACS is a long-time supporter 
and strong supporter of NSF and its merit review process, which 
is recognized globally as the gold standard for identifying the 
best research to fund. My testimony today will concentrate on 
chemistry and NSF's impact on our science.
    The Federal Government is an important source of support, 
particularly for basic research conducted within the chemical 
enterprise that also trains tomorrow's researchers. By 
stimulating innovation, the Federal Government empowers a 
competitive U.S. chemical enterprise, which contributes to U.S. 
economic growth.
    Mr. Chairman, I would like to focus on three areas from my 
written testimony that I believe are of particular value in 
these times of constrained resources and economic challenges. 
Those three areas are: one, balancing the NSF research 
portfolio; two, streamlining the merit review process; and 
three, measuring return on NSF investment.
    My first point has to do with portfolio management. As 
anyone with a retirement fund has been told, managing a 
portfolio is critical to its long-term strength, finding the 
right balance between proven performing stocks and riskier, 
higher-return investments. Managing a research portfolio is 
similar, striving to find the right balance between more 
incremental research that builds on previous successes and 
those ideas that are riskier and could lead to game-changing 
developments. In times of tight budgets and restricted funding, 
NSF mustn't become too conservative. The agency must ensure 
adequate attention to providing opportunities for young 
researchers as well as for out-of-the-box research that may 
create economic renewal, produce jobs and train the U.S. 
scientific workforce of the future.
    Secondly, NSF should be able to triage the least 
competitive submission from the review process. At the present 
time, all proposals submitted to NSF must go through an 
extensive peer review, even if they are intellectually weak 
with no hope of ultimately getting NSF funding. For every 100 
grant proposals a program officer reviews, a small number, 
perhaps around 10, will be of such high quality that it is 
obvious to everyone that it should be funded. Another 50 
proposals easily will be recognized as not competitive. 
However, in the current system, they all take considerable time 
to review and process. The real agonizing choices must be made 
concerning the remaining 40 proposals, all very good proposals. 
Under current funding, only 13 of those 40 proposals would be 
funded. The merit review process should be focused on these 
choices.
    Over the last 10 years, the number of grant proposals 
submitted to the NSF chemistry division has doubled while the 
size of the NSF chemistry division staff has remained the same, 
thereby presenting a significant challenge to NSF to continue 
to perform its job of identifying the best science.
    ACS is responsible for a research fund, albeit small 
compared to NSF. It is the ACS Petroleum Research Fund (PRF). 
It has seen the number of grant proposals skyrocket in recent 
years as well. To help manage this increase, a new policy was 
recently implemented that empowers PRF research managers to 
reject proposals at the outset that they deem to be poor in 
quality. The practical result has been that more time is freed 
up for PRF staff and the volunteer peer reviewers to focus on 
selecting the right balance of research from the strong 
proposals. Empowering NSF research managers to likewise would 
be a practical step to focus a steadily increasing NSF workload 
and maintain excellence in its merit review process.
    And that brings me to my third point. We need new tools and 
methods to evaluate the success of NSF investment. While ACS 
strongly supports NSF's broader impacts criteria, we need 
better tools to determine their overall effectiveness. 
Measuring how many fellowships or grants are funded is easy; 
measuring increased innovation, improved national security or 
broadening participation is complex and must take place over a 
longer time scale.
    It is difficult to single out how one individual effort has 
impacted a complex collection of national priorities. I 
understand that finding the right metrics for measuring these 
issues is very difficult but it is worthwhile. Empowering NSF 
to more easily gather data is necessary to ensure that research 
is successful in achieving the broader impacts and that the 
selection process is meeting its many goals.
    Chairman Brooks, I thank you for the opportunity to testify 
today and I would be happy to answer any questions you or your 
Subcommittee have.
    [The prepared statement of Dr. Jackson follows:]

                Prepared Statement of Dr. Nancy Jackson,
                  President, American Chemical Society
    As president of the American Chemical Society, or ACS, it is my 
great pleasure to address the Subcommittee this morning on the topic of 
the merit review process of the National Science Foundation (NSF).
    Founded in 1876, ACS has grown to be the world's largest scientific 
society with more than 163,000 members and one of the world's leading 
sources of authoritative scientific information. A nonprofit 
organization, ACS was chartered by the U.S. Congress in 1937 to advance 
chemistry in all its branches, promote scientific research and inquiry, 
and foster public welfare and education. ACS members work in industry, 
universities and colleges, and at national laboratories.
    ACS is at the forefront of the evolving worldwide chemical 
enterprise. It is the premier professional home for chemists, chemical 
engineers and related professionals around the world as well as a 
global leader in chemical information. ACS publishes 41 world-class 
scientific journals and operates the Chemical Abstracts Service, which 
provides the most comprehensive databases of disclosed research in 
chemistry and related sciences.
    Every year, ACS gives more than $11 million in grants for basic 
research in petroleum and related fields through the Petroleum Research 
Fund (PRF). Twenty-five researchers, who were recipients of these 
grants, later went on to become Nobel Laureates.
    The Society also plays a leadership role in educating and 
communicating with public policy makers and the general public about 
the importance of chemistry in our lives. This includes identifying new 
solutions to global challenges, improving public health, protecting the 
environment, and contributing to the economy.
    ACS has been a strong, long-time supporter of the National Science 
Foundation, which is of particular importance at this critical time in 
our nation's history. My testimony will concentrate on chemistry and 
NSF's impact on our science. While I think my observations and 
recommendations are broadly applicable, chemistry is my area of 
expertise.
    Chemistry is the fundamental science that is at the heart of 
processes and products that meet our most fundamental needs for food, 
shelter, and health, as well as developments and materials that are 
vital to advances in biotechnology, computing, and telecommunications. 
It is a keystone of U.S. manufacturing and is essential to a range of 
industries.
    America's $720 billion chemical industry is one of our nation's top 
exporters, with $171 billion in annual exports, which accounts for more 
than 10 cents of every dollar in total U.S. merchandise exports. Within 
the United States, the chemical industry employs 784,000 people and is 
a driver of innovation. The industry invests $55 billion in research 
and development annually, and one in five U.S. patents is chemistry 
related. In addition, the industry contributes to human and 
environmental health. Drug innovations, made possible through 
chemistry, have helped increase life expectancy in the United States by 
30 years over the past century.
    I mention this today because the success of the chemical enterprise 
is due largely to scientific and technological breakthroughs and 
advances made in industrial, academic, and government laboratories. 
Although much of the nation's chemical research is carried out by 
scientists, engineers, and technicians employed in industry and 
academia, the Federal Government is an important source of support, 
particularly for basic research conducted by our nation's universities 
and government laboratories. By stimulating the roots of innovation, 
the Federal Government plays a fundamental role in ensuring the ability 
of the U.S. chemical industry to stay competitive in the long term. And 
because so many other industries depend on chemicals, the federal 
investment enhances the ability of the United States to compete 
globally by enabling a high-tech, competitive chemical industry to 
supply new products at prices that give our nation's producers an edge.
    The NSF plays a unique role in the U.S. scientific enterprise. 
While other federal agencies have missions directed at advancing 
specific science and technology in health or energy, for example, the 
core mission of NSF is to foster a healthy scientific enterprise here 
in America. Supporting the best ideas and exploring new frontiers 
across research disciplines have been the hallmark of NSF and the 
backbone of the American research system.
    NSF has played a pivotal role in paving the way for scientific 
discovery, in large part, by awarding grants to members of the 
scientific research community that have demonstrated outstanding merit. 
The Foundation accomplishes its mission by supporting fundamental 
research and education in science and engineering. From aircraft 
design, pioneering medical tools and robotics, to discovering how 
children can learn chemistry better, NSF has played a key role in 
funding discoveries that have driven the nation's economy, improved our 
quality of life, and enhanced national security. It also supports high-
risk research and novel collaborations that could deliver exceptionally 
high rewards.
    NSF is not just about research. It's also about developing and 
training tomorrow's scientific workforce. There is a symbiotic 
relationship between research and education. When a graduate student or 
a post-doctoral student works with a researcher funded by NSF, the 
student is honing skills and adding new scientific knowledge. In this 
way, the torch is passed from one generation of researchers to the 
next. To put it another way, this is how we keep pushing the edge of 
the envelope. If the United States is to continue to be a leader in 
science and technology, then we need to have the trained workforce 
working in that space.
    NSF provides more than 20 percent of the federal support for basic 
research at academic institutions and supports roughly 10,000 new 
awards per year through the merit reviews of over 40,000 proposals 
received. Every year, an estimated 200,000 people, from undergraduates 
to senior faculty, participate directly in NSF research and education 
programs.
    The NSF merit review process is the gold standard worldwide, and is 
one of the reasons why U.S. science has been as successful as it is. 
When other countries seek to set up their own national research 
efforts, they often look to the U.S. NSF as the role model to emulate.
    At NSF, all proposals are evaluated for intellectual merit and 
broader impacts. NSF receives far more meritorious proposals than it 
could ever fund. While a proposal with weak intellectual merit has no 
hope of getting NSF funding, many proposals are rated ``excellent'' 
with strong intellectual merit and still do not get funded because of 
the stiff competition. The broader impacts criteria take into 
consideration which research is the most urgent or has the greatest 
relevance to improving the quality of life. This merit review process 
enables NSF to ensure that precious R&D money goes only to the most 
pressing R&D needs.
    As anyone with a retirement fund has been told, managing a 
portfolio is critical to its long-term strength. Financial advisors 
stress that it's important to find the right balance between solid 
performing stocks and riskier investments that may provide higher 
returns. Managing a research portfolio is similar: the research 
manager, whether working in industry or at NSF, strives to find the 
right balance between science that will deliver steady advances and 
ideas that are out of the box, but could result in game-changing 
developments. This point is especially important in times of restricted 
funding. It's human nature to make more conservative choices and be 
risk-aversive when times are tough. However, now more than ever, 
America needs pioneering research that will create economic renewal, 
produce jobs, and train the scientific workforce of the future. Extra 
efforts and attention must be paid to cultivating young researchers and 
game-changing ideas.
    One of the reasons why the merit review process is so successful is 
because it draws from the collective wisdom of the scientific 
community. Many NSF personnel come directly from the scientific 
community and will return to their research institutions at the end of 
their two- or-three-year rotations. Relying on rotating directors means 
the managers are up to date on the most recent scientific developments. 
The panels that perform the peer review of proposals are fellow 
researchers in the field, and as such, are also up to speed on the 
latest developments. This scientific community service, whether 
performed by grant proposal reviewers or NSF program officers, is an 
integral part of scientific culture. Many scientists dedicate their 
time in this way because it provides an opportunity to remain in touch 
with and influence the cutting edge, as well as because they understand 
that the system only works if everyone volunteers to play their part. 
In a way, it is the science community's way of 11paying it forward.''
    The merit review process requires significant efforts by both NSF 
employees and scientist volunteers. To better understand how the 
process plays out, consider this example from the NSF chemistry 
division. The division receives about 1,800 proposals annually. Each 
program officer in the division manages about 100 proposals a year. 
These managers are responsible for picking peer reviewers, and they 
must do so with an eye for diversity across a large number of factors 
such as ensuring that the reviewers reflect a balanced group based on 
type of institution (e.g., small undergraduate colleges vs. large 
research universities), geography, and racial and gender 
characteristics.
    Peer reviewers must also be experts within the proposal's subfield 
of chemistry. Generally, a program officer approaches three reviewers 
to find one who will accept the call to serve. Since each proposal 
requires three to five reviewers, this means the officers approach six 
to 10 reviewers for each proposal. Therefore, on the average, the NSF 
chemistry division approaches between 10,000 to 18,000 researchers to 
serve as peer reviewers for the proposals submitted.
    For every 100 grant proposals a program officer reviews, a small 
number (perhaps 10) will be of such high quality that it is obvious 
they should be funded. Another 50 proposals will be recognized as 
clearly not competitive; however, they still must be considered through 
the process. An agonizing choice must then be made over the 40 
proposals in the middle. These include proposals that may be considered 
excellent or very good. In fiscal year 2010, the NSF award rate was 23 
percent. In our analogy of 100 proposals, this would mean that 13 out 
of the remaining 40 would be funded.
    The broader impacts criteria include considerations about whether 
the research proposal would broaden underrepresented minorities' 
participation in science, strengthen U.S. infrastructure, improve 
national security, or foster innovation. Some of these impacts are the 
result of language in the America COMPETES bill enacted last year. The 
broader impacts criteria take into consideration which research is the 
most urgent or has the greatest relevance to improving the quality of 
life. The broader impacts criteria enables NSF to choose between 
meritorious and even more meritorious proposals, and is a way to ensure 
that precious R&D money goes to the most pressing R&D needs.
    It should be added that the number of grant proposals submitted to 
the NSF chemistry division steadily increases each year, more than 
doubling from levels 10 years ago. And while the number of proposals 
has doubled, the size of the NSF chemistry division staff has remained 
the same. NSF is challenged to continue to perform its job of 
supporting the best science, even as the sheer number of proposals 
competing for funding has ballooned.
    I mentioned in my introduction that ACS is responsible for the 
management and administration of the ACS Petroleum Research Fund (PRF), 
which was established in 1944 by seven oil companies as a perpetual 
trust to advance science education and fundamental research in the 
petroleum field. In 2010, the Fund provided $11.4 million for research 
grants.
    Although PRF is a small research fund, like the NSF, it has seen 
the number of submitted grant proposals skyrocket in recent years. 
Perhaps some insights gleaned from PRF would be useful in considering 
how to strengthen the NSF merit review process.
    Several years ago, to relieve the growing peer review burden on the 
science community and to lighten the administrative load on the PRF 
staff, a policy was implemented to withdraw proposals from 
consideration that were deemed to be ``poor'' from the get-go: This 
includes those that are poorly written, use bad science, or do not 
address the specific scientific areas that were eligible for funding. 
In practice, this means that the managers now triage approximately 20 
percent of the grants that come their way. These managers err on the 
side of caution: If there is any doubt that a proposal may have some 
merit, it is forwarded to the peer review panels for consideration and 
ranking. The result has been that, while some ``poor'' proposals are 
removed from the evaluation process, more time and energy is freed up 
for PRF staff and the volunteer peer reviewers to focus on selecting 
the right balance of research from the strongest proposals.
    Currently, NSF does not have the freedom to remove any proposals 
from the very bottom of the pile from consideration. Empowering NSF 
research managers to do so--provided that specific criteria are taken 
into consideration--would be a simple step to help NSF maintain 
excellence in its merit review process. If the average acceptance rate 
for an NSF proposal is about 23 percent, this means that 77 percent of 
funding proposals will be turned away. Enabling managers to remove the 
lowest 20 percent of those that would normally be rejected from 
consideration is highly unlikely to result in a potentially great 
proposal not getting its due consideration. Instead, this approach may 
be a practical step to balancing a steadily increasing NSF workload.
    I have mentioned the important role the broader impacts criteria 
play in the NSR merit review process. NSF promotes broadening 
participation of underrepresented minorities and women, and persons 
with disabilities. This also includes increasing diversity in the NSF 
portfolio with respect to types of institutions supported and the 
geographic regions represented. Broadening participation is one way to 
address the broader impacts criteria; however, other activities are 
also appropriate.
    The importance of NSF efforts to broaden participation of 
underrepresented minorities in science and engineering is well 
understood and supported by the scientific community. We know that if 
the scientific work force doesn't reflect the demographics of our 
country, we risk missing out on bringing the best minds and talents 
from every community to work on the scientific challenges that will 
impact all of our lives.
    In spite of NSF efforts in the broader impact criteria areas, we 
could use better tools to measure how effective these NSF efforts have 
been. One difficulty of measuring the long-term impact of the broader 
impact criteria is that it's easier to measure the inputs than the 
outputs. Measuring how many fellowships or grants are funded is easy. 
Measuring increased national innovation, improved national security, or 
broadening participating is complex and must take place over a longer 
time scale. These differences make it difficult to single out how one 
individual effort has impacted a complex collection of national 
priorities.
    As a scientist, I want these efforts to be successful. As a senior 
administrator, I recognize that it's nearly impossible to measure 
success if you can't measure long-term outputs. In industry, we 
understand that finding the right metrics is very difficult, but it's 
worthwhile to try. I do not know how to resolve this issue, but I do 
believe that empowering NSF to more easily gather the data needed to 
measure the success of the broader impacts criteria would be a 
necessary step to ensuring those efforts achieve their desired effect 
to the maximum extent possible.
    Chairman Brooks, I thank you for the opportunity to testify today 
and to share these thoughts with you. ACS believes the NSF is the 
cornerstone of the U.S. scientific enterprise, and we stand ready to 
assist you efforts to strengthen the agency for the benefit of the 
scientific community and the entire nation.

    Chairman Brooks. Thank you, Dr. Jackson.
    Next, we recognize our final witness, Dr. Jose.

                  STATEMENT OF DR. JORGE JOSE,

        VICE PRESIDENT FOR RESEARCH, INDIANA UNIVERSITY

    Dr. Jose. Good morning, Chairman Brooks, Ranking Member 
Lipinski and distinguished Members of the Subcommittee, and 
especially Representative Bucshon from Indiana. Thank you for 
allowing me the opportunity to speak with you today.
    My remarks arise from my experience as a primary 
investigator funded for many years by the National Science 
Foundation, as a member of NSF review panels, and an advisor 
for strategic planning at both the NSF and the National 
Institute of Health (NIH). I have been the Vice President for 
Research at two major research universities, for five years at 
the State University of New York in Buffalo, and since last 
August at Indiana University. Both institutions are members of 
the American Association for Universities and the Association 
for Public Land-Grant Universities.
    There are three points I would like to emphasize this 
morning. First, merit review is the most effective process we 
have for ensuring that federal funds are used to support the 
most important and far-reaching scientific research. Second, 
merit review is the best way to ensure impartiality in awarding 
funding to research. And finally, federal funding for 
scientific research awarded through rigorous process of merit 
review is a necessary and important component of continued 
American preeminence in the world and is the foundation of our 
Nation's long-term economic and national security.
    A very large number of scientific problems are interesting 
in principle but only a small number of those problems is 
important and deserves the investment necessary to search for 
and find solutions. I support what Dr. Jackson said about not 
all of the problems that people want to study are interesting 
or relevant for study. My broad experience in the process by 
which the Federal Government funds scientific research at 
universities leaves me to conclude while no review system is 
perfect, merit review is the best way to identify those 
problems that are important to support.
    One indication of the strength of the merit review system 
is the number of major scientific breakthroughs resulting from 
research that was awarded federal funding as a result of peer 
review by experts, people uniquely able to recognize the 
potential of proposed research. For instance, one can imagine 
how the average person in the street would respond if asked 
whether or not we should spend taxpayer dollars to study C. 
elegans, which is a tiny, transparent worm, a millimeter in 
size. Yet federally funding both at the NSF and the NIH on this 
worm has enabled scientists to identify key genes regulating 
organ development and programmed cell death and has shown that 
corresponding genes exist in high-end species including humans, 
of course. Nine recipients of Nobel Prizes since 2002 focused 
their work on C. elegans. Research on C. elegans has led to 
clinical trials on the treatment of macular degeneration, 
asthma, diabetes and brain diseases. These advances occurred 
because experts in the field recognized that this tiny worm 
would make a good model system to study and could help us learn 
important things about humans.
    I would like to comment on the broader impact criteria that 
are part of the merit review. In my experience, the broader 
impacts criteria are used only to decide between proposals of 
equal scientific merit, and this is entirely appropriate. 
Having said that, I would like to stress the importance of 
expanding the participation of underrepresented groups in the 
sciences, a goal which over the past 10 years has been the 
primary broader impact criteria for NSF. This is not a matter 
of establishing quotas or prioritizing this participation over 
the scientific importance of a proposal but the NSF emphasis on 
this is the demonstrated way of increasing the pool of talented 
and trained scientists which in turn enhances American economic 
competitiveness and leadership in scientific research, 
discovery and innovation.
    In these challenging times, it is more important than ever 
that federal funds are spent wisely. Federal research funding 
has for 70 years been the cornerstone of American economic 
security, scientific and educational preeminence, maintaining 
our competitiveness globally, continuing American leadership in 
scientific research and innovation and ensuring that our 
children and grandchildren enjoy a future in which American 
higher education and industry remain the envy of the world. All 
these rely on continued robust federal funding for scientific 
research and discovery, funding awarded through the merit 
review system as my colleagues have already stated.
    Thank you very much for inviting me here this morning and I 
look forward to answering any questions you may have. Thank 
you.
    [The prepared statement of Mr. Jose follows:]

                 Prepared Statement of Dr. Jorge Jose,
            Vice President for Research, Indiana University
    Chairman Brooks, Ranking Member Lipinski and distinguished Members 
of the Subcommittee, it is an honor to be here this morning to speak 
with you about the important topic of the merit review process in 
federal funding for scientific research, and in particular at the 
National Science Foundation. My remarks today arise from my experience 
as a primary investigator funded for many years by the NSF, as a member 
of NSF review panels, and as an advisor for strategic planning at both 
the National Science Foundation and the National Institutes of Health. 
I also speak from my experience as Vice President for Research at two 
major research universities, for five years at the State University of 
New York at Buffalo, and since last August, at Indiana University. Both 
institutions are members of the American Association of Universities 
and the Association of Public and Land-Grant Universities. This morning 
I would like to speak with you about the importance of the merit review 
process, specifically, its integral place in establishing American 
preeminence in hIgher education, scientific investigation, and economic 
innovation.
    My broad experience in different aspects of the process by which 
the federal government funds scientific research at universities leads 
me to conclude that while no system of review is perfect, nor 
guaranteed to fund only the best scientific research, the merit review 
system is the most effective process we have for ensuring that federal 
funds are used most effectively in support of scientific research, in 
particular at this time of limited resources when we need to prioritize 
how the taxpayer dollars are best invested.
    In some respects, the challenges facing federal funding agencies 
such as the NSF and the National Institutes of Health are very much 
like the challenges I face as Vice President for Research at Indiana 
University. As you may know, research universities often invest some of 
their limited resources to catalyze programs of research into issues 
that are of fundamental importance to our State, our Nation, and the 
world. My goal is to help our researchers identify and address the most 
important scientific, social and economic problems ofthe 21st century, 
such as energy security, health care, national security and our global 
competitiveness. Addressing these problems is not only valuable but is 
also a necessity because the problems will not solve themselves. 
Seemingly intractable problems can be solved only when the best minds 
with the appropriate expertise are brought together, and America's 
current and future well-being depends in an essential way upon the 
results of research into these problems.
    As important as this work is, research is just one among many 
important areas of the University's work and available resources--at IU 
as within the Federal Government--are limited. Given limited resources 
and given the importance of tbe problems, it is crucial that IU directs 
available resources to the projects with the strongest likelihood of 
being transformative and successful. I rely heavily on the ability of 
experts to assess each proposal, the work plan, and the potential of 
specific people to carry out a project successfully. This guidance 
helps to ensure that funding decisions are made on the basis of 
scientific merit rather than personal or political considerations. In 
short, merit review must be the foundation of funding decisions we make 
at IU because merit review enhances the likelihood that we will 
properly invest our limited available funds into research projects with 
the strongest potential for innovation, for transforming a field, or 
addressing successfully an important problem.
    We are, of course, following the path that was set up by Vannevar 
Bush in his ``Science--the endless frontier'' developed right after 
World War II, that led to the formation of the National Science 
Foundation. NSF introduced the merit review process as an essential 
component to assess and determine how tax dollars should be best 
allocated to scientific research. The merit review process has for the 
last 60 years led to many NSF notable successes. It is very important 
to recognize that a very large number of scientific problems are 
interesting in principle but a much smaller subset of those problems is 
important and deserves the investment necessary to search and find 
solutions.
    Merit review is the best way to identify the important problems. It 
is the best way to ensure that federal funds are invested in a healthy 
array of important problems, covering a breadth of areas and approaches 
within a particular field. And merit review is the best way to ensure 
impartiality, so that the best science and the best scientists are 
funded.
    In his planning for the NSF, Vannevar Bush drew on the experience 
of wartime scientific research, organized through the National Defense 
Research Council (NDRC). Wartime federal investment in scientific 
research resulted in the development of penicillin, the radar, and most 
famously, the atomic bomb. NDRC research brought together the very best 
scientific minds from Europe and the U.S. to work on the most important 
problems of the time--and resulted in discoveries that helped America 
become the pre-eminent economic, military, and scientific power of the 
20th century. Since the end of the Second World War, federal investment 
in research at American universities has been central to the 
development of universities that are the envy of the world, and merit 
review has been central to funding research that enables the United 
States to remain the leader in scientific inquiry, the development of 
new technologies, and the translation of fundamental research into 
applications that shape our lives every day. While there are many ways 
to assess and demonstrate the value of the merit review system, allow 
me to focus on one. The strength of relying upon merit review to 
determine what small percentage of proposals will be funded is 
demonstrated by its adoption by other countries across the world. 
Funding agencies in Europe, South America, and Asia, which are trying 
to emulate the research breakthroughs the U.S. has had in the last 60 
years, all rely on a merit review system in which experts assess 
proposed research much as we do in the United States.
    I can give you a long list of items we use every day which were 
often developed as a result of the research that was funded by the 
federal investments in scientific research. One which the large 
majority of Americans use is the cell phone; another is the Global 
Positioning System, which was developed by the Department of Defense 
but which was based on a trail of research discoveries that started 
with the work by Einstein on his mathematical theory of gravity. GPS 
use is so common that it is hard to imagine how we found our way 
anywhere before it became commercially viable! We also hear the weather 
report, in particular during these hot days; the food we eat has been 
produced using scientific breeding techniques. We often don't think 
about where all these things came from, but we do know that Americans 
as a whole are very proud of the long tradition of scientific research, 
inventiveness and innovation, a tradition which has made the U.S. the 
advanced technological society that it is today.
    It might be helpful to reflect for a moment on a couple of specific 
examples of the impact of federal funding for scientific research, 
because these examples point to the success of merit review. They also 
demonstrate that it is not always possible to anticipate what kind of 
impact research may have. For example, Caenorhabditis elegans (c. 
elegans), a transparent worm that most of us can go our whole lives 
without thinking too much about, but which has been the subject of 
significant research since the early 1960s. Only one millimeter in 
length, it was the C. elegans which first had its full genome 
sequenced, prior to the big achievement of decoding the human genome. 
For many reasons, C. elegans is a useful model organism, enabling 
researchers to learn about genetics, cell biology, and the pathogenesis 
that relate to many human diseases. Three times since 2002, Nobel 
Prizes have been awarded to researchers working with C. elegans. The 
2002 Nobel Prize in Physiology or Medicine was shared by three 
researchers, Sidney Brenner (Berkeley), John Sulston (Cambridge), and 
Robert Horvitz (MIT), whose work was funded largely by the National 
Institutes of Health through its processes of merit review. The 2002 
Laureates identified key genes regulating organ development and 
programmed cell death and showed that corresponding genes exist in 
higher species, including man. The 2006 Nobel Prize in Physiology or 
Medicine was awarded to longtime NIH grantees Andrew Fire (Stanford) 
and Craig C. Mello (U. Massachusetts) for their discovery of RNA 
interference in C. elegans--work which has led to clinical trials in 
the treatment of macular degeneration, asthma, diabetes, and brain 
diseases. Further, the 2008 Nobel Prize in Chemistry was awarded to 
Martin Chalfie (Columbia) for his work on green fluorescent protein in 
C. elegans (along with Roger Tsien and Osamu Shimomura, who studied 
these proteins in other contexts). I still read many C. elegans papers 
in the current scientific literature that keeps unraveling important 
new discoveries with very likely applications including useful drugs to 
treat diseases.
    You can imagine the response if you were to ask a regular person in 
the street if they felt that investing tax dollars would be justified 
to study a simple, almost insignificant worm. As the results show, this 
research has been a good investment--and it's an investment that relied 
on the experts who recognized that this organism was indeed an animal 
model system which had many properties in common with higher organisms 
like humans and that it was worth studying. It is important to note 
that a significant percentage of the Nobel Prize winners for the last 
50 years have been American or working in the U.S. This can be 
connected with the existence of our merit review system, which tries to 
fund and identify only the best and more promising ideas for funding.
    In the social and behavioral sciences, federal funding awarded 
through merit review has been at the foundation of substantial and 
important research. Elinor Ostrom (Indiana University) \1\ and Oliver 
Williamson (Berkeley) won the Nobel Prize for their work on ``Economic 
Governance.'' Ostrom explored how communities often govern shared 
property and common resources more effectively than institutions do. 
Her research not only challenges the logical assumption about the 
inefficiency of informal groups, but also demonstrates that economic 
analysis can help understand myriad forms of social organization. 
Williamson provided a theory of why some economic transactions take 
place within firms and other similar transactions take place between 
firms in the marketplace. Their work informs us about how to handle one 
ofthe most basic choices in human organization: When should decision 
power be controlled inside an organization, and when should decisions 
be left to the market or governments themselves. Ostrom is the only 
woman who has won the Nohel Prize in Economics, and she benefited from 
long-term funding from the Social and Behavioral Sciences section ofthe 
National Science Foundation. Those funds were awarded through merit 
review.
---------------------------------------------------------------------------
    \1\  Distinguished Professor of Political Science, and Senior 
Research Director, Workshop in Political Theory and Policy Analysis, 
Indiana University; Research Professor and Founding Director, Center 
for the Study of Institutional Diversity, Arizona State University.

    Why is the merit review process for awarding federal funds 
considered a strong or beneficial process? What is the impact of the 
merit review process on the breadth, type, and strength of research 
---------------------------------------------------------------------------
funded by the Federal Government?

    At the center of the merit review process is peer review, in which 
experts review each proposal for its importance, soundness, and 
possible transformational and broader impacts when considering if it 
should be funded or not. The experts who serve on review panels not 
only have expertise in specific research; they also have an 
understanding of the broader context of the discipline. They are 
therefore uniquely qualified to assess whether an idea is important as 
well as interesting; the extent to which the research methods proposed 
and the qualification of the researchers is appropriate for the problem 
under consideration; the degree to which a proposed research project 
has transformative potential. The strength of the merit review process 
rests largely on the service of subject experts, their willingness to 
read proposals carefully, often for free, so as to identify those most 
deserving of funding. It is important to recognize that the expected 
results from the proposed projects for funding are not known in 
advance; otherwise they would not be called research. It is for this 
reason that expert reviewers are able to best assess the probability of 
success of a project based on the previous track record of the 
investigators submitting the proposal and the track record of the 
reviewers on the subject matter under consideration.
    Of course, even the most well-intentioned group of subject experts 
is not immune from the biases, limitations, and agendas that are part 
and parcel of being human. However, in my experience as a reviewer for 
many years and after having submitted many proposals to the funding 
agencies, it is clear that the agencies try to reduce as much as they 
can all possible types of bias. By and large, most of the reviews by 
the experts are more than likely to arrive at the correct overall 
decision about whether or not it is appropriate to fund a proposal. 
There can be outliers to the general process, but reviewers for the 
most part are very strict about basing their decisions on scientific 
merit, which is why the percentage of proposals which are strongly 
recommended for funding is so low.
    At its best, the merit review process is a system of checks and 
balances not unlike our system of government, in that program officers 
can also offer a counterbalance to the limitations of the peer 
reviewers. Program officers bring to the decision-making process an 
awareness of agency priorities and funding trends that individual 
reviewers or review panels will not have. In selecting reviewers, 
framing the review process, and interpreting the panel's 
recommendations, program officers place funding decisions in a broader 
context than just subject expertise. Together, program officers and 
reviewers have been remarkably successful at identifying the best 
proposals across a wide variety of fields, basing their funding 
decisions first and foremost on the intellectual merit as assessed by 
the experts, the transformative potential, and intrinsic importance 
ofthe research proposed.

    With limited federal resources, what role does the merit review 
process play in ensuring that the best scientific and potentially 
transformative ideas receive funding? How do the broader impacts 
criteria requirements, in addition to intellectual merit, affect these 
funding decisions?

    The merit review process is the best way to ensure that limited 
resources are directed to the best ideas. As I stated above, the 
collaboration of subject experts and program officers enhances the 
likelihood that funded projects will be the ones addressing important 
questions, in ways that are methodologically appropriate and that can 
lead to transformative changes in the subject matter at hand. The merit 
review process is, in my view, the best way to minimize the potential 
for politicizing scientific research, and to ensure that limited funds 
are allocated as well as possible, in particular when the federal 
financial situation is as precarious as it is at the moment.
    The broader impacts considerations offer important additional 
criteria for funding. Allow me to stress that in my experience, the 
broader impacts criteria are used to decide between proposals of equal 
scientific merit--the contribution of a proposed project to achieving 
the extra-scientific goals included within these criteria does not 
outweigh scientific considerations. That is, assuming two proposals 
offer programs that are of equal importance and potential, then the 
broader impacts criteria should be used to distinguish them. In my 
view, funding decisions made in this way find the appropriate balance 
between intellectual merit and broader impacts of research.
    As members of the Subcommittee are likely aware, in response to the 
America COMPETES Act of 2010, the National Science Board recently 
proposed an expanded list of broader impact criteria for NSF proposals, 
a list that does give some cause for concern. I do not believe that 
this expanded list might alter the appropriate balance between 
intellectual merit and broader impacts in funding decisions. Rather, my 
concern is that the expanded list will diminish the National Science 
Foundation's admirable and necessary leadership in promoting the 
participation of underrepresented groups (women, racial, and ethnic 
minorities) in the sciences. Expanding the participation of 
underrepresented groups is an economic and intellectual necessity. 
Science and scientific innovation are increasingly important to a 
strong economy, and so American economic security as well as American 
pre-eminence in scientific inquiry and higher education depend upon 
expanding scientific education, at both the K 0912 and higher education 
levels. To the extent that the expanded list of broader impacts 
criteria diverts attention from this priority, it risks damaging our 
economic and national security. \2\
---------------------------------------------------------------------------
    \2\  See attachment A.
---------------------------------------------------------------------------
    Expanding the participation of members of underrepresented groups 
in the sciences is not a matter of establishing quotas or prioritizing 
this participation over the scientific importance of a proposal. The 
NSF emphasis upon this is a demonstrated way of increasing the pool of 
talented and trained scientists. This in turn enhances overall American 
competitiveness, and over the long run, it will ensure that the United 
States continues to be the world leader in scientific research, 
discovery, and innovation.

    How does the merit review process work to ensure review 
impartiality for all applicants while maintaining high standards of 
excellence?

    Among the strengths of the merit review process is a degree of 
flexibility and breadth that other possible review systems would be 
unable to match. By this I refer to the ways in which our proposal 
review processes recognize that merit has many facets. For instance, an 
investigator at an early stage in her/his career cannot be expected to 
have the track record that a more senior investigator has--and yet, the 
early career proposal may be as meritorious and promising as the ones 
submitted by the senior investigator earlier in their career. The NSF 
and other agencies have worked to develop programs that attend to 
differences in seniority, to allow the junior investigator to develop 
and flourish since they represent our future. CAREER Awards, available 
through the NSF, are targeted specifically at researchers early in 
their academic careers. But outside of these programs, the merit review 
system contributes to impartiality of reviews because it brings a group 
of scientific experts together to review proposals. Barring 
personalities or idiosyncratic agendas that subvert the process, merit 
review increases the likelihood of fair and open review of all 
proposals over any alternative review process I am aware of.

    Please discuss any potentially novel ideas that should be 
considered in order to strengthen the process.

    I am, as you have likely gathered, a strong proponent of the merit 
review system. I believe it is the process more likely to ensure that 
limited federal funds for research are awarded to the most significant 
scientific research, with the highest likelihood of long-term impact. 
Nonetheless, the process is not perfect. I have alluded to the 
possibility that review panels are not free from bias that can 
influence their funding recommendations. One weakness of the peer 
review process is that scientists can be somewhat conservative. By 
this, I mean that it is easier to identify and fund proposals with the 
likelihood of incremental scientific advances than it is to identify 
and fund proposals with the potential for transformative breakthroughs. 
To the extent that bias toward the incremental means that we fail to 
fund innovative, paradigm-shifting research, we may be missing 
opportunities for precisely the kind of scientific advances that 
characterized federally funded research in World War II.
    This conservatism is recognized by funding agencies, which have 
occasionally responded by altering the charge given to review panels. 
``For example, when NIH concluded that it was not awarding enough high 
risk/high payoff grants, it changed its charge to panels accordingly.'' 
\3\ Adapting guidelines given to review panels is one way to ensure 
flexibility and strength within merit review processes. ARPA 09E, the 
Department of Energy program that focuses on funding breakthrough 
research, offers another model that might strengthen the merit review 
process. By using a multi-part proposal and review process, ARPA 09E 
attempts to give reviewers a better understand of extraordinary 
proposals, and thus a better chance of recommending the best and most 
feasible proposals for funding. \4\ Similar multipart proposal and 
review processes might effectively be adopted by other funding 
agencies, in specific programs if not throughout all funding programs. 
This might strengthen the merit review process by making funding of 
extraordinary projects with transformative potential increasingly 
likely. Of course, scientific research advances slowly and having 
transformative discoveries is not as common as one would wish. Thus I 
am not advocating that funding be directed solely to risky research. 
Rather, I am suggesting that the overall research funding apparatus 
could be improved by a measured emphasis on potentially disruptive 
discoveries and having mechanisms that allow them to be funded.
---------------------------------------------------------------------------
    \3\  James Turner, ``Best Practices in Merit Review: A Report to 
the U.S. Department of Energy,'' APLU, December 2010, p. 7. 
www.aplu.org/document.doc?id=2948.
    \4\  Ibid., p. 10.
---------------------------------------------------------------------------
    In these challenging economic times, it is more important than ever 
that federal funds are spent wisely. Continued federal funding for 
scientific research remains an important priority. Federal research 
funding--via the National Defense Research Council during the Second 
World War, and via the NSF and other government agencies since then--
has for 70 years been a cornerstone of American economic security, 
scientific and educational preeminence. Maintaining our competitiveness 
globally, continuing American leadership in scientific research and 
innovation, and ensuring that our children and grandchildren enjoy a 
future in which American higher education and industry remain the envy 
of the world--all of these rely on continued, robust federal funding 
for scientific research and discovery. Yet each dollar we spend must be 
spent wisely--and a strong, flexible and rigorous merit review process 
is the best possible guarantee that American scientific research will 
continue to lead the way forward and that every taxpayer dollar is 
spent with the highest possible return on investment.
    I thank the Subcommittee on Research and Science Education for 
allowing me to express my opinions about the merit review systems used 
by the federal agencies in general but in particular by the National 
Science Foundation, which funded a significant portion of my research 
for close to 25 years. I will gladly respond to any questions you might 
have.

    Chairman Brooks. Thank you, Dr. Jose, and thank you, panel, 
for the information that you shared with us. I am reminding the 
Members that Committee rules limit questioning to five minutes. 
However, if time permits and there are Members who wish to 
engage in a second round of questions, we may do that. The 
Chair will at this point open the round of questions and as 
such I recognize myself.
    The first question is for Dr. Marrett. Based on fiscal year 
2010 data, please explain why over 2,700 proposals rated from 
poor to good to very good received federal funding over the 
8,000-plus proposals that received very good to excellent or 
excellent ratings and were not funded. More particularly, in 
looking at the graph that we have been provided, there were 
some proposals, one that had a poor to fair rating but was 
funded, we had 98 proposals that received a fair to good rating 
that were funded, while on the other hand we had 1,312 
proposals that received an excellent rating that were not 
funded, and similarly, 6,318 proposals that received a very 
good to excellent rating that were not funded. So generally 
speaking, why this kind of variation?
    Dr. Marrett. Yes. Now, I have to explain, to go back to one 
of my key points, the role of program officers who have to 
bring their expertise to bear. What you are reporting on are 
the results of the reviews of external panels. The external 
panels look at, as already indicated, very much the technical 
merit of proposals. There are other things that have to be 
taken into account, and they have been the founding of the 
Foundation. Thus, a program officer has to ask about what else 
might be known about the proposal. The program officer also 
pays a lot of attention to what Dr. Jackson mentioned, a 
portfolio. Thus, if we are to be concerned about how well we 
are serving the entire United States, how well a portfolio 
represents within a discipline the varying ways in which work 
might be done in that discipline so that there isn't complete 
overlap in what is being funded. So making sure that there this 
is this kind of balanced portfolio is a responsibility that 
leads then to the fact that our program officers who make their 
recommendations to the next levels of review in the process, 
program officers are not then bound by the kinds of assessments 
that might be given since those assessments are on technical 
merit. They are not bound to have to rely exclusively on those 
recommendations from the external reviewers.
    So let me close by saying, I will note again, those are 
recommendations that are made and they are not the final 
decisions or how the funding will be done.
    Chairman Brooks. You mentioned other things taken into 
account and then you gave as an example overlap. Are there any 
other factors taken into account that you have not shared?
    Dr. Marrett. Well, there can be a number of things. I think 
some of the questions from the Committee that we received asked 
about such questions as geographical balance. One of the 
questions commented on the fact that we know that talent is not 
limited to any specific part of the Nation and thus there is a 
question often of how are we looking across, but this is not to 
suggest that these are non-meritorious. You don't start with 
the assumption that you will do away with the merit but the 
meritoriousness of proposals. So there are numbers of other 
things. The potential contributions to innovation can be 
another criterion that a program officer looks at. We then do 
have--and these don't depart significantly from what will 
appear in all of the materials that we share with the 
community, with what will appear in the broader impacts 
criteria, but all of these can be brought to bear as 
assessments are made about how to use the funding in the most 
effective ways for the Nation.
    Chairman Brooks. These other criteria that you mentioned, 
are they required by statute, required by Code of Federal 
Regulations or internal to the way in which the NSF makes its 
decisions?
    Dr. Marrett. They are internal to NSF. We do abide by the 
code that affects all the federal agencies when it comes to 
merit review so merit review in a broad sense. They are sets of 
principle shared by all federal agencies because of the code.
    But then there are the specific criteria approaches that 
the Foundation has outlined and those are the ones that we try 
to make sure--not try--we actually make sure they are 
communicated, understood by the community, thus our Web sites, 
our communications always are very explicit about what is going 
to be expected of those who submit proposals and how the 
process will work.
    Chairman Brooks. And finally, the graph that I have talks 
about these proposals in terms of numbers but not in terms of 
dollars, so I know no score a number, poor to fair, a number of 
proposals that fit that criteria, fair to good, good to very 
good, very good to excellent, and excellent and so forth. Does 
the NSF have this information broken down by dollar figures or 
is it strictly on a number basis?
    Dr. Marrett. We can provide you the information about 
dollars if you would like to know.
    Chairman Brooks. Will you please provide our Subcommittee 
staff with that information?
    Dr. Marrett. We will.
    Chairman Brooks. Thank you.
    Next I will recognize Mr. Lipinski for his questions.
    Mr. Lipinski. Thank you, Mr. Chairman.
    Dr. Marrett, as I mentioned in my opening statement, I 
offered a provision in last year's COMPETES Act that authorized 
prize programs at all science agencies. This language began 
with a bill I introduced with Congressman Frank Wolf that would 
have created an innovation inducement prize pilot program at 
NSF. We introduced this legislation based on a 2007 National 
Academies report which concluded that ``an ambitious program of 
innovation inducement prize contests will be a sound investment 
in strengthening the infrastructure for U.S. innovation.'' I 
have also worked with Congressman Wolf to include report 
language for the fiscal year 2012 Commerce, Justice, Science 
appropriations bill that asks NSF to make use of its new 
authority, especially the mechanism for funding high-risk, 
high-reward research projects.
    Dr. Marrett, can you tell me whether NSF has had any 
discussions on this topic and whether it has any plans to 
administer a prize program?
    Dr. Marrett. Thank you, Mr. Lipinski. We have had extensive 
discussions on the topic and appreciate the intent, or we 
appreciate what has happened to give us the authority to try 
out other kinds of tools in the toolkit. I appreciate your 
earlier comment that we must make sure that what we do fits 
into the kind of support for fundamental work that lies at the 
heart of what the National Science Foundation does and so we 
have been looking at what would be appropriate in that context, 
and we will be launching a series of experimental efforts to 
try to see how these fit within our context and so we are--we 
haven't been ready to announce very broadly because these are 
highly experimental but we are giving a lot of attention to the 
use of prizes and awards as ways for advancing innovation and 
advancing knowledge in the Nation. So this would not be 
inconsistent with what our overall mission is but certainly we 
have paid close attention to the strong interest being 
expressed by this Subcommittee, members of the larger community 
in the fact that prizes and awards could be a mechanism for 
expanding our portfolio.
    Mr. Lipinski. Thank you.
    I want to turn now to Dr. Jose. One of the inherent 
challenges of NSF's merit review process is overcoming 
potential conflict that exists when an established researcher 
reviews a transformative proposal that may question that 
reviewer's work. Now, NSF proposes to address this challenge by 
awarding EAGER grants, retraining program officers and 
experimenting with alternative approaches to the merit review 
process. In your testimony, you mentioned a similar challenging 
experience by Indiana University and how crucial it is that IU 
directs available resources to projects with the strongest 
likelihood of being transformative and successful. Could you 
expand on how you ensure that funding decisions of 
transformative research are based on scientific merit rather 
than the personal considerations of reviewers, specifically, if 
such research conflicts with that of a reviewer? How does your 
process compare with NSF's process, and if it differs, how do 
you recommend incorporating Indiana University's process into 
NSF's process?
    Dr. Jose. Yes. Thank you very much for the question, Mr. 
Lipinski. One of the characteristics of a scientific endeavor 
is it moves slowly, usually moves slowly, and once in a while 
there is a big discovery and it has a big jump and then it 
continues. Scientists generally actually are conservative. When 
they review proposals, they are conservative and they are less 
willing often to risk the funds that they have to decide to 
award to a researcher because they are not sure that it is 
going to succeed. Now, doing research means that we don't know 
what the answer is going to be. That is why we call it research 
because we pose a question that we would like to find an 
answer. Often it is not easy to see how to get the answer. We 
have to have a plan of how we are going to do it.
    Now, when--we had a similar problem, as you mentioned, at 
Indiana University. We also have limited resources that we want 
to invest in trying to catalyze the research that the faculty 
may want to do that is risky and is transformative. Now, the 
way of doing that is by requiring from the reviewers to 
consider what will happen if this project will succeed, how 
much is it going to change the nature of the field in which the 
proposal is being submitted, is it going to be transformative. 
What does it mean to be transformative? It is really not going 
to be just small change in the way things have been done but is 
really going to be significant in the way that it is being 
done.
    The NSF and the NIH have recognized that that is a problem. 
In fact, I would say that the way they review proposals in the 
NSF is different than the way they review proposals in the NIH. 
NIH has review panels that are study sections that actually 
review panels for an extended period of time. In the National 
Science Foundation, the reviewers are usually confidential; 
their names are not known. And what happens is that the 
reviewers are often trying to make sure that the money is not 
going to be wasted so they are not as willing to risk funding a 
project that they are not so sure is going to work out. One 
needs to have some policies, and the NSF and NIH have done 
that.
    They have introduced projects, like the ones you have 
mentioned at the NSF, in which a certain amount of money is 
going to be given for projects that are risky but the payoff is 
going to be very high, if they succeed, it is going to be very 
high, money to be given for a project that actually going to be 
transformative. For example, the NIH has a program that is 
called Pioneer Awards. Pioneer Awards are given to projects 
that are very ambitious, very risky, that are really going to 
be transformative in the field in which they are given. So the 
agencies are trying to do that. It is not easy to do but it 
needs to be done. Having said that, funding proposals where the 
advance is slowly moving is also important because science 
doesn't advance in jumps suddenly. Thank you.
    Mr. Lipinski. Thank you, Dr. Jose. Thank you, Mr. Chairman.
    Chairman Brooks. The Chair recognizes next Mr. Bucshon from 
Indiana.
    Mr. Bucshon. Thank you, and welcome, Dr. Jose.
    Dr. Jose. Thank you.
    Mr. Bucshon. A couple comments. First of all, I was a 
cardiovascular surgeon in my previous career, and so I know a 
little bit about research and I would just like to say that a 
lot of discoveries, as everyone would recognize, are made by 
students and people who are just getting their careers started 
in science and so that is kind of the area that I am going to 
focus on a little bit.
    Some areas of research may only have a few experts in the 
field, so to speak, and the review process, even though 
technically external may actually be consistent with an in-the-
family-type review, and that happens a lot, as we know, in 
science. So I guess, Dr. Marrett, I would like to ask, how do 
we guard against the politics involved? Because everybody knows 
there is politics involved in this process. In a highly 
competitive academic environment, both at the institutional 
level and at the individual level, and is there a review 
process for the reviewers that shows their consistency and 
their willingness to assess projects that appear to be 
nonpartial, even in this setting?
    Dr. Marrett. Let me note again first that what the 
reviewers, the panels that we call in or the mail reviewers, 
those are again advisory, and when you mention, we are very 
concerned to make sure that the process is free of lots of 
conflicts. There are a couple of ways in which those are 
handled. One is by having all reviewers do have to sign the 
conflict forms to indicate that they are not in conflict. Our 
program officers do training for ethics and conflict training. 
In fact, interestingly, we also have that now, ethics training, 
for students, undergraduates, graduates, post-doctoral 
students. There is also a requirement for institutions, that 
institutions must assure that there isn't a conflict of 
interest that would be there.
    Even with all of that, we still have other sorts of 
processes to oversee how final awards are made and so the 
awards are not made exclusively on the basis of these 
recommendations. They go up through other levels for review. 
Then finally, if there still remain problems, those can be 
undertaken through our Office of the Inspector General. If 
there is evidence of waste, fraud or abuse, those we would 
report to the Inspector General and the Inspector General gets 
them through other processes. So we have had--we put in place 
checks and balances to reduce the likelihood that there will be 
the kind of personal considerations that I know would always be 
of concern, and that is why we have what I say again are the 
checks and balances to assist them that will not rely 
exclusively on any single individual.
    Mr. Bucshon. Thank you, because looking over the process, 
as you know, there is a disparity in the percentage of grants 
being made to new investigators versus established as well as 
universities that traditionally get federal funding versus 
those that may be new to the process, and I understand all the 
reasons why those may occur. People honestly are more 
experienced at submitting their proposals and so they may be 
more likely to be successful. But the reality is, there 
probably is a little bit of this internal family-type thing 
that does come into play. Again, saying that, you know, 
students and new researchers are ones that frequently make the 
most discoveries. I am glad to hear your explanation that these 
people, you know, they appear to have the same opportunity.
    Dr. Jose, I will ask you a question. In your experience, 
many years of experience, has the review--I mean, is there a 
lot of political interplay in the review process, and you are 
free to----
    Dr. Jose. I wouldn't call it political. I think that often 
people that have been working in the field for some time, they 
think that they have the right answer, and when someone else 
comes in and say well, maybe I have another way of looking at 
the problem with a different answer, they may say oh, no, no, 
no. There is some resistance at the beginning. But eventually 
one of the nice things about science and the review process is 
that other researchers will actually come up with results that 
will say yes, the new way of looking at the problem is the way 
to do it. But it is always, just as in human nature, I mean, 
all of a sudden you have a way of understanding a problem, you 
think you have the answer and someone else comes in with a more 
innovative way of doing it and says oh, maybe not, maybe it is 
not a good idea to support it but eventually I think the system 
works itself out to actually recognize new ways of doing 
things, and even when you have a group of people that are--what 
do you call them, in the family--that look at problems in one 
way, eventually they have to give credit to the new ways of 
looking at a problem and solving the problem.
    Now, let me say something about the young investigators. 
The NSF has a program for--and NIH too--for recognizing junior 
investigators that gives them some extra points, if you wish, 
when they start the running because they can't compete with the 
ones that have been in the field for 40 years. So there are 
programs that are working very well. They are not as good as 
they should be, but they have recognized those problems.
    Mr. Bucshon. Great. Thank you. I yield back.
    Chairman Brooks. Thank you, Mr. Bucshon.
    The Chair next recognizes Mr. Clarke.
    Mr. Clarke. Thank you, Mr. Chair. We are a few days from 
possible government default, and to me, I am the new guy here, 
relatively, like a few months here in Congress. Many years ago, 
I was a staffer. So I am going to say that maybe I have a 
perspective that could give us a clearer picture on things 
because I haven't been part of the system for a long time. But 
one underlying issue that I see that Members of Congress have 
regarding taxpayer-funded basic research as we are dealing with 
here is they want to make sure that we are not wasting money, 
that it also goes to make a constructive difference in our 
country, and the broader impacts criteria kind of lays out what 
our objectives would be.
    Dr. Jackson indicated, I think, a very important issue, 
though, is that how do we best measure whether that research is 
effectively achieving those broader impact goals, especially 
when it is research that is totally innovative and totally 
transformative. It is almost like, you know, asking the veteran 
Members of Congress to quit blaming each other and think of a 
different way of actually passing legislation by inspiring the 
public with an enlightened point of view. I am not sure if that 
would actually come from entrenched folks here in this system 
but it could.
    So in that sense, I am just posing this question to all of 
you. What type of measures do you think we should look at? Dr. 
Jackson indicated that maybe empowering the NSF to easily 
gather more data to measure the success of the research could 
be effective, but along those lines, we here in Congress want 
to make sure that when we are spending taxpayers' dollars that 
the taxpayers are getting the best benefit for those dollars. 
How do we measure it?
    Dr. Jose. If I may answer the question, I mean, there is a 
program now, a federal program that is called NSTAR where they 
are going to try to find out for every dollar that is spent by 
the Federal Government, how many jobs it will create, how many 
people are educated, what is the impact that those dollars have 
in the economy, for example, and the number of jobs and the 
competitiveness of the United States when comparing with other 
countries. It is a very difficult thing to do. It is a multi-
variable type of problem. One answer is not that simple. We had 
this question asked of us when the stimulus money was given to 
universities, and we were asked, and I was interviewed a few 
times, and people would ask me, well, you are going to get $20 
million so how many jobs will you create, how many jobs were 
created from that $20 million, and it was not an easy question 
to answer, particularly because the money had just arrived a 
few months before, and second, because in particular when it 
has to do with research, it takes some time actually to see the 
results.
    But I will say that in my view, the money that the United 
States has invested in funding research and particularly after 
the second World War has maintained the preeminence and the 
dominance of the United States economy. Furthermore, it has 
been the seed money that was used to create the best university 
system we have in the world and is the envy of the rest of the 
world. I would call that evidence that everything has been 
working out. Now, having said that, we need to review 
constantly how we are doing things, how we are investing the 
money, because times change and we should look at things in a 
different way like today with the budget problem that we have, 
the Federal Government budget problem that we need to be very 
careful about how we invest money, federal research taxpayer 
money.
    Dr. Marrett. I would just elaborate a bit on that. I think 
the points are very well taken. The National Science Foundation 
pays attention constantly to issues of metrics because we take 
seriously our responsibilities to be accountable to the larger 
public. Those metrics are reflected in many ways in the kinds 
of requirements for what must be done, broader impacts. There 
are the assessments that have to take place as people are 
reporting on the broader impacts. So we are constantly looking 
at, reviewing, examining what would tend to be approaches that 
are possible.
    I think the other issue, a key issue here is, it is much 
easier often to think about short-term returns for those things 
that might have--we can look at, then the longer-term returns, 
and it is the longer-term returns that are very important for 
an agency such as the National Science Foundation. We have done 
studies that have looked at--we take account of some of 
important developments and track them back. Sometimes those 
have been over an extended period of time. Thus, I think a part 
of Dr. Jackson's comments had to do with ensuring that our 
attention on returns would not focus us so heavily on immediate 
returns that we are unable to think about and invest in what 
will have to be the longer terms and the metrics appropriate 
for the longer life of activities.
    Dr. Jackson. And yes, I would say that is exactly true. 
When you analyze and measure the impact of fundamental and 
basic research, it takes a long time sometimes to see what 
those impacts are, so it is a long-term process, not just a 
short-term one.
    Dr. Yamamoto. I agree with that. I think you raise a really 
important question, and the way that I would look at it, as I 
said in my remarks, is that this must be viewed as a continuum 
that recognizes that application that really reaches a level of 
being evident in addressing national goals, requires a 
continuous input of fundamental information about complicated 
processes and that for any one grant application, it could be 
impossible actually to ascribe how they may be approaching 
national goals.
    Let me do one quick example. My colleague, Herb Boyer, in 
the early 1970s was working on a process that was sort of an 
esoteric process in which DNA that got into bacteria was broken 
down for some reason and he didn't understand, and this was 
called DNA restriction. The kind of DNA that would get into 
bacteria would be restricted by breaking it down. And very few 
people, very few scientists were interested in this process. I 
would wager that if this were put up against the test of 
addressing national goals, it might not do very well. That 
discovery of what DNA restriction modification was turned into 
the capacity to make recombinant DNA, which then turned into 
the biotechnology industry worth billions and billions of 
dollars and leading the world. So it was impossible to ascribe 
with that particular grant what its impact would be but there 
it was.
    So I think that the goal would be in fact not to look at 
these things individually, to ask collectively whether the 
federally funded scientific research endeavor in this country 
is actually moving things toward national goals, to recognize 
that we understand so little about these processes that a 
continued input of investigation at the fundamental level is 
essential and that that continuum really does work.
    The NSF is the basic research engine of the Federal 
Government and as such really needs to be celebrated and 
protected. There are mission-oriented agencies including the 
NIH in fact, the DOE, the DOD, Department of Agriculture, 
Commerce, elements of Commerce and so forth that have specific 
missions and I think that those should be tested in the short 
term against whether they are addressing national goals but 
always to recognize that input of fundamental science from the 
National Science Foundation and elsewhere is always going to be 
essential.
    Chairman Brooks. The Chair next recognizes Mr. Hultgren.
    Mr. Hultgren. Thank you, Mr. Chairman, and thank you all 
for being here.
    I have a question for Dr. Yamamoto and Dr. Jackson and Dr. 
Jose. Each of you has testified about broader impacts imposed 
on the Foundation by Congress in the 2010 American COMPETES 
Reauthorization Act and that it will adversely affect funding 
decisions by requiring reviewers to consider metrics far 
outside the scope of the intellectual merit. Dr. Jose suggests 
that rather than enhancing competitiveness and national 
security, they risk damaging our economic and national 
security. I wondered, are reviewers even privy to the national 
priorities prior to the meeting of a panel, and just asking if 
you would comment, if each of you would comment on this.
    Dr. Jose. I would like to say that all of us are aware of 
our environment. All of us are aware about what is going on 
with Congress today. For example, all of us read the news, all 
of us know what is important. All of us are very aware about 
what is happening in the growth of the economies in the rest of 
the world and how the United States is having to compete very 
strongly with China, which in particular is growing very fast 
and we know or we feel that one of the ways in which the United 
States is going to succeed is by innovation, by being 
innovative, by being creative and introducing new knowledge and 
new ideas, and I think that we should not limit the imagination 
of investigators and researchers by having a large number of 
restrictions about how the money is going to be awarded. I 
mean, we put too many restrictions that constrain the freedom 
that researchers have that often leads to the creation of an 
industry like biotechnology, as Dr. Yamamoto just said. It is 
the essence actually of the freedom of thinking freely that 
creates new innovations that were totally unexpected.
    Mr. Hultgren. Dr. Yamamoto or Dr. Jackson?
    Mr. Yamamoto. As I said in my comments, I think that one of 
the critical features of merit review is a singular focus on 
scientific merit and that anything outside of that is valuable 
and important and in fact essential as it may be such as 
addressing broader national goals should reside outside of that 
process and that asking merit reviewers to make an assessment 
of how well the fit is of a given project to national goals I 
think is inappropriate. I think the agency individually and 
collectively, as I said, the federal scientific research 
endeavor needs to be paying attention to whether national goals 
are being addressed but that neither reviewers nor applicants, 
in my opinion, should be mandated to make such a statement. If 
there is a clear application, of course, it would be to the 
advantage of the applicant to make that clear, but if not, and 
it is a fundamental question, then you are really asking 
reviewers to make guesses, and I think one of the great things 
about the merit review process is that there is not a lot of 
guesswork there and that we shouldn't be asking reviewers to 
step outside of their expertise.
    Dr. Jackson. As I mentioned in my testimony, there are a 
lot of proposals that are sent to NSF that are very, very good, 
excellent proposals that just cannot get funded because there 
is not enough money so that when the program officer at NSF 
looks at 40 proposals that are all great and could have a huge 
impact on this country and they have to decide which of those 
40 to choose, then the broader impact does come as an advantage 
to choose between all these great proposals, and that is an 
important role, I think, for the broader impacts criteria and 
should be kept as such because there are clearly far more many 
proposals that are excellent that can be funded.
    Mr. Jose. May I say one more thing?
    Mr. Hultgren. Certainly.
    Mr. Jose. Just going along with what you said, I just came 
to Indiana University last year, and we have limited resources 
but we introduced a seed funding program for people that would 
be willing in interdisciplinary research. We actually asked 
people first to have new projects, new collaborations and so on 
and so forth. We know of about 165 proposals and we could only 
fund 10 percent. There were many that were recommended for 
funding but we had to make a decision at the end. We decided 
never to fund something that was not recommended on merit, but 
then we have to use some criteria to decide which projects were 
more likely to be good and producing new results or new 
discoveries and so we had to reduce the total number of 
projects that we were going to fund using other criteria. So it 
is important to have a combination perhaps of the two.
    Mr. Hultgren. My time is winding down. If I could just ask 
Dr. Marrett just to follow up quickly, Mr. Chairman?
    Chairman Brooks. We are going to do a second round, given 
our time availability. Go ahead and----
    Mr. Hultgren. Let me just quickly, Dr. Marrett, I know this 
is something that Congress has directed you to do. It wasn't 
necessarily something internal that you decided. I wonder if 
you would have any comment on what your colleagues at the table 
stated, what your feelings are about this.
    Dr. Marrett. Well, it is not quite true that this is only, 
that we only are talking about broader impacts because of 
Congressional activity. No, from the outset, from the time that 
NSF was founded and started making awards, it has always 
considered both the technical side and how we address the 
national goals. The idea, the terminology of broader impacts 
was really brought in in 1998 when the Foundation said we are 
now going to collapse several criteria down to the two 
criteria, intellectual merit and broader impacts. The broader 
impacts statement or the expectation is that the goals, these 
do have to reflect some of the kinds of goals that are expected 
by the Nation of the investments in science. So they would 
include economic competitiveness, development of a STEM 
workforce, increased participation by underrepresented groups, 
partnerships between academia and industry. In other words, we 
have seen these as very critical for the Nation and reasonable 
as an agency thinks about how it will allocate resources. I 
think the only difference between the NIH and NSF approach, NIH 
does much of the same kind of a thing, it is at a different 
level. So the advisory councils take up the questions of how 
should the portfolio and how should these other national 
considerations be taken into account. They happen to be done as 
a part of our regular merit review process in NSF. But this is 
not a new idea.
    Mr. Hultgren. But wouldn't you say, though, it is an--this 
expansion is mandated by COMPETES? Isn't that true and that 
makes it unique?
    Dr. Marrett. No, what COMPETES did was to say be much 
clearer and there are other things with reference to broader 
impacts, and so that is what the Science Board is looking at, 
that is what the expectation of the Foundation, and that is 
what we do believe we can be much clearer on what broader 
impacts should mean and how it would come up with indications 
appropriate metrics, but again, this was a part of an ongoing 
process and we appreciate then the interest of COMPETES, the 
interest of others in helping us continue to move forward in 
explaining and being accountable, again, to the Nation.
    Mr. Hultgren. Mr. Chairman, you have been very gracious. I 
yield back. Thank you.
    Chairman Brooks. Now for the second round of questions. The 
Chair will start first.
    Dr. Jose's testimony stated that the merit review process 
is at best a system of checks and balances, and this question 
is for all of you. Being a member of a government body built to 
provide a system of checks and balances, I am interested in 
this theory. In the context of the financial circumstances that 
Congressman Clarke so notably described that we face here in 
Washington, DC, do you or any of your colleagues sitting with 
you today have any additional suggestions on how to strengthen 
the system based on our need for a balanced approach to federal 
funding, particularly to ensure the advancement of science as 
our sole or primary funding criteria?
    Dr. Jose. Actually, let me actually say something. I mean, 
NSF has no specific mandate to do other than basic research and 
try to uncover new laws of biology or physics. The NIH has a 
very clear mandate to try to find the best way to improve 
American citizens' health. The Department of Energy has very 
special goals, the Department of Defense, each one of them has 
a merit review system that works differently than the one at 
the NSF. The NSF is to some extent unique.
    Having said that, I think that when I said checks and 
balances, what I mean is that the reviewing process of papers, 
not just proposals for funding--I mean, we are supposed to 
publish, many of us that work in science, our research, our 
results. It is very hard to get results published in the best 
journals if they are not of the highest quality. The same thing 
happens in the merit review process. You are not going to get 
funded unless you have the highest quality. So in a sense, we 
have like a voting system. Our colleagues actually vote about 
what we do, if they think it is good or not, as we get elected 
or reelected in Congress or not, I mean, the voters have to 
decide if we are doing a good job or not. As researchers, if we 
are doing a good job, our colleagues will vote that yes, get 
funded, yes, get published, yes, get evaluated for lectures and 
so on and so forth. So that is what I mean in the sense that we 
have checks and balances.
    When a recent result is wrong, the community will find out 
that it is wrong because when they try to reproduce that 
discovery and they cannot reproduce it, like cold fusion, for 
example, it just will die immediately. That is what I meant by 
checks and balances. That is how science advances, not in a 
straight line continuum.
    Chairman Brooks. Thank you for the additional information 
with respect to checks and balances. Do you have any specific 
suggestions on how we can strengthen or improve the process or 
system in order to help ensure that these scarce federal 
dollars are being spent on basic science or research rather 
than being affected by the criteria that are extraneous to 
those primary functions?
    Dr. Jose. Yes, of course. I think that, as I said, it is a 
human endeavor when people try to review proposals. As 
Representative Bucshon said, I mean, there is some element of 
human nature that interferes and how do we review proposals, 
who is reviewing the proposals, is the best investment that one 
can make or not. But we recognize that is a problem we have. We 
develop all kinds of checks and balances from the agencies, 
from you. You all the time are telling us you have to convince 
us that you are doing the right thing, otherwise we won't give 
you money to do the research, and it is not perfect and we keep 
changing it and correcting it as we go.
    Chairman Brooks. Well, my question is, do you have any 
specific suggestions on how to improve the process?
    Dr. Jose. Yes. I think that--let me--yes. I think that--
actually I have thought a lot about that. There are blind types 
of reviews because often the name of the scientist plays a role 
in if it is going to get funded or not going to get funded. 
Perhaps even the institution, perhaps the project and the area. 
I think that--let me see. How do improve the system? I think 
that we have to make sure that we choose the right reviewers. 
We have to make sure that there is no conflict of interest with 
the reviewers being part of the same group of people that are 
funding each other. Other than that, I think I have many things 
to say but it would take a while to describe.
    Chairman Brooks. Well, thank you, Dr. Jose. You mentioned 
blind reviews. What are those?
    Dr. Jose. Well, for example, if you submit a proposal, you 
don't know the name of the author of the proposal, the 
institution of the proposal, as you don't know the name of the 
referees, and then you will just blindly decide if this is 
really good or not whereas the way we do it now, I mean, we 
know the name of the reviewer. We can know if they are very 
good or not very good and decide--that may influence it a 
little bit.
    Chairman Brooks. Are you recommending that we have blind 
reviews?
    Dr. Jose. No, no. We have tried to do that. In fact, there 
is a journal trying to do that and they have mixed reviews, 
mixed results. I mean, they tried to do that, not having the 
name of the author, not having the name of the reviewers, and 
they just review it blindly. I am not sure that the end result 
was satisfactory as expected.
    Chairman Brooks. Dr. Jackson or Dr. Yamamoto or Dr. 
Marrett, do you all have any specific suggestions that you 
think we should look at in order to try to improve the process?
    Dr. Jackson. Well, I do think--I will go back to what I 
mentioned in my testimony about giving NSF program officers the 
ability to triage proposals. At this point in time, all these 
proposals have to go through merit review. That taxes the 
scientific community, that taxes the program officers, and we 
need to make this--since we can't fund all the proposals, and 
there are a lot of excellent proposals--there are more 
excellent proposals than we can fund--it makes sense for us to 
spend our time thinking about which of those excellent 
proposals we should fund rather than spreading our time across 
the whole gamut of proposals.
    Chairman Brooks. I am sorry. I understand what you have 
already testified to, and my question should have been better. 
Do you have anything specific in addition to what you have 
already informed us of?
    Dr. Jose. I am sorry. I must say something about that, 
about what Dr. Jackson has. I mean, at the NSF, there are 
increasing number of proposals that have been submitted to the 
NSF for review and they have decided to assign to the 
university to have a limited submission-type thing. You can 
only submit two or three grants when there is several million 
dollars rather than having 10 or 15. That is a change that was 
made and it is a change also that has been made in the NIH as 
well.
    Chairman Brooks. Thank you.
    Dr. Jackson, I don't know if you have thought of anything 
in addition to what you have previously shared with us?
    Dr. Jackson. It is a messy human process and it would be 
difficult to make it more efficient. It is always something you 
are struggling to improve.
    Chairman Brooks. Well, I have to admit that when I heard it 
compared to the election process, that raised some concern.
    Dr. Yamamoto or Dr. Marrett, do you all have any other 
specific suggestions on what we can do to improve the process?
    Dr. Yamamoto. Let me make two comments that I want to frame 
as an expansion of what President Obama said when he addressed 
the National Academy of Sciences in April 2009. He said that a 
basic research project may not work for a year or a decade or 
ever, and it is for that reason that public funds, government 
funds, should support basic research because the private sector 
will always underinvest in that side because when a project 
works, it works for everyone. When it doesn't work, of course, 
the private sector loses. Maintaining that basic research and 
engine is really critical, and so let me make one suggestion, 
which is an expansion of what I said about transformative 
research, and I will try to frame briefly why I think it really 
needs to be a distinct sector of research that is supported in 
this country, and then a second comment that really approaches 
this question of broader impacts.
    So transformative research, I suggested in my comments, and 
I will expand very briefly, should be viewed as a different 
kind of research. It is destructive of current views rather 
than extending them or deepening our understanding of current 
paradigms, and to do that requires that we recognize that 
special characteristic and have a review process that is very 
distinctive. I proposed the transformative research track at 
the NIH in 2006. It has been installed in a limited way. I 
chaired the first two rounds of review of those applications 
when they came in, and an essential feature of I think making 
that program successful is actually carving out a different 
mode of merit review, one that is driven by reviewers that are 
not deep content experts in the areas that are being proposed 
but rather are generalists who think deeply, who are able to 
recognize really high-impact ideas when they see them and are 
able to celebrate the fact that those ideas may actually bring 
down their current ways of thinking about a process.
    Luckily, such scientists exist in this country. They can be 
found. They can be identified. They are willing to participate 
in this way. They are the kinds of people that are finally 
getting challenge grants from the Gates Foundation that are 
proposing--making proposals to the transformative track and are 
willing to review in that way. So it takes a different kind of 
review that recognizes that you don't bring in content experts 
who made the paradigms because they are going to be immediately 
critical of someone who comes in and says I think the current 
paradigm is wrong. I could expand on that further.
    Let me move to the second point, something I didn't 
mention, didn't talk about at all, and in a sense, you could 
view this as being outside of the merit review process, but I 
think not, and that is the idea of recognizing, especially with 
this broader impacts mandate, that basic research has advanced 
to a point where it can increasingly be applied, and since I 
work in the biomedical sphere, I see this all of the time with 
the opportunities of our faculty in my own laboratory to be 
able to move fundamental research to application, but there is 
the so-called valley of death that I am sure you have heard 
about, that especially in the current economy has made it very 
difficult to be able to do the few experiments that are needed 
to make it evident to pharmaceutical companies and the 
biomedical sphere that there is a worthy investment there. I 
think that we have an opportunity to be able to extend the 
continuum to be able to build new interfaces between academic 
basic research funded by the NSF and other agencies and build 
public-private partnerships that can more effectively move 
ideas into application, and in that way be able to really 
directly approach this challenge of addressing broader impacts 
that the COMPETES Act really challenged all of us to be able to 
do.
    Chairman Brooks. Thank you, Dr. Yamamoto.
    And Dr. Marrett, I don't know if you have any suggestions. 
It occurred to me that if there some, you probably would have 
already implemented them as Deputy Director and Acting Director 
but nonetheless, if you have any other suggestions on how you 
think we can improve the process, please share it.
    Dr. Marrett. And I take your question to mean, what 
especially might the subcommittee do, and I have two 
recommendations there. One is to share broadly the 
understanding of the process of merit review. I am not sure how 
widely the process is understood by, I wouldn't just say 
Congress, or general public, yet if we are drawing on public 
resources, we need to make sure that that process is 
understood. So this hearing represents a good approach for 
broadening knowledge about the process.
    The second recommendation I would make for the Subcommittee 
is, I hope you would be open to our sharing with you the 
experiments, the pilots that we are trying. We do say that 
there are areas where there can be improvement. Let me take the 
case of potentially transformative research. We are not all 
that satisfied but we are experimenting with, do we advance the 
potentially transformative research by identifying proposals 
once they have come in, by soliciting new ideas, by thinking of 
the sorts of experience from NIH. We are trying all of that. We 
would welcome then coming back as we have learned from those 
experiments to say more about what should be advanced. We are 
not quite ready yet to say these are the things that we think 
ought to be put in place. And finally there, I would say 
whatever, we think that at the core, that core has to be 
maintained. I mean by that core that commitment to merit and so 
whatever the sorts of changes, the changes are around the 
margins in many respects. They never should undermine the 
importance of relying on the meritoriousness of the idea and 
the approach that is being proposed. So thank you.
    Chairman Brooks. Thank you, Dr. Marrett.
    Mr. Lipinski, thank you for being patient.
    Mr. Lipinski. Thank you, Mr. Chairman. I think that--I am 
glad we went to a second round of questions because I think we 
really got into some really good ideas here about how, you 
know, to make improvements to the system, and I think actually 
I would ask--the Chairman asked Dr. Marrett if we could 
possibly do, maybe a tutorial, and get into how exactly this 
process does work, because we are talking about it here but 
even someone who has presented a proposal and has gotten a 
grant, I would like to really know better how the whole system 
actually works and maybe the opportunity for members to sit 
down and go through a tutorial. I think that would be a great 
idea for all of us.
    Chairman Brooks. That is a good suggestion, and I welcome 
it.
    Mr. Lipinski. Because I think there is a lot to the 
process, and we are talking about the process but I think there 
is a lot of details to the process and things that probably 
each of us doesn't completely understand and might be able to 
get a better understanding of that.
    Two things I want to get into. The first one, Dr. Jackson 
had--one of the main proposals you had to sort of lessen the 
work is for the program officers to be able to essentially 
triage and put aside proposals that come in that the program 
officer doesn't see there being merit to. I wanted to get a 
better understanding of what you are saying how that would be 
done, and then I will get Dr. Marrett's response on that. Are 
you saying then that immediately set aside these proposals 
without doing any sort of response to them? Because right now 
all the proposals get a response from the reviewers. It would 
seem the only way you could save time really is if the program 
officer just put these aside without any comments and just send 
it back and say rejected right off the top. In that way, you 
would not be getting--you would not be giving the researchers 
any feedback at all. I was just wondering how exactly you saw 
that working.
    Dr. Jackson. Right, and you are right. That is the downside 
to triaging proposals that clearly do not--clearly aren't up to 
snuff, so to speak, is that they miss the opportunity for a 
certain amount of feedback, and there are several ways you can 
handle that. Certainly for the petroleum research fund at ACS, 
there are certain ways we deal with that. One is that you can 
ask for a review from the program officer who in our case and 
in NSF's case is an expert in the field so that program officer 
does have to write up something about what is a problem with 
the project and this sort of thing. Another thing that we do 
with the Petroleum Research Fund is to provide workshops 
particularly focused at younger researchers, but anybody can 
come, that talk about what makes for a good proposal and so we 
are training the new generation and new people to the field how 
to write proposals, what we are looking for and what needs to 
be done, so we are doing this to counteract maybe some of the 
individual feedback that could be--that would be given if it 
was sent out to review.
    But you have to understand that a proposal goes out not 
just to one reviewer but to a number of reviewers, so there are 
possibilities too you could have; if you realized that this is 
not going to be funded, you could ask one reviewer perhaps to 
provide a review, to provide advice to the proposal writer, or 
there are a number of other things you can do that would 
decrease the amount of work for each proposal. But in the PRF, 
we recognize that these are just not going to get funded and so 
we spend a limited amount of time with them and try to do our 
education separately.
    Mr. Lipinski. Yeah, I understand the need to save time and 
I think we all want to find ways, because there is--reviewers 
are overburdened and it is hard to get people to serve in that 
capacity. I just want to give--and I am not, you know, being 
critical of that proposal. I want to see what Dr. Marrett would 
have to say about that.
    Dr. Jackson. Well, and we also--one more thing. We also 
encourage those proposal writers to call our program managers 
and talk to them about their proposal so there can be some kind 
of exchange and learning going on.
    Dr. Marrett. We are absolutely intrigued by anything that 
could help reduce the burdens on program officers, on 
reviewers. The number of proposals we are getting, the number 
continues to climb and it is very difficult. On the other hand, 
there is the kind of view, a very pervasive view that a part of 
the whole process is to help give good feedback, especially 
help in the cultivation of new investigators and thus our 
program officers are hesitant simply to say let us try to move 
this as quickly as possible. Because I think there is a 
responsibility to help people understand where the problems 
might have been, to help cultivate the ways in which they might 
develop stronger proposals. It is a dilemma then of how to have 
an effective, efficient strategy that is at the same time an 
educational strategy that we are all seeking to accomplish.
    So as I said, we welcome ideas but we are trying to examine 
them in the context of the larger kinds of things that we seek 
to do in the whole development of the science and engineering 
community of the United States.
    Mr. Lipinski. And I wanted to go on to another, if the 
Chairman will indulge me.
    Chairman Brooks. Feel free.
    Mr. Lipinski. I wanted to ask about conducting the 
committee reviews in virtual environments. It is something I 
mentioned in my opening statement. I personally have had no 
experience at all. I have no avatar. I have not gotten into 
this so I can't really speak from any expertise here, but I 
know this is something that has been looked at, and I was 
wondering if--I will start with Dr. Marrett. I want to ask all 
the other panelists about their opinions on using this as a way 
to make it easier in some ways in doing this in a virtual 
environment. You don't have to fly people into Washington to do 
this, and other possible merits to using this system. I would 
also think that the blind review process--and also having 
people come together in a way that they don't know each other 
that could impact, change the dynamics of the group. So, Dr. 
Marrett, we will start with you.
    Dr. Marrett. Yes, we are looking very much at virtual 
environments. There are experiments that have been taking 
place, and these environments, thinking of virtual panels 
especially, we have been thinking of them for two reasons of 
reducing some of the cost, the cost for reviewers, and for 
potentially expanding the pool of reviewers. In other words, 
there are some people who would find it difficult to do the 
traveling since most of these are here in this area. We thought 
this would be a way then to bring more people into the process.
    What we are looking at, what are the sorts of things that 
have to be taken into account. There are matters of security, 
for example, that you have to make sure that you have been 
abiding by. You raised the questions about the kind of 
interpersonal dynamics. There, we are actually drawing some of 
the research being sponsored by our Directorate for Social 
Behavioral Economic Sciences where there are studies of human 
dynamics, enough for us to understand how those dynamics might 
come into play and actual studies specifically of virtual 
environments. All of this then we think we need to take into 
account and that is why we are experimenting but we are trying 
to bring the knowledge to bear as much as we can to ask what 
would be the consequences and where there might be some savings 
but where there might be some losses if we are not careful 
about how we try to institute the idea of virtual environments.
    Mr. Lipinski. Any other panelists have any opinions here? I 
will start with Dr. Jose.
    Dr. Jose. Yeah, let me say that since I am quite old and I 
was funded for many years, I can see the evolution of how 
proposals are reviewed by the NSF. For a long time, there was 
only by mail review, and they only waited until the reviews 
were done and they analyzed what the kind of grade each one of 
the reviewers gave. Then they went into having a combination of 
having mail and panel reviews. And I can tell you that by 
having done both, there is a significant change in the actual 
assessment of the proposals that arises from meeting with 
people live than just from reviewing it by yourself.
    Okay. Now, with the virtual analysis they have to do, it is 
very important to know what kind of human dynamics are going to 
influence to some extent the reviewing process, and the reason 
I am saying this is that one of the committees that I was a 
member of, the mail reviewers gave much better grades to the 
proposals than when we sat down and discussed the proposals and 
interacted and listened to everybody else live, okay, so what 
effect would it have to have it virtually and not face to face, 
one has to wonder. But it is course a very useful and important 
thing to explore, yes.
    Mr. Lipinski. Who else? Dr. Jackson?
    Dr. Jackson. Living in Albuquerque, New Mexico, which seems 
to have no direct flights to anywhere, the idea of having 
virtual meetings is very exciting to me and to my colleagues 
because we are more likely to be able to be involved in these 
sort of--you know, become more--participate more in these sort 
of activities. But also because of that, I have done a lot more 
of this sort of virtual meetings and this sort of thing and I 
would say that yes, perhaps when you first start doing it, it 
is a little unusual but it is remarkable how with time you can 
become very accustomed to it, and I highly encourage NSF to 
consider these sorts of possibilities because I think the kind 
of expertise they could tap into will only become greater and 
richer.
    Mr. Lipinski. Yeah, obviously I brought up avatars. You 
don't have to do it that way, but Dr. Yamamoto?
    Dr. Yamamoto. I agree. I think there are two drivers for 
this increased--rapid evolution toward electronic environment 
for review. The first is that those of us who didn't grow up 
using those media are disappearing and being replaced 
fortunately by people who have screens in front of them their 
entire lives and are very comfortable working in that 
environment. And the second is the technology development is 
marching along very quickly and there are electronic video 
conferencing modes that you are looking across the room at a 
screen and it looks exactly like we are looking across the room 
at you. And so the opportunity for interactions, which I agree 
with Dr. Jose, is essential in being able to come to this sort 
of agreement about the scientific merit of an application that 
can really occur increasingly seamlessly in this sort of 
environment.
    Let me make one further comment, because you asked, 
Congressman Lipinski, about this idea of blind reviews that Dr. 
Jose had mentioned, and just say there again here is the 
spectrum of experiments ongoing in different agencies. You may 
know that the Howard Hughes Medical Institute does actually the 
opposite of a blind review. In fact, rather than choosing 
scientific projects as the agencies that we all represent and 
are working and do, they simply choose people, and so the 
identification of the person is the whole ballgame for the 
Howard Hughes Institute. The Gates Foundation in their initial 
round of reviews for their Grand Challenges grants, which I 
participated in several times, is blinded to the identity of 
the applicants, and after they make it through the first round, 
then that identity is revealed, and then NSF and NIH, the 
federal agencies that I am familiar with, the identity is 
always known.
    There is a risk of conflict of interest or other kinds of 
biases that can be introduced at that point, but let me say 
that having been experienced in review for at least NSF and NIH 
extensively that I don't actually see that actually occurring, 
and instead knowing the identity of the investigator, in fact, 
building it in as an explicit criterion for the merit review 
process, turns out to be very important. There have been 
studies done and arguments made that in fact the strongest 
predictor of success of a given scientific proposal is in fact 
the track record of the investigator who is making the 
proposal. So in that sense, knowing that person, their past 
capacity for doing innovative work--and let me just insert here 
that this does not mean--because very often that statement that 
I just made raises concern that this would bias against young 
investigators. In fact, it does not, and I think the clearest 
view of that is that I have been involved, and I am sure my 
colleagues have as well, for years in making judgments about 
graduate fellowships, postdoctoral fellowships, young trainees 
who have very little in the way of conventional track records 
but it is very easy, I think leaving those meetings, those of 
us who are on those panels, feel very confident that we have 
been able to make the right choices for identifying the best 
candidates based on what they have done before, even though 
they have been in a training mode and very different from being 
an independent investigator, and data prove that out that those 
people that win such fellowships and awards go on to be able to 
do very good work.
    And so my view is that having that identify early on at 
least--I think the Gates Foundation strategy actually works 
quite well--but being able to unmask those individuals early 
on, to be able to see their track record and build it into the 
merit review process is important.
    Mr. Lipinski. Well, let me throw my own two cents in here. 
I am not always certain from--I think it depends on the field 
possibly. In my background as a political scientist, I was not 
always convinced that it was helpful to the advancement of 
knowledge to know who was proposing something because sometimes 
I think that had an influence that wasn't good on review 
processes, and I am talking more about journal reviews. But 
that is my own two cents. It is very interesting to hear, 
though, Dr. Yamamoto's and everyone else's experiences with 
this, and I thank all of you very much and I especially thank 
the Chairman for his indulgence here and for this hearing.
    Chairman Brooks. My pleasure, but we are not finished yet. 
I am going to follow up on something that Mr. Lipinski noted, 
and that is, we have talked about the virtual panels but what 
is the cost of in-person panels?
    Dr. Marrett, do we have any kind of transportation costs or 
hotel costs or is there a budget line item, or how much does it 
cost per panel or how many panels are there per year? Can you 
share some information with us about that?
    Dr. Marrett. We will get that information to you. We 
actually have done the calculations of what the costs are 
associated with panels, and those do include the travel costs, 
of course, and there are hotel issues there, and we had started 
as we were looking at what would be the savings from virtual 
panels, and I should mention that in the case of avatars, we 
have had reviews done in Second Life. But in terms of the sorts 
of costs associated with virtual panels, we are not quite sure 
because there are other kinds of things. We just haven't 
explored enough now. We can't do the comparative information 
but we can get to you how many panels, what the cost is of an 
average panel for the National Science Foundation.
    Chairman Brooks. And if you can do a comparison format, 
too, I would anticipate that the virtual panels are 
significantly less expensive. But if we have some kind of 
comparison model, that would be beneficial.
    Dr. Marrett. We will try that, but why I say it is not 
necessarily going to be much cheaper because there is the 
question of the technology that we might have it on our side, 
but it is also got to be at the other side, and one of the 
things that has been talked about will be using regional panels 
where there will be cost of travel to the regional panels as 
well, the enhanced security costs that come into play, so we 
are not quite sure at the outset that there would be lots of 
savings that could happen over time in a number of ways. That 
is why I said we won't have for you--it will be very rough 
kinds of estimates when it comes to virtual panels but we can 
tell you about the investments currently made in the in-person 
panels.
    Chairman Brooks. Thank you, Dr. Marrett.
    As a short concluding statement, I don't know if you all 
have access to this, perhaps Dr. Marrett already has it, but 
some kind of compilation of success stories with NSF, the kind 
of information that this Subcommittee can share with the public 
or the Members can be familiar with so when asked, we can share 
it with the public. I am sure you are all familiar with the 
relatively famous shrimp on a treadmill commercial that has 
been running over the last month or two, and I am afraid that 
that may give the general public some dissatisfaction with the 
way their tax dollars are being spent, so I think it is 
important to also have information that we can share with the 
public that shows success stories and the kinds of returns on 
the investments that we are asking taxpayers to make.
    Having said that, I want to thank the witnesses for their 
valuable testimony and the Members for their questions. If the 
Members of the Subcommittee have additional questions for the 
witnesses, you are free to submit those, and we will ask you to 
respond to those in writing. The record will remain open for 
two weeks for additional comments from the Members.
    The witnesses are excused and this hearing is adjourned.
    [Whereupon, at 11:43 a.m., the Subcommittee was adjourned.]
                              ----------                              


                   Answers to Post-Hearing Questions




                   Answers to Post-Hearing Questions
Responses by Dr. Cora Marrett,
Deputy Director, National Science Foundation

Questions submitted by Chairman Mo Brooks

Q1. As requested during the hearing, please provide the total amount of 
federal funding awarded to those proposals rated from ``Poor'' to 
``Good or Very Good'' for FY 10. While you touched on it briefly at the 
hearing, please also expand on why those proposals received funding 
over proposals rated ``Very Good to Excellent'' and ``Excellent.''

A1.  All funded proposals are determined to be highly meritorious based 
on a combination of reviews by individuals, panel deliberations and 
program officer evaluation. On average, NSF proposals are reviewed by 
four to six individuals, depending on the type of review. All reviewers 
are chosen for their specific expertise related to the subject, and the 
collection of persons brings different points of view to the decision-
making process. When the average reviewer score is in the ``good'' 
range, it often represents a split of ``excellent'' or ``very good'' 
reviews with some ``fair'' or ``poor'' review scores that lowered the 
average. It is important to note that the proposal rating data included 
in the annual NSB Merit Review Report reflects proposal ratings before 
panel deliberations and, therefore, not the final panel evaluation. The 
panel evaluation is based on a thorough discussion of the proposal's 
strengths and weaknesses in the context of the full set of proposals 
being reviewed; this discussion forms the basis for placing a proposal 
in a particular category. These in-depth discussions can often clarify 
perceived weaknesses and result in a proposal being recommended for 
funding despite the initial average review score. Likewise, some 
proposals with high average review scores are not recommended by panels 
as a result of a detailed discussion that uncovers weaknesses that 
might not have been reflected in the initial reviews.
    The expertise of the NSF Program Officer making the final 
recommendation is also an important voice in the process. Program 
Officers take into consideration other factors that might not have been 
considered by expert reviewers. For example, proposals for innovative 
new ideas often use unproven methods or techniques that might be 
considered risky by reviewers and panelists. Risky proposals often 
result in transformative research that accelerates the pace of 
discovery. Although Program Officers consider concerns about risk 
expressed by panels, they also see the value of funding potentially 
transformative research. Proposals that do not review well at panel 
because the methods are unproven or risky can be given small awards to 
allow enough work for a ``proof of concept.'' Program Officers will 
also consider broader impacts that might not be obvious to reviewers, 
such as an infrastructure need that will serve a large number of 
people. There are also many dimensions of portfolio balance that 
influence the final recommendation. In addition to maintaining a 
diverse scientific portfolio, Program Officers strive to fund proposals 
from diverse institution types across the U.S., and from both young and 
experienced investigators.
    As explained above, the reviewer rating data reported in the Merit 
Review Report are only initial reviewer ratings, which is just the 
beginning of the merit review process leading to a final dertermination 
of whether any given proposal should be funded. Initial reviewer 
ratings do not reflect panel deliberations or Program Officer input. In 
FY 2010, NSF funded approximately $46K in proposals initially rated as 
poor, $21M in proposals initially rated as fair, $818M in proposals 
initially rated as good, and $1.6B in proposals initially rated as very 
good. Following panel discussion and analysis, all of these proposals 
were determined to be highly meritorious nothwithstanding their initial 
rankings.

Q2. In your testimony, you described experimenting with innovative 
approaches to identify potentially transformative research. Please 
expand on the ``ideas factory sandpit'' approach and tell us what you 
are learning from it and other novel approaches.

A2.  NSF has experimented with an approach to identifying potentially 
transformative high-risk research that it is now calling ``Ideas Lab.'' 
The Ideas Lab is closely modeled on the ``sandpit'' process developed 
by the UK's Engineering and Physical Sciences Research Council (EPSRC). 
The essential element of the Ideas Lab is an intensive interactive 
residential workshop involving 20-30 participants, with the aim of 
developing bold, often risky, new approaches to grand challenge 
questions in areas that could benefit from creative ``out-of-the-box'' 
thinking. A fundamental aspect of the EPSRC sandpit that has been 
incorporated into the Ideas Lab is the use of a highly 
multidisciplinary mix of participants (including active researchers 
from diverse fields and potential users of research outcomes) to 
address specific research challenges. A description of the process used 
follows. Slight variations should be anticipated as NSF gains 
experience with the process and as it is adapted to different topics.
    To identify potential participants, a solicitation is issued that 
includes an open call for participants. Interested individuals submit 
short preliminary proposals that include concise descriptions of their 
pertinent experience and expertise as well as their communication 
skills, collaborative activities, and creative abilities. A panel of 
reviewers evaluates the applications and identifies a pool of potential 
participants from a range of disciplines and backgrounds who have high 
potential to work at the interface between disciplines and to develop 
new and highly original research ideas. NSF Program Officers make the 
final selection from the pool to ensure a diverse mix of participants. 
Industrial psychologists provide advice that guides but does not decide 
participant selection.
    During the multi-day Ideas Lab workshop, participants interact in 
unconventional new ways to develop innovative research project ideas on 
the selected topic area. Professional facilitators, experienced in 
sandpit-like activities, integrate creative problem-solving techniques, 
iterative project-development activities, and real-time peer review by 
both participants and a resident panel of experts (called the mentor 
group) to advance the most innovative ideas. Outcomes at the end of the 
workshop are research project concepts that vary in scale and scope in 
addressing the grand challenge topic of the Ideas Lab. At the end of 
the Ideas Lab, the panel of reviewers provides a consensus report 
summarizing its evaluation of each project concept. Based on this 
review, the Program Officers invite the submission of full proposals 
for some, none, or all of the project concepts. The invited groups have 
six to eight weeks to submit full proposals, which are then reviewed by 
the same panel of mentors using NSF's two merit review criteria. Based 
on that review, NSF then makes a decision whether to fund some or all 
of the proposals. Taking part in the Ideas Lab does not mean that the 
participant is guaranteed to be funded under an award resulting from 
the Ideas Lab process.
    Experimentation with the Ideas Lab is still at an early stage. A 
total of four Ideas Labs have been conducted to date. The first three 
resulted in 12 awards and the fourth is currently awaiting full 
proposal submissions. Feedback from participants in the Ideas Labs has 
been positive. However, the resulting funded projects are still in 
their beginning phases. As they progress, NSF will look at the outcomes 
of these projects to evaluate whether they resulted in transformative 
research.

Q3. Researchers will send in proposals whenever they have an idea that 
they would like to have funded. However, NSF also puts out 
solicitations for specific areas of research. Please explain how 
decisions are made on what types of research areas warrant a specific 
solicitation from the Foundation? What happens if the Foundation does 
not receive high quality proposals for a solicitation? Do you pick from 
what you have or do you rework the solicitation?

A3.  Solicitations are formal NSF publications that encourage the 
submission of proposals in specific program areas of interest to NSF. 
Solicitations are generally more focused than program announcements, 
and normally apply for a limited period of time. Ideas for new 
solicitations can be initiated within Divisions by Division Directors, 
at the Directorate level by Assistant Directors, or by groups of 
Assistant Directors who see the need for a new cross-cutting activity. 
Program Directors also commonly suggest ideas for new initiatives. The 
initial spark for a new initiative often comes from interaction with 
the scientific community through scientific meetings or other 
communications. When an idea for a new solicitation is suggested, a 
working group is formed that includes Program Officers who are expert 
in the research area. The working group collaborates on a detailed plan 
for the new solicitation, which is then discussed and reviewed by 
various levels of leadership before approval.
    Specific factors that are considered when deciding whether to 
develop a new solicitation include the following:

     the intellectual reason for the Program, activity, or 
initiative;

     whether the new activity(ies) will generate sufficient 
interest in the targeted community;

     whether the Program, activity, or initiative is new, how 
it supports the long-range goals of the Directorate and/or NSF;

     whether the size of the effort justifies a separate 
announcement and/or competition;

     the total funding available for the proposal competition, 
including estimated proposal receipts and anticipated number of awards 
and funding levels;

     cross-Directorate participation (and implications) in the 
Program.

    Program Solicitations often specify submission limits, award 
conditions or reporting requirements, and provide supplemental proposal 
preparation guidance in addition to what is in the Grant Proposal 
Guide. Program solicitations also provide specific review criteria when 
reviewing proposals. In cases where the Foundation does not receive 
high quality proposals for a solicitation, we decline the proposals 
that are not of high quality. The solicitation could be revised and 
recompeted to attract high quality proposals. Because NSF funding 
opportunities generally attract more high quality proposals that we can 
fund, this would be a rare occurrence.

Q4. After reviewing the flow chart for the proposal and award process 
and timeline, the Directorate Assistant Director seems conspicuously 
absent from the process. Please describe what the role and 
responsibilities of the Assistant Director are in the funding process, 
both from a programmatic and overall agency funding priorities 
perspective.

A4.  While Assistant Directors (ADs) are not involved in the day-to-day 
review and processing of proposals submitted to the Foundation, as 
described in the NSF Grant Proposal Guide Exhibit III 091 (http://
www.nsf.gov/pubs/policydocs/pappguide/nsf1100/gpg-3ex1.pdf), 
they fulfill a vital role in the overall funding process. (See the 
attached referenced flowchart.)
    ADs are knowledgeable about the award portfolios in their 
directorates, but they are not involved in the decision-making process 
itself, because their role is to set the vision and strategic goals and 
objectives for the divisions/offices that report to them.
    Assistant Directors also play an important role in the formal 
reconsideration process. If a PI is dissatisfied with the explanation 
they receive for why a proposal has been declined, he/she may request a 
reconsideration of the decision. ADs/Office Heads are responsible for 
responding to these requests, and review the proposal record to 
determine whether NSF's review of a declined proposal was fair and 
reasonable, substantively and procedurally. If they were involved in 
the decision-making process, the would have a conflict of interest in 
responding to any official reconsideration request.

Q5. How does the Foundation leadership ensure that Program Officers 
``produce and manage a balanced portfolio of awards that address a 
variety of considerations and objectives'' as the FY 10 NSB Report 
states?

A5.  Portfolio balance is reviewed at a variety of levels at different 
times during the decision-making process. Program Officers consider 
many dimensions of portfolio balance when they are making decisions 
about what proposals should be recommended for awards. Some of the 
factors that are considered include: balance across disciplines and 
subdisciplines, award size and duration, awards to new investigators, 
geographical distribution of awards, awards to different types of 
institutions, innovative/potentially transformative projects, projects 
with elements of risk, inter- and multidisciplinary projects, projects 
that integrate research and education, and projects that are relevant 
to agency mission or national priorities. Division Directors review the 
recommendations by Program Officers for portfolio balance before they 
concur with the award recommendations. Portfolio balance is also 
reviewed by our Committees of Visitors who review programs at three-
year intervals. Some programs also contract for external evaluations of 
their portfolio periodically to inform how they might make changes to 
their programs. The results of both COV reports and external portfolio 
analyses are reveiwed by Directorate Advisory Comittees.

Q6. According to the FY 10 Board Report, NSF awarded approximately five 
percent of its annual budget to federal agencies and laboratories. What 
kinds of awards were these and did they go through the formal merit 
review process?

A6.  The 2010 Merit Review Report to the National Science Board 
reported that NSF funded $351.2M in awards to Federally Funded Research 
and Development Centers (FFRDCs). The majority of this funding went to 
two organizations that build and manage large astronomy facilities for 
university consortia: Associated Universities Inc. (AUI) and the 
Association of Universities for Research in Astronomy (AURA). AUI 
received $111M in funding associated with the National Radio Astronomy 
Observatory, the Atacama Large Milimeter Array (ALMA), and other 
related projects. The Association for Universities for Research in 
Astronomy (AURA) received $234.3M for a number of projects including 
building the Advanced Technology Solar Telescope and the Advanced 
Technology Solar Telescope and operations and management of the Gemini 
Observatory, the National Optical Astronomy Observatory, and the 
National Solar Observatory. In addition, $0.4M in funding went to fund 
several much smaller projects through another FFRDC, Aerospace 
Corporation. Proposals submitted to NSF by FFRDCs go through the same 
merit review process as other proposals. The large awards for building 
and operating large facilities go through a very lengthy and detailed 
review process that includes site visits, cost reviews, design reviews, 
and approval by the National Science Board.
    The $351.2M reported in the Merit Review Report also includes $5M 
in contracts to fund the Science and Technology Policy Institute (STP) 
operated by the Institute for Defense Analyses. STPI provides rigorous 
and objective analysis of science and technology (S&T) policy issues 
for the White House Office of Science and Technology Policy (OSTP) and 
other offices and councils within the Executive Branch of the U.S. 
Government and federal agencies. IDA was selected to operate STPI in 
2003 following a competition and undergoes reviews at five-year 
intervals.
    Note that the funding to FFRDCs described in the FY 2010 Merit 
Review Report did not include contract funds to or from other federal 
agencies through interagency agreements.

Q7. What kind of peer reviewers are coming from industry, non-profits, 
and government? Do they all have Ph.D.s? What role does a panelist from 
the government play? What qualifications do they have?

A7.  Reviewers are chosen for their expertise in areas covered by the 
proposals that they are asked to review. For research proposals, 
reviewers are typically researchers in domains of science relevant to 
the topic of the proposal. In the review of proposals for facilities, 
in addition to the reviewers who can provide input on the research 
impacts, technical feasibility and soundness of the faciltity's design, 
Program Officers may also include reviewers with expertise in other 
relevant fields such as project management, systems engineering, 
complex acquisition processes, architectural design, etc.
    In many scientific and engineering disciplines, some of the leading 
researchers work in industry, non-profits, government laboratories and 
FFRDCs. Examples include computer science and a number of others. Such 
researchers are very much the peers of their academic counterparts and 
are included in NSF's pool of peer reviewers. Some of these researchers 
may have spent part of their research careers in academia and some in 
industry or an FFRDC, allowing them to bring important perspectives on 
the state of the art in the different environments and their potential 
broader impacts. These reviewers tend to have the qualifications that 
are typical for the research communities to which they belong. In many 
disciplines, this is often a Ph.D., although occasionally it is simply 
long experience doing cutting-edge research. Typically, what signals 
the expertise of a researcher is his or her record of research 
achievement, including significant publication in peer-reviewed 
journals and conference proceedings.
    Reviewers from government and industry are often more familiar with 
project management and complex acquisition processes than some of their 
academic counterparts, and so such individuals are sometimes asked to 
bring this expertise to review teams looking at proposals for research 
facilities. Such individuals may or may not have Ph.D.s.
    In general, what NSF looks for in its choices of reviewers is 
expertise in the topics under review.

Q8. How does the Foundation train reviewers to prevent the phenomenon 
of implicit bias?

A8.  The frontline of the merit review process are the approximately 
520 NSF Program Officers (POs) who select experts who can provide the 
information needed to make a recommendation in accordance with the 
National Science Board (NSB) approved criteria for selection of 
projects. Program Officers are trained on conflicts of interest, the 
importance of getting a diversity of perspectives, and guarding against 
the influence of subjective or biased input.
    Proposals submitted to NSF receive rigorous and objective treatment 
and POs ensure that this takes place. Proposals are evaluated by 
independent reviewers consisting of scientists, engineers, and 
educators who do not work at NSF or for the institution that employs 
the proposing researchers. NSF selects the reviewers from among a pool 
of experts in each field, and their evaluations are anonymous. On 
average, about 50,000 experts give their time to serve on review panels 
each year. POs ensure that there is diverse representation within the 
review group. The goal is to achieve a balance among various 
characteristics, including type of organization represented, reviewer 
diversity, age distribution and geographic balance.
    The reviewer's job is to provide advice to NSF on which projects 
are the highest priorities. This competitive process ensures that many 
voices are heard and that only the best projects make it to the funding 
stage. When someone is asked to review a proposal (either as an ad hoc 
or panel reviewer), they are provided with information on the 
confidentiality of the process and the potential for conflicts of 
interest. Panelists sign a ``Conflict-of-interests and Confidentiality 
Statement'' whenever they participate in a panel. For ad hoc reviewers, 
by submitting their review, they are acknowledging that they've been 
informed of such policies. Again, NSF POs are responsible for assuring 
that appropriate, qualified merit reviewers are selected and the entire 
process is overseen by Section Heads and/or Division Directors who 
supervise the Program Officers.

Q9. The 2010 reauthorization of the America COMPETES Act required the 
Foundation to ``Apply a Broader Impacts Review Criterion to achieve'' 
various goals. Witnesses at the hearing raised some concerns with the 
draft criteria that is currently being weighed by the Board. Have the 
goals, now specified in statute, been considered in the past when 
making funding decisions? Are the peer reviewers taking these goals 
into consideration during their review or are the Program Officers 
simply tasked with this responsibility? Based on the work being 
conducted by NSB and NSF and your experience with the merit review 
process, is the legislative requirement achiveable and is it necessary?

A9.  NSF strives to invest in a robust and diverse portfolio of 
projects that creates new knowledge and enables braekthroughs in 
understanding across all areas of science and engineering research and 
education. To identify which projects to support, NSF relies on a a 
merit review process that incorporates consideration of both the 
technical merits of a proposed project and its potential to contribute 
more broadly to advancing NSF's mission ``to promote the progress of 
science; to advance the national health, prosperity, and welfare; to 
secure the national defense; and for other purposes.'' In 1997, these 
considerations were put into action through the two primary merit 
review criteria of Intellectual Merit and Broader Impacts. Each 
reviewer must consider, and address, both merit review criteria for 
each proposal.
    As noted in your question, the importance of incorporating 
consideration of potential broader impacts in deciding which projects 
to fund was re-emphasized in the America COMPETES Reauthorization Act 
of 2010. Having the reinforcement of Congressional support on the 
fundamental nature of the Foundation's Organic Act is always an 
important, and appreciated, development. However, there is a danger of 
viewing the Broader Impacts criterion as a ``one size fits all'' 
checklist, which would be a mistake.
    This COMPETES Reauthorization identified a number of societally 
relevant outcomes that may result as a consequence of NSF-funded 
research. Stated more broadly, these outcomes include (but are not 
limited to) increased participation of women, persons with 
disabilities, and underrepresented minorities in STEM; improved STEM 
education at all levels; increased public scientific literacy and 
public engagement with science and technology; improved well being of 
individuals in society; development of a globally competitive STEM 
workforce; increased partnerships between academia, industry, and 
others; increased national security; increased economic competitiveness 
of the United States; and enhanced infrastructure for research and 
education. These represent examples of societally relevant outcomes. 
The NSF will strive to clarify that these examples should not be 
considered either comprehensive or prescriptive, and that investigators 
may include appropriate outcomes not covered by these examples.



Responses by Dr. Keith Yamamoto,
Vice Chancellor for Research, University of California San Francisco

Questions submitted by Chairman Mo Brooks

    Thank you for these insightful questions, Chairman Brooks.

Q1. In basic science research, when one cannot possibly know what the 
outcome may be, how can one establish what the broader impacts will be?

A1.  Basic science research, by definition, is untargeted, driven only 
by the curiosity, intuition, background and expertise of the 
investigator(s) who define a question and develop a plan to answer it. 
For such investigations, it is impossible to ascribe broader impacts at 
the time of merit review. Only after the work has been completed, often 
long after its completion, can the wisdom of retrospection be applied 
to define broader impacts. Thus, inclusion of the broader impacts 
criterion in the merit review of grant applications is misguided and 
damaging to the integrity of the merit review process.

Q2. Having sat on review panels, please tell me how heavily the broader 
impacts of a proposal are weighed when rating the proposals you were 
considering? Did the evaluation of broader impacts have any effect on 
the evaluation of the intellectual merit of the proposals?

A2.  My experience on NSF review panels predated the 1998 development 
of the broader impacts criterion, so I cannot provide experience-based 
perspective here. However, as indicated in my testimony, any departure 
from singular focus on scientific merit, as mandated by this assessment 
of broader impacts, erodes the evaluation of the intellectual merit of 
the proposals.
    In this regard, I wish to comment in particular on the point raised 
by Ms. Jackson that the broader impact criterion provides a metric to 
distinguish proposals that are equally meritorious on scientific 
grounds. She viewed this metric as ``an important role for the broader 
impacts criteria and should be kept as such because there are clearly 
far more proposals that are excellent than can be funded.'' With due 
respect, my opinion is sharply different. Application of this approach 
would discriminate against true basic science research as defined in my 
response to your first question above, and favor funding of work with a 
clear application. To the extent that this evaluation metric would 
become known in the scientific community, this approach would lead to a 
decline in basic science proposals. Fundamental discovery remains 
absolutely essential for scientific progress, and would suffer 
significantly from negative discrimination applied at the merit review 
stage.
    In NIH peer review, as noted by Dr. Marrett, this conflict is 
avoided by placing broader impact considerations in a separate review 
process carried out by separate review committees, thus maintaining the 
singular focus of merit review.
Responses by Dr. Nancy B. Jackson, 
President, American Chemical Society

Questions submitted by Chairman Mo Brooks

Q1. In basic science research, when one cannot possibly know what the 
outcome may be, how can one establish what the broader impacts will be?

A1.  At the National Science Foundation (NSF), all proposals are 
evaluated for intellectual merit and broader impacts. Two major 
determinate factors in estimating the long-term broad impacts of 
scientific proposals are the amount of resources available to fund 
grants and the standards of the Request for Proposal (RFP).
    NSF receives many more meritorious proposals than it could ever 
fund. While a proposal with weak intellectual merit has no hope of 
getting NSF funding, many proposals are rated ``excellent'' with strong 
intellectual merit and still do not get funded because of the stiff 
competition. The broader impacts criteria allow decision makers to 
gauge which research is the most urgent or has the greatest relevance 
to improving the quality of life. This merit review process enables NSF 
to ensure that precious R&D money goes only to the most relevant R&D 
needs.
    As anyone with a retirement fund has been told, managing a 
portfolio is critical to its long-term strength. Financial advisors 
stress that it's important to find the right balance between solid 
performing stocks and riskier investments that may provide higher 
returns. Managing a research portfolio is similar.
    The research manager, whether working in industry or at a 
government agency, strives to find the right balance between science 
that will deliver steady advances and ideas that are out of the box, 
but could result in game-changing developments. This point is 
especially important in times of restricted funding. It's human nature 
to make more conservative choices and be risk-aversive when times are 
tough. Expanding resources allows grant reviewers to ``take a chance'' 
on promising but less established scientific ideas that may result in a 
major scientific leap when compared to a safer but more conservative 
proposal.
    For every 100 grant proposals a program officer reviews, a small 
number (perhaps 10) will be of such high quality that it is obvious 
they should be funded. Another 50 proposals will be recognized as 
clearly not competitive; however, they still must be addressed through 
the process. An agonizing choice must then be made over the 40 
proposals in the middle. These include proposals that may be considered 
excellent or very good. In fiscal year 2010, the NSF award rate was 23 
percent. In our analogy of 100 proposals, this would mean that 13 out 
of the remaining 40 would be funded.
    Unfortunately, it has proven difficult to pinpoint from where the 
next game-changing scientific moment will occur. In 1994, when NSF 
funded the Stanford Integrated Digital Library Project, the Web was a 
tiny portion of the Internet, which was dominated by file sharing and 
gopher sites. This innovative grant to fund research into developing 
methods for searching Internet databases eventually lead to the 
creation of a $200 billion dollar company (Google, Inc.), which 
revolutionized almost every aspect of the way humans interact with each 
other and information. However, it would have been difficult to 
pinpoint that particular project as the most promising grant NSF would 
fund in 1994.
    Similarly, Dr. Robert Grubbs won the 2005 Nobel Prize in chemistry 
for his work in olefin metathesis. This complex organic reaction 
essentially allows molecules to swap components and is broadly used 
today in the pharmaceutical industry. The basic understanding of olefin 
metathesis dates back to the 1950s; however, at that time it was deemed 
expensive and dangerous for industrial use. For four decades corporate 
and academic researchers labored to refine and improve the reaction, 
which did not become industrially productive until the late 1990s. NSF 
was a major contributor to Dr. Grubbs' work, which is revolutionizing 
the way drug companies improve the environmental footprint of chemical 
reactions. Without NSF's long-term dedication to the rigor and quality 
of Dr. Grubbs' (along with many others) work, safe and economically 
viable olefin metathesis may not have occurred.
    These two examples (one based on a single grant that revolutionized 
the information age, the other, the result of four decades of research 
by multiple teams of researchers) show the need for sustained, 
predictable research funding. Ensuring a steady stream of research 
funds would allow NSF to fund not only the most rigorous and reliable 
research, but also take risks in cutting-edge ideas based on promising 
data. By doing so, the government would be encouraging the broadest 
possible impacts of taxpayers' precious dollars.
    One of the reasons why the merit review process is so successful is 
because it draws from the collective wisdom of the scientific 
community. Many NSF personnel come directly from the scientific 
community and return to their research institutions at the end of their 
two- or three-year rotations. Relying on rotating directors means the 
managers are up to date on the most recent scientific developments. The 
panels that perform the peer review of proposals are fellow researchers 
in the field, and as such, are also up to speed on the latest 
developments. This scientific community service, whether performed by 
grant proposal reviewers or NSF program officers, is an integral part 
of scientific culture. Many scientists dedicate their time in this way 
because it provides an opportunity to remain in touch with and 
influence the cutting edge, as well as because they understand that the 
system only works if everyone volunteers to play their part. In a way, 
it is the science community's way of ``paying it forward.''
    The broader impacts criteria include considerations about whether 
the research proposal would broaden underrepresented minorities' 
participation in science, strengthen U.S. infrastructure, improve 
national security, or foster innovation. Including some of these 
impacts is the result of language in the America COMPETES bill enacted 
last year. The broader impacts criteria take into consideration which 
research is the most urgent or has the greatest relevance to improving 
the quality of life. The broader impacts criteria enables NSF to choose 
between meritorious and even more meritorious proposals and is a way to 
ensure that precious R&D money goes to the most pressing R&D needs.
Responses by Dr. Jorge Jose,
Vice President for Research, Indiana University

Questions submitted by Chairman Mo Brooks

Q1. In basic science research, when one cannot possibly know what the 
outcome may be, how can one establish what the broader impacts will be?

A1.  Thank you, Chairman Brooks, for the question. As you correctly 
note, in basic science research, it is not possible to know in advance 
what the outcome of research will be. We pose a question for which we 
would like to find an answer, and we lay out a research plan that in 
principle should lead us to an answer. The research plan is based on 
prior knowledge and an educated hypothesis of the expected results. A 
researcher may not know for certain that a particular plan will yield 
the results they hope for, but they do know aspects of a plan that is 
likely to do so. Reviewers, in fact, will base their review on whether 
the hypothesis and the research plan seem to be an appropriate way to 
ascertain the ansers to the question being posed. In the course of an 
investigation, other questions or an unexpected discovery may arise 
that may lead the researcher to significantly change the direction of 
the research.
    My sense is that a similar awareness of cause and effect can shape 
our expectations for what the broader impacts of any given research 
project might be. Take, for example, the broader impact goal of 
expanding the participation of underrepresented minorities in science, 
technology, engineering, and mathematics (STEM). I cannot know for 
certain that a research project will have that result, but I can know 
some aspects of a research plan that are likely to do so. If my 
research plan, for instance, includes some form of collaboration with 
colleagues from an HBCU, then it is more likely to expand the 
participation of underrepresented minorities than if it does not 
include that collaboration. Of course the collaboration is not a 
failsafe guarantee what the broader impact will be, but it increases 
the likelihood--just as certain elements of the scientific research 
plan will increase the likelihood of certain experimental outcomes.

Q2. Having sat on review panels, please tell me how heavily the broader 
impacts of a proposal are weighed when rating the proposals you were 
considering. Did the evaluation of broader impacts have any effect on 
the evaluation of the intellectual merit of the proposals?

A2.  Thank you, Chairman Brooks, for that question. As Dr. Marrett 
noted, the NSF has many processes in place to guard against bias within 
review committees--to make sure, in other words, that as far as is 
possible, the process identifies the projects with the most significant 
intellectual merit. Ideally, questions of broader impact are used only 
to decide afterwards between proposals of equal intellectual merit. 
Certainly at Indiana University, when we are allocating resources, we 
make every effort to do the same thing--to remove conflicts of 
interest, to focus on the intellectual merit of competing proposals, 
and to leave ``broader impact'' and strategic considerations for later.
    Realistically, of course, it is impossible to guard against all 
variables that might prioritize some other aspect of a proposal over 
its intellectual merit. Indeed, one might worry that the expansion of 
broader impact goals as laid out in the America Competes Act increases 
that difficulty. But my experience is that more often than not, the 
intellectual merit of a proposal is the issue that is most 
determinative in funding recommendations, since consideration of 
broader impacts is generally a second-level concern.
