[Federal Register Volume 61, Number 237 (Monday, December 9, 1996)]
[Notices]
[Pages 64922-64930]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 96-31214]
=======================================================================
-----------------------------------------------------------------------
NATIONAL SCIENCE FOUNDATION
Proposed Data Collection: Comment Request
Title of Proposed Collection: National Science Board and National
Science Foundation Staff Task Force on Merit Review Discussion
Report
Merit Review at NSF
For every proposal that receives funding from the National Science
Foundation, two do not. To determine which get funded and which do not,
NSF relies on a rigorous, competitive process of merit review based on
peer evaluation.
Merit review is the cornerstone of the NSF's work. Virtually all of
the 30,000 new proposals submitted to NSF annually undergo external
merit review. NSF receives over 170,000 reviews each year to help
evaluate these proposals. Through the use of merit review, NSF seeks to
maintain the high standards of excellence and accountability for which
it is known around the world.
Why Consider Changing NSF's Merit Review Criteria?
NSF's current criteria were adopted by the National Science Board
in 1981. They remain an effective means for determining the optimal
allocation of NSF's valuable resources. From time to time, it is
neverless prudent to examine the review criteria--in the spirit of
improving an already outstanding system.
Furthermore, there are also a number of important factors that
deserve consideration in any assessment of NSF's review criteria:
--First, NSF's 1994 strategic plan established long-range goals and
core strategies for the Foundation.
--Second, several studies suggest that there is room for improvement in
NSF's highly successful system of merit review. For example, surveys of
reviewers and program officers have revealed that the current criteria
are not always well understood and often ignored.
--Third, seminal events over the past fifteen years--notably the end of
the Cold War and the rise of global economic competition--have altered
the context for public support of research and education. It is now
more important than ever to highlight and document the returns to
society on NSF's investments in research and education.
It is worth noting in addition that maintaining flexibility in the
application of criteria may be as important as the criteria themselves.
Most reviewers will only address those elements that they feel they are
capable of judging. Similarly, NSF also does not pre-assign weights to
the criteria; given the variation across NSF's many different programs,
any such ``one size fits all'' approach would be counterproductive.
Overall, excellence will continue to be the hallmark of all NSF-
sponsored activities.
Furthermore, NSF will continue to employ special criteria when
proposals are expected to respond to the specific objectives of certain
programs and activities. Examples include teacher training projects and
the development of large research facilities.
Opportunity for Input and Comments
At the November 1996 meeting of the National Science Board, the
Board's Merit Review Task Force recommended that the current merit
review criteria be simplified and that the language be harmonized with
the NSF strategic plan. The current criteria and the Task Force's
recommended criteria are shown below.
With the release of the Task Force's discussion report, NSF and the
Board aim to stimulate discussion within and outside the Foundation.
NSF is seeking input and comments from all interested persons--
especially current and potential grant applicants and reviewers, as
well as informed observers and followers of science and engineering
research and education. To encourage the broadest possible comment and
discussion, we have posted a summary of this document along with a
comparison of current and proposed merit review criteria on our
homepage (http://www.nsf.gov). The summary includes ``hotlinks'' to the
full NSB Task Force report, NSF strategic plan, and other related
documents. Most important, there is a response box for you to provide
the agency with your feedback electronically.
[[Page 64923]]
We hope you will provide us with your thoughts on the proposed
criteria. Comments on any aspect of the merit review criteria are
welcome. In particular, we are interested in your views on questions
such as:
--Are the proposed criteria clear? Would they be easier to use than the
current criteria?
--Would the proposed criteria elicit useful input and comments from
reviewers?
--Would the proposed criteria improve NSF's ability to foster linkages
(e.g. across disciplines and between academe and industry)?
--Would the proposed criteria contribute to the integration of research
and education?
--Are there further improvements to the criteria that you would
recommend?
Thank you for taking the time to share your ideas with us. Please
feel free to raise any specific questions or concerns you may have
regarding the proposed criteria or the merit review process generally.
(A set of FAQs (frequently asked questions) is available for your
reference.)
Also, please let us know via the response forms if you would like
to receive information describing what changes to the criteria (if any)
are eventually adopted by the Board. A final decision is expected by
the summer of 1997.
Send comments via the feedback mechanisms provided on the NSF
homepage at (http://www.nsf.gov). Comments also can be mailed to Office
of Policy Support, National Science Foundation, 4201 Wilson Boulevard,
Room 1205, Arlington, VA 22230.
All comments should be received by January 31, 1997.
Dated: December 4, 1996.
George T. Mazuzan,
Acting Director, Office of Legislative and Public Affairs.
Current and Proposed Merit Review Criteria
Current Criteria (adopted in 1981)
1. Research performer competence--This criterion relates to the
capability of the investigators, the technical soundness of the
proposed approach, and the adequacy of the institutional resources
available.
2. Intrinsic merit of the research--This criterion is used to
assess the likelihood that the research will lead to new discoveries or
fundamental advances within its field of science or engineering, or
have substantial impact on progress in that field or in other science
and engineering fields.
3. Utility or relevance of the research--This criterion is used to
assess the likelihood that the research can contribute to the
achievement of a goal that is extrinsic or in addition to that of the
research itself, and thereby serves as the basis for new or improved
technology or assist in the solution of societal problems.
4. Effect on the infrastructure of science and engineering--This
criterion relates to the potential of the proposed research to
contribute to better understanding or improvement of the equality,
distribution, or effectiveness of the nation's scientific and
engineering research, education, and manpower base.
Proposed Criteria
1. What is the intellectual merit and quality of the proposed
activity?
The following are suggested questions to consider in assessing how
well the proposal meets the criterion: What is the likelihood that the
project will significantly advance the knowledge base within and/or
across different fields? Does the proposed activity suggest and explore
new lines of inquiry? To what degree does the proposer's documented
expertise and record of achievement increase the probability of
success? Is the project conceptually well designed? Is the plan for
organizing and managing the project credible and well conceived? And,
is there sufficient access to resources?
2. What are the broader impacts of the proposed activity?
The following are suggested questions to consider in assessing how
well the proposal meets the criterion: How well does the activity
advance discovery and understanding while concurrently promoting
teaching, training, and learning? Will it create/enhance facilities,
instrumentation, information bases, networks, partnerships, and/or
other infrastructure? How well does the activity broaden the diversity
of participants? Does the activity enhance scientific and technological
literacy? And, what is the potential impact on meeting societal needs?
[NSB/MR-96-15]
National Science Board and National Science Foundation Staff
Task Force on Merit Review; Discussion Report
November 20, 1996.
National Science Board
DR. F. ALBERT COTTON, Distinguished Professor, Department of
Chemistry, Texas A&M University
* DR. CHARLES E. HESS, Director of International Programs,
University of California--Davis
---------------------------------------------------------------------------
* Member, Executive Committee
---------------------------------------------------------------------------
* DR. JOHN E. HOPCROFT, Joseph Silbert Dean of Engineering, Cornell
University
* DR. SHIRLEY M. MALCOM, Head, Directorate for Education and Human
Resources Programs, American Association for the Advancement of
Science
* DR. JAMES L. POWELL, President & Director, Los Angeles Museum of
Natural History
DR. FRANK H.T. RHODES, President Emeritus, Cornell University
DR. IAN M. ROSS, President-Emeritus, AT&T Bell Laboratories
* DR. RICHARD N. ZARE (Chairman), Professor, Department of
Chemistry, Stanford University
DR. SANFORD D. GREENBERT, Chairman & CEO of TEI Industries, Inc.
DR. EVE L. MENGER, Director, Characterization Sciences & Services,
Corning Incorporated
DR. CLAUDIA I. MITCHELL-KERNAN, Vice Chancellor, Academic Affairs
and Dean, Graduate Division, University of California
* DR. DIANA S. NATALICIO (Vice Chairman), President, The University
of Texas at El Paso
DR. ROBERT M. SOLOW, Institute Professor Emeritus, Massachusetts
Institute of Technology
DR. WARREN M. WASHINGTON, Senior Scientists and Head, Climate Change
Research Section, National Center for Atmospheric Research
DR. JOHN A. WHITE, JR., Regents' Professor and Dean of Engineering,
Georgia Institute of Technology
** DR. JOHN A. ARMSTRONG, IBM Vice President for Science &
Technology (Retired)
---------------------------------------------------------------------------
** NSB nominee pending U.S. Senate confirmation
---------------------------------------------------------------------------
** DR. MARK K. GAILLARD, Professor of Physics, University of
California, Berkeley
** DR. M.R.C. GREENWOOD, Chancellor, University of California, Santa
Cruz
** DR. STANLEY V. JASKOLSKI, Vice President, Eaton Corporation
** DR. EAMON M. KELLY, President, Tulane University
** DR. JANE LUBCHENCO, Wayne and Gladys Valley Professor of Marine
Biology and Distinguished Professor of Zoology, Oregon State
University
** DR. VERA RUBIN, Staff Member (Astronomy), Department of
Terrestrial Magnetism, Carnegie Institution of Washington
** DR. BOB H. SUZUKI, President, California State Polytechnic
University
** DR. RICHARD TALIA, Professor, Department of Computational &
Applied Mathematics, Rice University
* DR. NEAL F. LANE (Chairman, Executive Committee), Director, NSF
DR. MARTA CEHELSKY, Executive Officer
Members of the Task Force
National Science Board Members:
Dr. Warren M. Washington, Chair, Dr. Shirley M. Malcom.
[[Page 64924]]
National Science Foundation Staff
Dr. Mary E. Clutter, Dr. John B. Hunt.
Mr. Paul J. Herer, Executive Secretary.
I. Context of the Report
The merit review process is the modus operandi for the evaluation
of proposals at the National Science Foundation (NSF). While almost all
of the 30,000 proposals submitted to NSF annually undergo external
merit review, NSF has the resources to fund only about one third of
them. NSF receives over 170,000 reviews each year to help evaluate
these proposals. Through the use of merit review, NSF seeks to maintain
its high standards of excellence and accountability for which it is
known around the world.
In 1981, the National Science Board (NSB) adopted four generic
criteria for the selection of research projects, titled: (1) research
performance competence, (2) intrinsic merit of the research, (3)
utility or relevance of the research, and (4) effect of the research on
the infrastructure of science and engineering. (A detailed description
of these criteria may be found in Appendix A.) Because education
programs had been eliminated from the budget at that time, the 1981
criteria addressed on research proposals. In the 1980s, they were
adapted to suit education programs as those were reestablished.
Also, since 1981, the portfolio of projects solicited and supported
by NSF has expanded to include, among other things, broad education
initiative and focused center-based activities. Further, the NSF
Strategic Plan (NSF95-24) embraces new long-range goals and core
strategies, and the Government Performance and Results Act (GPRA)
emphasizes the importance of linking NSF's goals and strategies to the
results of its portfolio of investments in science and engineering. In
light of these changes, an assessment of the appropriateness of the NSB
criteria seems warranted.
At its May 1995 meeting, the NSB stated that re-examining the
criteria in light of the new Strategic Plan was a matter of high Board
interest. Subsequently, an NSF staff task group on review criteria,
formed by the Deputy Director, found that the criteria are unevenly
applied by reviewers and NSF staff in the proposal review and selection
process, and reported that, ``The NSB criteria are in need of
clarification and should be rewritten.'' The task group also
recommended that options be explored for more effective application of
the criteria.
In May 1996, the Board established the NSB-NSF Staff Task Force on
Merit Review, and charged it with examining the Board's generic review
criteria and making recommendations on retaining or changing them,
along with providing guidance on their use. This paper presents the
Task Force's deliberations and findings. It is not intended as a final
set of recommendations but as a means of stimulating discussion within
and outside of the Foundation.
II. Task Force Membership and Activities
The Task Force has the following membership:
National Science Board Members
Dr. Warren M. Washington, Chair, Senior Scientist, Climate and Global
Dynamics Division, National Center for Atmospheric Research, Boulder,
Colorado
Dr. Shirley M. Malcom, Head, Directorate for Education and Human
Resources Programs, American Association for the Advancement of
Science, Washington, D.C.
National Science Foundation Staff
Dr. Mary E. Clutter, Assistant Director for Biological Sciences
Dr. John B. Hunt,* Acting Assistant Director for Mathematical and
Physical Sciences
---------------------------------------------------------------------------
* Replaced Dr. William Harris.
---------------------------------------------------------------------------
Executive Secretary
Mr. Paul J. Herer, Senior Advisor for Planning and Technology
Evaluation, Directorate for Engineering
The Task Force met several times for extensive discussions, and
reviewed a number of previous studies, surveys and reports, including
the following:
(1) Criteria for the Selection of Research Projects by the National
Science Foundation, adopted by the National Science Board at its 228th
meeting on August 20-21, 1981.
(2) Federally Funded Research: Decisions for a Decade. U.S.
Congress, Office of Technology Assessment (1991).
(3) The Track Record of NSF Proposal Review: Reviewers Rate the
Process. NSF Program Evaluation Staff and Science Resources
International (SRI) (1991).
(4) Peer Review. Reforms Needed to Ensure Fairness in Federal
Agency Grant Selection, United States General Accounting Office (1994).
(5) Report of the NIH Committee on Improving Peer Review (1996).
(6) NSF Proposal Review Project Reports (1996, by internal teams):
Task Group on Review Criteria (P. Stephens, Chair)
Task Group on Review Variations (D. Schindel/D. Chubin)
Task Group on Calibration and Disaggregated Ratings (C. Eavey)
III. Current Criteria and Their Use
The four generic criteria established by the NSB in 1981 for the
selection of projects are: (1) research performance competence, (2)
intrinsic merit of the research, (3) utility or relevance of the
research, and (4) effect of the research on the infrastructure of
science and engineering. For reference, the full NSB guidance for these
criteria are provided in Appendix I.
The table below summarizes the results of two surveys and
highlights some of the problems with the current criteria from two
different perspectives.
A cross-section of reviewers in a 1991 NSF/SRI survey
(first column) considered the first two NSB criteria (intrinsic merit
and PI competence) to be considerably more important than the last two.
Less than half of the respondents said they usually commented on all
four criteria; as many as 20% said they ignored the NSB criteria
altogether.
A 1995 electronic survey of NSF program officers (P.O.) in
35 divisions on reviewer responsiveness (second column) revealed that
program officers experience difficulty in obtaining useful input from
reviewers with respect to criterion 3 (utility/relevance) and criterion
4 (infrastructure).
Perceived Importance and Usefulness of Current Review Criteria
[In percent]
------------------------------------------------------------------------
1991 SRI 1995 survey
survey of of NSF
reviewers P.O.\2\
----------------------------------------------------\1\-----------------
(1) Competence............................... 94 0
(2) Instrinisic Merit........................ 98 2
(3) Utility/Relevance........................ 56 31
(4) infrastructure........................... 26 46
------------------------------------------------------------------------
\1\ Percent of reviewers who said criterion was ``extremely important''.
\2\ Percent of program officers expressing difficulty in obtaining
useful input.
In addition to these surveys, the NSF Office of Policy Support OPS)
recently conducted an informal content analysis on a small sample of
reviews of research project proposals to gain an empirical perspective
of how reviewers use the four NSB criteria. By far the criterion most
frequently used by reviewers was research performance competence.
Almost every reviewer commented on some variation of competence. The
intrinsic merit of the proposed research was addressed in about 80% of
the reviews; utility/relevance in about 40%; and infrastructure in
about a third of the reviews. For criterion 4, reviewers
[[Page 64925]]
referred to such potential ``products'' as trained researchers/graduate
students, hardware, and information data bases. The goals and core
strategies in NSF's strategic plan, such as the integration of
education and research, were rarely mentioned in the reviews.
These studies imply that there are a number of problems with the
current NSB generic criteria, including:
Lack of clarity in wording encourages the use of
``unwritten'' criteria.
Reviewers and Program Officers do not apply to the current
criteria uniformly (e.g., criterion #3 and #4 are not well understood
and often ignored).
Criteria do not easily encompass non-research activities,
e.g., education and human resources, large-scale facilities, and
centers.
Criteria do not track very will with NSF Strategic Plan.
Considerable variation exists in use of criteria across
NSF.
In February 1996, the NSF staff Task Group on Review Criteria
(Chair, Pamela Stephens) reported that, ``The NSB criteria are in need
of clarification and should be rewritten'', with consideration given
to: (a) making the criteria clearer to evaluators; (b) emphasizing
important attributes such as innovation, clarity of thought and
soundness of approach; and (c) encouraging substantive comments on the
quality of proposals. The Task Group further recommended that NSF
explore more effective ways to apply the infrastructure criterion, and
should continue the practice of allowing programs to employ additional
specific criteria as needed.
The staff Task Group suggested a number of interrelated components
that contribute to the evaluation of a proposal's overall merit,
including: Intrinsic Merit, Significance, Innovative, Approach,
Feasibility, and Effect on Infrastructure. This served as a starting
point for the NSB-NSF Task Force.
IV. Revised Generic Merit Review Criteria
The Task Force recommends the two generic criteria (below) to
replace the current four NSB criteria. Within each criterion is a set
of contextual elements, defined by questions to assist the reviewer in
understanding their intent. These elements are non-inclusive; i.e. it
is recognized that, for some programs, other considerations not
identified below may be important for the evaluation of proposals.
Further, reviewers are requested to address only those elements that
they consider relevant to the proposal at hand and that they feel
qualified to make judgments on.
#1 What is the intellectual merit and quality of the proposed activity?
The following are suggested questions to consider in assessing how
well the proposal meets the criterion: What is the likelihood that the
project will significantly advance the knowledge base within and/or
across different fields? Does the proposed activity suggest and explore
new lines of inquiry? To what degree does the proposer's documented
expertise and record of achievement increase the probability of
success? Is the project conceptually well designed? Is the plan for
organizing and managing the project credible and well conceived? And,
is there sufficient access to resources?
#2 What are the broader impacts of the proposed activity?
The following are suggested questions to consider in assessing how
well the proposal meets the criterion: How well does the activity
advance discovery and understanding while concurrently promoting
teaching, training, and learning? Will it create/enhance facilities,
instrumentation, information bases, networks, partnerships, and/or
other infrastructure? How well does the activity broaden the diversity
of participants? Does the activity enhance scientific and technological
literacy? And, what is the potential impact on meeting societal needs?
The NSB-NSF Task Force believes that the proposed new criteria
offer several advantages over the existing criteria, such as:
NSF is increasingly asked to connect its investments to
societal value, while preserving the ability of the merit review system
to select excellence within a portfolio that is rich and diverse.
Having two criteria, one for intellectual quality and the other for
societal impact, should serve to reveal the situations where proposals
have high quality but minimal potential impact (and vice-versa).
Quality will continue to be the threshold criterion, but will come to
be seen as not sufficient by itself for making an award.
The two new criteria are more clearly related to the goals
and strategies in the NSF Strategic Plan. For example, ``NSF in a
Changing World'' states (page 31) that: ``We rely on our proven system
of merit review, which weighs each proposal's technical merit,
creativity, educational impact, and its potential benefits to
society.''
The criteria are simplified by reducing their number from
four to two, and are defined for reviewers and proposers by a set of
suggested contextual elements. Reviewers are asked to describe the
proposal's ``strengths and weaknesses'' with respect to each criterion
using only those contextual elements that they consider relevant to the
proposal at hand.
V. Application of the Proposed Generic Criteria
The Task Force was charged not only with examining the Board's
generic review criteria but also recommending accompanying guidance on
their use. There are a number of important ``process'' issues that help
to frame this guidance.
Because of the great range and diversity of activities supported by
NSF, it is evident that maintaining flexibility in the application of
criteria is as important as the criteria themselves. Most reviewers
will only address those elements that they feel they are capable of
judging. Asking proposers and reviewers to address all of the
contextual elements in each and every proposal, regardless of the
nature of the proposed activity, is not only unrealistic but, in fact,
may be counterproductive. Also, pre-assigning weights to the criteria
will, if applied to all proposals, incorrectly appraise some of them.
It is important to take into account the relative roles of the
external expert reviewers and the NSF program staff. Specifically, NSF
proposals are evaluated by the Program Officer and other NSF staff with
the help of the written reviews from expert peers. These external
reviews are always advisory; the final funding decision rests with the
NSF staff. Hence, while the external reviewer applies the review
criteria to the individual proposal, the Program Officer must evaluate
the proposal within the context of managing a balanced portfolio of
projects that will achieve the program's objectives and contribute to
NSF's overall mission. In particular, reviewer assessment of criterion
#2 (potential impact and societal value) is intended to provide NSF
with input from reviewers, but the ultimate responsibility for judging
the potential impact of the investment of public funds must rest with
NSF. Hence, the Task Force recommends that the NSF staff be provided
flexibility and discretion in the application and weighting of
criteria.
The Use of Special Criteria
NSF supports an extremely diverse set of activities ranging from
individual investigator projects to teacher training to large research
facilities. Many of these activities have special objectives
[[Page 64926]]
and require proposals that are responsive to them. Program
solicitations and announcements are frequently used to solicit
proposals from the community, and, in some cases, the NSB generic
criteria are modified or augmented to make the review process
responsive to the special objectives.
For example, the CISE Minority Institutions Infrastructure Program
Announcement (NSF 96-15) lists nine additional factors that will be
used to evaluate the proposals, including such factors as: (1)
institutional cost-sharing, commitment, and related support to the
projects, and (2) institutional track record in graduating minority
scientists and engineers.
The EHR/CISE Networking Infrastructure for Education Program
Solicitation (NSF 93-13) adds six additional criteria, including:
``Sustainability: The Potential to leverage the ability of the
education community to carry out full scale, self-sustaining and
scaleable educational networking models.''
In other cases, a set of criteria are provided in-lieu of the NSB
generic criteria. For example, the Academic Research Infrastructure
(ARI) Program (NSF 96-12) specifies the following criteria headings:
Research and Research Training Merit; Infrastructure Need; Project
Impacts; and Plans & Funding. Under the latter category, ``the
institutional management plan for maintenance and operation of the
requested facility'' is cited.
Revising the NSB generic criteria will lessen but not eliminate the
need for special criteria. However, it is important that the additional
or replacement criteria be consistent with the intent and spirit of the
NSB generic criteria. Since each new program announcement or
solicitation receives considerable NSF internal review before it is
issued, it is appropriate that this be considered during the
publication's clearance process.
Options for Rating Proposals
Whatever the criteria, reviewers and panelists must be encouraged
to provide substantive comments on proposals, not merely ``check
boxes'' to satisfy some proposal rating scheme. Moreover, NSF should
not impose a rigid system of multiple criteria and sub-criteria, each
with a separate score. The end result is often a review with too much
weight given to less significant aspects of the proposal.
In terms of adjectival proposal ratings and numerical scoring, the
Task Force extensively discussed the pros and cons of several options,
including the following:
1. No ratings or scores. Reviewer comments on proposal's strengths
and weaknesses; then provides a summary narrative statement.
Pros:
Encourages more substantive reviewer comments while
avoiding ``box checking''.
Avoids dependence on ``uncalibrated'' scores.
Results in fewer NSF staff callbacks to reviewers to
clarify ratings and reconcile comments with ratings.
Encourages reviewer to give equal attention to both
criteria.
Makes it easier for program officer to go against the
``collective wisdom''; i.e., to recommend ``high risk'' proposals that
may not be as highly rated as some ``low risk'' proposals.
Cons:
More difficult to ``bin'' proposals (i.e., into categories
such as those that definitely should be funded, those that might be
funded, and those that definitely should not be funded).
More difficult to evaluate the effectiveness and fairness
of the merit review system (i.e., cannot compare ratings scores with
proposal decisions).
Introduces more subjectively into the review process
because of difficulty in interpreting the narrative statement alone.
2. Separate rating for each of the two criterion.
Pros:
Sends message to community that both criteria are
important.
NSF program staff has flexibility to determine relative
application (weighting) of the two criteria to the funding decision.
Provides program officers with better information for
making funding decisions and can provide more precise feedback to
applicants.
Eliminates mere averaging of ratings as a means of ranking
proposals.
Ends semantic arguments about whether a proposal is, e.g.,
``excellent'' or merely ``outstanding'', or somewhere in between.
Cons:
May complicate the ranking of proposals in the panel
review process and lead to proposal ranking that do not reflect
consensus.
May encourage even greater degree of ``box checking'' in
place of substantive comments, i.e., could result in shorter and less
detailed written comments.
3. Single composite rating (for the two criteria).
Pros:
Simplest to understand and use.
Easy to relate proposal ratings to proposal decisions.
Cons:
Reviewers will implicitly weigh each criterion; may not
give much attention to criterion #2 in assigning overall rating.
Encourages ``box checking'' rather than substantive
comments.
Scores may be arbitrary or uncalibrated (i.e., too lenient
or strict).
In order to determine which is the most effective rating scheme
(i.e., one that optimizes rationality, excellence, and fairness) the
Task Force encourages the Foundation to experiment with various
options. In designing these experiments, NSF should be fully cognizant
of recent NIH efforts to redesign its peer review system.
NSF instructions and guidance to reviewers are very important. The
system will be improved only if the reviewer use the criteria when
evaluating the proposal. Thus, whatever criteria the NSB decides upon,
they must be formatted for maximum use. This means redesigning the
review form and the Grant Proposal Guide so that both the P.I.'s and
reviewers understand what is to be evaluated. In fact, it may be
advisable to design different review forms for different classes of
proposals; for example, for investigator initiated research proposals,
for large facility proposals, for systemic education reform projects,
etc.
In order to illustrate how the new criteria might be presented to
the merit reviewer, a sample draft NSF Proposal Review Form is provided
in Appendix B. While option #2 (i.e., provide a rating for each
criterion) is being used in this case for illustration purposes, this
does not imply that it is the recommendation of the Task Force.
A draft one-page synopsis of NSF's strategic plan, NSF in a
Changing World, is also provided in the Appendix C. This plan provides
a context for shaping the Foundation's future through a set of
principles, goals, and core strategies that are aimed at developing a
greater sense of interdependence between the research and education
communities and the public. While a one-to-one mapping of the generic
review criteria to the NSF strategic plan is not necessary, the Task
Force believes that outside expert reviewers should be exposed to at
least a summary of the strategic plan. This may be accomplished by
attaching the synopsis to the proposal review form.
The new criteria imply that changes to NSF's guidelines for
preparing proposals are needed. This should be carefully looked at by
NSF management. At the very least changes will have to be made in the
Grant Proposal Guide. Additionally, in all NSF program
[[Page 64927]]
solicitations and announcements, NSF should carefully explain the full
set of criteria that will be used to evaluate the proposal, including
those related to the program's investment portfolio.
VI. Future Action
On October 17, 1996, the National Science Board approved the
release of the Task Force Discussion Report, subject to final clearance
by the Executive Committee, not as NSB policy, but as a proposal for
broader discussion inside and outside of the Foundation. Specifically,
the Director, NSF, is authorized to: ``share the report with the
Nation's research and education community for comment, for the purpose
of informing the Task Force on Merit Review''. The NSB also requested
the Task Force to provide its recommendations at the March 1997 Meeting
of the National Science Board, with respect to the nature and content
of the new general criteria for review of proposals submitted to NSF
(see Appendix D).
Note. To encourage the broadest possible comment and discussion,
NSF has posted a summary of this document along with a comparison of
the current and proposed merit review criteria on its homepage
(http://www.nsf.gov). Most important, there is a response box for
you to provide the agency with your feedback electronically. NSF
wants to hear your views and specific suggestions on this report.
Appendices
Appendix A--Current Criteria (adopted in 1981)
1. Research performer competence--relates to the capability of the
investigators, the technical soundness of the proposed approach, and
the adequacy of the institutional resources available.
2. Intrinsic merit of the research--the likelihood that the
research will lead to new discoveries or fundamental advances within
its field of science or engineering, or have substantial impact or have
substantial impact on progress in that field or in other science and
engineering fields.
3. Utility or revlenace of the research--the likelihood that the
research can contribute to the achievement of a goal that is extrinsic
or in addition to that of the research itself, and thereby serves as
the basis for new or improved technology or assist in the solution of
societal problems.
4. Effect on the infrastructure of science and engineering--the
potential of the proposed research to contribute to better
understanding or improvement of the quality, distribution, or
effectiveness of the nation's scientific and engineering research,
education, and manpower base.
BILLING CODE 7555-01-M
[[Page 64928]]
[GRAPHIC] [TIFF OMITTED] TN09DE96.001
[[Page 64929]]
[GRAPHIC] [TIFF OMITTED] TN09DE96.002
BILLING CODE 7555-01-C
[[Page 64930]]
Appendix C--Synopsis of NSF Strategic Plan; NSF in a Changing World
(NSF 95-24)
In 1995, the National Science Foundation issued its strategic plan,
NSF in a Changing World, which reiterated the Foundation's mission and
established its strategic goals. The National Science Foundation Act of
1950 (Public Law 81-507) set forth NSF's mission and purpose.
To promote the progress of science: to advance the national health,
prosperity, and welfare: to secure the national defense * * *
As described in NSF in a Changing World, the National Science
Foundation has three long-range goals:
Enable the U.S. to uphold a position of world leadership
in all aspects of science, mathematics, and engineering. This grows
from the conviction that a position of world leadership in science,
mathematics, and engineering provide the Nation with the broadest range
of options in determining the course of our economic future and our
national security.
Promote the discovery, integration, dissemination, and
employment of new knowledge in service to society. This goal emphasizes
the connection between world leadership in science and engineering on
the one hand and contributions in the national interest on the other.
Achieve excellence in U.S. science, mathematics,
engineering, and technology education at all levels. This goal is
worthy in its own right, and also recognizes that the first two goals
can be met only by providing educational excellence. It requires
attention to needs at every level of schooling and access to science,
mathematics, engineering, and technology educational opportunities for
every member of society.
To move toward the achievement of these goals, NSF employs a set of
core strategies. These strategies reaffirm the Foundation's traditions,
especially its reliance on merit review of investigator-initiated
proposals, yet at the same time point to new directions for the
Foundation.
Develop intellectual capital. Selecting the best ideas in
research and education and the most capable people to carry them out is
at the heart of NSF's programmatic activities and the merit review
system with which we implement those programs. Opening opportunities
for all Americans to participate fully in an increasingly technological
society is an essential part of NSF's mission.
Strengthen the physical infrastructure. NSF's programs
support investments in new windows on the universe, through facilities
planning and modernization, instrument acquisition, design and
development, and shared-use research platforms.
Integrate research and education. NSF aims to infuse
education with the joy of discovery and to bring an awareness of the
needs of the learning process to research, creating a rich environment
for both.
Promote partnerships. For NSF, success requires
collaboration with many different partners, including universities,
industry, elementary and secondary schools, other Federal agencies,
state and local governments, and other institutions. We also carry out
partnerships across national boundaries.
The Foundation's general goals and strategies are translated into a
diverse portfolio of activities, which often embody more than one
strategy and contribute to more than one goal. In turn, NSF's efforts
interact with those of other Federal agencies, state and local
governments, school districts, schools, and partners in the private
sector to produce progress toward the three goals. NSF does not itself
conduct research or educate students. Instead, it invests the Nation's
resources in a portfolio of projects and activities performed by
universities, schools, nonprofit institutions, and small businesses.
NSF balances its investments among three broad program functions,
research, projects, facilities, and education and training.
Appendix D--Resolution Approved by the National Science Board at its
339th Meeting, on October 17, 1996
[NSB-96-182]
October 17, 1996.
Whereas, competive merit review, with peer evaluation, is the National
Science Foundation's accepted method of informing its proposal decision
processes;
Whereas, the Board requested that the general review criteria adopted
by the Board in 1981 be re-examined in light of the Strategic Plan
entitled ``NSF in a Changing World,'' as approved by the Board in
October 1994;
Whereas, a joint Task Force of Board members and Foundation staff,
having reviewed a number of studies, surveys and reports and engaged in
extensive discussions of criteria and related matters, have produced a
report containing proposed new general criteria for the review of NSF
proposals;
Whereas, NSF works in partnership with the Nation's research and
education community in all its endeavors;
Now therefore be it resolved, that the National Science Board:
Receives the report of its Task Force on Merit Review containing
proposed new general criteria for review of proposals submitted to NSF;
Authorizes the Director, NSF, to share the report with the Nation's
research and education community for comment, for the purpose of
informing the Task Force on Merit Review;
And asks the Task Force on Merit Review to provide its
recommendations at the March 1997 Meeting of the National Science
Board, with respect to the nature and content of any such criteria.
[FR Doc. 96-31214 Filed 12-6-96; 8:45 am]
BILLING CODE 7555-01-M