[Federal Register Volume 86, Number 21 (Wednesday, February 3, 2021)]
[Notices]
[Pages 8045-8047]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 2021-02240]


=======================================================================
-----------------------------------------------------------------------

NATIONAL SCIENCE FOUNDATION


Agency Information Collection Activities: Comment Request; Merit 
Review Survey--2021 and 2023 Assessment of Applicant and Reviewer 
Experiences

AGENCY: National Science Foundation.

ACTION: Submission for OMB review; comment request.

-----------------------------------------------------------------------

SUMMARY: The National Science Foundation (NSF) has submitted the

[[Page 8046]]

following information collection requirement to OMB for review and 
clearance under the Paperwork Reduction Act of 1995. This is the second 
notice for public comment; the first was published in the Federal 
Register, and no comments were received. NSF is forwarding the proposed 
submission to the Office of Management and Budget (OMB) for clearance 
simultaneously with the publication of this second notice.

DATES: Written comments and recommendations for the proposed 
information collection should be sent within 30 days of publication of 
this notice to www.reginfo.gov/public/do/PRAmain. Find this particular 
information collection by selecting ``Currently under 30-day Review--
Open for Public Comments'' or by using the search function.

FOR FURTHER INFORMATION CONTACT: Suzanne H. Plimpton, Reports Clearance 
Officer, National Science Foundation, 2415 Eisenhower Avenue, 
Alexandria, VA 22314, or send email to [email protected]. Individuals 
who use a telecommunications device for the deaf (TDD) may call the 
Federal Information Relay Service (FIRS) at 1-800-877-8339, which is 
accessible 24 hours a day, 7 days a week, 365 days a year (including 
federal holidays).
    Copies of the submission may be obtained by calling 703-292-7556.

SUPPLEMENTARY INFORMATION: NSF may not conduct or sponsor a collection 
of information unless the collection of information displays a 
currently valid OMB control number and the agency informs potential 
persons who are to respond to the collection of information that such 
persons are not required to respond to the collection of information 
unless it displays a currently valid OMB control number.
    Title of Collection: Merit Review Survey--2021 & 2023 Assessment of 
Applicant and Reviewer Experiences.
    OMB Number: 3145-NEW.
    Type of Request: Request for approval to establish an information 
collection.
    Proposed Project: The National Science Foundation (NSF) receives 
close to 50,000 proposals for funding annually, each of which undergoes 
a rigorous merit review process that is designed to ensure all 
proposals are fairly and thoroughly reviewed. The merit review process 
comprises three phases:
    1. NSF announces funding opportunities on the NSF website and 
Grants.gov. Applicants prepare proposals in response to these 
opportunities and submit their proposals via FastLane (NSF's web-based 
system for proposal submission and review) or Grants.gov.
    2. Proposals are assigned to the appropriate program(s) for review. 
Each proposal is assigned a Program Officer (PO) who selects external 
reviewers to evaluate the proposal according to the two NSF merit 
review criteria, Intellectual Merit and Broader Impacts. The 
Intellectual Merit criterion encompasses the potential to advance 
knowledge. The Broader Impacts criterion encompasses the potential to 
benefit society and contribute to the achievement of specific, desired 
societal outcomes. Programs may have additional review criteria 
particular to the goals and objectives of the program. The NSF 
guidelines for the selection of reviewers are designed to ensure 
selection of experts who can give program officers the proper 
information needed to make a recommendation in accordance with the 
merit review criteria. POs utilize the proposal's reference list, the 
investigator's suggested reviewers, and personal knowledge of 
individual reviewers to identify a pool of diverse experts with respect 
to type of organization represented, demographics, experience, and 
geographic balance, selecting appropriate reviewers with no apparent 
potential conflicts. Most proposals are reviewed by three to ten 
content expert reviewers who provide written feedback on the proposal 
through FastLane. POs synthesize reviewer comments and issue a 
recommendation to either decline or award funding based on reviewer 
feedback, panel discussions, the amount of available funding, and 
portfolio balances (i.e., the diversity of a portfolio, including 
factors such as award type, career stage, demographic characteristics, 
geographic location, institution type, research topic, laboratory 
funding status, and intellectual risk). The proposal and PO 
recommendation is then forwarded to the appropriate Division Director 
or other NSF official for additional review and action to either 
decline or award.
    3. Each proposal recommended for award undergoes an administrative 
review conducted by NSF's Office of Budget, Finance, and Award 
Management. If it passes this review, the proposal is awarded.
    Through this review process, NSF aims to identify the highest 
quality proposals to receive funding. The success of this process 
hinges on the assumptions that applicants will continue to submit to 
NSF their ideas for cutting-edge research and that experts in their 
respective fields will continue to provide high-quality reviews of 
those proposals.
    The goal of this data collection is to assess the experiences of 
applicants and reviewers and their satisfaction with NSF's merit review 
process. The data collection for which this OMB approval is requested 
includes a Web-based survey that will be administered to all applicants 
and reviewers who participated in the merit review process between 
fiscal years (FY) 2018 and FY 2020 (2021 survey) and between FY 2020 
and FY 2022 (2023 survey).
    The specific research objectives are to--
    1. Assess applicant and reviewer perceptions of, and satisfaction 
with, various aspects of the merit review process.
    2. Document the time burden the merit review process places on 
reviewers and applicants.
    3. Examine applicant and reviewer perceptions of the quality of 
reviews and of proposals.
    4. Assess the changes in applicant and reviewer perceptions of 
burden, satisfaction, and quality between the 2019 and 2021 surveys and 
the 2021 and 2023 surveys.
    5. Examine the variation of applicant and reviewer perception of 
satisfaction, burden, and quality by key population subgroups, 
including race/ethnicity, gender, and disability.
    6. Describe the extent to which NSF's reviewer orientation video is 
correlated with awareness of different types of cognitive biases and 
the use of strategies to reduce cognitive bias and to provide 
constructive feedback.
    7. Describe the extent to which the elimination of annual proposal 
deadlines affected reviewer and applicant burden, perceptions of 
proposal and review quality, and satisfaction with the merit review 
process.
    8. Describe applicants and reviewers experiences with student 
support programs as well as what NSF application and funding support is 
associated with the receipt of financial support from NSF as an 
undergraduate or graduate student.
    Data from the survey will be used to improve NSF's implementation 
of the merit review process.
    Use of the information: The primary purpose of collecting this 
information is program evaluation. The data collected will enable NSF 
to assess the satisfaction, including perceptions of burden and 
quality, of applicants and reviewers who participate in the merit 
review process in order to monitor and improve the program and assess 
its implementation. Findings will inform continual improvement 
activities related to the merit review process.

[[Page 8047]]

    Expected respondents: All applicants who have submitted proposals 
and reviewers who have reviewed NSF proposals between FY 2018 and 2020 
will be invited to participate in the 2021 survey and comparable 
individuals who participated between FY 2020 and FY 2022 will be 
invited to participate in the 2023 survey. This is estimated to be 
approximately 87,000 individuals per survey round.
    Average time per reporting: The online survey is comprised 
primarily of close-ended questions and is designed to be completed by 
respondents in approximately 20 minutes.
    Frequency: Eligible applicants and reviewers will be asked to the 
complete the 2021 Merit Review survey one time in fall 2021. For the 
2023 survey, eligible applicants and reviewers will be asked to 
complete the survey one time in fall 2023.
    Estimate of burden: It is estimated the survey will require 
approximately 20 minutes (on average) to complete. The anticipated 
universe size for each survey cycle is 87,000 individuals, which 
includes all applicants who submitted proposals and all reviewers 
between FY 2018 and FY 2020 (for the 2021 survey) and between FY 2020 
and FY 2022 (for the 2023 survey). The estimated survey response rate 
for each the 2021 and 2023 survey rounds is 40 percent. Therefore, the 
total burden is 23,200 hours; this is a respondent burden of 11,600 
hours per survey year (2021 and 2023).
    Based on 2019 merit review survey data, it is anticipated that most 
survey respondents will be working at an academic institution, likely 
in a teaching and/or research capacity. Therefore, for the purpose of 
burden estimates, we have used the annual mean wage for postsecondary 
teachers from Bureau of Labor Statistics, which is $79,540.\1\ Assuming 
a 40-hour workweek over the course of 52 weeks annually, the hourly 
wage for this occupation is approximately $38.00. Therefore, the 
overall cost to survey respondents for each survey year (2021 and 2023) 
would be approximately $440,800 (11,600 burden hours x $38.00 per 
hour), as shown in table A.12.1 below.
---------------------------------------------------------------------------

    \1\ Source: https://www.bls.gov/ooh/education-training-and-library/postsecondary-teachers.htm.

                                              Table A.12.1--Estimate of Respondent Burden and Cost by Year
--------------------------------------------------------------------------------------------------------------------------------------------------------
                                                                               Number of     Average burden
                           Year                                Number of     responses per    per response    Total burden     Average      Total cost
                                                              respondents     respondent        (hours)           hours      hourly wage
--------------------------------------------------------------------------------------------------------------------------------------------------------
2021......................................................          34,800               1          0.33333          11,600          $38        $440,800
2022......................................................               0               0                0               0            0               0
2023......................................................          34,800               1          0.33333          11,600           38         440,800
                                                           ---------------------------------------------------------------------------------------------
    Total.................................................          69,600               1          0.33333          23,200           38         881,600
--------------------------------------------------------------------------------------------------------------------------------------------------------

    Comments: Comments are invited on (a) whether the proposed 
collection of information is necessary for the proper performance of 
the functions of the Agency, including whether the information shall 
have practical utility; (b) the accuracy of the Agency's estimate of 
the burden of the proposed collection of information; (c) ways to 
enhance the quality, utility, and clarity of the information on 
respondents, including through the use of automated collection 
techniques or other forms of information technology; and (d) ways to 
minimize the burden of the collection of information on those who are 
to respond, including through the use of appropriate automated, 
electronic, mechanical, or other technological collection techniques or 
other forms of information technology.

    Dated: January 29, 2021.
Suzanne H. Plimpton,
Reports Clearance Officer, National Science Foundation.
[FR Doc. 2021-02240 Filed 2-2-21; 8:45 am]
BILLING CODE 7555-01-P