[Federal Register Volume 80, Number 127 (Thursday, July 2, 2015)]
[Notices]
[Pages 38236-38238]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 2015-16369]


-----------------------------------------------------------------------

NATIONAL SCIENCE FOUNDATION


Agency Information Collection Activities: Comment Request

AGENCY: National Science Foundation.

ACTION: Submission for OMB review; comment request.

-----------------------------------------------------------------------

SUMMARY: The National Science Foundation (NSF) has submitted the 
following information collection requirement to OMB for review and 
clearance under the Paperwork Reduction Act of 1995, Public Law 104-13 
(44 U.S.C. 3501 et seq.). This is the second notice for public comment; 
the first was published in the Federal Register at 80 FR 10724 on 
February 27, 2015, and no comments were received. Comments regarding 
(a) whether the collection of information is necessary for the proper 
performance of the functions of the NSF, including whether the 
information will have practical utility; (b) the accuracy of the NSF's 
estimate of the burden of the proposed collection of information; (c) 
ways to enhance the quality, utility and clarity of the information to 
be collected, including through the use of automated collection 
techniques or other forms of information technology; (d) ways to 
minimize the burden of the collection of information on those who are 
to respond, including through the use of appropriate automated or other 
forms of information technology should be addressed to: Office of 
Information and Regulatory Affairs of OMB, Attention: Desk Officer for 
National Science Foundation, 725 7th Street NW., Room 10235, 
Washington, DC 20503, and to Suzanne H. Plimpton, Reports Clearance 
Officer, National Science Foundation, 4201 Wilson Boulevard, Suite 
1265, Arlington, Virginia 22230 or send email to [email protected]. 
Comments regarding these information collections are best assured of 
having their full effect if received within 30 days of this 
notification. Copies of the submission may be obtained by calling 703-
292-7556. Individuals who use a telecommunications device for the deaf 
(TDD) may call the Federal Information Relay Service (FIRS) at 1-800-
877-8339, which is accessible 24 hours a day, 7 days a week, 365 days a 
year (including federal holidays).
    NSF may not conduct or sponsor a collection of information unless 
the collection of information displays a currently valid OMB control 
number and the agency informs potential persons who are to respond to 
the collection of information that such persons are not required to 
respond to the collection of information unless it displays a currently 
valid OMB control number.

SUPPLEMENTARY INFORMATION: 
    Title of Collection: Generic Clearance of Survey Improvement 
Projects from the National Science Foundation.
    OMB Number: 3145--NEW.
    Type of Request: Intent to seek approval to establish a generic 
clearance for survey improvement projects for the National Science 
Foundation.
    Abstract:

[[Page 38237]]

Proposed Project

    The National Science Foundation (NSF) requests that the Office of 
Management and Budget (OMB) grant a generic clearance that will allow 
NSF to rigorously develop, test, and evaluate its survey instruments 
and methodologies. As part of the execution of its strategic plan, NSF 
has proposed several core strategies of which the following are related 
to eliciting information from entities outside of NSF ``Maintain 
extensive documentation, tracking, and public dissemination of 
performance indicators.'' and ``Develop, where appropriate, 
quantitative or evidence-based evaluation of outcomes.'' This request 
is part of an ongoing initiative to improve NSF surveys as a mechanism 
to develop appropriate high quality instruments to collect quantitative 
information for evidence-based decision-making and evaluation as 
recommended by both its own guidelines and those of OMB.\1\
---------------------------------------------------------------------------

    \1\ NSF Information Quality Guidelines are available on http://www.nsf.gov/policies/infoqual.jsp. OMB Information Quality 
Guidelines are available on http://www.whitehouse.gov/omb/inforeg/infopoltech.html. OMB standards and guidelines for statistical 
surveys are available on http://www.whitehouse.gov/omb/inforeg/statpolicy/standards_stat_surveys.pdf.
---------------------------------------------------------------------------

    In the last decade, state-of-the art data collection and analysis 
methods have been increasingly instituted by NSF and other federal 
agencies, and are now routinely used to improve the quality and 
timeliness of data and analyses. These new methods or techniques many 
times help reduce respondents' cognitive workload and burden. The 
purpose of this generic clearance is to allow NSF to continue to adopt 
and use these methods or techniques to improve its current data 
collections on science, engineering, and technology inputs, outputs and 
outcomes. They will be used to improve the content of existing surveys, 
to aid in the development of new data collections to capture the impact 
of NSF funding on the U.S. science and engineering (S&E) enterprise, 
and inform the existing NSF portfolio.
    Following standard OMB requirements, NSF will submit to OMB an 
individual request for each survey improvement project it undertakes 
under this generic clearance. NSF will request OMB approval in advance 
and provide OMB with a copy of the questionnaire (if one is used) and 
materials describing the project.
    NSF envisions using a variety of survey improvement techniques, as 
appropriate to the individual projects, such as focus groups, cognitive 
and usability laboratory and field techniques, exploratory interviews, 
behavior coding, respondent debriefing, pilot studies, pretests and 
split-panel tests. NSF has used such techniques in previous activities 
conducted under generic clearances granted to individual divisions.
    a. Focus Groups. A qualitative methodology that brings together a 
small number of relatively homogenous subjects to discuss pre-
identified topics. A protocol containing questions or topics focused on 
a particular issue or issues is used to guide these sessions, and is 
administered by a trained facilitator. Focus groups are useful for 
exploring and identifying issues with either respondents or 
stakeholders. Focus groups are a good choice during the development of 
a survey or survey topic, when a pre-existing questionnaire or survey 
questions on the topic do not yet exist.
    NSF has used focus groups for several projects under the Science 
Resources Statistics generic clearance (OMB Clearance Number 3145-0174) 
to assist with redesign of surveys when it became evident that the 
content of a survey was outdated and did not reflect current issues or 
the context that respondents were facing.
    2. Cognitive and Usability Laboratory and Field Techniques. A 
qualitative methodology that refers to a set of tools employed to study 
and identify errors that are introduced during the survey process. 
These techniques are generally conducted by a researcher with an 
individual respondent, though observers may sometimes be present. 
Cognitive techniques are generally used to understand the question-
response process, whereas usability is generally used to understand 
respondent reactions to the features of an electronic survey 
instrument, for instance, its display and navigation. In concurrent 
interviews, respondents are asked to think aloud as they actually 
answer the survey. In retrospective interviews, respondents answer the 
survey as they would normally, then `think aloud' afterwards. Other 
techniques, which are described in the literature and which will be 
employed as appropriate include: follow-up probing, memory cue tasks, 
paraphrasing, confidence rating, response latency measurements, free 
and dimensional sort classification tasks, and vignette 
classifications. The objective of all of these techniques is to aid in 
the development of surveys that work with respondents' thought 
processes, thus reducing response error and burden. These techniques 
are generally very useful for studying and revising a pre-existing 
questionnaire. NSF has used cognitive and usability testing in previous 
generic clearance projects (OMB Control Numbers 3145-0157 and 3145-
0174) to improve existing survey items, to develop and refine new 
content on existing surveys, and to explore content for new surveys.
    c. Exploratory Interviews. A technique where interviews are 
conducted with individuals to gather information about a topical area. 
These may be used in the very early stages of developing a new survey. 
They may cover discussions related to administrative records, subject 
matter, definitions, etc. Exploratory interviews may also be used to 
investigate whether there are sufficient issues related to an existing 
data collection to consider a redesign.
    NSF has used such interviews extensively in recordkeeping studies 
with respondents to several of its establishment surveys to determine 
both what types of records institutions keep (and therefore what types 
of information they can supply), as well as where and in what format 
such records are kept.
    3. Respondent Debriefing. A technique in which individuals are 
queried about how they have responded to a particular survey, question, 
or series of questions. The purpose of the debriefing is to determine 
if the original survey questions are understood as intended, to learn 
about respondents' form filling behavior and recordkeeping systems, or 
to elicit respondents' satisfaction with the survey. This information 
can then be used (especially if it is triangulated with other 
information) to improve the survey. This technique can be used as a 
qualitative or quantitative measurement, depending on how it is 
administered. This technique has been employed in NSF generic clearance 
projects (OMB Clearance Number 3145-0174) to identify potential 
problems with existing survey items both quantitatively (response 
behavior study, or RBS, using web survey questions with respondents to 
the Survey of Graduate Students and Post-doctorates in Science and 
Engineering, or GSS) and qualitatively (interviews using semi-
structured protocols with Higher Education R&D Survey respondents).
    4. Pilot Studies/Pretests. These methodologies are used to test a 
preliminary version of the data collection instrument, as was done with 
the Early Career Doctorate Project.
    Pretests are used to gather data to refine questionnaire items and 
scales and assess reliability, validity, or other survey measurement 
issues. Pilot studies are also used to test aspects of implementation 
procedures. The sample may be purposive in nature, or limited to 
particular groups for whom the

[[Page 38238]]

information is most needed. Alternatively, small samples can be 
selected to statistically represent at least some aspect of the survey 
population.
    5. Split Panel Tests. A technique for controlled experimental 
testing of alternatives. Thus, they allow one to choose from among 
competing questions, questionnaires, definitions, error messages, 
surveys, or survey improvement methodologies with greater confidence 
than other methods alone. Split panel tests conducted during the actual 
fielding of the survey are superior in that they support both internal 
validity (controlled comparisons of variables under investigation) and 
external validity (represent the population under study). Nearly any of 
the previously mentioned survey improvement methods can be strengthened 
when teamed with this method.
    6. Behavior Coding. A quantitative technique in which a standard 
set of codes is systematically applied to respondent/interviewer 
interactions in interviewer-administered surveys or respondent/
questionnaire interactions in self-administered surveys. Though this 
technique can quantifiably identify problems with the wording of 
questions, it does not necessarily illuminate the underlying causes.
    Use of the Information: The information obtained from these efforts 
will be used to develop new NSF surveys and improve current ones. These 
surveys will generally be used to monitor outputs and outcomes of NSF 
funding over time (particularly data that is not being collected in 
annual and final reports), and manage and improve programs. Data 
collected through survey questionnaires can be used in program 
evaluation studies and can be matched to administrative data to 
understand NSF's portfolio of investments. Specifically, the 
information from the survey questionnaire improvement projects will be 
used to reduce respondent burden and to improve the quality of the data 
collected in these surveys. These objectives are met when respondents 
are presented with plain, coherent, and unambiguous questionnaires 
asking for data compatible with respondents' memory and/or current 
reporting and recordkeeping practices. The purpose of the survey 
improvement projects will be to ensure that NSF surveys are 
continuously attempting to meet these standards of excellence.
    Improved NSF surveys will help policy makers make decisions on R&D 
funding, STEM education, scientific and technical workforce, 
innovation, as well as contribute to increased agency efficiency and 
reduced survey costs. In addition, methodological findings have broader 
implications for survey research and may be presented in technical 
papers at conferences or published in the proceedings of conferences or 
in journals.

Estimate of Burden

    NSF estimates that a total reporting burden of 171,000 hours over 
the three years of the requested generic clearance is possible from 
working to evaluate/improve existing surveys and to develop new ones. 
This includes both the burden placed on respondents participating in 
each activity as well as burden imposed on potential respondents during 
screening activities. Table 1 provides a list of potential improvement 
projects for which generic clearance activities might be conducted, 
along with estimates of the number of respondents and burden hours that 
might be involved in each.

                 Table 1--Potential Improvement Projects
------------------------------------------------------------------------
                                             Number of
        Improvement project type            respondents        Hours
                                                \2\
------------------------------------------------------------------------
Cognitive Testing.......................           5,000          15,000
Focus Groups............................           5,000          10,000
Card Sorting............................           5,000           5,000
Interviews..............................           5,000           5,000
Panelist Survey.........................           7,000          12,000
Past Awardee Survey.....................           9,000          14,000
Usability Testing.......................           5,000          10,000
Additional surveys not specified........          35,000         100,000
                                         -------------------------------
    Total...............................          76,000         171,000
------------------------------------------------------------------------

Respondents

    The respondents are PIs, program coordinators, or participants in 
NSF-funded activities.
---------------------------------------------------------------------------

    \2\ Number of respondents listed for any individual survey may 
represent several methodological improvement projects.
---------------------------------------------------------------------------

Estimates of Annualized Cost to Respondents for the Hour Burdens

    The cost to respondents generated by the list of potential projects 
is estimated to be $7,212,780 over the three years of the clearance. No 
one year's cost would exceed $7,212,780. In other words, if all work 
were done in one year, costs in that one year would be $7,212,780 and 
the costs in each of the other 2 years would be zero. As in previous 
requests for generic clearance authority, the total cost was estimated 
by summing all the hours that might be used on all projects over the 
three years (171,000) wage amount is the May 2011 national cross-
industry estimate of the mean hourly wage for a financial analyst, or 
Job Category 13-2051, by the Bureau of Statistics. http://www.bls.gov/oes/#data. The total hours are based on similar NSF projects over the 
past few years.
    There are no capital, startup, operation or maintenance costs to 
the respondents. The costs generated by future data collections will be 
described in the clearance request for each specific data collection. 
NSF does not anticipate any capital, startup, operation, or maintenance 
costs for future surveys.

    Dated: June 29, 2015.
Suzanne H. Plimpton,
Reports Clearance Officer, National Science Foundation.
[FR Doc. 2015-16369 Filed 7-1-15; 8:45 am]
 BILLING CODE 7555-01-P