[Federal Register Volume 79, Number 6 (Thursday, January 9, 2014)]
[Notices]
[Pages 1628-1630]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 2014-00147]


=======================================================================
-----------------------------------------------------------------------

DEPARTMENT OF COMMERCE


Submission for OMB Review; Comment Request

    The Department of Commerce will submit to the Office of Management 
and Budget (OMB) for clearance the following proposal for collection of 
information under the provisions of the Paperwork Reduction Act (44 
U.S.C. chapter 35).
    Agency: U.S. Census Bureau.
    Title: Generic Clearance for Internet Nonprobability Panel 
Pretesting.
    Form Number(s): None.
    OMB Control Number: None.
    Type of Request: New collection.
    Burden Hours: 8,334.
    Number of Respondents: 50,000.
    Average Hours Per Response: 10 minutes.
    Needs and Uses: The U.S. Census Bureau is requesting a new OMB 
generic clearance to conduct a variety of medium-scale iterative 
Internet research pretesting activities. We will dedicate a block of 
hours to these activities for each of the next three years. OMB will be 
informed in writing of the purpose and scope of each of these 
activities, as well as the time frame and number of burden hours used. 
The number of hours used will not exceed the number set aside for this 
purpose.
    The Census Bureau is committed to conducting research in a cost 
efficient manner. Currently, several stages of testing occur in 
research projects at the Census Bureau. As a first stage of research, 
the Census Bureau pretests questions on surveys or censuses and 
evaluates the usability and ease of use of Web sites using a small 
number of subjects during focus groups, usability and cognitive 
testing. These projects are in-person and labor-intensive, but 
typically only target samples of 20 to 30 respondents. This small-scale 
work is done through an existing OMB generic clearance. Often the 
second stage is a larger-scale field test with a split-panel design of 
a survey or a release of a Census Bureau data dissemination product 
with a feedback mechanism. The field tests often involve a lot of 
preparatory work and often are limited in the number of panels tested 
due to

[[Page 1629]]

the cost considerations. They are often targeted at very large sample 
sizes with over 10,000 respondents per panel. These are typically done 
using stand-alone OMB clearances.
    Cost efficiencies can occur by testing some research questions in a 
medium-scale test, using a smaller number of participants than what we 
typically use in a field test, yet a larger and more diverse set of 
participants than who we recruit for cognitive and usability tests. 
Using Internet panel pretesting, we can answer some research questions 
more thoroughly than in the small-scale testing, but less expensively 
than in the large-scale field test. This clearance seeks to establish a 
medium-scale (defined as having sample sizes from 100-2000 per study), 
cost-efficient method of testing questions and contact strategies over 
the Internet through different types of nonprobability samples.
    For example, email has been identified as a possible cost-effective 
notification strategy for online data collection. Email has not been 
used extensively as a notification mode for past censuses nor other 
government surveys. (Please see ``Supporting literature'' section at 
the end of this section.) Prior to implementing an email strategy, the 
Census Bureau needs to determine the best email invitation to maximize 
the likelihood that someone will open the email and initiate the 
survey. Assessment of numerous email variations in a large-scale test 
would be cost-prohibitive. Medium-scale testing of email variations is 
more efficient. This research will be used to answer some fundamental 
questions about how to optimize email (and possibly text message) 
contacts.
    This research program will be used by the Census Bureau and survey 
sponsors to test alternative contact methods, including emails and text 
messages (via an opt-in strategy), improve online questionnaires and 
procedures, reduce respondent burden, and ultimately increase the 
quality of data collected in the Census Bureau censuses and surveys. We 
will use the clearance to conduct pretesting of decennial and 
demographic census and survey questionnaires prior to fielding them as 
well as communications and/or marketing strategies and data 
dissemination tools for the Census Bureau. The primary method of 
identifying measurement problems with the questionnaire or survey 
procedure is split panel tests. This will encompass both methodological 
and subject matter research questions that can be tested on a medium-
scale nonprobability panel.
    This research program will also be used by the Census Bureau for 
remote usability testing of electronic interfaces and to perform other 
qualitative analyses such as respondent debriefings. An advantage of 
using remote, medium-scale testing is that participants can test 
products at their convenience using their own equipment, as opposed to 
using Census Bureau-supplied computers. A diverse participant pool 
(geographically, demographically, or economically) is another 
advantage. Remote usability testing would use click through rates and 
other paradata, accuracy and satisfaction scores, and written 
qualitative comments to determine optimal interface designs and to 
obtain feedback from respondents.
    The public will be offered an opportunity to participate in this 
research remotely, by signing up for an online research panel. If a 
person opts in, the Census Bureau will occasionally email (or text, if 
applicable) the person an invitation to complete a survey for one of 
our research projects. Invited respondents will be told the topic of 
the survey, and how long it will take to complete it. Under this 
clearance, we will also conduct similar-scale and similarly designed 
research using other email lists to validate preliminary findings and 
expand the research.
    One of the testing methodologies to be used is Split sample 
experiments. This involves testing alternative versions of 
questionnaires, invitations to questionnaires (e.g., emails or text 
messages), or Web sites, at least some of which have been designed to 
address problems identified in draft versions or versions from previous 
waves. The use of multiple questionnaires, invitations, or Web sites, 
randomly assigned to permit statistical comparisons, is the critical 
component here; data collection will be via the Internet. Comparison of 
revised questionnaires (or invitations) against a control version, 
preferably, or against each other facilitates statistical evaluation of 
the performance of alternative versions of the questionnaire (or 
invitation or Web site).
    The number of versions tested and the number of cases per version 
will depend on the objectives of the test. We cannot specify with 
certainty a minimum panel size, although we would expect that no 
questionnaire versions would be administered to less than fifty 
respondents.
    Split sample tests that incorporate methodological questionnaire 
design experiments will have a larger maximum sample size (up to 
several hundred cases per panel) than other pretest methods. This will 
enable the detection of statistically significant differences, and 
facilitate methodological experiments that can extend questionnaire 
design knowledge more generally for use in a variety of Census Bureau 
data collection instruments.
    Another testing methodology is Usability Interviews. This method 
involves getting respondent input to aid in the development of 
automated questionnaires and Web sites and associated materials. The 
objective is to identify problems that keep respondents from completing 
automated questionnaires accurately and efficiently with minimal 
burden, or that prevent respondents from successfully navigating Web 
sites and finding the information they seek. Remote usability testing 
may be conducted under this clearance, whereby a user would receive an 
invitation to use a Web site or survey, then answer targeted questions 
about that experience.
    This clearance will only cover pretests primarily conducted 
remotely, via the Internet. Since the types of surveys included under 
the umbrella of the clearance are so varied, it is difficult to specify 
at this point what kinds of activities would be involved in any 
particular test, but a key component will be the comparison of one 
invitation, questionnaire or Web site to another.
    We will provide OMB with a copy of questionnaires and invitations 
in advance of any testing activity. Depending on the stage of 
development, this may be the printed material from the last round of a 
survey or a revised draft based on analysis of other evaluation data. 
For a test of alternative procedures, the description and rationale for 
the procedures would be submitted. We will also provide a description 
of the sample design and the planned administration. OMB will endeavor 
to provide comments on substantive issues within 10 working days of 
receipt.
    The Census Bureau will consult with the Economics and Statistics 
Administration (ESA) and OMB prior to submission on the appropriateness 
of submissions under this clearance that may raise policy or 
substantive issues. With respect to ESA, this will include all research 
and testing related to the American Community Survey (ACS) and the 2020 
decennial census. In addition, the Census Bureau will consult with ESA 
on any research and testing proposals that are presented to the Data 
Stewardship Executive Policy (DSEP) Committee. Consultation with ESA 
includes the Census Bureau providing copies of the materials to be 
tested in advance of any testing.

[[Page 1630]]

    The Census Bureau will send ESA and OMB an annual report at the end 
of each year summarizing the number of hours used, as well as the 
nature and results of the activities completed under this clearance.
    The information collected in this program of developing and testing 
questionnaires will be used by staff from the Census Bureau and 
sponsoring agencies to evaluate and improve the quality of the data in 
the surveys and censuses that are ultimately conducted. Because the 
questionnaires being tested under this clearance are still in the 
process of development, the data that result from these collections are 
not considered official statistics of the Census Bureau or other 
Federal agencies. Data will be included in research reports prepared 
for sponsors inside and outside of the Census Bureau. The results may 
also be prepared for presentations related to survey methodology at 
professional meetings or publications in professional journals.
    Affected Public: Individuals or households.
    Frequency: On occasion.
    Respondent's Obligation: Voluntary.
    Legal Authority: Data collection for this project is authorized 
under the authorizing legislation for the questionnaire being tested. 
This may be Title 13 U.S.C., Sections 131, 141, 161, 181, 182, 193, and 
301 for Census Bureau-sponsored surveys, and Title 13 and 15 for 
surveys sponsored by other Federal agencies. We do not now know what 
other titles will be referenced, since we do not know what survey 
questionnaires will be pretested during the course of the clearance.
    OMB Desk Officer: Brian Harris-Kojetin, (202) 395-7314.
    Copies of the above information collection proposal can be obtained 
by calling or writing Jennifer Jessup, Departmental Paperwork Clearance 
Officer, (202) 482-0336, Department of Commerce, Room 6616, 14th and 
Constitution Avenue NW., Washington, DC 20230 (or via the Internet at 
[email protected]).
    Written comments and recommendations for the proposed information 
collection should be sent within 30 days of publication of this notice 
to Brian Harris-Kojetin, OMB Desk Officer either by fax (202-395-7245) 
or email ([email protected]).

    Dated: January 6, 2014.
Glenna Mickelson,
Management Analyst, Office of the Chief Information Officer.
[FR Doc. 2014-00147 Filed 1-8-14; 8:45 am]
BILLING CODE 3510-07-P