Survey Methodology: An Innovative Technique for Estimating Sensitive
Survey Items (Staff Study, 11/01/1999, GAO/GGD-00-30).
GAO provided information on an innovative technique (called the
"three-card method") for collecting data on sensitive policy-relevant
topics.
GAO noted that: (1) the three-card method is designed to collect
sensitive data in large-scale surveys; it is intended to allow
estimation of the needed statistics while maximizing response privacy
and reducing "question threat"; (2) GAO originally devised this
technique to collect data on immigration, but it believes that the
technique might also prove useful in a variety of other sensitive policy
areas where the collection of relevant information has, thus far, proved
elusive; (3) a scientific survey involves a representative sample of the
population of interest; (4) GAO's technique extends this approach to
select three independent representative samples, each composed of
completely different persons; (5) all persons are asked the same
potentially sensitive question; (6) all are presented with answer
alternatives printed on an 8 1/2 by 11 card, using a design that
arranges the answer alternatives in different boxes on the card; (7)
this arrangement avoids zeroing in on the sensitive answer category; (8)
the logic of the technique involves: (a) a three-box answer format,
which is used on each card; (b) three slightly different cards (cards 1,
2, and 3), one for each sample; and (c) indirect estimation of the
sensitive category; (9) no respondent is ever directly asked whether he
or she is in the sensitive category; (10) each sample provides a
different piece of less sensitive information--a different piece of the
puzzle; and (11) the outlines of the missing sensitive piece are
apparent for the population as a whole when all other pieces are in
place.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: GGD-00-30
TITLE: Survey Methodology: An Innovative Technique for Estimating
Sensitive Survey Items
DATE: 11/01/1999
SUBJECT: Statistical data
Data integrity
Labor statistics
Statistical methods
Surveys
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** <[email protected]> **
** **
** with the message 'info' in the body. **
******************************************************************
United States General Accounting Office
GAO
Staff Study
November 1999
GAO/GGD-00-30
SURVEY METHODOLOGY
An Innovative Technique for Estimating Sensitive
Survey Items
Ordering Information
The first copy of each GAO report and testimony
is free. Additional copies are $2 each. Orders
should be sent to the following address,
accompanied by a check or money order made out
to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are
accepted, also. Orders for 100 or more copies to
be mailed to a single address are discounted 25
percent.
Order by mail:
U.S. General Accounting Office
P.O. Box 37050
Washington, DC 20013
or visit:
Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC
Orders may also be placed by calling (202) 512-
6000 or by using fax number (202) 512-6061, or
TDD (202) 512-2537.
Each day, GAO issues a list of newly available
reports and testimony. To receive facsimile
copies of the daily list or any list from the
past 30 days, please call (202) 512-6000 using a
touch-tone phone. A recorded menu will provide
information on how to obtain these lists.
For information on how to access GAO reports on
the INTERNET, send e-mail message with "info" in
the body to:
[email protected]
or visit GAO's World Wide Web Home Page at:
http://www.gao.gov
United States
General Accounting Office
Washington, D.C. 20548-0001
Official Business
Penalty for Private Use $300
Address Correction Requested
Bulk Rate
Postage & Fees Paid
GAO
Permit No. G100
(966710)
Contents
Page 2GAO/GGD-00-30 Estimating Sensitive Survey Items
Preface 1
Overview 5
Background 5
The Logic of the Three-Card Method 7
The Three-Box Answer Format 7
Cards 1, 2, and 3 Yield Direct 8
Estimates for Less Sensitive
Categories
The Sensitive Category 9
Detailed Information 10
Introductory Material 10
Preliminary Testing on Legal Status 10
Instrument Development 11
Sensitivity and Willingness to Answer 11
Other Potential Applications 11
Using the Method in a Large-Scale 12
Survey
Organization of This Study 13
Chapter 1 14
The Three-Card Method:
Its Logic and Key
Features
Answer Format, Samples, and Cards 14
Logic of the Basic Three-Card Method 15
Follow-up Questions 21
Estimating Length of Stay 22
Tracking Trends Across Time 23
Using "Trainer" Cards Is Key 23
Separately Develop Each Legal Status 27
Card
Chapter 2 28
Phases of Development
and Testing
Survey Instruments 28
Methodology for Phase 1: Initial 29
Pretests
Methodology for Phase 2: Field Test 30
With Farmworkers
Methodology for Phase 3: Cognitive 35
Interviews
Limitations of Results 37
Chapter 3 38
Results of Preliminary
Development and Testing
Results on the Three-Box Answer Format 39
Results on the Appropriateness of 40
Icons
Results on Legal Status Categories 41
Results on Sensitivity of the Legal 46
Status Question
Chapter 4 48
Estimating Other
Sensitive Items With the
Three-Card Method
Violence 48
Sensitive Personal Choices 49
Business and Tax Issues 50
Government Workers' Noncompliance 51
Chapter 5 54
Challenges and Potential
Solutions in Using the
Three-Card Method in a
Large-Scale Survey
Interviews Using Flashcards 54
Screening for the Relevant 55
Subpopulation
Efficient Estimation 56
Minimizing the Bias of Nonresponse 58
Issues in Surveying Special 59
Populations
Appendixes Appendix I: Variance of the Estimates 62
Appendix II: Different Forms of the 65
Questionnaire Used in the Field Test
Appendix III: Estimates for 79
Demographic Subgroups
Appendix IV: Data From Respondents Who 80
Participated in the Cognitive
Interviews
Bibliography 81
Tables Table II.1: Variations in Wording of 65
Survey Questions in the Nine
Questionnaires We Field-Tested
Figures Figure 1: Illustration of Card 1 for 8
Estimating a Less Sensitive Category
Using the Three-Card Method
Figure 2: Illustration of Cards 2 and 9
3 for Estimating Less Sensitive
Categories Using the Three-Card
Method
Figure 1.1: Legal Status Card 1 16
Figure 1.2: Legal Status Card 2 18
Figure 1.3: Legal Status Card 3 20
Figure 1.4: Introductory Flashcard 24
With Types of Houses
Figure 1.5: Introductory Flashcard 26
With Types of Transportation
Figure 2.1: Introductory Card With 32
Household Appliances
Figure 2.2: Introductory Card With 33
Modes of Transportation
Figure 2.3: Early Version of Legal 34
Status Card
Figure 3.1: Legal Status Card Used in 43
the Cognitive Interviews
Figure I.1: Statistical Expression of 64
the Three-Card Estimator, Its
Variance, and Technique Effect
Figure II.1: Form 1 of the 66
Questionnaire
Abbreviations
ACASI Audiocomputer-assisted self-
interviewing
EWI Entered without inspection
INS Immigration and Naturalization Service
NAWS National Agricultural Workers Survey
OMB Office of Management and Budget
Preface
Page 1GAO/GGD-00-30 Estimating Sensitive Survey It
ems
This staff study provides information on an
innovative technique for collecting data on
sensitive policy-relevant topics.1 If successful,
this technique might eventually help fill key data
gaps and improve statistical information relevant
to the national decision-making process. Our main
goal is to stimulate interest in-and where
appropriate, encourage further development and
testing of-this promising, but as yet not fully
validated, technique.
The "three-card method" is designed to collect
sensitive data in large-scale surveys; it is
intended to allow estimation of the needed
statistics while maximizing response privacy and
reducing "question threat."2 We originally devised
this technique to collect data on immigration, but
we believe it might also prove useful in a variety
of other sensitive policy areas where the
collection of relevant information has, thus far,
proved elusive. We encourage readers to consider
whether developing a new application of the three-
card method might prove worthwhile.
For questions about the material contained in this
staff study, please contact me at (202) 512-4128
or Judith A. Droitcour, who served as project
director, at (202) 512-7997. Other key
contributors to this assignment were Eric M.
Larson, Ruth B. McKay, and Maria P. Vargas.
Susan S. Westin
Associate Director
International Relations and Trade Issues
_______________________________
1GAO staff studies are typically prepared to
present background information and intended to
contribute to a specific body of knowledge.
2"Question threat" refers to questions the
respondent may perceive as threatening or
incriminating. In such cases, the respondent may
not answer the question or may provide distorted
answers (Rossi et al., 1983; and Bradburn and
Sudman, 1979).
Overview
Page 6GAO/GGD-00-30 Estimating Sensitive Survey It
ems
In some areas of public policy, there has been
repeated evidence that statistics to inform key
debates are lacking or inadequate. When a
sensitive topic area is at issue, the reason for
the information gap may simply be that relevant
questions are deemed too threatening to ask in
large-scale surveys.
The purpose of this staff study is to report on an
innovative questionnaire survey technique: the
"three-card method." Though not yet fully
validated, the new technique may prove to be
applicable across a range of policy-relevant
topics.
Background
We originally devised the three-card method as a
way to survey foreign-born respondents about their
immigration status. As summarized below, we were
motivated by data gaps that are highly relevant to
immigration policy. During the early to mid-1990s,
immigration laws were debated and changed.1 More
recently, proposals for revisions have continued
to be put forward.2 But as the U.S. Commission on
Immigration Reform recognized in 1994, without
reliable data, it is difficult to assess the
impact of immigration policies-or of immigration
itself-on American society.
Our 1998 report on the quality of immigration
statistics (GAO/GGD-98-164, July 1998) found that
to be relevant to current laws and debates,
information on foreign-born residents of the
United States should be broken out by legal
status. Five key legal statuses3 are
� legal permanent residents (persons with
official green cards);
� refugees and asylees (persons granted
asylum);
� persons admitted temporarily who stay
(legally) for more than a year;
� illegal immigrants who remain for more than a
year;4 and
� naturalized U.S. citizens.
The same 1998 report found that neither records
from the Immigration and Naturalization Service
(INS) nor data from current surveys provide
reliable estimates of these legal status groups.
Naturalized citizens (only) can be identified in
some current surveys, although even here we found
there may be some problems (GAO/GGD-98-164, July
1998).5
Surveys of the general population do provide
information on foreign-born persons-but not by
legal status. The reason is that questions on
legal status have been deemed too threatening to
ask.6 As a result, policy researchers cannot track
trends in employment for legal immigrants, illegal
aliens, or persons in other immigration statuses.
Similar gaps occur for other important outcomes.7
Immigration status is also important for policy-
related studies of immigrant health.8 As a first
step toward filling these gaps, we devised a new
survey technique that was described briefly in our
1998 report.9
This staff study reviews the logic of the three-
card method and describes key procedures of the
technique. It also provides more complete
information on our development and testing efforts
than was previously reported, indicates
possibilities for future applications of the three-
card method in a variety of sensitive areas, and
discusses challenges in fielding a national survey
using this innovative technique.
The Logic of the Three-Card Method
A scientific survey involves a representative
sample of the population of interest. Our
technique extends this approach to select three
independent representative samples, each composed
of completely different persons.
All persons are asked the same potentially
sensitive question. All are presented with answer
alternatives printed on an 8-1/2" by 11" card,
using a design that arranges the answer
alternatives in different boxes on the card. As
explained below, this avoids zeroing in on the
sensitive answer category. The logic of the
technique involves
� a three-box answer format, which is used on
each card;
� three slightly different cards (cards 1, 2,
and 3)-one for each sample; and
� indirect estimation of the sensitive
category.
No respondent is ever directly asked whether he or
she is in the sensitive category. Each sample
provides a different piece of less sensitive
information-a different piece of the puzzle. The
outlines of the missing sensitive piece are
apparent for the population as a whole when all
other pieces are in place.
The Three-Box Answer Format
The key to asking respondents a potentially
sensitive question, without zeroing in on the
sensitive answer category, is the three-box
format. For example, figure 1 shows a card that
has three boxes: Box A, Box B, and Box C. Box A
contains one of the less sensitive answer
categories. Box B combines the sensitive category
with a number of other less sensitive categories.
Box C is all other categories-that is, any answer
category not covered in Box A or Box B. Each box
is an answer alternative.
Using the example of immigration status, the boxes
might be
� Box A: Legal permanent resident with a valid
and official green card issued to me by the U.S.
government;
� Box B: U.S. citizen; student, work, or
tourist visa; undocumented10 (do not have my own
valid official green card); and refugee or asylee
(without a green card); and
� Box C: Some other category not in Box A or
Box B (specify).
Respondents are asked to report which Box applies
to them. They are told that if it is Box B, we do
not want to know which specific category applies
to them.
Cards 1, 2, and 3 Yield Direct Estimates for Less
Sensitive Categories
Respondents in sample 1 are shown a card like that
in figure 1 (card 1). The purpose of gathering
information with card 1 is to estimate the percent
of persons in the first "less sensitive" category.
Cards 2 and 3 are illustrated in figure 2. These
cards are for samples 2 and 3, respectively.
Comparison of the three cards shown in figures 1
and 2 indicates that the less sensitive categories
are rotated between Box A and Box B. When a
category appears in Box A, direct estimation is
possible. Thus, the percentages of the population
in each less sensitive (Box A) category are
estimated directly by separate samples. The
sensitive category always remains in Box B,
together with other less sensitive categories. The
sensitive category is not estimated directly.
Figure 1: Illustration of Card 1 for Estimating a
Less Sensitive Category Using the Three-Card
Method
Source: GAO. (The actual size of the card is 8-
1/2" by 11.")
Figure 2: Illustration of Cards 2 and 3 for
Estimating Less Sensitive Categories Using the
Three-Card Method
Source: GAO. (The actual size of each card is 8-
1/2" by 11.")
The Sensitive Category
Although the sensitive category always appears
together with other less sensitive categories in
Box B, an indirect (and unbiased) estimate of the
sensitive category can be obtained by putting
together the various pieces of less sensitive
information. The size of the missing piece is
calculated by subtraction. If the categories
listed in Box A, B, and C are mutually exclusive
and, taken together, are exhaustive, they should
total 100 percent. Subtracting the percentage
estimates of the less sensitive categories from
100 percent yields a remainder that represents an
indirect estimate of the percentage in the
sensitive category. (Note: This procedure is
appropriate only when the sensitive item is not
too rare. Attempting to obtain an indirect
estimate of a rare category would be like trying
to find a "needle in a haystack.")
Detailed Information
Detailed information on Box A categories is
obtained with follow-up questions for only the
respondents who choose Box A. Because the
sensitive category is never asked about directly,
there are no follow-up questions for those who
choose Box B. (The purpose of the technique is to
avoid zeroing in the sensitive category.)
If respondents are asked other policy-relevant
questions (e.g., questions about employment and
income), then correlates for each less sensitive
category may be obtained directly. The logic of
estimating correlates of the sensitive category
(indirectly) is explained in appendix III.
Briefly, separate indirect estimates can be
obtained for major (broad) subgroups defined, for
example, by income category.
Introductory Material
We used introductory or training cards to
familiarize respondents with the three-box answer
format when we applied the technique to legal
status. These training cards dealt with relatively
innocuous topics. They were intended to cue
respondents to the fact that the interviewer would
not zero in on any specific Box B category. (See
ch. 1 for examples of training cards.)
The early portions of the interview-even before
presenting respondents with the training cards-can
focus on obtaining various kinds of general and
potentially policy-relevant information, such as
household size, age, employment, income, and so
forth.
Preliminary Testing on Legal Status
We conducted a series of development and testing
efforts in which the three-card method was used to
ask foreign-born respondents about their legal
status. These efforts consisted of three main
phases, each of which focused on foreign-born
Hispanic respondents: pretests (27 respondents),11
a field test with 81 farmworkers,12 and cognitive
interviews (8 respondents).13 We were only able to
test in one foreign language, and we selected
Spanish because it is the most prevalent foreign
language in the United States. Respondents were
selected from groups, such as farmworkers and
those seeking help from legal clinics, that are
thought to overrepresent illegal immigrants.
Instrument Development
There were iterative revisions of the survey
instruments based on each set of results. The
cards featured icons to illustrate the categories
(e.g., a picture of a green card was used to
illustrate the legal permanent resident category.)
However, we found that some field test respondents
and interviewers reacted negatively to the early
version of the icons; that is, field test
respondents perceived at least some of the icons
as childish or indicated that some seemed
ambiguous.14 One of the main reasons for conducting
the subsequent cognitive interviews was to improve
the icons. By the conclusion of the cognitive
interviews, respondents viewed each card as
appropriate for adults, and most thought each icon
was recognizable without seeing its label.
Sensitivity and Willingness to Answer
All 116 respondents in the three phases of testing
answered the legal status question using a three-
box card; there were no refusals. Overall, two-
thirds of respondents chose Box B, which contained
the sensitive category. In the cognitive
interviewing phase only, respondents were
specifically asked about the sensitivity of the
legal status question. The majority of these
respondents (five of eight) thought the legal
status card would be regarded as sensitive by some
persons, even when using the three-box format;
however, all eight respondents chose Box B.
It is important to note that a validity study,
conducted under conditions similar to that of a
large-scale survey, would be needed to determine
whether-or to what extent-respondents actually
chose the appropriate boxes.
Other Potential Applications
We believe that the three-card method might be
applied to a variety of sensitive subject areas
other than immigration. To judge whether the
technique might be appropriate for a specific
question area, one should consider three
requirements:
Potential answers to the key question must be
organized in a set of mutually exclusive and
exhaustive categories. (For some applications,
drafting an appropriate set of categories might
require some creativity.)
Only one of the answer categories can be
sensitive.
Most important, the sensitive item should not be
rare within the population surveyed. Some items
that appear to be rare at first glance may not be
rare within certain definable populations. Thus,
in some cases, one must first define a population
or group at risk, then screen for that population,
and finally survey its members.15
With these considerations in mind, we brainstormed
potential applications ranging from violence
(e.g., road rage among young male drivers) and
sensitive personal choices (abortion, drug use) to
organizational actions (such as the ways
manufacturers in key industries dispose of
hazardous waste). Chapter 4 discusses these and
other examples-providing possible approaches to
wording sensitive and less sensitive categories.16
Using the Method in a Large-Scale Survey
Assuming that a specific application of the three-
card method is deemed appropriate for use in a
large-scale survey, there would be a number of
challenges. Some of these involve efficiency. For
example, the cards rule out the option of
telephone interviews, and fielding a large-scale
personal interview survey would be costly. Low-
cost options that might be feasible in some
instances include (1) "piggyback" insertion of
items in an ongoing in-person survey; (2) group
administration (e.g., a survey of high school
students); and (3) an Internet survey, in which
the cards are shown on the computer screen.17
Other challenges include finding cost-effective
ways to screen for the relevant subpopulation and
keeping the margin of error at an acceptably low
level (see ch. 5). Still other challenges may
occur, depending on the context of the particular
application. A number of special challenges
pertain when planning a survey of the foreign-born
population. For example, our 1998 report discussed
the issue of possible undercoverage of the foreign-
born in censuses and surveys and suggested ways of
estimating the foreign-born undercount.18
Organization of This Study
The logic and procedures of the three-card method
are discussed in detail in chapter 1, using the
example of legal status for foreign-born
respondents. The methods we used for the
preliminary development and testing and the
results we obtained are presented in chapters 2
and 3. Examples of potential applications in a
variety of sensitive topic areas are explored in
chapter 4. Lastly, in chapter 5, challenges in
fielding the three-card method in a national
survey and potential solutions are discussed.
Appendixes I through IV provide details on a
variety of relevant technical topics. These
include estimating the variance, procedures for
estimating sensitive characteristics for
demographic subgroups, the questionnaire used in
the field test, and detailed data from the
cognitive interviews. Lastly, we have included a
bibliography listing the major sources we used.
_______________________________
1Major legislation includes the Immigration Act of
1990 (P.L. 101-649), Violent Crime Control and Law
Enforcement Act of 1994 (P.L. 103-322), Personal
Responsibility and Work Opportunity Reconciliation
Act of 1996 (P.L. 104-193), and the Illegal
Immigration Reform and Immigrant Responsibility
Act of 1996 (P.L. 104-208).
2Bills introduced in the 106th Congress that would
amend current immigration law include H.R. 1399,
Fairness for Legal Immigrants Act of 1999; S.
1227, Immigrant Children's Health Improvement Act
of 1999; H.R. 2698, New Workers for Economic
Growth Act; and S. 455, Nursing Relief for
Disadvantaged Areas Act of 1999. Other proposals
have involved rules for the admission of temporary
workers, including temporary agricultural workers
(see GAO/HEHS-98-20, Dec. 1997) and temporary
skilled workers (see Hearing before the House
Subcommittee on Immigration and Claims, Committee
on the Judiciary, Aug. 5, 1999).
3Virtually all laws regarding foreign-born persons
apply to these five major legal statuses (or to
subcategories of these statuses). Legal categories
of foreign-born persons are defined under the
Immigration and Nationality Act of 1952, as
amended (8 U.S.C. 1101 et seq.). The terms and
length of their admission to the United States,
entitlements, rights, and benefits are addressed
in the act and other legislation.
4Illegal immigrants include persons who entered
without inspection (EWI)-that is, entered the
United States surreptitiously-and overstays.
Overstays (also called nonimmigrant overstays and
visa overstays) are persons who entered the United
States legally for a temporary period, but
illegally stayed beyond the required departure
date.
5INS has provided some composite estimates of the
size of certain other legal status groups, but
these estimates rely on major assumptions and may
not be statistically reliable.
6For other problems with census and survey data on
the foreign-born, including undercount and
nonresponse, see GAO/GGD-98-164, July 1998.
7Other important outcomes that follow legal and
illegal immigrants' entry into the United States
include educational attainments, poverty status,
and family formation.
8The implementation of the Personal Responsibility
and Work Opportunity Reconciliation Act of 1996
(P.L. 104-193) and the Health Insurance
Portability and Accountability Act of 1996 (P.L.
104-191) requires the reassessment of national
survey questions that obtain information on
citizenship status, as well as welfare programs
and health insurance coverage (Loue and Bunde,
1999).
9Earlier indirect survey-based estimation
techniques, including "randomized response"
(Warner, 1965) and the "item count" (Droitcour et
al., 1991), seemed inappropriate for the foreign-
born population or for asking about legal status.
The three-card method builds on these earlier
survey techniques as well as demographic methods
of residual estimation (see Schryock and Siegel
and Associates, 1980).
10Undocumented refers to illegal immigrants. This
is the sensitive category.
11By "pretesting" we mean trying out questions on a
relatively small number of respondents and making
corrections and revisions on the basis of problems
that surface in those interviews. Pretests include
(1) asking the interview questions and (2) asking
respondents about problems they had in
understanding the questions or in answering them.
12A field test is a small-scale preliminary survey,
typically conducted under conditions as similar as
possible to those to be used in an eventual full-
scale survey.
13Cognitive interviewing focuses on the mental
processes of the respondent while he or she is
answering survey questions. (See ch. 2 for a
fuller discussion of this method.)
14Icons are drawings we used to illustrate the
categories in Box A and Box B, as shown in ch. 1,
figs. 1.1-1.3. The U.S. citizen category, e.g., is
accompanied by a drawing of a U.S. flag.
15Otherwise, the indirect-estimation effort could
amount to "looking for a needle in a haystack."
(See app. I for a discussion of the variance of
the indirect estimate.)
16When choosing these examples, we tried to keep
all three considerations in mind. However, it was
not always clear to us how prevalent or rare a
particular sensitive behavior might be within a
defined population.
17Some surveys are already conducted entirely on
the Internet; e.g., we used e-mail to administer a
questionnaire to some 350 EPA cleanup managers
responsible for the 609 sites where cleanups are
still under way (see GAO/RECD-99-245, July 1999).
We also reported the results of the survey for
each site on the Internet at
http//www.gao.gov/RCED-99-245/.
18See GAO/GGD-98-164, July 1998, pp. 57-58.
Chapter 1
The Three-Card Method: Its Logic and Key Features
Page 15GAO/GGD-00-30 Estimating Sensitive Survey I
tems
The three-card method is designed to encourage
more truthful responses to sensitive questions
while protecting respondent privacy. In its first
application, the technique was used to ask foreign-
born respondents about their legal status in such
a way that a truthful response would not reveal
whether the respondent (or anyone else) is here
illegally.1
No one could ever discover, on the basis of these
survey data-not even through a series of
deductions-that any respondent is an illegal
immigrant. But when all data are combined, survey-
based estimates can be achieved for all legal
status groups-including the sensitive illegal
immigrant category. The logic of the three-card
method involves
� a three-box answer format,
� selection of three random samples of foreign-
born persons, and
� three different cards-one for each sample.
Key features of the method include
� the ability to ask some respondents detailed
follow-up questions to estimate length of stay and
to obtain trend data;
� introductory "trainer" cards, which
familiarize respondents with the three-box answer
format and with the use of icons to identify
specific categories; and
� the inclusion of different details on
different legal status cards, to maximize
effective and efficient communication.
Answer Format, Samples, and Cards
The "three-box answer format" groups various
answer categories in three boxes (Box A, Box B,
and Box C). The respondent answers by picking one
of the boxes. Categories are arranged so that the
sensitive category (here, illegal or
"undocumented" status) appears in a box that
includes several less sensitive categories (such
as refugee or asylee, naturalized citizen, etc.).2
Icons are used to help identify the various legal
status categories.
Each of the three samples consists of different
foreign-born persons. The three legal status cards
differ only in terms of which categories are
grouped in which box.3 Each card is shown to
respondents in one sample only. Thus, each
respondent sees only one legal status card.
The key to the new method is that respondents in
each sample provide partial information (a
different "piece of the puzzle"). By accessing
data from all three samples (all three legal
status cards), direct estimates can be made for
most categories. The sensitive category is
estimated indirectly, utilizing data from all
three samples. The following section describes how
this is accomplished.
Logic of the Basic Three-Card Method
The logic of the basic three-card method can be
explained using a hypothetical example that is
carried through to explain each of three samples.
Respondents in the first sample of foreign-born
respondents are shown a legal status card (card 1)
with categories arranged in a three-box format, as
shown in figure 1.1.4 The card is shown in Spanish
because all testing to date has been conducted in
Spanish. The English translation of card 1 is
� Box A: Legal permanent resident with a valid
and official green card issued to me by the U.S.
government;
� Box B: U.S. citizen; student, work, or
tourist visa; undocumented5 (do not have my own
valid official green card); and refugee or asylee
(without a green card); and
� Box C: Some other category not in Box A or
Box B (specify).
Respondents are asked to report which box applies
to them and are told that if it is Box B, we do
not want to know which specific category applies
to them. The purpose of interviews with this first
sample is to obtain a valid estimate of the
percent of foreign-born who have officially
obtained green cards. (Hypothetical example: 35
percent of the resident foreign-born population
have officially obtained green cards.)
Figure 1.1: Legal Status Card 1
Source: GAO. (The actual size of the card is 8-
1/2" by 11.")
Respondents in the second sample (completely
different individuals) are shown a legal status
card (card 2) with a different arrangement of the
categories in the same three-box format (see fig.
1.2). This time, Box A contains the status of a
naturalized U.S. citizen, whereas Box B includes
legal permanent resident along with other
immigration statuses. Respondents are asked to
pick the box that applies to them and are told
that if it is Box B, we do not want to know which
category in Box B applies to them. The purpose of
interviewing this sample with this card is to
obtain a valid estimate of the percentage of the
foreign-born who are naturalized citizens.
(Hypothetical example: 30 percent are naturalized
citizens.)
Figure 1.2: Legal Status Card 2
Source: GAO. (The actual size of card is 8-1/2" by
11.")
Respondents in the third sample (again, different
persons) are shown a legal status card (card 3)
with yet another arrangement of the categories in
the three-box format (see fig. 1.3). This time,
Box A features refugees and persons granted asylum
as well as those here legally with temporary
visas.6 Respondents are told that if they are in
Box B, we do not want to know which category in
Box B applies to them. The purpose of interviewing
this sample is to get an estimate of the
percentage in Box A categories-refugees, asylees,
and persons here on a legal temporary basis.
(Hypothetical example: 10 percent pick Box A.)
Figure 1.3: Legal Status Card 3
Source: GAO. (The actual size of card is 8-1/2" by
11.")
Thus, each of the legal status categories-except
for the sensitive category (illegal
status)-alternatively appears in Box A. Assuming
that the estimates for the legal status categories
are mutually exclusive and that these categories
together with Box C represent an exhaustive set of
possible legal statuses, it is possible to obtain
an indirect estimate of illegal immigrants. That
is, extending the hypothetical examples above, we
would estimate that 75 percent of the foreign-born
are here legally in the four major statuses (35%
with green card + 30% naturalized U.S. citizens +
10% refugees, asylees, and persons here on a legal
temporary basis7 = 75%). Suppose also that 1
percent picked Box C (some other category).
Subtracting these hypothetical estimates from 100
percent yields 24 percent (100% - 75% - 1% = 24%).
Thus, our estimate would be that 24 percent are
here illegally.
The three-card method also allows detailed
information to be gathered on subcategories of
legal statuses. As described in the following
section, this is accomplished through a special
strategy for follow-up questions that is designed
to avoid increasing question threat.8
Follow-up Questions
Respondents who choose Box B on the legal status
card shown to them are not asked any further
questions because Box B always contains the
sensitive category. However, respondents who
choose Box A can be asked a set of follow-up
questions about the details of their (Box A) legal
status.9
Follow-up questions are essential for sample 3
respondents who choose box A, because these
respondents are shown legal status card 3, which
includes two categories in Box A. Follow-up
questions can determine which of the two legal
status categories applies to the respondent.
Follow-up questions might also help confirm the
validity of answers for respondents choosing Box A
or, alternatively, provide the information needed
to reclassify certain respondents into Box B or
Box C. For example, suppose a person in sample 1
chooses Box A, apparently claiming to have a green
card. If follow-up questions reveal that although
the respondent had applied for green card status,
he or she has not actually received that status
yet, then this sample 1 respondent would be
recoded out of Box A. Specific follow-up questions
used for legal status card 1 are shown in appendix
II.10
For sample 3 (legal status card 3), follow-up
questions could help clarify whether a respondent
chose Box A incorrectly.11
Estimating Length of Stay
We defined two of the five major legal status
categories of foreign-born persons, in part, by
length of stay.12 The reason is that they reflect
temporary admissions or illegal presence in the
United States, and these conditions are different
from an admission that is intended to be
permanent, such as for legal permanent residence.
The two length-of-stay statuses are (1) persons
who were legally admitted here on a temporary
basis who remain (legally) longer than 1 year and
(2) persons here illegally who remain longer than
1 year.
We based the foregoing length-of-stay criterion on
the U.N.-recommended definition of "permanent"
immigration and on the Immigration and
Naturalization Service's (INS) practices regarding
its overstay estimates.13
Estimates of foreign-born persons here for any
time period can be made by isolating the data for
foreign-born respondents who have been here for a
specific length of time, and then using these data
to derive the relevant estimate. This is
especially important for two categories-illegals
and persons here on temporary visas.14
Estimates for demographic subgroups can be
obtained in a similar manner (see app. III).
Tracking Trends Across Time
Through repeatedly applying the three-card method
in subsequent surveys, it would be possible to
track trends across time for various legal status
groups. For example, trends in the number of
foreign-born persons here on temporary visas-or
the number illegally residing here-could be
tracked for all such persons or separately for
males and for females or for other major
demographic groups (e.g., Mexican origin). To
illustrate, the analyst would first isolate data
on Mexican-born respondents, then follow the
estimation and subtraction process previously
described.
Using "Trainer" Cards Is Key
Introductory or "trainer" cards are a key feature
of the three-card method. Before showing a
respondent the legal status card assigned to his
or her sample, the interviewer asks questions on
less sensitive topics using cards that also have
words, icons, and boxes. The purpose of these
introductory trainer cards is to familiarize
respondents with the general format-including the
task of answering by choosing a box-before they
are shown the legal status card. Figures 1.4 and
1.5 show introductory cards, each of which has a
three-box format and icons to help identify each
category.
The first introductory card (fig. 1.4) shows
different categories of houses or dwellings. Box A
shows a farmhouse, whereas Box B includes various
categories-apartments, single-family houses, and
so forth. Box C refers to other types of housing
(not shown in Box A or Box B). This card is used
with a question such as: "In what kind of house do
you think most of the people in your home country
live?" Before respondents actually voice an
answer, they are instructed to answer by just
picking a box-and are also told that if the answer
is Box B, we do not want to know which specific
category applies to them.
Figure 1.4: Introductory Flashcard With Types of
Houses
Source: GAO. (The actual size of card is 8-1/2" by
11.")
The second introductory card (fig. 1.5) also has
three boxes. It presents various modes of
transportation: Box A contains a boat; Box B
contains four other modes of transportation-plane,
train, auto, or on foot; and Box C refers to any
type of transportation not shown in either Box A
or Box B. This card is used together with
questions such as: "What kind of transportation
did you use the most recent time you traveled from
your home country to the United States?" Or, "If
you were to travel to your home country sometime
within the next 12 months, what kind of
transportation do you think you would use?" Again,
before the respondent answers, he or she is asked
to just pick a box and is told that if the answer
is in Box B, we do not want to know which specific
category applies.
Figure 1.5: Introductory Flashcard With Types of
Transportation
Source: GAO. (The actual size of card is 8-1/2" by
11.")
We recommend using both of these introductory
cards so that respondents will be thoroughly
familiar with the three-box format before they are
shown a legal status card. The logic is that
respondents can freely ask questions about how the
format works on the more innocuous introductory
trainer cards and will know, in advance of seeing
the legal status card, that they will not have to
identify a specific category in Box B.
Separately Develop Each Legal Status Card
Each legal status card must be separately
developed and tested before being used in a
survey. It may be necessary to develop specific
instructions for each card to enable a better
presentation by the interviewer and to facilitate
more accurate answers by the respondent. In such a
case, we believe only those detailed instructions
needed for a particular card should be included on
that card. The reason is that instructions
relevant to other cards would clutter the card and
make it more difficult to take in at a glance.
Because Box A of each card features a different
category (or categories), the cards differ in
terms of the key instructions and explanatory
details needed to help respondents determine
whether they belong in Box A, Box B, or Box C. It
is not necessary for respondents to distinguish
between categories listed within Box B.
For example, a respondent with an expired visa who
is shown legal status card 1 should know that he
or she belongs in Box B. The reason is that on
legal status card 1, Box B contains both the
temporary visa category and the undocumented
category. The respondent need only choose Box B;
he or she does not have to distinguish between
categories within Box B. But a similar respondent
in sample 3 would be shown legal status card 3,
which includes the temporary visa category in Box
A and the undocumented category in Box B. A
respondent with an expired visa might be unsure
which box to choose-unless legal status card 3 has
language making it clear that persons with expired
visas belong in Box B.
It is important to note that all three legal
status cards are shown in here to illustrate the
logic of the three-card method. However, only one
of those cards-legal status card 1-was subjected
to development and testing.15
_______________________________
1This extends our earlier work on the three-card
method, which was introduced in our report on the
quality of immigration statistics available to
policymakers (GAO/GGD-98-164, Jul. 1998). Our
other work in this area includes evaluating the
quality of data available to evaluate the effects
of immigration (GAO/PEMD-89-8, Dec. 1988),
projecting future legal immigration (GAO/PEMD-88-
7, Jan. 1988; GAO/T-PEMD-89-1, Mar. 1989; GAO/PEMD-
89-12, Apr. 1989; GAO/PEMD-90-5, Nov. 1989),
nonimmigrant alien workers in the United States
(GAO/PEMD-92-17, Apr. 1992), and problems with
INS' overstay estimation methods (GAO/PEMD-95-20,
Sept. 1995).
2Each sample is shown only one of these three-box
legal status cards. The categories on the cards
must be mutually exclusive and exhaustive (i.e.,
each person belongs in one and only one category).
3Each of the three legal status cards must be
individually developed and pretested, as explained
later in this chapter.
4The legal status card shown in fig.1.1 is a
revised version we developed after the 1998 report
was released. The various development and testing
efforts completed to date are discussed in ch. 2.
5Undocumented refers to illegal immigrants. This
is the sensitive category.
6Using two categories in Box A lowers variance
costs because three, rather than four, samples
would be required. Those who pick Box A would be
asked follow-up questions to determine their exact
legal status, which is also the case with the
other two legal status cards.
7Refugees, asylees, and persons here on a legal
temporary basis could be further disaggregated by
using follow-up questions to determine their
status.
8"Question threat" refers to questions the
respondent may perceive as threatening or
incriminating. In such cases, the respondent may
not answer the question or may provide distorted
answers (Rossi et al., 1983; and Bradburn and
Sudman, 1979).
9This could include, for example, a question about
when the Box A legal status was officially
obtained (e.g., green card status) and the basis
upon which the respondent applied for the green
card (e.g., as the spouse of a citizen, relative
of a permanent resident alien, and so forth). In
this way, detailed information on legal status can
be attained for most legal status subcategories.
10However, as described in ch. 3, in a field test
with 81 respondents, no one was reclassified on
the basis of these follow-up questions.
11For example, some persons who entered as refugees
may still self-identify generally as refugees even
if they later obtained green cards or became
naturalized U.S. citizens. Such errors could, in
some cases, then be recoded to the correct box
(B).
12The other three are legal permanent residents,
refugees and asylees, and naturalized U.S.
citizens. The term "asylees" refers to persons
granted asylum, without legal permanent resident
status.
13The U.N. definition (Shryock and Siegel and
Associates, 1980), has been applied to INS data on
arrivals of foreign-born persons to the United
States (Kraly and Warren, 1992); also, INS'
definition of overstays counts only those who
resided illegally in the United States for more
than 1 year.
14Some foreign-born workers who are here on
temporary visas are currently of particular policy
interest, in part because of their length of stay.
(See GAO/PEMD-92-17, Apr. 1992, and "Hearing on
the H-1B Temporary Professional Worker Visa
Program and Information Technology Workforce
Issues," hearing before the Committee on the
Judiciary, Subcommittee on Immigration and Claims,
House of Representatives, Aug. 5, 1999.)
15As described in the following chapter, the
developmental testing consisted of three phases:
initial pretesting by our staff who are fluent in
Spanish, a field test, and formal cognitive
interviewing. Results are presented in ch. 3.
Chapter 2
Phases of Development and Testing
Page 29GAO/GGD-00-30 Estimating Sensitive Survey I
tems
Our preliminary development and testing effort was
conducted in three phases:1 (1) initial pretesting
of instruments, (2) a field test with farmworkers,
and (3) subsequent cognitive interviews.2 This
three-phase effort covered two introductory cards
as well as legal status card 1. The third phase
was designed, in large part, to address problems
that surfaced in the field test.
Each phase built on the results of the previous
phase. That is, on the basis of phase 1 results,
instruments were revised before beginning phase 2;
and on the basis of phase 2, instruments were
revised at the outset of phase 3.3 Revisions were
also made during phase 3.
In each phase, foreign-born Hispanic respondents
were selected in such a way that illegal residents
would likely be heavily represented. Generally,
Hispanics in professional and semiprofessional
occupations were not included, and it is likely
that long-term Hispanic residents were
underrepresented. We did not attempt to interview
non-Hispanic foreign-born residents (Asian,
African, or European).
The end of this chapter describes important
limitations on the results of this effort.
Survey Instruments
Throughout the development and testing of the
three-card method, the following survey
instruments were used:
� introductory trainer cards intended to
familiarize respondents with icons and train them
in the use of the three-box answer format before
exposing them to a legal status card and
� legal status card 1, in which Box A features
the legal permanent resident category.4
All respondents also answered a lead-in
questionnaire that obtained demographic
information and provided context for the specific
questions and cards we were testing. Additionally,
the lead-in questionnaire may help establish
rapport and motivate respondents to provide
accurate answers on the legal status card.5
Respondents who picked Box A on legal status card
1 (i.e., those who claimed legal permanent
resident or green card status) were also asked
follow-up questions concerning details of their
legal status.6
At various points during our work on the three-
card method, we asked experts to review legal
status card 1. These included GAO colleagues;
officials and staff at the Immigration and
Naturalization Service (INS), the Bureau of the
Census, and the Office of Refugee Resettlement at
the Department of Health and Human Services; and
private-sector experts in immigration, statistics,
and survey methods.7 Revisions were made, as
needed.
All survey materials were available in Spanish.
All interviewers were fluent in both Spanish and
English, and almost all interviews were conducted
in Spanish.8
Methodology for Phase 1: Initial Pretests
Phase 1 of the development and testing consisted
of a series of pretests and iterative revision of
survey instruments. Four bilingual GAO staff
members conducted 27 pretest interviews with
foreign-born Hispanics at four locations. These
included a Los Angeles "drop-in" center frequented
by young Hispanic males and operated by a
charitable Catholic group; a legal aid clinic in
Arlington, Virginia, that specializes in helping
Hispanic immigrants; and various facilities in
farm areas in Colorado and Pennsylvania during the
harvest of labor-intensive crops, such as apples
and peaches. Most of the pretest respondents were
from Mexico, and most were males.
At each location, we first met with and explained
our project to directors or owners of the facility
we wished to enter. Subsequently, we introduced
ourselves to prospective respondents (foreign-born
Hispanics) and gained their cooperation. The
support of the directors or owners was critical
for gaining access to the facilities, and in some
cases, but not all, the directors or owners also
served as "opinion leaders," who validated our
activities to prospective respondents.
Methodology for Phase 2: Field Test With
Farmworkers
Phase 2, following the initial pretests, consisted
of a field test in which a contractor (Aguirre
International) conducted survey interviews with 81
farmworkers in six states. As a first step, the
questionnaires and cards we developed were
submitted to the contractor for review. The
contractor suggested minor changes, and we revised
the questionnaires as needed.9 The interviewers
employed by the contractor were subsequently
debriefed by our staff.
The contractor conducted our 81 interviews as an
add-on to the National Agricultural Workers Survey
(NAWS).10 The NAWS is an annual survey, and the
Aguirre interviewers are experienced in developing
rapport and communicating with Hispanic
farmworkers.
We chose the NAWS population because it contains a
high percentage of foreign-born workers, almost
all foreign-born respondents are Hispanic and
speak Spanish, and a high percentage are believed
to be illegal immigrants. Specifically, our
expectation was that most foreign-born farmworkers
would either be working here illegally or would
have obtained, as a result of the amnesty period
authorized by the Immigration Reform and Control
Act of 1986, a valid green card.11 The majority of
farmworkers were male, and nearly all were from
Mexico.
The cards used to interview 81 Hispanic
farmworkers were as follows:
� The first introductory card featured
household appliances (see fig. 2.1) and was
intended to familiarize respondents with the use
of cards showing words and pictures.12
� The second introductory card featured types
of transportation shown in a three-box format
(fig. 2.2, which is the same as the card shown in
ch. 1, fig. 1.5).
� Legal status card 1 featured the legal
permanent resident category in Box A-specifically,
the early version of that card (see fig. 2.3).13
Figure 2.1: Introductory Card With Household
Appliances
Source: GAO. (The actual size of card is 8-1/2" by
11.")
Figure 2.2: Introductory Card With Modes of
Transportation
Source: GAO. (The actual size of card is 8-1/2" by
11.")
Figure 2.3: Early Version of Legal Status Card
Source: GAO. (The actual size of card is 8-1/2" by
11.")
The contractor field-tested these cards and
related questions from November 1997 to February
1998.14 The 81 interviews by four interviewers were
conducted at multiple sites in six states
(Arizona, California, Florida, Georgia, Kentucky,
and Texas). All respondents were interviewed in
Spanish. The final portion of the field test
consisted of our debriefing the Aguirre
interviewers.
The reaction of some respondents to specific icons
suggested there was a need for focus groups or
other qualitative work aimed at evaluating and
possibly revising the icons. For example,
interviewers reported that some respondents
thought the icons were childish and unnecessary,
and even that the card was unnecessary because
they had understood the verbal instructions and
felt that they did not need help.
Methodology for Phase 3: Cognitive Interviews
Phase 3 consisted of two sets of formal cognitive
interviews with small samples of respondents.
Cognitive interviewing is a method that focuses on
the mental processes of the respondent while he or
she is answering the survey questions.15 The
cognitive model asks
� What does the respondent think the question
is asking?
� What do specific words and phrases in the
question mean to the respondent?
For the three-card research, the model also asks
� What do the icons in the survey material mean
to the respondent?
A variety of methods are used in cognitive
interviews. The research protocol we developed
included scripted retrospective probes (questions)
and a vignette. In the former, the respondent is
asked standardized questions about the difficulty
and sensitivity of survey items in a debriefing
session following completion of the formal
questionnaire. The vignette is designed to provide
information about the respondent's decision-making
process in making a judgment about a hypothetical
situation.
The respondent is asked several scripted questions
during a debriefing at the conclusion of the
survey questionnaire. Three questions were
developed for each icon, as follows. First, the
interviewer asked: "Do you think that anyone would
have difficulty recognizing the (first, second,
etc.) category in (Box A, B)?" Next, the
interviewer covered the written label for each
icon on the card and asked, "What does this icon
mean to you?"16 Lastly, if the icon did not convey
the intended meaning, the interviewer asked, "How
can we change the icon to make the intended
meaning more clear?"
Cognitive interview research is an iterative
process in which the findings on problems
identified in each set of interviews are used to
modify the questions to be tested in the next set
of interviews. Because each research interview
provides a great deal of information about the
mental processes utilized by the respondent, a
relatively small number of cognitive interviews is
required in this type of research.
One of our pretest interviewers, who is bilingual
in English and Spanish, conducted the cognitive
interviews after training by another of our staff,
who is a cognitive interviewing expert with
knowledge of Spanish and who also participated in
the actual interviews. The cognitive interviews
were conducted in two sets: The first four
respondents were adult males from El Salvador,
Bolivia, Mexico, and the Dominican Republic.17
Following these interviews, materials were
revised. The second set of four cognitive
interviews, using the revised survey materials,
was conducted with four additional respondents in
the same location. Two of these respondents, one
female and one male, were Salvadoran; the other
two, one female and one male, were from Peru and
Nicaragua, respectively.
All of these interviews were conducted at an
employment and training facility sponsored by a
local, church-based organization (Hispanic
Outreach Center). The center has a reputation in
the community as being supportive of immigrants,
offering legal and material assistance (e.g.,
food) to local residents.
Limitations of Results
The results of this work are, by definition,
limited to the Hispanic groups we interviewed.18 We
also note that it is particularly difficult to
generalize about respondent perceptions of
question sensitivity, as such perceptions
logically depend on many variables. (These would
include the setting of the interview, the level of
trust inspired by the particular interviewer, the
social or political climate prevailing at the time
a survey is conducted, and possibly other
factors.) Finally, levels of respondent
comprehension, acceptance, and sensitivity all
depend on the specific instruments used, and it is
possible that future developmental work can yield
improvement. Moreover, results may vary depending
on whether the method is used to estimate legal
status among foreign-born persons or whether it is
used to estimate a different sensitive item in a
different population.
For all these reasons, as stated in our 1998
report (GAO/GGD-98-164), additional development
and testing would be needed before using the three-
card method in a large-scale national survey.
_______________________________
1The development and testing was conducted between
July 1997 and September 1998: initial pretesting
of instruments (Jul.-Aug., 1997), application in a
field test with farmworkers (Nov. 1997-Feb. 1998),
and cognitive interviewing (Sept. 1998).
2Cognitive interviewing involves examining mental
processes of respondents as they answer key
questions; i.e., questioning them about what the
question meant to them and about how they arrived
at their answer.
3Problems clearly observable to the interviewers,
given the normal question asking-and-answering
process, were identified in the field test.
4We did not test the alternative legal status
cards-i.e., legal status cards 2 and 3, which show
different categories in Box A. (See figs. 1.2 and
1.3 in ch. 1.)
5In the field test with farmworkers, the lead-in
questionnaire covered work-related health issues,
such as injuries at work, use of pesticides in the
fields, drinking water, etc.
6A copy of the three-card segment of the
questionnaire used in the field test is shown in
app. II.
7Our nonagency reviewers included included Norman
Bradburn, Senior Vice President for Research,
National Opinion Research Center, Chicago, IL;
Robert Groves, Director, Joint Program for Survey
Methodology, University of Maryland and University
of Michigan; Professor David M. Heer, Population
Research Laboratory, University of Southern
California, Los Angeles, CA; Graham Kalton, Senior
Vice President, Westat, Inc., Rockville, MD;
Charles B. Keely, Chairman, Department of
Demography, Georgetown University, Washington,
D.C.; Mary Grace Kovar, National Opinion Research
Center, Washington, D.C.; Demetrios G.
Papademetriou, Senior Associate and Director,
International Migration Policy Program, Carnegie
Endowment for International Peace, Washington,
D.C.; Jeffrey S. Passel, The Urban Institute,
Washington, D.C.; Donald B. Rubin, Chairman,
Department of Statistics, Harvard University,
Cambridge, MA; Fritz Scheuren, The Urban
Institute, Washington, D.C.; Michael S.
Teitelbaum, Co-Chair, U.S. Commission on
Immigration Reform; and Professor Alan Zaslavsky,
Department of Health Care Policy, Harvard Medical
School, Cambridge, MA.
8One respondent requested to be interviewed in
English; although Spanish was his native language,
he wanted to practice his English.
9Nine forms of this questionnaire were developed
(see app. II), so that each could be administered
to nine respondents.
10Data from our interviews were not included as
part of the regular NAWS survey data.
11The Immigration Reform and Control Act was passed
Nov. 6, 1986, in order to control and deter
illegal immigration to the United States. Its
major provisions stipulate legalization of
undocumented aliens, legalization of certain
agricultural workers, sanctions for employers who
knowingly hire undocumented workers, and increased
enforcement at U.S. borders.
12We eliminated this card following the field test.
13We modified this card in the cognitive testing
phase, which addressed problems with the icons
that were encountered by the contractor during the
field test. The most recent version is shown in
ch. 1.
14Nine slightly different forms of the three-card
question series were tested. The contractor tested
each form using nine different respondents-for a
total of 81 interviews. Consistent with the NAWS
procedures, farmworker respondents were paid $10
cash as an incentive to participate.
15The source materials upon which we based our work
include DeMaio, et al., 1993; Polivka and Rothgeb,
1993; and Willis, n.d.
16The need for icons that convey the conceptual
categories without written labels is because not
all Hispanic adults are literate in Spanish.
Results of the March 1995 Current Population
Survey showed that about 11 percent of Hispanics
age 25 or older had less than a 5th grade
education.
17App. IV contains a chart that includes
demographic characteristics of the Hispanic
respondents interviewed at the suburban Maryland
Employment Center between Sept. 1 and 3, 1998.
18As noted at the outset of this chapter, we did
not attempt to cover Asian, African, or European
immigrants. Moreover, the Hispanics we interviewed
generally did not include professionals or
semiprofessionals, and they probably
overrepresented new arrivals and undocumented
workers.
Chapter 3
Results of Preliminary Development and Testing
Page 46GAO/GGD-00-30 Estimating Sensitive Survey I
tems
As explained in the previous chapter, our
development and testing effort consisted of three
forms of research-pretests, a field test with
farmworkers, and subsequent cognitive interviewing
to correct problems identified in the field test.
We recognize that these forms of research cannot
indicate whether respondents provide accurate
answers. A validity study would be needed for such
an assessment, and none has been conducted.1
However, preliminary development and testing has
provided some relevant information.
Logically, the three-card method's potential to
elicit accurate responses on legal status depends
on whether respondents
� comprehend and accept the three-box format
for answering a question;
� generally view the icons as appropriate
(because we believe the icons are essential aids
to some respondents' comprehension);
� comprehend the various legal status
categories, as represented by words and icons; and
� perceive the three-box question on legal
status as sensitive-and if so, whether they are
nevertheless willing to answer the question and,
indeed, choose a box containing the sensitive
item.2
Our preliminary development and testing has
provided some information on each of these four
points.
As detailed below, most respondents appeared to
comprehend the three-box format for answering; and
they also appeared to accept this answer format at
face value. Respondents found the final version of
the icons to be appropriate for adults. With
respect to comprehension of the legal status
categories and related icons, the final version of
the instruments seemed to communicate
effectively-although some further improvement of
specific icons may be possible.
Over the three phases of research, perceptions of
the sensitivity of the legal status question were
mixed, but no respondent refused to answer it, and
two-thirds picked Box B (which includes the
sensitive item). In the cognitive interviewing
phase, five of eight respondents identified the
legal status question as sensitive but all picked
Box B. Based on these results, we believe that
further development and testing efforts are
justified.
Results on the Three-Box Answer Format
In the pretests and the field test with
farmworkers, the three-box format was used for the
second introductory card (modes of transportation)
and for legal status card 1. In the third phase,
the three-box format was used for both
introductory cards and for legal status card 1.
In the initial pretests, most respondents appeared
to understand the three-box answer format. But two
needed more specific instructions. One respondent,
in particular, alerted us to the need for improved
instructions: When using the transportation card,
she did not seem to understand the logic of the
boxes. We therefore revised the instructions for
the three-box transportation card to more
specifically describe the logic of the three-box
format and to emphasize to the respondent that we
were interested only in which box he or she was
in-not any specific category. We also added
assurances, just before asking about legal status
card 1, that our purpose was not to identify any
individual person who may be an illegal immigrant.
Subsequently, in another pretest location, two
respondents, who were administered the
questionnaire together, appeared to understand the
logic of the three-box cards, but falsely claimed
to be in Box A of legal status card 1. Box A
contains the green card category, and in
debriefing, both respondents admitted to being
illegal aliens. They explained that they had
picked Box A because they wanted to have a green
card. We therefore revised the instructions for
the legal status card by adding the following
statement: "We are only interested in which BOX
you are in right now-and not a box you want to be
in, or have applied to be in, in the future.
So . . . are you actually in Box A, B, or C right
now?" The revised wording also emphasized (even
more strongly than before) that we did not intend
to identify any particular category in Box B.3
In the field test, no problems surfaced with
respect to comprehension of the instructions for
the three-box answer format (phase 2). That is,
interviewers did not report problems of this type
in the debriefing sessions.
In the cognitive interviewing (phase 3), the first
introductory card (showing types of houses) used a
three-box format, and most respondents answered by
pointing to a specific type of house within Box B.
However, by the third card (on legal status), one
respondent appeared not to understand the task; he
pointed to the specific undocumented category,
instead of just indicating Box B. But before being
asked about his legal status, this respondent had
volunteered to the interviewer that he was
undocumented.4 Thus, there is some question as to
whether he misunderstood the task or not. It may
be that some further testing or revision of the
instructions is needed to ensure that every
respondent fully comprehends the task.
Interviewers in the initial pretests did not
report any respondent questions about why three
boxes were being used. Similarly, in the in-depth
debriefing sessions following the farmworker field
test, interviewers did not report having to
provide special explanations about why the three-
box format was used.5 Because respondents did not
raise questions about the use of the three-box
answer format, they did not appear to have a
problem with it. Therefore, specific questions on
acceptance of the three-box format were not
included in the cognitive interviewing. However,
future cognitive work might be designed to learn
how respondents perceive the three-box format.
Results on the Appropriateness of Icons
The initial pretests did not indicate problems
with the appropriateness of the icons. However,
the field test with farmworkers produced mixed
results. Some interviewers had positive comments
about the icons,6 but others reported that some
respondents viewed the icons as childish and
unnecessary-and even that the cards were
unnecessary because the respondents felt they had
understood the verbal instructions and thus did
not need help.
Some field-test interviewers indicated that the
major problem seemed to be the first introductory
card, which featured appliances. (The appliance
card is a simple card that was intended to
familiarize respondents with icons and cards
before introducing the three-box format. The
appliance card is shown as fig. 2.1. in ch. 2.)
To improve the appropriateness of the icons, we
decided to change the subject matter and design of
the first introductory card. In preparation for
the first set of cognitive interviews, we
developed a new introductory card to replace the
appliance card. The new card showed different
housing categories; the housing categories and
icons were grouped into three boxes. (The final
version of the new introductory card featuring
types of homes is shown in fig. 1.4 of ch. 1.) The
respondent was asked to indicate which box
contained the type of house he or she lived in at
age five.7
Changing the first introductory card meant that
now both introductory cards would be training
cards, in which respondents could practice
answering by selecting a box, rather than by
indicating individual items or categories.8 The
transportation card was retained in the same form
as in the field test (see fig. 1.5 in ch. 1). The
icons on the legal status card were revised as
described in the following sections.
In our debriefing we asked about icons for each
card, taken as a whole, as follows: "Do you think
that the icons on this card are appropriate for
adults?" All considered the icons appropriate for
use with adults.
Results on Legal Status Categories
From the outset, icons were used to aid
respondents' comprehension of legal status
categories. In phase 3, to further enhance
comprehension of these categories, we added a more
detailed introduction to legal status card 1, in
which the interviewer gave a verbal summary of the
written descriptions of the various immigration
categories represented by the icons. This was to
ensure that respondents with limited Spanish
literacy would be provided with the information
written under the various immigration categories.
The Green Card Category: Box A, Legal Status Card
1
Box A of legal status card 1 contains the category
"legal permanent resident" with an icon consisting
of a picture of a green card. The respondent's
crucial task in picking a box is determining
whether or not he or she belongs in Box A.
Problems in the wording of Box A cropped up in the
field test and were addressed in the cognitive
interviews.
Specifically, in the field test with farmworkers,
most respondents did not appear to be confused
about which box to choose, but a few respondents
were uncertain-at least initially. These
respondents told interviewers that they had fake
green cards or had borrowed a green card from
someone-or had "border-crossing cards"9-and that
they were unsure whether or not Box A might apply
to them.10 (Fig. 3.1 shows the version of legal
status card 1 that was used in the cognitive
interviews. This is the same card that was used in
the field test.)
Interviewers had been trained to help respondents,
when necessary, by restating or paraphrasing
questions and providing explanations. With help
from interviewers, all 81 field-test respondents
were able to pick a box. No respondent was coded
"not sure." In fact, all chose either Box A or Box
B. None were coded Box C. Nevertheless, it was
unclear whether every respondent who selected Box
A was indicating that he or she actually possessed
a green card issued in his or her name.
Figure 3.1: Legal Status Card Used in the
Cognitive Interviews
Source: GAO. (The actual size of card is 8-1/2" by
11.")
Rewording the Green Card Item
In the field test instruments, the green card
category in Box A was labeled (translating from
the Spanish that appeared on the card): "Legal
foreign resident with a certificate of permanent
immigration (also known as the green card)
obtained officially." This definition allowed some
ambiguity, and therefore, before conducting the
cognitive interviews, we reworded it as follows
(again translating from the Spanish that appeared
on the card): "Legal foreign resident with a valid
and official card (also called green card) issued
for this person by the U. S. Government." The
cognitive interviewing then tested whether this
version communicated effectively.
Testing the Reworded Item
Specifically, the cognitive interview included a
vignette designed to provide information on the
decision-making process used by respondents for
including an individual in the revised green card
category in Box A. Respondents were asked their
opinion about whether a person who possessed an
official green card that he or she had borrowed
from a friend or relative belonged in that
category.
In the first set of interviews, the wording of the
one-sentence vignette proved to be awkward to read
and difficult for the respondents to understand.
But after the interviewer paraphrased the
vignette, each of the respondents understood and
answered the vignette without difficulty. All four
stated that an immigrant who possessed a borrowed
official green card did not belong in the legal
permanent resident category in Box A.
With respect to the icon for this category
(picture of a green card), one respondent
suggested enlarging it somewhat for greater
visibility. We made this change. We also reworded
the vignette for the second set of cognitive
interviews.
The revised vignette was readily understood. As
was found in the first day's interviews, all four
respondents stated that an immigrant with a
borrowed official green card did not belong in the
more explicitly defined Box A green card category.
Other Categories: Box B on Legal Status Card 1
The farmworker field test yielded criticisms of
some of the legal status icons in Box B. More
importantly, some field-test respondents told
interviewers that the icon for refugees (which
shows a person running toward the United States)
appeared to them to represent an undocumented
worker running away from U.S. Border Patrol or
other law enforcement officers. The icon for
student, worker, or tourist visa-a suitcase with
various stickers-was also viewed as problematic.
Additionally, one of the interviewers recommended
replacing the icon for the undocumented category
with a picture of a green card with a diagonal
line through the card.
We changed each of these Box B icons as part of
the preparation for cognitive interviewing. To
better represent the idea of someone fleeing
danger by coming to the United States, we added a
symbol for an explosion behind the running figure
and had the figure's arms stretch forward with his
hands touching the map of the United States. We
also developed a new icon for the undocumented
category using the same green card image as
appears in Box A, but drawing a circle around it
and a diagonal line through it. In addition, the
original icon showing a suitcase with stickers was
replaced by the plainer suitcase icon that is
routinely used in transportation terminals.
In the first set of cognitive interviews, the new
icon for a suitcase was identified as such by the
respondents, as was the new icon for the
undocumented category. But only one of the four
recognized the refugee or asylee category from the
icon alone. Two respondents perceived the figure
either as a refugee being pushed away by the map
of the United States or holding the map up-or as a
terrorist.11 One respondent recommended that the
arms of the fleeing figure in the refugee or
asylee icon extend into the map of the United
States, thereby dispelling the perception that the
figure was either holding up the map, or was a
terrorist being forced away from the country. One
respondent did not interpret the drawing as
showing an explosion.
Before the next set of cognitive interviews, we
made a number of changes to the cards. The size of
the green card was increased in the two icons
using this image. The fleeing figure in the
refugee or asylee icon was changed to two fleeing
figures, thereby increasing the suggestion of many
persons fleeing to safety. The drawing of the
explosion was enlarged and the word "Boom" added
to the middle of the image. This style of drawing
is used in fotonovelas, vividly illustrated
pictorial storybooks similar to U.S. comic books,
which are popular among Hispanic adults.
In the second set of cognitive interviews, all
four respondents recognized the concept of refugee
or asylee from the icon alone.
Cognitive debriefing revealed that, although the
respondents understood the concepts symbolized by
the flag (citizen) and suitcase (temporary visa
status), there was nothing inherent in the flag or
the suitcase to represent citizenship or temporary
visa status. While it is true that interviewers
can explain the icons for respondents who still
experience some uncertainty, further cognitive
work might improve the recognizability of the
icons without the interviewer's help.
Results on Sensitivity of the Legal Status
Question
None of the 116 foreign-born Hispanics interviewed
across all three phases, refused to answer when
using the three-box format, and 81 (about 70
percent) chose the box (Box B) that included a
sensitive category. However, a number of
respondents perceived the question as sensitive.
In the field test with 81 farmworkers, 30 picked
Box A-that is, claimed to have a valid green card.12
The remaining 51 respondents (63 percent) selected
Box B, which contained the undocumented worker
category. The population of foreign-born
farmworkers is thought to contain few naturalized
citizens, few refugees, and very few workers here
with temporary work visas.13 Thus, we believe that
most of the farmworkers who selected Box B were
probably working without legal authorization.
There were apparent differences in how sensitive
the legal-status question appeared to various
respondents.14 Although the field test did not
include specific questions on sensitivity,
interviewers reported their impressions, and these
showed considerable variation, as follows:
� At some locations, there were instances where
respondents voiced hostility, seemed to identify
the survey with INS, or initially mistook the
interviewer for a tax collector. Notably, in
Florida, where the interviewer's impression was
that all respondents were illegal, respondents
reportedly appeared fearful of INS and nervous
about the survey, and some respondents at first
hesitated to select a box. Nevertheless, all
Florida respondents picked Box B.
� At other locations, respondents appeared to
be relaxed, and some even volunteered to the
interviewer that they were here illegally. In
Arizona, California, and Kentucky, interviewers
reported that respondents did not appear to find
the question on legal status intrusive.
By contrast to the field test, the cognitive
interviews specifically asked each respondent
about the sensitivity of the legal status
question. That is, a two-part question on the
sensitivity of each three-box card was asked at
the end of the cognitive interview, as follows:
"Do you think that anyone would find the question
for this flashcard to be sensitive? If yes, why is
that?"
In the first set of cognitive interviews, two of
the four respondents stated that the question on
immigration status was not sensitive because the
question was asked in an indirect manner. The
other two indicated that some would find the
question sensitive. One said this was because
legal status is confidential information, and some
people would not want to be asked about it. The
other respondent said that people do not want to
tell the truth about their legal status. (Yet this
respondent had volunteered that he was
"undocumented" early in the interview.) All four
respondents in this set of interviews chose Box B,
which includes the sensitive item.
In the second set of interviews, one of the four
respondents did not find the question about
immigration status sensitive; the other three did
report sensitivity associated with this question.
One respondent said that people would be fearful
of being turned in to the INS. Another said the
question is sensitive because the majority of
foreigners he knows are undocumented, and they
would not want to answer this question. A third
respondent reported that it is not a bad question,
but there are some people who would find it
sensitive.15 Again, however, all four respondents
chose Box B.
_______________________________
1One possibility for a validity study would be to
survey the clients of legal aid clinics that
specialize in immigration law, with the
cooperation of clinic administrators. The
administrators (who are bound by confidentiality)
could determine whether each client had chosen the
correct box. To safeguard client privacy, the
clinic administrators would disclose the
percentage of clients who incorrectly chose Box
A-and also the percentage who incorrectly chose
Box B (if any)-but no specific individual results.
2Sensitivity is also of interest from a survey
planning point of view. For example, sensitive
questions would probably not be "piggybacked" onto
a general purpose survey.
3Also, on six of the nine forms of the field-test
instrument, we added a follow-up question that
provided respondents who chose Box A on the legal
status flashcard with an opportunity to change
their answers, asking them to "Please recheck Box
B carefully, because some people pick Box A by
mistake" (see app. II).
4If this respondent did not understand, he may not
be typical because his first language was neither
Spanish nor English, but was a South American
Indian language, and the interviewer believed that
this respondent had not been paying careful
attention.
5The debriefing sessions did not include a
question that specifically asked whether
respondents had asked about why the three-box
format was used. However, repeated questions asked
interviewers to report problems that respondents
had or other information on respondent reactions
to the cards.
6The Florida interviewer in the field test
reported that some respondents said they found the
icons (pictures) on the cards useful. The
California interviewer indicated that respondents
with less education tended to like the card more
than those with more education-perhaps reflecting
reactions to the icons.
7One respondent pointed out that some respondents
who, as children, had lived in a "campo" (hut)
might find the housing question sensitive or
embarrassing. An alternative would be to ask
instead about the type of house that the
respondent believes is most typical for families
in their home country.
8The new introductory card, on house types, was
pretested for acceptability of the icons before
being used in the formal cognitive interviewing.
On Aug. 25, 1998, interviews on this card were
conducted in Spanish with four Hispanic
respondents, one Nicaraguan and three Salvadoran,
at a Hispanic Community Center in Silver Spring,
MD. All four found the icons on this card
appropriate for adults, and all could recognize
the type of house from the icon alone. They
suggested changing the name for the rural dwelling
category from "casa de compag�a" to "casa de
campo." Respondents also suggested adding a thatch
roof to the rustic house. We drew a thatch roof on
the rural house icon and changed the name of this
house type from "casa de compag�a" to "casa de
campo."
9The term "border crossing identification card"
means "a document of identity bearing that
designation issued to an alien who is lawfully
admitted for permanent residence, or to an alien
who is a resident in foreign contiguous territory,
by a consular officer or an immigration officer
for the purpose of crossing over the borders
between the United States and foreign contiguous
territory in accordance with such conditions for
its issuance and use as may be prescribed by
regulations." 8 U.S.C. 1101(a)(6).
10In addition, one farmworker who could not read
English or Spanish had a difficult time deciding
in which box he belonged; he repeatedly stated
that he had married a U.S. citizen and finally
chose Box A.
11Two U.S. embassies had been bombed by terrorists
in the previous week, making headlines in the news
media.
12Each field-test respondent who selected Box A was
able to identify a specific category or program
under which he or she obtained a green card. Most
of these (19 out of 30, or about 63 percent) said
they had obtained their green cards through the
amnesty under the Immigration Reform and Control
Act of 1986 (P.L. 99-603), which included
provisions for Seasonal Agricultural Workers.
13During 1994-95, only about 3 percent of this
population claimed to be naturalized citizens
(Mines et al., 1997), and very few are here on
legal temporary farmworker visas. For additional
related work in this area, see GAO/HEHS-98-20,
Dec. 1997. About one million farmworkers acquired
valid green cards through the amnesty under the
Immigration Reform and Control Act of 1986,
although many of them have now moved to other
types of work.
14We believe the fact that respondents were not
afraid to pick a box containing the sensitive item
justifies further development and testing. In
particular, a validity study would address the
issue of possible systematic bias in the
responses.
15The interviewer's impression is that this
respondent was a U.S. citizen who believed that
undocumented persons would find the question
sensitive.
Chapter 4
Estimating Other Sensitive Items With the Three-
Card Method
Page 52GAO/GGD-00-30 Estimating Sensitive Survey I
tems
The three-card method was originally applied to a
personal characteristic (immigration status). This
chapter explores whether the technique might also
be applied to various sensitive behaviors and
practices. These include
� violence;
� sensitive personal choices;
� business and tax issues; and possibly,
� government workers' noncompliance with
important reporting requirements or with certain
other laws and regulations.
Logically, the three-card method is relevant to
any sensitive topic where (1) answers can be
categorized in a set of mutually exclusive and
exhaustive categories, (2) one of the answer
categories is sensitive, and (3) the sensitive
category is not too rare-at least not within a
defined survey population.
This chapter provides illustrations of how the
three-card method might be applied in each of the
areas listed above. However, we caution the reader
that these illustrations are, for the most part,
the result of brainstorming. They have not been
subjected to empirical development or testing and
are merely intended to be suggestive of
possibilities.
Some of the areas discussed are more sensitive
than others, and we also note that, logically, the
more sensitive the topic, the greater the need to
couple the three-card method with assurances of
anonymity-such as administering the technique via
pen-and-pencil "secret ballots" rather than in a
personal interview situation.
Violence
Violent behaviors that the three-card method might
be used to estimate include road rage and other
"anger responses," such as spouse abuse.
Road Rage
A survey of road rage among young male drivers
might first ask drivers about incidents (e.g.,
being cut off) that angered them during a
specified period (past year, past 5 years) and
then ask how they reacted the time they expressed
the most anger.1
The sensitive answer category would involve
physical actions by the respondent-such as
attempting to force the other driver off the road;
getting out of the car to accost the other driver
at a stop light; brandishing a weapon; or becoming
involved in actual physical contact, such as
shoving. The less sensitive categories might
include
� did nothing, muttered under one's breath, or
made remarks to passengers in one's car;
� honked horn or flashed lights at the other
car (but did not yell or gesture); and
� yelled or gestured angrily at the other
driver.
Drivers would specifically be told that we are
interested in the category that represents their
most extreme anger response. The categories could
be arranged on a card with one of the less
sensitive categories in Box A and the rest of the
categories in Box B. Box C would represent some
other response (not included in Box A or Box B).
Using a multicard, multisample approach, direct
data would be separately collected on each of the
less sensitive categories; then an indirect
estimate of the most sensitive category would be
obtained by subtraction.
Other Violent Behaviors
Spouse abuse and child abuse-as well as police
brutality and elder abuse in nursing homes-are
areas where survey questions might be framed in a
similar fashion to the road rage questions
described above. For example, a survey of police
officers might ask them about suspects or
perpetrators who angered them during the past year
and ask about their most extreme anger response.
Less sensitive categories would include ignoring
the provocation, going by the strictest rules,
speaking harshly to the suspect, and so on. The
sensitive category would involve some form of
physical abuse.
It also might be possible to use this approach to
study high school students' responses to other
students who anger them.
Sensitive Personal Choices
Sensitive personal behavior choices-such as
abortion or drug use-might also be studied using
the three-card method.
Abortion
Though legal, abortion is a sensitive topic for
many women. Some who have chosen to terminate a
recent pregnancy might deny ever having been
pregnant-or might report on the outcome of an
earlier pregnancy.2 A three-card approach might be
used to make the question less threatening.
For example, in asking about the outcome of the
respondent's most recent pregnancy, one option
would be to define answer categories such as
hospital birth, home birth, miscarriage,
unavoidable termination for medical reasons, and
abortion (for nonmedical reasons).3 The less
sensitive categories would appear in Box A of
alternate cards, and the percent whose most recent
pregnancy ended with an abortion would be
calculated indirectly.
Marijuana Use
A simple direct question on current (past month)
marijuana use is potentially sensitive.4 One
alternative would be to use the three-card method
to estimate past-month marijuana use-in
conjunction with estimating use of less sensitive
substances (such as beer, wine, and hard liquor).
Here, the less sensitive categories might be
� drank beer or wine but no hard liquor and no
marijuana,
� drank hard liquor (with or without beer or
wine), and
� no use of alcohol or marijuana.
Each of the less sensitive categories would appear
in Box A on alternative cards and the sensitive
category-marijuana use (with or without alcohol
use)-would be estimated indirectly.
Business and Tax Issues
There are many illegal, disapproved, or sensitive
business and tax practices. Examples include
manufacturers' disposal of hazardous wastes and
the failure to file tax returns.
Disposal of Hazardous Wastes
It might be possible to develop a survey of
manufacturers in an industry that routinely deals
with hazardous materials. The sensitive category
would be the illegal disposal of specific
hazardous wastes. The less sensitive answer
categories might include disposal of the specific
hazardous wastes
� in all instances using a contractor;
� partly using a contractor and partly self-
disposal in accordance with EPA guidelines; and
� in all instances, self-disposal in accordance
with EPA guidelines.
The sensitive category would encompass all direct
forms of illegal dumping by the manufacturer. An
actual application of the method would have to be
developed by working with persons knowledgeable in
this area.5
Failure to File Tax Return
Tax nonfiling is difficult to estimate for self-
employed persons. To create a number of less
sensitive categories, the act of filing or not
filing might be combined with asking about the
type of return filed and who prepared the return.
The less sensitive categories might be
� my spouse prepared a joint return for us,
which has been filed;
� I prepared a joint return for myself and my
spouse, and it has been filed;
� I prepared my own return, and filed it as an
individual; and
� a tax preparer prepared the return
(individual or joint), and it has been filed.
The sensitive category would be: I have not sent
in a return for last year.
Government Workers' Noncompliance
The extent to which certain government
employees-at federal, state, and local levels-fail
to comply with reporting requirements or various
other laws or regulations is of interest from a
managerial or oversight point of view. For
example, it would be desirable to monitor whether
certain types of noncompliance rise above
threshold levels, thereby signaling the need for
remedies such as agency-wide ethics training.
In more serious instances, prevalence data
might-if available-help define the kinds of
investigative approaches that would be most
appropriate. (For example, in defense-related
work, such data might help determine whether or
not widespread polygraph tests should be required
for scientists working at government
laboratories.)
In the following sections, two examples explore
whether noncompliance might be asked about by
using the three-card method in anonymous surveys
of government workers.
Failure to Report Offers of Bribes
Police officers and certain other government
workers (for example, at the federal level, e.g.,
those working for the Customs Service, the Border
Patrol, or the Postal Service) may be expected or
required to report offers of bribes or incentives
that might be offered in hope that the official
would overlook certain occurrences (such as
illegal imports or the possession of illegal
substances). At some agencies or departments,
regulations or laws require employees to report
all bribery attempts.6
It might be that a technique such as the three-
card method would facilitate asking about
government workers' compliance or noncompliance
with these reporting requirements. One possibility
might be to define the sensitive category as
having failed to report an offer of an
inducement-within a specified time period. The
less sensitive categories might refer to
� having been offered inducements both on duty
and off duty, and reporting all cases;
� having experienced one or more such
occurrences while on official duty only (no
similar occurrences while off duty), and reporting
all cases; and
� having experienced one or more such
occurrences while off duty only, and reporting all
cases.
Violations of National Security
Turning to the area of defense-related government
work, in the fall of 1999, concerns surfaced about
U.S. scientists' contacts with foreign agents and
the possibly unauthorized passing of information
by federal employees. These concerns were aired
before Congress (Loeb, 1999; and Pincus and Loeb,
1999).7 Currently, the charged atmosphere might
prevent any attempt to conduct a survey of
unauthorized exchanges of information-even if
perceived as minor or as justified in the
particular instance. However, in the future, if a
technique such as the three-card method were used
to survey these workers, the sensitive category
might be defined as "gave some unauthorized
information to a foreign national-not as part of
an authorized U.S. counterintelligence operation."
The less sensitive categories might indicate
whether the employee
� had been approached by a foreign national
seeking unauthorized information, but did not
provide it;
� had been approached by a foreign national,
and-as part of an authorized operation-gave out
information provided by U.S. counterintelligence;
or
� had not been approached by a member of a
foreign intelligence service.
Again, all categories would be linked to a
specific time frame (e.g., within the past 5
years).
_______________________________
1For the driving population as a whole, road rage
expressed in "physical confrontation" within the
past 5 years is estimated at 4 percent. (This
statistic is from a 1997 Gallup poll, in which
respondents who drive were asked if they had
"honked horn," "shouted," "slowed down," "flashed
high-beams," or engaged in "verbal exchange" or
"physical confrontation." See Gallup Poll Monthly,
No. 383 (Aug. 1997), p. 60). We would anticipate
that levels of physical confrontation would be
considerably higher among young male drivers than
in the driving population as a whole.
2Researchers have found that underreporting
abortion "is a persistent problem in studies
conducted in the United States and elsewhere,
irrespective of the research design or study
population" (Udry et al., 1996). These same
researchers point out that the underreporting of
abortion also causes biased estimates of
contraceptive failure, miscarriage, and
reproductive histories that rely on self-reports.
3A different set of categories would be birth of a
girl, birth of a boy, multiple birth, miscarriage,
unavoidable termination for medical reasons, and
abortion (for nonmedical reasons).
4This potential sensitivity may be heightened in
certain circumstances, such as a workplace survey
of employees or a survey of military personnel,
or, students at the service academies.
5Separately developed cards-with different
categories-might be used for surveys of trucking
companies that remove hazardous wastes from
manufacturers in the relevant industry and
landfill, incinerator, or other companies that
receive and dispose of the relevant wastes.
6For example, U.S. Postal Service employees are
required to report any bribe or attempted bribe,
"undue influence or coercion to induce or attempt
to induce the employee to act or neglect to act in
regard to his official responsibilities." 39
C.F.R. 447.61.
7These issues concerned certain Department of
Energy national laboratories.
Chapter 5
Challenges and Potential Solutions in Using the
Three-Card Method in a Large-Scale Survey
Page 59GAO/GGD-00-30 Estimating Sensitive Survey I
tems
Often, surveying a new subject or topical area or
using a new method requires considerable
development and cost-as well as ingenuity-but the
payoff can be improved information for the policy-
making process. Assuming that a specific
application of the three-card method is fully
tested, validated, and deemed appropriate for use
in a large-scale survey, a number of challenges
would apply. These challenges include
� obtaining reasonably low-cost (efficient)
interviews that include flashcards;
� efficiently screening for members of key
populations (e.g., foreign-born persons, young
male drivers);
� efficiently estimating three-card items
despite the need to draw three different samples
and to estimate the sensitive item indirectly;
� avoiding nonresponse bias across the three
samples; and
� dealing effectively with issues involved in
surveying special groups, such as the foreign-
born.
Interviews Using Flashcards
Thus far, all development and testing of the new
method has been conducted in person, in face-to-
face interviews. Personal interviews are
considerably more expensive than telephone
interviews. But we believe the flashcards are
essential, and therefore that the three-card
method is not appropriate for a telephone survey.1
Adding a three-card format to an ongoing, in-
person survey could be much less expensive than
conducting a new in-person survey.2 The three
samples could be selected for the three-card
portion of the interview only. But because
piggybacking onto an existing in-person survey may
not always be an option,3 other potential low-cost
alternatives should be considered. Three
alternative strategies are as follows:
� group administration, which has been used to
survey high school students, for example;4
� mail surveys; and
� Internet surveys where a computer screen
displays the equivalent of a three-box card, have
been conducted with certain populations. Of
course, many households are not connected to the
Internet, so for general population surveys a
feasible option could be a dual-frame or dual-mode
survey involving the use of personal interviews
where computers are not available.5
Applications such as these-which do not involve
answering aloud-might have an added advantage in
encouraging more truthful responses to sensitive
items.6
It seems likely, however, that many applications
would require a new in-person survey, because the
cards are essential and the options outlined above
may not be feasible. In these instances, the cost
of a new large-scale in-person survey would be
considerable.
Screening for the Relevant Subpopulation
Some sensitive items are relevant only to a
specific subpopulation; e.g., legal status is
relevant only to the foreign-born. Other sensitive
items may be of most interest in a
subpopulation-and rare in other groups; for
example, road rage could be more relevant to young
male drivers than others-possibly it might be most
relevant to young males who have been ticketed for
moving violations in the past 5 years.
In some cases, the subpopulation of interest might
be concentrated in one specific location or a
number of specific locations.7 This could allow
reasonably efficient direct sampling. In other
cases, it would be necessary to first screen more
general populations to identify members of the
relevant group.8
A main option here is to use an ongoing survey as
a "screener" (even if that survey cannot be used
to ask the three-card questions themselves). It
may be necessary to "piggyback" screening
questions onto the existing survey. But a special
purpose-or even a general purpose-survey may
already include questions that could serve to
screen for the subpopulation in question.9 For
purposes of screening, a telephone survey is often
as appropriate as an in-person survey. Once
members of the subpopulation of interest are
identified, a "linked survey" could be conducted.
That is, interviewers would revisit identified
households within 1 to 3 months after the main
survey to ask a short set of additional questions
using the three-box format.
Another option that might be relevant in certain
circumstances is to use existing records. For
example, it might be possible to use drivers'
license records to sample young male drivers (for
a survey of road rage)-or even to use records of
moving violations to sample only those with moving
violations in recent years.
Efficient Estimation
The direct estimate of each less sensitive
category that is featured in Box A is based on
just one sample (i.e., only those respondents
answering the card that features that particular
category in Box A). Obviously, the number of
respondents in a particular sample is less than
the total number of respondents in all three
samples. All else being equal, the fewer
respondents answering a specific item, the higher
is the variance-or the margin of error-for the
resulting estimate.
Variance is further increased for the indirect
estimate of the sensitive item (which never
appears in Box A). This is because the indirect
estimate is obtained by a linear combination of
the three direct estimates-and the variance is
increased by each additional estimate (see app.
I). Finally, as always, subgroup estimates (e.g.,
illegal residents who are male or those who are
female) have a higher variance than total
estimates (e.g., all illegals) because of the
reduced number of qualifying respondents.
The most direct way to reduce the variance of an
estimate is to increase the sample size; however,
the dollars needed for additional interviews
increase costs. Four efficiency-minded
options-that is, strategies to reduce variance
without the added cost of increasing the sample
size-can potentially be used. These strategies,
some of which can be used in tandem with each
other, are:
� using information about the subpopulation (if
it is known) to stratify the subpopulation in
advance;
� using a principle of "optimal allocation" to
determine the relative sizes of the three samples
(see app. II);
� setting sample sizes from every block or
cluster in proportion to counts of the relevant
subpopulations in these "blocks" or clusters; and
� obtaining some of the three-card information
by inserting one of the cards in an ongoing
survey.
Using a Principle of Optimal Allocation
The variance can be reduced by setting or choosing
the relative sizes of the three samples based on
one of the principles of optimal allocation. As
background, we note that, given three samples of
the same size, the variances associated with the
direct estimates of Box A categories will very
likely not be the same. (To illustrate this for
legal status, the percentage of foreign-born who
possess a green card is probably not the same as
the percentage who are naturalized citizens.10 This
is important because, as explained in standard
statistical texts, a very low or very high
percentage is associated with lower variance than
is one near 50 percent.) Depending on the
population being surveyed, it seems likely that
with equal sample sizes, some cards would be "high
variance" cards and others would be "low variance"
cards.
To minimize the variance associated with the
indirect estimate of illegals, which is based on
three direct estimates, the principle of optimal
allocation we are using suggests that larger
number(s) of sample cases be allocated to estimate
the proportion of the population represented by
the high variance cards and fewer numbers of
sample cases be assigned to estimate the
proportion of the population represented by the
lower variance cards (see Cochran, 1977, pp.
96ff).
Setting Sample Sizes Proportional to Block Sizes
To ensure that the sampling rates of the relevant
subpopulation are equal across small clusters
(i.e., blocks), the sample sizes for the three
surveys can be allocated proportionally to the
number of people in the relevant subpopulation in
each block. For example, suppose that, by using
the principle of optimal allocation, the sizes of
the three samples are 500, 300, and 200,
respectively. Also, assume that the sample area
has two blocks, the first of which has four times
as many members of the relevant subpopulation as
the second. Then the allocation of the three
samples would be 400, 240, and 160, respectively,
in the first block, and 100, 60, and 40,
respectively, in the second block.
Obtaining Information From Respondents in an
Ongoing Survey
Even where it is not possible to use the three-
card method in an ongoing survey, it may still be
possible to insert a question on one of the cards
(e.g., a portion of foreign-born respondents in a
general purpose survey might be asked whether they
are U.S. citizens, using a three-box card).11 These
data could be combined with the full set of three-
card responses separately obtained in a different
survey (perhaps a linked follow-back survey as
discussed above), thus lowering the variance.
Ideally, there would be tests for possible
response differences between various survey modes.
Minimizing the Bias of Nonresponse
Person (unit) nonresponse could vary across the
three samples, raising the possibility of
noncomparability and biased estimates. This might
be the case, for example, if interviewers
perceived a particular legal status card as more
sensitive than the others; that is, they might not
try as hard to "get the interview" when assigned
the card perceived as more sensitive. This
possibility can be avoided by utilizing a "blind"
randomization procedure, such as a sealed envelope
that is opened only after contacting the
respondent or after cooperation has been obtained.
If a linked "screener" survey is used, the impact
of person nonresponse might be minimized through
imputations based on results from the initial
survey.12 It also seems advisable to sample
households that were coded as nonresponses in the
initial survey; screen for target-population
residents in those households; and conduct three-
card interviews, as appropriate.
Turning to item question nonresponse among survey
participants, we have thus far not encountered any
instances of item nonresponse to the three-card
questions-whether in our pretests, the field test,
or the cognitive interviews. Moreover, although we
have only tested one card, we believe it is the
most sensitive of the three because it asks
whether a respondent has a green card or not.
If a small amount of item nonresponse were to
occur, it might be possible to minimize its impact
by using imputation methods. However, substantial
amounts of item nonresponse would affect the
usability of the results.
Issues in Surveying Special Populations
There are special challenges in surveying various
special populations-foreign-born persons, teens,
the elderly, or other specialized groups. The
foreign-born population is of special interest in
this report because the initial application of the
three-card method applies to that population. The
problem of combating incomplete coverage of this
subpopulation was discussed in our 1998 report
(GAO/GGD-98-164). The following section discusses
challenges in interviewing members of the foreign-
born population.
Interviewing the Foreign-Born
Efforts to survey foreign-born individuals
necessarily involve (1) building trust and
communicating with foreign-born groups and, if
using the three-card method, (2) selecting icons
that are effective across diverse groups.
Building Trust and Communicating With Diverse
Groups
Trust and communication are important in all
surveys, and in all cases, the ability of the
interviewer to establish rapport is essential.
However, building trust and communicating
accurately may be more challenging in surveys that
focus on a special population-such as teens, the
elderly, or the foreign-born. In particular, the
foreign-born population seems to present a unique
set of challenges because, for example,
respondents and interviewers may be from very
different cultures and might not speak the same
language.
Logically, matching interviewers to respondents on
foreign-born status and "home country" is one
method of maximizing trust and communication. In
Los Angeles, surveys that targeted residents from
selected nations-Mexico, El Salvador, and the
Philippines (DaVanzo et al., 1994; Heer, 1990, pp.
88-97; Bustamante et al., 1996)-were generally
able to employ and assign interviewers to match
respondents' home countries.13 (In our field test
of one Salvadoran and 80 Mexican farmworkers, the
interviewers were all fluent in Spanish. Although
not all were from Mexico, they were experienced in
interviewing this population.)
A strategy of "matching on home country" might not
be feasible for a nationwide survey covering
immigrant groups from all nations of the globe. A
national effort to approach the matching condition
might be possible using three different tactics,
as follows.
� If a linked survey is used,14 and if the
initial survey identifies the respondent's home
country, then recontact could begin with an
explanatory telephone call, and the survey workers
placing those calls from a central phone bank
could be immigrants who are matched to each
respondent on home country. Subsequently, an
interviewer would make a personal visit. If it is
known that language assistance will be required,
the interviewer might carry a cell-phone and
contact headquarters for help from a survey worker
from the respondent's home country.
� Even if a linked survey is not used, it is
possible to identify areas according to ethnic
settlements and to try to provide appropriate
bilingual interviewers to the extent possible.
Cards and interview materials can be translated
and tested in as many languages as possible.15 All
interviewers could carry cell-phones and contact a
central phone-bank for help from an interviewer
who speaks the respondents' home language.
� Alternatively, it might be possible to
develop a self-administered version of the
immigration status question. That is, instead of
showing the respondent a card, the interviewer
would give the respondent an "answer sheet" that
looks like one of the legal status cards shown at
the outset of this report. Instructions at the top
of the answer sheet would tell the respondent, in
his or her own language, to circle the letter
beside the box that applies to him or her. The
instructions would also make clear that if it is
Box B, we do not want to know which specific
category applies. This approach might be useful
for immigrants who are literate in the language of
their home country.
Technology can also help overcome communication
barriers. Audio computer-assisted self-
interviewing (ACASI) is a technology that allows
for the administration of questionnaires in
multiple languages-even when survey field staff
are monolingual. It has been used successfully to
interview older Korean immigrants who do not speak
English. With this technology, "Subjects hear the
questions through headphones and enter responses
directly into the computer using the keyboard"
(Hendershot et al., 1996, p. 165). With the latest
system, the field interviewer simply plugs
headphones directly into a port on the laptop
(Turner et al., 1996, p. 174). Additional benefits
of ACASI are that visuals can be provided and that
the respondent's privacy is enhanced vis-�-vis
interviewers and other members of the household.
This can further increase the accurate reporting
of sensitive items (Turner et al., 1998).
Selection of Icons That Are Effective Across
Diverse Groups
Finally, the icons used on the cards deserve
special attention. The cards and icons shown in
earlier chapters were pilot-tested with Hispanic
respondents. It may be necessary to identify icons
that are more appropriate or effective for other
cultures (e.g., Asian) or to identify icons that
transcend cultural barriers. Only cross-cultural
testing can determine this.
Challenges Differ Across Targeted Subpopulations
Because potential applications of the three-card
method may involve different subpopulations (see
ch. 4), it can be reasonably expected that there
may be a variety of different challenges in
fielding a specific survey. Interviewing a
nationwide sample of foreign-born persons about
their legal status, for example, might be
considerably more difficult than interviewing
young male drivers about "road rage." The reason
is that for young male drivers, a language barrier
may occur in relatively few cases. For this
reason, generalizations in this area are difficult
at best.
_______________________________
1We believe that the visual display of the boxes
and the combination of categories in Box B is
essential. The reason is that this helps convey
the privacy protection to respondents ("If you're
in Box B, we do not want to know which category
applies to you.")
2Some existing national surveys are conducted
exclusively face-to-face. One example is the
National Health Interview Survey, which involves
about 100,000 in-person interviews each year.
Certain other large-scale surveys conduct only a
portion of their interviews face-to-face, and this
creates a problem for the use of cards.
3Sponsors of the ongoing survey might not wish to
add a somewhat sensitive question series.
4See "The Monitoring The Future Study, 1975-1997,"
a survey of drug use among secondary school
students (Johnston, O'Malley, and Bachman, 1998).
5Some segments of the foreign-born population may
be unlikely to own computers with an Internet
connection, but other segments may be likely to
have them. The Wall Street Journal recently
published an article that discusses the pros and
cons of doing surveys using the Internet,
including methodological issues (Simons, 1999).
6Self-administered questionnaires and answer
sheets have been associated with more accurate
responses because interviewers need not know the
sensitive reply. Moreover, if certain kinds of
interviews are conducted in a household, answering
aloud may be sensitive because other members of
the household may overhear the response (see,
e.g., Gfroerer, 1985, p. 22).
7To cite one example, a population of farmworkers
with a large subpopulation of foreign-born
individuals would probably be located on farms.
8For a discussion of alternative "sampling
strategies and sources of recruitment" in
surveying the foreign-born population, see Loue
and Bunce (1999).
9For example, the Current Population Survey asks
about nativity; this represents a screen for
foreign-born.
10For example, it may be that 40 percent of foreign-
born farmworkers have green cards, whereas less
than 5 percent are naturalized citizens.
11Some national surveys currently include a
question on citizenship. Chip Alexander, Assistant
Division Chief for Longitudinal and Expenditure
Surveys Design, Bureau of the Census, pointed out
a technique of drawing the sample ("double
sampling") that would minimize the variance.
Essentially, this would involve piggybacking the
least sensitive card onto the "screener" survey,
for a subsample of relevant respondents; a follow-
on survey, using all three cards in alternate
samples, would be limited to those not shown a
card in the screener survey.
12Suppose, e.g., that of 500 Mexican males who
participated in the initial survey, 480 responded
in the follow-up. The responses of the 20 Mexican
male nonrespondents might be imputed based on (1)
characteristics reported by all 500 in the initial
survey, such as occupation, age, and time residing
in the United States and (2) the three-card
responses of the 480 Mexican males who did
participate in the follow-up survey.
13These surveys apparently had some success in
asking direct questions about immigration status.
14That is, if, as discussed above, foreign-born
respondents are identified in an initial general
purpose survey and then recontacted for questions
about immigration.
15The pilot for the "new immigrant survey" (in
which new green-card holders from various
countries are interviewed) included questionnaire
translations for 6 languages; interviewers
speaking 11 other languages were hired (Jasso,
1999).
Appendix I
Variance of the Estimates
Page 64GAO/GGD-00-30 Estimating Sensitive Survey I
tems
The statistical expression of the three-card
estimator of the percent of foreign-born who are
here illegally, the variance of this estimator,
and the "technique effect" are shown in figure
I.1. (See also GAO/GGD-98-164, July 1998.)
Clearly, there are variance costs associated with
using an indirect method. First, each estimate of
legal status is based on one sample, rather than
all three samples. Second, the indirect estimate
of illegal immigrants is affected by the variance
of each of the estimates that is included in its
calculation. Assuming that the three samples are
of equal size and that for each card 25 percent of
respondents belong in Box A, the confidence
interval for a three-card method estimate of
illegal immigrants would be three times as large
as a corresponding direct estimate. Keep in mind,
however, that variance costs depend heavily upon
the distribution of immigration status groups in
the population surveyed and the relative sizes of
the three samples.
Two examples are presented below to give the
reader a flavor of what real-world precision might
be.
� Example 1: In a population of foreign-born
agricultural workers, the distribution of
immigration status might be 55 percent illegal; 36
percent legal permanent residents; 3 percent U.S.
citizens; and 6 percent temporary workers,
refugees, or asylees. Assuming this distribution
and a total sample size of only 1,000, allocated
with 100 respondents to answer the card with U.S.
citizen in Box A; 200 to answer the card with
temporary visas, refugees, or asylees in Box A;
and 700 to answer the card with legal permanent
resident in Box A, the 95-percent confidence
interval for an estimate of 55 percent illegal
would be 49 to 61 percent.
� Example 2: In the residential foreign-born
population of the United States, taken as a whole,
the distribution might be 22 percent illegal; 30
percent U.S. citizens; 38 percent legal permanent
residents; and 10 percent temporary workers,
refugees, or asylees (without green cards).
Assuming this distribution and a total sample size
of 13,000-the approximate number of foreign-born
in the Current Population Survey supplement-with
6,000 respondents allocated to answer the card
with legal permanent resident in Box A; 5,550
allocated to answer the card with U.S. citizen in
Box A; and 1,500 to answer the card with temporary
workers, refugees, or asylees in Box A, the 95-
percent confidence interval for an estimate of 22
percent illegal would be 20 to 24 percent.
We note that in the foregoing examples, the sizes
of the three samples were chosen by adapting one
of the principles of optimal allocation.1 The
confidence intervals in the examples would have
been larger if three equal-sized samples had been
used.
The formula for calculating the variance of the
estimates is shown in figure I.1. To estimate the
percentage of a foreign-born subgroup who are here
illegally (e.g., percent of foreign-born
California residents who are here illegally),
calculations identical to those shown in the
formula would be performed using only the relevant
data (e.g., the data for all foreign-born
residents of California). The variance of such a
subgroup estimate would depend on the size of the
sample for that subgroup and the distribution of
legal status in that group.
However, to derive an estimate such as the
percentage of illegal immigrants living in
California implies a ratio estimate. That is, both
the numerator and the denominator would involve
indirect estimates of illegal immigrants (the
numerator, illegal immigrants in California; the
denominator, illegal immigrants in the United
States). Specifically, the numerator would consist
of the number of foreign-born Californians times
the estimated percentage of that group who are
here illegally. The denominator would consist of
the number of foreign-born residents of the United
States times the estimated percentage who are here
illegally. Assuming that the number of foreign-
born residents is known (e.g., from the census),
the percentage of illegal immigrants living in
California reduces to the product of two factors:
� a constant, consisting of the number of
foreign-born living in California divided by the
number of foreign-born living in the United
States; and
� a ratio estimate-specifically, the ratio of
the estimate of the percent of foreign-born
California residents who are here illegally to the
estimate of the percentage of foreign-born U.S.
residents who are here illegally.
Deriving the variance of such an estimate is
complex and beyond the scope of this paper.
However, with respect to the first factor, it is
clear that the constant factor tends to reduce the
variance for states with relatively few foreign-
born. With respect to the second factor, a formula
is available (see Hansen, Hurwitz, and Madow,
1953, vol. II, p. 107). The overall variance of
such a ratio may be substantial because the two
estimates involved in the ratio both carry
variance costs. Special designs to minimize
variance costs may be feasible, however, depending
on the application.
Figure I.1: Statistical Expression of the Three-
Card Estimator, Its Variance, and Technique Effect
Source: GAO/GGD-98-164, July 1998, p. 75.
_______________________________
1The key guideline of optimal allocation that we
followed minimizes the variance of an estimate by
assigning a somewhat greater proportion of the
sample where the variance is highest-and a
somewhat smaller proportion where the variance is
lowest. (Optimal allocation is usually used to
determine sample size for strata rather than for
subsamples.)
Appendix II
Different Forms of the Questionnaire Used in the
Field Test
Page 78GAO/GGD-00-30 Estimating Sensitive Survey I
tems
The 81 interviews were conducted as an add-on to
the National Agricultural Workers Survey (NAWS).
We used nine different forms of the three-card
segment in the field test. The purposes included
(1) varying certain questions used as structured
probes to obtain additional information and (2)
conforming to Office of Management and Budget
(OMB) policy, which applies to the NAWS survey.
Specifically, OMB requires a survey form to be
formally reviewed if it is to be used to survey 10
or more people. We developed different forms
because the review time would have exceeded our
available time.
The NAWS is a survey that gathers information
about farmworkers in the United States by asking
them questions about their basic demographics,
legal status, education, family size and household
composition, and wages and working conditions in
agricultural jobs. Demographic characteristics
include their ethnic composition, their age and
gender distribution, and the division among
immigration and citizenship categories. Our 81
interviews were conducted using an abridged
version of the NAWS questionnaire, with our three-
card series added on.
Form 1 of the three-card question series has been
reproduced in figure II.1. The differences among
the nine forms are explained in table II.1
Table II.1: Variations in Wording of Survey
Questions in the Nine Questionnaires We Field-
Tested
The structured probe portion of
question GAO-4a
Included as Not Included with
shown on includ additional
form 1 (fig. ed explanationb
II.1)
Contains all Form 1 Form 2 Form 3
questions
Omits question Form 4 Form 5 Form 6
GAO-1b onlyc
Omits question Form 7 Form 8 Form 9
GAO-3b onlyd
aSee p. 8, fig. II.1. The multi-paragraph
structured probe follows the instruction, "If
respondent chooses Box A, ask: . . . ."
bInterviewer instructed to provide ad hoc
explanations if respondent does not understand
question.
c As shown on p. 2, fig. II.1, question GAO-1b
asks about household appliances not present at the
location where the respondent is living or staying
now.
dAs shown on p. 5, fig. II.1, question GAO-3b asks
if the respondent lives permanently in the United
States or just works here temporarily.
Source: GAO.
Figure II.1: Form 1 of the Questionnaire
Appendix III
Estimates for Demographic Subgroups
Page 79GAO/GGD-00-30 Estimating Sensitive Survey I
tems
Estimates of legal permanent residents (Box A for
legal status card 1) can be directly obtained for
demographic subgroups, such as male or female;
residents of various geographic areas (e.g.,
California); and age or income groups.
The only proviso is that demographic information
must be obtained for all respondents earlier in
the questionnaire. Subgroup estimates can be
obtained simply by isolating, for example, male
respondents in sample 1 and then calculating the
percentage who chose Box A. Then, perform the same
procedures separately for female respondents in
sample 1. Of course, how refined a subgroup can be
reliably estimated would depend on the size of the
sample and other aspects of sample design.
Estimates of foreign-born U.S. citizens in
different subgroups can be obtained in a similar
fashion, using data for sample 2 respondents. The
situation is analogous for the legal statuses that
appear in Box A of the legal status card shown to
sample 3.
The three-card method can also be used to
indirectly obtain separate subgroup estimates for
illegal immigrants-at least for sizable subgroups
of illegals, such as males and females, or
California residents. This can be done by
isolating the data for foreign-born respondents in
the first subgroup of interest (e.g., foreign-born
males), obtaining separate estimates for Box A in
each of the three samples, and then subtracting
percents estimated from each legal status from 100
percent. The procedure would then be repeated for
the next subgroup of interest (e.g., isolate data
for foreign-born females and repeat calculations).
Such procedures work best for large subgroups
within the foreign-born population.
Given a large enough sample, indirect
estimates of illegals could be obtained for
several key geographic areas-for example, key
immigration states like New York and Texas. In
some cases, it may be desirable to produce
estimates for smaller areas, such as a county
(e.g., Los Angeles County) or a group of counties
(i.e., the counties of Texas that border Mexico).
Similarly, a separate estimate of illegal
immigrants might be obtained for a particular
demographic subgroup (such as women of
childbearing age or persons below the poverty
line). Of course, whether a subgroup of illegals
could be reliably estimated would depend on the
size of the sample and other aspects of sample
design.
Appendix IV
Data From Respondents Who Participated in the
Cognitive Interviews
Page 80GAO/GGD-00-30 Estimating Sensitive Survey I
tems
Gende Age Nativit U.S. job Primar Understo Found Respondent comments
r y experience y od card 3
langua refugee sensit
ge icon? ive?
Male 38 El Auto Spanis No No Use bigger green card icon.
Salvado factory h It looks like the person in
r assembly, the refugee icon is holding
cleaning, up a map, and the explosion
bus driver looks like a hole in the
paper. Immigration question
is good because it is not
personalized.
Male 49 Bolivia Lawn care, Quechu Yes Yes Add a fake green card to the
carpet a undocumented icon. Some would
cleaner, find the immigration question
electricia sensitive because they do not
n want to tell the truth about
their status.
Male 60 Mexico Constructi Spanis No Yes Immigration question is
on, h sensitive because some do not
milking want to be asked about that.
cows That is confidential
information.
Male 30 Dominic Mowing Spanis No No Refugee icon is confusing. It
an lawns h looks like a terrorist being
Republi pushed out of the United
c States.
Femal 20 Peru Cleaning Spanis Yes No The respondent said that she
e and child h recognized the icons for
care "Undocumented,"
suitcase/visa, and the
Refugee categories. Change
diagonal line to crossed
lines in the " undocumented"
icon.
Femal 34 El Factory Spanis Yes Yes Use an additional line to
e Salvado machine h and no cross out the undocumented
r operator, icon. Change the refugee
cleaning symbol to a thatch roof
without walls; that's the
symbol for refugee in my
country. The immigration
question is not bad, I do not
find it sensitive, but some
people would.
Male 34 Nicarag Painter Spanis Yes Yes Add visa to suitcase icon.
ua h and no Add U.S. seal to citizen
icon. Immigration question is
sensitive because many people
are undocumented and would
not want to answer.
Male 22 El Painter, Spanis Yes Yes Some compesinos (rural
Salvado carpet h people) wouldn't recognize
r installer, the undocumented icon. Add
commercial trees and a "danger" sign to
cleaning the refugee icon. Some would
find the immigration question
sensitive because they would
fear being turned in to INS.
Source: GAO compilation of data from cognitive
interviews.
Bibliography
Page 84GAO/GGD-00-30 Estimating Sensitive Survey I
tems
Bradburn, Norman M., and Seymour Sudman. Improving
Interview Method and Questionnaire Design:
Response Effects to Threatening Questions in
Survey Research. San Francisco: Jossey-Bass, 1979.
Bustamente, J.A., et al. "Characteristics of
Mexican Immigrants, Legal and Undocumented, in Los
Angeles County: A Pilot Study." Research Report.
El Colegio de la Frontera Norte and University of
Southern California, July 1996.
Cochran, William G. Sampling Techniques. 3rd ed.
New York: John Wiley & Sons, 1977.
DaVanzo, Julie, Jennifer Hawes-Dawson, R. Burciaga
Valdez, and Georges Vernez. Surveying Immigrant
Communities: Policy Imperatives and Technical
Challenges, Santa Monica, CA: RAND, 1994.
DeMaio, Theresa, et al. "Protocol for Pretesting
Demographic Surveys at the Census Bureau." Report
of the Pretesting Committee. 1993.
Droitcour, Judith, et al. "The Item Count
Technique as a Method of Indirect Questioning: A
Review of Its Development and a Case Study
Application. In Measurement Errors in Surveys, by
Paul P. Biemer et al. New York: John Wiley & Sons,
Inc., 1991, pp. 185-210.
Ferguson, George A. Statistical Analysis in
Psychology and Education. 4th ed. New York: McGraw-
Hill Book Company, 1976.
Gallup Poll Monthly, No. 383, August 1997.
Gfroerer, Joseph. "Influence of Privacy on Self-
Reported Drug Use by Youths," in Self-Report
Methods of Estimating Drug Use: Meeting Current
Challenges to Validity, edited by Beatrice A.
Rouse, Nicholas J. Kozel, and Louise G. Richards.
NIDA Research Monograph 57. Rockville, MD:
National Institute on Drug Abuse, 1985, pp. 22-30.
Hansen, Morris H., William N. Hurwitz, and William
G. Madow. Sample Survey Methods and Theory. Volume
II: Theory. New York: John Wiley & Sons, 1953.
Heer, David M. Undocumented Mexicans in the United
States. New York: Cambridge University Press,
1990.
Hendershot, Tabitha P., et al. "Multilingual
ACASI: Using English-Speaking Interviewers to
Survey Elderly Members of Korean-Speaking
Households." In Richard Warnecke, Health Survey
Research Methods Conference Proceedings, DHHS
Publication No. (PHS) 96-1013. Hyattsville, MD:
National Center for Health Statistics, 1996, pp.
165-69.
Jasso, Guillermina. Department of Sociology, New
York University. Personal communication, March 16,
1999.
Johnston, Lloyd D., Patrick M. O'Malley, and
Jerald G. Bachman. National Survey Results on Drug
Use from The Monitoring The Future Study, 1975-
1997, Vol. 1. Secondary School Students. National
Institute on Drug Abuse. NIH Publication No. 98-
4345. Washington, D.C.: U.S. Government Printing
Office, 1998.
Kraly, Ellen Percy, and Robert Warren. "Estimates
of Long-Term Immigration to the United States:
Moving U.S. Statistics toward United Nations
Concepts." Demography, 29 (Nov. 1992), pp. 613-26.
Levine, Daniel B., Kenneth Hill, and Robert Warren
(eds.). Immigration Statistics: A Story of
Neglect. Washington, D.C.: National Academy Press,
1985.
Locander, W., S. Sudman, and N. Bradburn. "An
Investigation of Interview Method, Threat and
Response Distortion." Journal of the American
Statistical Association,Vol. 71(1976), pp. 269-75.
Loeb, Vernon. "Senators Challenge Energy Polygraph
Plan: Nuclear Workers' Reputations Could Be at
Risk in Unreliable Testing, Officials Say."
Washington Post, Sept. 26, 1999, p. A20.
Loue, Sana and Arwen Bunce. "The Assessment of
Immigration Status in Health Research." Vital and
Health Statistics. Series 2: Data Evaluation and
Methods Research No. 127. National Center for
Health Statistics, Hyattsville, MD, Sept. 1999.
DHHS Publication No. (PHS) 99-1327. Washington,
D.C.: U.S. Government Printing Office, 1999.
Mines, Richard, et al. A Profile of U.S.
Farmworkers: Demographics, Household Composition,
Income and Use of Services. Washington, D.C.:
Department of Labor, Office of the Assistant
Secretary for Policy, 1997.
Passel, Jeffrey S., and Rebecca L. Clark.
Immigrants in New York: Their Legal Status,
Incomes, and Taxes. Washington, D.C.: The Urban
Institute, 1998.
Pincus, Walter, and Vernon Loeb. "Lie Tests Anger
Lab Scientists: DOE Workers Say Morale
Threatened." Washington Post, Sept. 23, 1999, pp.
A3, A14.
Polivka, Anne E. and Jennifer M. Rothgeb.
"Overhauling the Current Population Survey:
Redesigning the Questionnaire." Monthly Labor
Review 116: 9 (Sept. 1993), pp. 10-28.
Rossi, Peter H., et al. Handbook of Survey
Research. Orlando, FL: Academic Press, Inc., 1983.
Simons, John. "Is a Web Political Poll Reliable?
Yes? No? Maybe?" Wall Street Journal, April 13,
1999, p. B1.
Schryock, Henry S., and Jacob S. Siegel and
Associates. The Methods and Materials of
Demography. 4th ed. Washington, D.C.: U.S.
Government Printing Office, 1980.
Turner, Charles F., et al. "Impact of ACASI on
Reporting of Male-Male Sexual Contacts:
Preliminary Results From the 1995 National Survey
of Adolescent Males." In Richard Warnecke, Health
Survey Research Methods Conference Proceedings,
DHHS Publication No. (PHS) 96-1013. Hyattsville,
MD: National Center for Health Statistics, 1996,
pp. 171-76.
Turner, C. F., et al. "Adolescent Sexual Behavior,
Drug Use, and Violence: Increased Reporting With
Computer Survey Technology." Science, Vol. 280
(1998), pp. 867-73.
Udry, J. Richard, et al. "A Medical Record Linkage
Analysis of Abortion Underreporting." Family
Planning Perspectives, Vol. 28, No. 5,
September/October 1996.
U.S. House of Representatives. "Omnibus Oversight
Hearing on Illegal Immigration." Committee on the
Judiciary, Subcommittee on Immigration and Claims,
Mar. 18, 1999.
U.S. House of Representatives. "Hearing on the H-
1B Temporary Professional Worker Visa Program and
Information Technology Workforce Issues."
Committee on the Judiciary, Subcommittee on
Immigration and Claims, Aug. 5, 1999.
U.S. Immigration and Naturalization Service.
Statistical Yearbook of the Immigration and
Naturalization Service, 1996. Washington, D.C.:
U.S. Government Printing Office, 1997.
U.S. General Accounting Office. Superfund: Half
the Sites Have All Cleanup Remedies in Place or
Completed (GAO/RCED-99-245). Washington, D.C.:
GAO, July 1999.
U.S. General Accounting Office. Immigration
Statistics: Information Gaps, Quality Issues Limit
Utility of Federal Data to Policymakers (GAO/GGD-
98-164). Washington, D.C.: GAO, July 1998.
U.S. General Accounting Office. H-2A Agricultural
Guestworker Program: Changes Could Improve
Services to Employers and Better Protect Workers
(GAO/HEHS-98-20). Washington, D.C.: GAO, Dec.
1997.
U.S. General Accounting Office. Illegal
Immigration: INS Overstay Estimation Methods Need
Improvement (GAO/PEMD-95-20). Washington, D.C.:
GAO, Sept. 1995.
U.S. General Accounting Office. Immigration and
the Labor Market: Nonimmigrant Alien Workers in
the United States. (GAO/PEMD-92-17). Washington,
D.C.: GAO, Apr. 1992.
U.S. General Accounting Office. Immigration
Reform: Major Changes Likely Under S. 358.
(GAO/PEMD-90-5). Washington, D.C.: GAO, Nov. 1989.
U.S. General Accounting Office. Immigration:
Projected Immigration Under S. 448 and Recent
Trends in Legal Immigration. (GAO/PEMD-89-12).
Washington, D.C.: GAO, Apr. 1989.
U.S. General Accounting Office. Immigration: Data
Not Sufficient for Proposed Legislation. (GAO/PEMD-
89-8). Washington, D.C.: GAO, Dec. 1988.
U.S. General Accounting Office. Immigration: S.
358 Would Change the Distribution of Immigrant
Classes. (GAO/T-PEMD-89-1). Washington, D.C.: GAO,
Mar. 1989.
U.S. General Accounting Office. Immigration: The
Future Flow of Legal Immigration to the United
States. (GAO/PEMD-88-7). Washington, D.C.: GAO,
Jan. 1988.
Warner, Stanley. "Randomized Response: A Survey
Technique for Eliminating Evasive Answer Bias."
Journal of the American Statistical Association,
Vol. 60 (1965), pp. 63-69.
Warren, Robert, and Jeffrey S. Passel. "A Count of
the Uncountable: Estimates of Undocumented Aliens
Counted in the 1980 United States Census."
Demography, Vol. 24:3(1987), pp. 375-93.
Willis, Gordon B. "Cognitive Interviewing and
Questionnaire Design: A Training Manual." Office
of Research and Methodology, National Center for
Health Statistics, Hyattsville, MD., n.d.
*** End of Document ***