    Survey Methodology: An Innovative Technique for Estimating Sensitive
    Survey Items (Staff Study, 11/01/1999, GAO/GGD-00-30).

    GAO provided information on an innovative technique (called the
    "three-card method") for collecting data on sensitive policy-relevant
    topics.

    GAO noted that: (1) the three-card method is designed to collect
    sensitive data in large-scale surveys; it is intended to allow
    estimation of the needed statistics while maximizing response privacy
    and reducing "question threat"; (2) GAO originally devised this
    technique to collect data on immigration, but it believes that the
    technique might also prove useful in a variety of other sensitive policy
    areas where the collection of relevant information has, thus far, proved
    elusive; (3) a scientific survey involves a representative sample of the
    population of interest; (4) GAO's technique extends this approach to
    select three independent representative samples, each composed of
    completely different persons; (5) all persons are asked the same
    potentially sensitive question; (6) all are presented with answer
    alternatives printed on an 8 1/2 by 11 card, using a design that
    arranges the answer alternatives in different boxes on the card; (7)
    this arrangement avoids zeroing in on the sensitive answer category; (8)
    the logic of the technique involves: (a) a three-box answer format,
    which is used on each card; (b) three slightly different cards (cards 1,
    2, and 3), one for each sample; and (c) indirect estimation of the
    sensitive category; (9) no respondent is ever directly asked whether he
    or she is in the sensitive category; (10) each sample provides a
    different piece of less sensitive information--a different piece of the
    puzzle; and (11) the outlines of the missing sensitive piece are
    apparent for the population as a whole when all other pieces are in
    place.

    --------------------------- Indexing Terms -----------------------------

     REPORTNUM:  GGD-00-30
         TITLE:  Survey Methodology: An Innovative Technique for Estimating
    	     Sensitive Survey Items
          DATE:  11/01/1999
       SUBJECT:  Statistical data
    	     Data integrity
    	     Labor statistics
    	     Statistical methods
    	     Surveys

    ******************************************************************
    ** This file contains an ASCII representation of the text of a  **
    ** GAO report.  Delineations within the text indicating chapter **
    ** titles, headings, and bullets are preserved.                 **
    **                                                              **
    ** No attempt has been made to display graphic images, although **
    ** figure captions are reproduced.  Tables are included, but    **
    ** may not resemble those in the printed version.               **
    **                                                              **
    ** Please see the PDF (Portable Document Format) file, when     **
    ** available, for a complete electronic file of the printed     **
    ** document's contents.                                         **
    **                                                              **
    ** A printed copy of this report may be obtained from the GAO   **
    ** Document Distribution Center.  For further details, please   **
    ** send an e-mail message to:                                   **
    **                                                              **
    **                    <info@www.gao.gov>                        **
    **                                                              **
    ** with the message 'info' in the body.                         **
    ******************************************************************

    United States General Accounting Office
    GAO

    Staff Study

    November 1999

    GAO/GGD-00-30

    SURVEY METHODOLOGY
    An Innovative Technique for Estimating Sensitive

    Survey Items

    Ordering Information

    The first copy of each GAO report and testimony
    is free. Additional copies are $2 each. Orders
    should be sent to the following address,
    accompanied by a check or money order made out
    to the Superintendent of Documents, when
    necessary. VISA and MasterCard credit cards are
    accepted, also. Orders for 100 or more copies to
    be mailed to a single address are discounted 25
    percent.

    Order by mail:

    U.S. General Accounting Office
    P.O. Box 37050
    Washington, DC 20013

    or visit:

    Room 1100
    700 4th St. NW (corner of 4th and G Sts. NW)
    U.S. General Accounting Office
    Washington, DC

    Orders may also be placed by calling (202) 512-
    6000 or by using fax number (202) 512-6061, or
    TDD (202) 512-2537.

    Each day, GAO issues a list of newly available
    reports and testimony. To receive facsimile
    copies of the daily list or any list from the
    past 30 days, please call (202) 512-6000 using a
    touch-tone phone. A recorded menu will provide
    information on how to obtain these lists.

    For information on how to access GAO reports on
    the INTERNET, send e-mail message with "info" in
    the body to:

    info@www.gao.gov

    or visit GAO's World Wide Web Home Page at:

    http://www.gao.gov

    United States
    General Accounting Office
    Washington, D.C. 20548-0001
    Official Business
    Penalty for Private Use $300
    Address Correction Requested

                        Bulk Rate
                   Postage & Fees Paid
                           GAO
                     Permit No. G100

    (966710)

    Contents
    Page 2GAO/GGD-00-30 Estimating Sensitive Survey Items
    Preface                                                                     1

    Overview                                                                    5
                               Background                                       5
                               The Logic of the Three-Card Method               7
                               The Three-Box Answer Format                      7
                               Cards 1, 2, and 3 Yield Direct                   8
                               Estimates for Less Sensitive
                               Categories
                               The Sensitive Category                           9
                               Detailed Information                            10
                               Introductory Material                           10
                               Preliminary Testing on Legal Status             10
                               Instrument Development                          11
                               Sensitivity and Willingness to Answer           11
                               Other Potential Applications                    11
                               Using the Method in a Large-Scale               12
                               Survey
                               Organization of This Study                      13

    Chapter 1                                                                  14
    The Three-Card Method:
    Its Logic and Key
    Features
                               Answer Format, Samples, and Cards               14
                               Logic of the Basic Three-Card Method            15
                               Follow-up Questions                             21
                               Estimating Length of Stay                       22
                               Tracking Trends Across Time                     23
                               Using "Trainer" Cards Is Key                    23
                               Separately Develop Each Legal Status            27
                               Card

    Chapter 2                                                                  28
    Phases of Development
    and Testing
                               Survey Instruments                              28
                               Methodology for Phase 1: Initial                29
                               Pretests
                               Methodology for Phase 2: Field Test             30
                               With Farmworkers
                               Methodology for Phase 3: Cognitive              35
                               Interviews
                               Limitations of Results                          37

    Chapter 3                                                                  38
    Results of Preliminary
    Development and Testing
                               Results on the Three-Box Answer Format          39
                               Results on the Appropriateness of               40
                               Icons
                               Results on Legal Status Categories              41
                               Results on Sensitivity of the Legal             46
                               Status Question

    Chapter 4                                                                  48
    Estimating Other
    Sensitive Items With the
    Three-Card Method
                               Violence                                        48
                               Sensitive Personal Choices                      49
                               Business and Tax Issues                         50
                               Government Workers' Noncompliance               51

    Chapter 5                                                                  54
    Challenges and Potential
    Solutions in Using the
    Three-Card Method in a
    Large-Scale Survey
                               Interviews Using Flashcards                     54
                               Screening for the Relevant                      55
                               Subpopulation
                               Efficient Estimation                            56
                               Minimizing the Bias of Nonresponse              58
                               Issues in Surveying Special                     59
                               Populations

    Appendixes                 Appendix I: Variance of the Estimates           62
                               Appendix II: Different Forms of the             65
                               Questionnaire Used in the Field Test
                               Appendix III: Estimates for                     79
                               Demographic Subgroups
                               Appendix IV: Data From Respondents Who          80
                               Participated in the Cognitive
                               Interviews

    Bibliography                                                               81

    Tables                     Table II.1: Variations in Wording of            65
                               Survey Questions in the Nine
                               Questionnaires We Field-Tested

    Figures                    Figure 1: Illustration of Card 1 for             8
                               Estimating a Less Sensitive Category
                               Using the Three-Card Method
                               Figure 2: Illustration of Cards 2 and            9
                               3 for Estimating Less Sensitive
                               Categories Using the Three-Card
                               Method
                               Figure 1.1: Legal Status Card 1                 16
                               Figure 1.2: Legal Status Card 2                 18
                               Figure 1.3:  Legal Status Card 3                20
                               Figure 1.4: Introductory Flashcard              24
                               With Types of Houses
                               Figure 1.5: Introductory Flashcard              26
                               With Types of Transportation
                               Figure 2.1: Introductory Card With              32
                               Household Appliances
                               Figure 2.2: Introductory Card With              33
                               Modes of Transportation
                               Figure 2.3: Early Version of Legal              34
                               Status Card
                               Figure 3.1: Legal Status Card Used in           43
                               the Cognitive Interviews
                               Figure I.1: Statistical Expression of           64
                               the Three-Card Estimator, Its
                               Variance, and Technique Effect
                               Figure II.1:  Form 1 of the                     66
                               Questionnaire

    Abbreviations

    ACASI     Audiocomputer-assisted self-
    interviewing
    EWI       Entered without inspection
    INS       Immigration and Naturalization Service
    NAWS      National Agricultural Workers Survey
    OMB       Office of Management and Budget

    Preface
    Page 1GAO/GGD-00-30 Estimating Sensitive Survey It
    ems
    This staff study provides information on an
    innovative technique for collecting data on
    sensitive policy-relevant topics.1 If successful,
    this technique might eventually help fill key data
    gaps and improve statistical information relevant
    to the national decision-making process. Our main
    goal is to stimulate interest in-and where
    appropriate, encourage further development and
    testing of-this promising, but as yet not fully
    validated, technique.

    The "three-card method" is designed to collect
    sensitive data in large-scale surveys; it is
    intended to allow estimation of the needed
    statistics while maximizing response privacy and
    reducing "question threat."2 We originally devised
    this technique to collect data on immigration, but
    we believe it might also prove useful in a variety
    of other sensitive policy areas where the
    collection of relevant information has, thus far,
    proved elusive. We encourage readers to consider
    whether developing a new application of the three-
    card method might prove worthwhile.

    For questions about the material contained in this
    staff study, please contact me at (202) 512-4128
    or Judith A. Droitcour, who served as project
    director, at (202) 512-7997. Other key
    contributors to this assignment were Eric M.
    Larson, Ruth B. McKay, and Maria P. Vargas.

    Susan S. Westin
    Associate Director
    International Relations and Trade Issues

    _______________________________
    1GAO staff studies are typically prepared to
    present background information and intended to
    contribute to a specific body of knowledge.
    2"Question threat" refers to questions the
    respondent may perceive as threatening or
    incriminating. In such cases, the respondent may
    not answer the question or may provide distorted
    answers (Rossi et al., 1983; and Bradburn and
    Sudman, 1979).

    Overview
    Page 6GAO/GGD-00-30 Estimating Sensitive Survey It
    ems
    In some areas of public policy, there has been
    repeated evidence that statistics to inform key
    debates are lacking or inadequate. When a
    sensitive topic area is at issue, the reason for
    the information gap may simply be that relevant
    questions are deemed too threatening to ask in
    large-scale surveys.

    The purpose of this staff study is to report on an
    innovative questionnaire survey technique: the
    "three-card method." Though not yet fully
    validated, the new technique may prove to be
    applicable across a range of policy-relevant
    topics.

    Background
    We originally devised the three-card method as a
    way to survey foreign-born respondents about their
    immigration status. As summarized below, we were
    motivated by data gaps that are highly relevant to
    immigration policy. During the early to mid-1990s,
    immigration laws were debated and changed.1 More
    recently, proposals for revisions have continued
    to be put forward.2 But as the U.S. Commission on
    Immigration Reform recognized in 1994, without
    reliable data, it is difficult to assess the
    impact of immigration policies-or of immigration
    itself-on American society.

    Our 1998 report on the quality of immigration
    statistics (GAO/GGD-98-164, July 1998) found that
    to be relevant to current laws and debates,
    information on foreign-born residents of the
    United States should be broken out by legal
    status. Five key legal statuses3 are

    �    legal permanent residents (persons with
    official green cards);
    �    refugees and asylees (persons granted
    asylum);
    �    persons admitted temporarily who stay
    (legally) for more than a year;
    �    illegal immigrants who remain for more than a
    year;4 and
    �    naturalized U.S. citizens.

    The same 1998 report found that neither records
    from the Immigration and Naturalization Service
    (INS) nor data from current surveys provide
    reliable estimates of these legal status groups.
    Naturalized citizens (only) can be identified in
    some current surveys, although even here we found
    there may be some problems (GAO/GGD-98-164, July
    1998).5

    Surveys of the general population do provide
    information on foreign-born persons-but not by
    legal status. The reason is that questions on
    legal status have been deemed too threatening to
    ask.6 As a result, policy researchers cannot track
    trends in employment for legal immigrants, illegal
    aliens, or persons in other immigration statuses.
    Similar gaps occur for other important outcomes.7
    Immigration status is also important for policy-
    related studies of immigrant health.8 As a first
    step toward filling these gaps, we devised a new
    survey technique that was described briefly in our
    1998 report.9

    This staff study reviews the logic of the three-
    card method and describes key procedures of the
    technique. It also provides more complete
    information on our development and testing efforts
    than was previously reported, indicates
    possibilities for future applications of the three-
    card method in a variety of sensitive areas, and
    discusses challenges in fielding a national survey
    using this innovative technique.

    The Logic of the Three-Card Method
    A scientific survey involves a representative
    sample of the population of interest. Our
    technique extends this approach to select three
    independent representative samples, each composed
    of completely different persons.

    All persons are asked the same potentially
    sensitive question. All are presented with answer
    alternatives printed on an 8-1/2" by 11" card,
    using a design that arranges the answer
    alternatives in different boxes on the card. As
    explained below, this avoids zeroing in on the
    sensitive answer category. The logic of the
    technique involves

    �    a three-box answer format, which is used on
    each card;
    �    three slightly different cards (cards 1, 2,
    and 3)-one for each sample; and
    �    indirect estimation of the sensitive
    category.

    No respondent is ever directly asked whether he or
    she is in the sensitive category. Each sample
    provides a different piece of less sensitive
    information-a different piece of the puzzle. The
    outlines of the missing sensitive piece are
    apparent for the population as a whole when all
    other pieces are in place.

    The Three-Box Answer Format
    The key to asking respondents a potentially
    sensitive question, without zeroing in on the
    sensitive answer category, is the three-box
    format. For example, figure 1 shows a card that
    has three boxes: Box A, Box B, and Box C. Box A
    contains one of the less sensitive answer
    categories. Box B combines the sensitive category
    with a number of other less sensitive categories.
    Box C is all other categories-that is, any answer
    category not covered in Box A or Box B. Each box
    is an answer alternative.

    Using the example of immigration status, the boxes
    might be

    �    Box A: Legal permanent resident with a valid
    and official green card issued to me by the U.S.
    government;
    �    Box B: U.S. citizen; student, work, or
    tourist visa; undocumented10 (do not have my own
    valid official green card); and refugee or asylee
    (without a green card); and
    �    Box C: Some other category not in Box A or
    Box B (specify).

    Respondents are asked to report which Box applies
    to them. They are told that if it is Box B, we do
    not want to know which specific category applies
    to them.

    Cards 1, 2, and 3 Yield Direct Estimates for Less
    Sensitive Categories
    Respondents in sample 1 are shown a card like that
    in figure 1 (card 1). The purpose of gathering
    information with card 1 is to estimate the percent
    of persons in the first "less sensitive" category.
    Cards 2 and 3 are illustrated in figure 2. These
    cards are for samples 2 and 3, respectively.

    Comparison of the three cards shown in figures 1
    and 2 indicates that the less sensitive categories
    are rotated between Box A and Box B. When a
    category appears in Box A, direct estimation is
    possible. Thus, the percentages of the population
    in each less sensitive (Box A) category are
    estimated directly by separate samples. The
    sensitive category always remains in Box B,
    together with other less sensitive categories. The
    sensitive category is not estimated directly.

    Figure 1: Illustration of Card 1 for Estimating a
    Less Sensitive Category Using the Three-Card
    Method

    Source: GAO. (The actual size of the card is 8-
    1/2" by 11.")

    Figure 2: Illustration of Cards 2 and 3 for
    Estimating Less Sensitive Categories Using the
    Three-Card Method

    Source: GAO. (The actual size of each card is 8-
    1/2" by 11.")

    The Sensitive Category
    Although the sensitive category always appears
    together with other less sensitive categories in
    Box B, an indirect (and unbiased) estimate of the
    sensitive category can be obtained by putting
    together the various pieces of less sensitive
    information. The size of the missing piece is
    calculated by subtraction. If the categories
    listed in Box A, B, and C are mutually exclusive
    and, taken together, are exhaustive, they should
    total 100 percent. Subtracting the percentage
    estimates of the less sensitive categories from
    100 percent yields a remainder that represents an
    indirect estimate of the percentage in the
    sensitive category. (Note: This procedure is
    appropriate only when the sensitive item is not
    too rare. Attempting to obtain an indirect
    estimate of a rare category would be like trying
    to find a "needle in a haystack.")

    Detailed Information
    Detailed information on Box A categories is
    obtained with follow-up questions for only the
    respondents who choose Box A. Because the
    sensitive category is never asked about directly,
    there are no follow-up questions for those who
    choose Box B. (The purpose of the technique is to
    avoid zeroing in the sensitive category.)

    If respondents are asked other policy-relevant
    questions (e.g., questions about employment and
    income), then correlates for each less sensitive
    category may be obtained directly. The logic of
    estimating correlates of the sensitive category
    (indirectly) is explained in appendix III.
    Briefly, separate indirect estimates can be
    obtained for major (broad) subgroups defined, for
    example, by income category.

    Introductory Material
    We used introductory or training cards to
    familiarize respondents with the three-box answer
    format when we applied the technique to legal
    status. These training cards dealt with relatively
    innocuous topics. They were intended to cue
    respondents to the fact that the interviewer would
    not zero in on any specific Box B category. (See
    ch. 1 for examples of training cards.)

    The early portions of the interview-even before
    presenting respondents with the training cards-can
    focus on obtaining various kinds of general and
    potentially policy-relevant information, such as
    household size, age, employment, income, and so
    forth.

    Preliminary Testing on Legal Status
    We conducted a series of development and testing
    efforts in which the three-card method was used to
    ask foreign-born respondents about their legal
    status. These efforts consisted of three main
    phases, each of which focused on foreign-born
    Hispanic respondents: pretests (27 respondents),11
    a field test with 81 farmworkers,12 and cognitive
    interviews (8 respondents).13 We were only able to
    test in one foreign language, and we selected
    Spanish because it is the most prevalent foreign
    language in the United States. Respondents were
    selected from groups, such as farmworkers and
    those seeking help from legal clinics, that are
    thought to overrepresent illegal immigrants.

    Instrument Development
    There were iterative revisions of the survey
    instruments based on each set of results. The
    cards featured icons to illustrate the categories
    (e.g., a picture of a green card was used to
    illustrate the legal permanent resident category.)
    However, we found that some field test respondents
    and interviewers reacted negatively to the early
    version of the icons; that is, field test
    respondents perceived at least some of the icons
    as childish or indicated that some seemed
    ambiguous.14 One of the main reasons for conducting
    the subsequent cognitive interviews was to improve
    the icons. By the conclusion of the cognitive
    interviews, respondents viewed each card as
    appropriate for adults, and most thought each icon
    was recognizable without seeing its label.

    Sensitivity and Willingness to Answer
    All 116 respondents in the three phases of testing
    answered the legal status question using a three-
    box card; there were no refusals. Overall, two-
    thirds of respondents chose Box B, which contained
    the sensitive category. In the cognitive
    interviewing phase only, respondents were
    specifically asked about the sensitivity of the
    legal status question. The majority of these
    respondents (five of eight) thought the legal
    status card would be regarded as sensitive by some
    persons, even when using the three-box format;
    however, all eight respondents chose Box B.

    It is important to note that a validity study,
    conducted under conditions similar to that of a
    large-scale survey, would be needed to determine
    whether-or to what extent-respondents actually
    chose the appropriate boxes.

    Other Potential Applications
    We believe that the three-card method might be
    applied to a variety of sensitive subject areas
    other than immigration. To judge whether the
    technique might be appropriate for a specific
    question area, one should consider three
    requirements:

    Potential answers to the key question must be
    organized in a set of mutually exclusive and
    exhaustive categories. (For some applications,
    drafting an appropriate set of categories might
    require some creativity.)

    Only one of the answer categories can be
    sensitive.

    Most important, the sensitive item should not be
    rare within the population surveyed. Some items
    that appear to be rare at first glance may not be
    rare within certain definable populations. Thus,
    in some cases, one must first define a population
    or group at risk, then screen for that population,
    and finally survey its members.15

    With these considerations in mind, we brainstormed
    potential applications ranging from violence
    (e.g., road rage among young male drivers) and
    sensitive personal choices (abortion, drug use) to
    organizational actions (such as the ways
    manufacturers in key industries dispose of
    hazardous waste). Chapter 4 discusses these and
    other examples-providing possible approaches to
    wording sensitive and less sensitive categories.16

    Using the Method in a Large-Scale Survey
    Assuming that a specific application of the three-
    card method is deemed appropriate for use in a
    large-scale survey, there would be a number of
    challenges. Some of these involve efficiency. For
    example, the cards rule out the option of
    telephone interviews, and fielding a large-scale
    personal interview survey would be costly. Low-
    cost options that might be feasible in some
    instances include (1) "piggyback" insertion of
    items in an ongoing in-person survey; (2) group
    administration (e.g., a survey of high school
    students); and (3) an Internet survey, in which
    the cards are shown on the computer screen.17

    Other challenges include finding cost-effective
    ways to screen for the relevant subpopulation and
    keeping the margin of error at an acceptably low
    level (see ch. 5). Still other challenges may
    occur, depending on the context of the particular
    application. A number of special challenges
    pertain when planning a survey of the foreign-born
    population. For example, our 1998 report discussed
    the issue of possible undercoverage of the foreign-
    born in censuses and surveys and suggested ways of
    estimating the foreign-born undercount.18

    Organization of This Study
    The logic and procedures of the three-card method
    are discussed in detail in chapter 1, using the
    example of legal status for foreign-born
    respondents. The methods we used for the
    preliminary development and testing and the
    results we obtained are presented in chapters 2
    and 3. Examples of potential applications in a
    variety of sensitive topic areas are explored in
    chapter 4. Lastly, in chapter 5, challenges in
    fielding the three-card method in a national
    survey and potential solutions are discussed.

    Appendixes I through IV provide details on a
    variety of relevant technical topics. These
    include estimating the variance, procedures for
    estimating sensitive characteristics for
    demographic subgroups, the questionnaire used in
    the field test, and detailed data from the
    cognitive interviews. Lastly, we have included a
    bibliography listing the major sources we used.

    _______________________________
    1Major legislation includes the Immigration Act of
    1990 (P.L. 101-649), Violent Crime Control and Law
    Enforcement Act of 1994 (P.L. 103-322), Personal
    Responsibility and Work Opportunity Reconciliation
    Act of 1996 (P.L. 104-193), and the Illegal
    Immigration Reform and Immigrant Responsibility
    Act of 1996 (P.L. 104-208).
    2Bills introduced in the 106th Congress that would
    amend current immigration law include H.R. 1399,
    Fairness for Legal Immigrants Act of 1999; S.
    1227, Immigrant Children's Health Improvement Act
    of 1999; H.R. 2698, New Workers for Economic
    Growth Act; and S. 455, Nursing Relief for
    Disadvantaged Areas Act of 1999. Other proposals
    have involved rules for the admission of temporary
    workers, including temporary agricultural workers
    (see GAO/HEHS-98-20, Dec. 1997) and temporary
    skilled workers (see Hearing before the House
    Subcommittee on Immigration and Claims, Committee
    on the Judiciary, Aug. 5, 1999).
    3Virtually all laws regarding foreign-born persons
    apply to these five major legal statuses (or to
    subcategories of these statuses). Legal categories
    of foreign-born persons are defined under the
    Immigration and Nationality Act of 1952, as
    amended (8 U.S.C. 1101 et seq.). The terms and
    length of their admission to the United States,
    entitlements, rights, and benefits are addressed
    in the act and other legislation.
    4Illegal immigrants include persons who entered
    without inspection (EWI)-that is, entered the
    United States surreptitiously-and overstays.
    Overstays (also called nonimmigrant overstays and
    visa overstays) are persons who entered the United
    States legally for a temporary period, but
    illegally stayed beyond the required departure
    date.
    5INS has provided some composite estimates of the
    size of certain other legal status groups, but
    these estimates rely on major assumptions and may
    not be statistically reliable.
    6For other problems with census and survey data on
    the foreign-born, including undercount and
    nonresponse, see GAO/GGD-98-164, July 1998.
    7Other important outcomes that follow legal and
    illegal immigrants' entry into the United States
    include educational attainments, poverty status,
    and family formation.
    8The implementation of the Personal Responsibility
    and Work Opportunity Reconciliation Act of 1996
    (P.L. 104-193) and the Health Insurance
    Portability and Accountability Act of 1996 (P.L.
    104-191) requires the reassessment of national
    survey questions that obtain information on
    citizenship status, as well as welfare programs
    and health insurance coverage (Loue and Bunde,
    1999).
    9Earlier indirect survey-based estimation
    techniques, including "randomized response"
    (Warner, 1965) and the "item count" (Droitcour et
    al., 1991), seemed inappropriate for the foreign-
    born population or for asking about legal status.
    The three-card method builds on these earlier
    survey techniques as well as demographic methods
    of residual estimation (see Schryock and Siegel
    and Associates, 1980).
    10Undocumented refers to illegal immigrants. This
    is the sensitive category.
    11By "pretesting" we mean trying out questions on a
    relatively small number of respondents and making
    corrections and revisions on the basis of problems
    that surface in those interviews. Pretests include
    (1) asking the interview questions and (2) asking
    respondents about problems they had in
    understanding the questions or in answering them.
    12A field test is a small-scale preliminary survey,
    typically conducted under conditions as similar as
    possible to those to be used in an eventual full-
    scale survey.
    13Cognitive interviewing focuses on the mental
    processes of the respondent while he or she is
    answering survey questions. (See ch. 2 for a
    fuller discussion of this method.)
    14Icons are drawings we used to illustrate the
    categories in Box A and Box B, as shown in ch. 1,
    figs. 1.1-1.3. The U.S. citizen category, e.g., is
    accompanied by a drawing of a U.S. flag.
    15Otherwise, the indirect-estimation effort could
    amount to "looking for a needle in a haystack."
    (See app. I for a discussion of the variance of
    the indirect estimate.)
    16When choosing these examples, we tried to keep
    all three considerations in mind. However, it was
    not always clear to us how prevalent or rare a
    particular sensitive behavior might be within a
    defined population.
    17Some surveys are already conducted entirely on
    the Internet; e.g., we used e-mail to administer a
    questionnaire to some 350 EPA cleanup managers
    responsible for the 609 sites where cleanups are
    still under way (see GAO/RECD-99-245, July 1999).
    We also reported the results of the survey for
    each site on the Internet at
    http//www.gao.gov/RCED-99-245/.
    18See GAO/GGD-98-164, July 1998, pp. 57-58.

    Chapter 1
    The Three-Card Method: Its Logic and Key Features
    Page 15GAO/GGD-00-30 Estimating Sensitive Survey I
    tems
    The three-card method is designed to encourage
    more truthful responses to sensitive questions
    while protecting respondent privacy. In its first
    application, the technique was used to ask foreign-
    born respondents about their legal status in such
    a way that a truthful response would not reveal
    whether the respondent (or anyone else) is here
    illegally.1

    No one could ever discover, on the basis of these
    survey data-not even through a series of
    deductions-that any respondent is an illegal
    immigrant. But when all data are combined, survey-
    based estimates can be achieved for all legal
    status groups-including the sensitive illegal
    immigrant category. The logic of the three-card
    method involves

    �    a three-box answer format,
    �    selection of three random samples of foreign-
    born persons, and
    �    three different cards-one for each sample.

    Key features of the method include

    �    the ability to ask some respondents detailed
    follow-up questions to estimate length of stay and
    to obtain trend data;
    �    introductory "trainer" cards, which
    familiarize respondents with the three-box answer
    format and with the use of icons to identify
    specific categories; and
    �    the inclusion of different details on
    different legal status cards, to maximize
    effective and efficient communication.

    Answer Format, Samples, and Cards
    The "three-box answer format" groups various
    answer categories in three boxes (Box A, Box B,
    and Box C). The respondent answers by picking one
    of the boxes. Categories are arranged so that the
    sensitive category (here, illegal or
    "undocumented" status) appears in a box that
    includes several less sensitive categories (such
    as refugee or asylee, naturalized citizen, etc.).2
    Icons are used to help identify the various legal
    status categories.

    Each of the three samples consists of different
    foreign-born persons. The three legal status cards
    differ only in terms of which categories are
    grouped in which box.3 Each card is shown to
    respondents in one sample only. Thus, each
    respondent sees only one legal status card.

    The key to the new method is that respondents in
    each sample provide partial information (a
    different "piece of the puzzle"). By accessing
    data from all three samples (all three legal
    status cards), direct estimates can be made for
    most categories. The sensitive category is
    estimated indirectly, utilizing data from all
    three samples. The following section describes how
    this is accomplished.

    Logic of the Basic Three-Card Method
    The logic of the basic three-card method can be
    explained using a hypothetical example that is
    carried through to explain each of three samples.
    Respondents in the first sample of foreign-born
    respondents are shown a legal status card (card 1)
    with categories arranged in a three-box format, as
    shown in figure 1.1.4 The card is shown in Spanish
    because all testing to date has been conducted in
    Spanish. The English translation of card 1 is

    �    Box A: Legal permanent resident with a valid
    and official green card issued to me by the U.S.
    government;
    �    Box B: U.S. citizen; student, work, or
    tourist visa; undocumented5 (do not have my own
    valid official green card); and refugee or asylee
    (without a green card); and
    �    Box C: Some other category not in Box A or
    Box B (specify).

    Respondents are asked to report which box applies
    to them and are told that if it is Box B, we do
    not want to know which specific category applies
    to them. The purpose of interviews with this first
    sample is to obtain a valid estimate of the
    percent of foreign-born who have officially
    obtained green cards. (Hypothetical example: 35
    percent of the resident foreign-born population
    have officially obtained green cards.)

    Figure 1.1: Legal Status Card 1

    Source: GAO. (The actual size of the card is 8-
    1/2" by 11.")

    Respondents in the second sample (completely
    different individuals) are shown a legal status
    card (card 2) with a different arrangement of the
    categories in the same three-box format (see fig.
    1.2). This time, Box A contains the status of a
    naturalized U.S. citizen, whereas Box B includes
    legal permanent resident along with other
    immigration statuses. Respondents are asked to
    pick the box that applies to them and are told
    that if it is Box B, we do not want to know which
    category in Box B applies to them. The purpose of
    interviewing this sample with this card is to
    obtain a valid estimate of the percentage of the
    foreign-born who are naturalized citizens.
    (Hypothetical example: 30 percent are naturalized
    citizens.)

    Figure 1.2: Legal Status Card 2

    Source: GAO. (The actual size of card is 8-1/2" by
    11.")

    Respondents in the third sample (again, different
    persons) are shown a legal status card (card 3)
    with yet another arrangement of the categories in
    the three-box format (see fig. 1.3). This time,
    Box A features refugees and persons granted asylum
    as well as those here legally with temporary
    visas.6 Respondents are told that if they are in
    Box B, we do not want to know which category in
    Box B applies to them. The purpose of interviewing
    this sample is to get an estimate of the
    percentage in Box A categories-refugees, asylees,
    and persons here on a legal temporary basis.
    (Hypothetical example: 10 percent pick Box A.)

    Figure 1.3:  Legal Status Card 3

    Source: GAO. (The actual size of card is 8-1/2" by
    11.")

    Thus, each of the legal status categories-except
    for the sensitive category (illegal
    status)-alternatively appears in Box A. Assuming
    that the estimates for the legal status categories
    are mutually exclusive and that these categories
    together with Box C represent an exhaustive set of
    possible legal statuses, it is possible to obtain
    an indirect estimate of illegal immigrants. That
    is, extending the hypothetical examples above, we
    would estimate that 75 percent of the foreign-born
    are here legally in the four major statuses (35%
    with green card + 30% naturalized U.S. citizens +
    10% refugees, asylees, and persons here on a legal
    temporary basis7 = 75%). Suppose also that 1
    percent picked Box C (some other category).
    Subtracting these hypothetical estimates from 100
    percent yields 24 percent (100% - 75% - 1% = 24%).
    Thus, our estimate would be that 24 percent are
    here illegally.

    The three-card method also allows detailed
    information to be gathered on subcategories of
    legal statuses. As described in the following
    section, this is accomplished through a special
    strategy for follow-up questions that is designed
    to avoid increasing question threat.8

    Follow-up Questions
    Respondents who choose Box B on the legal status
    card shown to them are not asked any further
    questions because Box B always contains the
    sensitive category. However, respondents who
    choose Box A can be asked a set of follow-up
    questions about the details of their (Box A) legal
    status.9

    Follow-up questions are essential for sample 3
    respondents who choose box A, because these
    respondents are shown legal status card 3, which
    includes two categories in Box A. Follow-up
    questions can determine which of the two legal
    status categories applies to the respondent.

    Follow-up questions might also help confirm the
    validity of answers for respondents choosing Box A
    or, alternatively, provide the information needed
    to reclassify certain respondents into Box B or
    Box C. For example, suppose a person in sample 1
    chooses Box A, apparently claiming to have a green
    card. If follow-up questions reveal that although
    the respondent had applied for green card status,
    he or she has not actually received that status
    yet, then this sample 1 respondent would be
    recoded out of Box A. Specific follow-up questions
    used for legal status card 1 are shown in appendix
    II.10

    For sample 3 (legal status card 3), follow-up
    questions could help clarify whether a respondent
    chose Box A incorrectly.11

    Estimating Length of Stay
    We defined two of the five major legal status
    categories of foreign-born persons, in part, by
    length of stay.12 The reason is that they reflect
    temporary admissions or illegal presence in the
    United States, and these conditions are different
    from an admission that is intended to be
    permanent, such as for legal permanent residence.
    The two length-of-stay statuses are (1) persons
    who were legally admitted here on a temporary
    basis who remain (legally) longer than 1 year and
    (2) persons here illegally who remain longer than
    1 year.

    We based the foregoing length-of-stay criterion on
    the U.N.-recommended definition of "permanent"
    immigration and on the Immigration and
    Naturalization Service's (INS) practices regarding
    its overstay estimates.13

    Estimates of foreign-born persons here for any
    time period can be made by isolating the data for
    foreign-born respondents who have been here for a
    specific length of time, and then using these data
    to derive the relevant estimate. This is
    especially important for two categories-illegals
    and persons here on temporary visas.14

    Estimates for demographic subgroups can be
    obtained in a similar manner (see app. III).

    Tracking Trends Across Time
    Through repeatedly applying the three-card method
    in subsequent surveys, it would be possible to
    track trends across time for various legal status
    groups. For example, trends in the number of
    foreign-born persons here on temporary visas-or
    the number illegally residing here-could be
    tracked for all such persons or separately for
    males and for females or for other major
    demographic groups (e.g., Mexican origin). To
    illustrate, the analyst would first isolate data
    on Mexican-born respondents, then follow the
    estimation and subtraction process previously
    described.

    Using "Trainer" Cards Is Key
    Introductory or "trainer" cards are a key feature
    of the three-card method. Before showing a
    respondent the legal status card assigned to his
    or her sample, the interviewer asks questions on
    less sensitive topics using cards that also have
    words, icons, and boxes. The purpose of these
    introductory trainer cards is to familiarize
    respondents with the general format-including the
    task of answering by choosing a box-before they
    are shown the legal status card. Figures 1.4 and
    1.5 show introductory cards, each of which has a
    three-box format and icons to help identify each
    category.

    The first introductory card (fig. 1.4) shows
    different categories of houses or dwellings. Box A
    shows a farmhouse, whereas Box B includes various
    categories-apartments, single-family houses, and
    so forth. Box C refers to other types of housing
    (not shown in Box A or Box B). This card is used
    with a question such as: "In what kind of house do
    you think most of the people in your home country
    live?" Before respondents actually voice an
    answer, they are instructed to answer by just
    picking a box-and are also told that if the answer
    is Box B, we do not want to know which specific
    category applies to them.

    Figure 1.4: Introductory Flashcard With Types of
    Houses

    Source: GAO. (The actual size of card is 8-1/2" by
    11.")

    The second introductory card (fig. 1.5) also has
    three boxes. It presents various modes of
    transportation: Box A contains a boat; Box B
    contains four other modes of transportation-plane,
    train, auto, or on foot; and Box C refers to any
    type of transportation not shown in either Box A
    or Box B. This card is used together with
    questions such as: "What kind of transportation
    did you use the most recent time you traveled from
    your home country to the United States?" Or, "If
    you were to travel to your home country sometime
    within the next 12 months, what kind of
    transportation do you think you would use?" Again,
    before the respondent answers, he or she is asked
    to just pick a box and is told that if the answer
    is in Box B, we do not want to know which specific
    category applies.

    Figure 1.5: Introductory Flashcard With Types of
    Transportation

    Source: GAO. (The actual size of card is 8-1/2" by
    11.")

    We recommend using both of these introductory
    cards so that respondents will be thoroughly
    familiar with the three-box format before they are
    shown a legal status card. The logic is that
    respondents can freely ask questions about how the
    format works on the more innocuous introductory
    trainer cards and will know, in advance of seeing
    the legal status card, that they will not have to
    identify a specific category in Box B.

    Separately Develop Each Legal Status Card
    Each legal status card must be separately
    developed and tested before being used in a
    survey. It may be necessary to develop specific
    instructions for each card to enable a better
    presentation by the interviewer and to facilitate
    more accurate answers by the respondent. In such a
    case, we believe only those detailed instructions
    needed for a particular card should be included on
    that card. The reason is that instructions
    relevant to other cards would clutter the card and
    make it more difficult to take in at a glance.

    Because Box A of each card features a different
    category (or categories), the cards differ in
    terms of the key instructions and explanatory
    details needed to help respondents determine
    whether they belong in Box A, Box B, or Box C. It
    is not necessary for respondents to distinguish
    between categories listed within Box B.

    For example, a respondent with an expired visa who
    is shown legal status card 1 should know that he
    or she belongs in Box B. The reason is that on
    legal status card 1, Box B contains both the
    temporary visa category and the undocumented
    category. The respondent need only choose Box B;
    he or she does not have to distinguish between
    categories within Box B. But a similar respondent
    in sample 3 would be shown legal status card 3,
    which includes the temporary visa category in Box
    A and the undocumented category in Box B. A
    respondent with an expired visa might be unsure
    which box to choose-unless legal status card 3 has
    language making it clear that persons with expired
    visas belong in Box B.

    It is important to note that all three legal
    status cards are shown in here to illustrate the
    logic of the three-card method. However, only one
    of those cards-legal status card 1-was subjected
    to development and testing.15

    _______________________________
    1This extends our earlier work on the three-card
    method, which was introduced in our report on the
    quality of immigration statistics available to
    policymakers (GAO/GGD-98-164, Jul. 1998).  Our
    other work in this area includes evaluating the
    quality of data available to evaluate the effects
    of immigration (GAO/PEMD-89-8, Dec. 1988),
    projecting future legal immigration (GAO/PEMD-88-
    7, Jan. 1988; GAO/T-PEMD-89-1, Mar. 1989; GAO/PEMD-
    89-12, Apr. 1989; GAO/PEMD-90-5, Nov. 1989),
    nonimmigrant alien workers in the United States
    (GAO/PEMD-92-17, Apr. 1992), and problems with
    INS' overstay estimation methods (GAO/PEMD-95-20,
    Sept. 1995).
    2Each sample is shown only one of these three-box
    legal status cards. The categories on the cards
    must be mutually exclusive and exhaustive (i.e.,
    each person belongs in one and only one category).
    3Each of the three legal status cards must be
    individually developed and pretested, as explained
    later in this chapter.
    4The legal status card shown in fig.1.1 is a
    revised version we developed after the 1998 report
    was released. The various development and testing
    efforts completed to date are discussed in ch. 2.
    5Undocumented refers to illegal immigrants. This
    is the sensitive category.
    6Using two categories in Box A lowers variance
    costs because three, rather than four, samples
    would be required. Those who pick Box A would be
    asked follow-up questions to determine their exact
    legal status, which is also the case with the
    other two legal status cards.
    7Refugees, asylees, and persons here on a legal
    temporary basis could be further disaggregated by
    using follow-up questions to determine their
    status.
    8"Question threat" refers to questions the
    respondent may perceive as threatening or
    incriminating. In such cases, the respondent may
    not answer the question or may provide distorted
    answers (Rossi et al., 1983; and Bradburn and
    Sudman, 1979).
    9This could include, for example, a question about
    when the Box A legal status was officially
    obtained (e.g., green card status) and the basis
    upon which the respondent applied for the green
    card (e.g., as the spouse of a citizen, relative
    of a permanent resident alien, and so forth). In
    this way, detailed information on legal status can
    be attained for most legal status subcategories.
    10However, as described in ch. 3, in a field test
    with 81 respondents, no one was reclassified on
    the basis of these follow-up questions.
    11For example, some persons who entered as refugees
    may still self-identify generally as refugees even
    if they later obtained green cards or became
    naturalized U.S. citizens. Such errors could, in
    some cases, then be recoded to the correct box
    (B).
    12The other three are legal permanent residents,
    refugees and asylees, and naturalized U.S.
    citizens. The term "asylees" refers to persons
    granted asylum, without legal permanent resident
    status.
    13The U.N. definition (Shryock and Siegel and
    Associates, 1980), has been applied to INS data on
    arrivals of foreign-born persons to the United
    States (Kraly and Warren, 1992); also, INS'
    definition of overstays counts only those who
    resided illegally in the United States for more
    than 1 year.
    14Some foreign-born workers who are here on
    temporary visas are currently of particular policy
    interest, in part because of their length of stay.
    (See GAO/PEMD-92-17, Apr. 1992, and "Hearing on
    the H-1B Temporary Professional Worker Visa
    Program and Information Technology Workforce
    Issues," hearing before the Committee on the
    Judiciary, Subcommittee on Immigration and Claims,
    House of Representatives, Aug. 5, 1999.)
    15As described in the following chapter, the
    developmental testing consisted of three phases:
    initial pretesting by our staff who are fluent in
    Spanish, a field test, and formal cognitive
    interviewing. Results are presented in ch. 3.

    Chapter 2
    Phases of Development and Testing
    Page 29GAO/GGD-00-30 Estimating Sensitive Survey I
    tems
    Our preliminary development and testing effort was
    conducted in three phases:1 (1) initial pretesting
    of instruments, (2) a field test with farmworkers,
    and (3) subsequent cognitive interviews.2 This
    three-phase effort covered two introductory cards
    as well as legal status card 1. The third phase
    was designed, in large part, to address problems
    that surfaced in the field test.

    Each phase built on the results of the previous
    phase. That is, on the basis of phase 1 results,
    instruments were revised before beginning phase 2;
    and on the basis of phase 2, instruments were
    revised at the outset of phase 3.3 Revisions were
    also made during phase 3.

    In each phase, foreign-born Hispanic respondents
    were selected in such a way that illegal residents
    would likely be heavily represented. Generally,
    Hispanics in professional and semiprofessional
    occupations were not included, and it is likely
    that long-term Hispanic residents were
    underrepresented. We did not attempt to interview
    non-Hispanic foreign-born residents (Asian,
    African, or European).

    The end of this chapter describes important
    limitations on the results of this effort.

    Survey Instruments
    Throughout the development and testing of the
    three-card method, the following survey
    instruments were used:

    �    introductory trainer cards intended to
    familiarize respondents with icons and train them
    in the use of the three-box answer format before
    exposing them to a legal status card and
    �    legal status card 1, in which Box A features
    the legal permanent resident category.4

    All respondents also answered a lead-in
    questionnaire that obtained demographic
    information and provided context for the specific
    questions and cards we were testing. Additionally,
    the lead-in questionnaire may help establish
    rapport and motivate respondents to provide
    accurate answers on the legal status card.5

    Respondents who picked Box A on legal status card
    1 (i.e., those who claimed legal permanent
    resident or green card status) were also asked
    follow-up questions concerning details of their
    legal status.6

    At various points during our work on the three-
    card method, we asked experts to review legal
    status card 1. These included GAO colleagues;
    officials and staff at the Immigration and
    Naturalization Service (INS), the Bureau of the
    Census, and the Office of Refugee Resettlement at
    the Department of Health and Human Services; and
    private-sector experts in immigration, statistics,
    and survey methods.7 Revisions were made, as
    needed.

    All survey materials were available in Spanish.
    All interviewers were fluent in both Spanish and
    English, and almost all interviews were conducted
    in Spanish.8

    Methodology for Phase 1: Initial Pretests
    Phase 1 of the development and testing consisted
    of a series of pretests and iterative revision of
    survey instruments. Four bilingual GAO staff
    members conducted 27 pretest interviews with
    foreign-born Hispanics at four locations. These
    included a Los Angeles "drop-in" center frequented
    by young Hispanic males and operated by a
    charitable Catholic group; a legal aid clinic in
    Arlington, Virginia, that specializes in helping
    Hispanic immigrants; and various facilities in
    farm areas in Colorado and Pennsylvania during the
    harvest of labor-intensive crops, such as apples
    and peaches. Most of the pretest respondents were
    from Mexico, and most were males.

    At each location, we first met with and explained
    our project to directors or owners of the facility
    we wished to enter. Subsequently, we introduced
    ourselves to prospective respondents (foreign-born
    Hispanics) and gained their cooperation. The
    support of the directors or owners was critical
    for gaining access to the facilities, and in some
    cases, but not all, the directors or owners also
    served as "opinion leaders," who validated our
    activities to prospective respondents.

    Methodology for Phase 2: Field Test With
    Farmworkers
    Phase 2, following the initial pretests, consisted
    of a field test in which a contractor (Aguirre
    International) conducted survey interviews with 81
    farmworkers in six states. As a first step, the
    questionnaires and cards we developed were
    submitted to the contractor for review. The
    contractor suggested minor changes, and we revised
    the questionnaires as needed.9 The interviewers
    employed by the contractor were subsequently
    debriefed by our staff.

    The contractor conducted our 81 interviews as an
    add-on to the National Agricultural Workers Survey
    (NAWS).10 The NAWS is an annual survey, and the
    Aguirre interviewers are experienced in developing
    rapport and communicating with Hispanic
    farmworkers.

    We chose the NAWS population because it contains a
    high percentage of foreign-born workers, almost
    all foreign-born respondents are Hispanic and
    speak Spanish, and a high percentage are believed
    to be illegal immigrants. Specifically, our
    expectation was that most foreign-born farmworkers
    would either be working here illegally or would
    have obtained, as a result of the amnesty period
    authorized by the Immigration Reform and Control
    Act of 1986, a valid green card.11 The majority of
    farmworkers were male, and nearly all were from
    Mexico.

    The cards used to interview 81 Hispanic
    farmworkers were as follows:

    �    The first introductory card featured
    household appliances (see fig. 2.1) and was
    intended to familiarize respondents with the use
    of cards showing words and pictures.12
    �    The second introductory card featured types
    of transportation shown in a three-box format
    (fig. 2.2, which is the same as the card shown in
    ch. 1, fig. 1.5).
    �    Legal status card 1 featured the legal
    permanent resident category in Box A-specifically,
    the early version of that card (see fig. 2.3).13

    Figure 2.1: Introductory Card With Household
    Appliances

    Source: GAO. (The actual size of card is 8-1/2" by
    11.")

    Figure 2.2: Introductory Card With Modes of
    Transportation

    Source: GAO. (The actual size of card is 8-1/2" by
    11.")

    Figure 2.3: Early Version of Legal Status Card

    Source: GAO. (The actual size of card is 8-1/2" by
    11.")

    The contractor field-tested these cards and
    related questions from November 1997 to February
    1998.14 The 81 interviews by four interviewers were
    conducted at multiple sites in six states
    (Arizona, California, Florida, Georgia, Kentucky,
    and Texas). All respondents were interviewed in
    Spanish. The final portion of the field test
    consisted of our debriefing the Aguirre
    interviewers.

    The reaction of some respondents to specific icons
    suggested there was a need for focus groups or
    other qualitative work aimed at evaluating and
    possibly revising the icons. For example,
    interviewers reported that some respondents
    thought the icons were childish and unnecessary,
    and even that the card was unnecessary because
    they had understood the verbal instructions and
    felt that they did not need help.

    Methodology for Phase 3: Cognitive Interviews
    Phase 3 consisted of two sets of formal cognitive
    interviews with small samples of respondents.
    Cognitive interviewing is a method that focuses on
    the mental processes of the respondent while he or
    she is answering the survey questions.15 The
    cognitive model asks

    �    What does the respondent think the question
    is asking?
    �    What do specific words and phrases in the
    question mean to the respondent?

    For the three-card research, the model also asks

    �    What do the icons in the survey material mean
    to the respondent?

    A variety of methods are used in cognitive
    interviews. The research protocol we developed
    included scripted retrospective probes (questions)
    and a vignette. In the former, the respondent is
    asked standardized questions about the difficulty
    and sensitivity of survey items in a debriefing
    session following completion of the formal
    questionnaire. The vignette is designed to provide
    information about the respondent's decision-making
    process in making a judgment about a hypothetical
    situation.

    The respondent is asked several scripted questions
    during a debriefing at the conclusion of the
    survey questionnaire. Three questions were
    developed for each icon, as follows. First, the
    interviewer asked: "Do you think that anyone would
    have difficulty recognizing the (first, second,
    etc.) category in (Box A, B)?" Next, the
    interviewer covered the written label for each
    icon on the card and asked, "What does this icon
    mean to you?"16 Lastly, if the icon did not convey
    the intended meaning, the interviewer asked, "How
    can we change the icon to make the intended
    meaning more clear?"

    Cognitive interview research is an iterative
    process in which the findings on problems
    identified in each set of interviews are used to
    modify the questions to be tested in the next set
    of interviews. Because each research interview
    provides a great deal of information about the
    mental processes utilized by the respondent, a
    relatively small number of cognitive interviews is
    required in this type of research.

    One of our pretest interviewers, who is bilingual
    in English and Spanish, conducted the cognitive
    interviews after training by another of our staff,
    who is a cognitive interviewing expert with
    knowledge of Spanish and who also participated in
    the actual interviews. The cognitive interviews
    were conducted in two sets: The first four
    respondents were adult males from El Salvador,
    Bolivia, Mexico, and the Dominican Republic.17
    Following these interviews, materials were
    revised. The second set of four cognitive
    interviews, using the revised survey materials,
    was conducted with four additional respondents in
    the same location. Two of these respondents, one
    female and one male, were Salvadoran; the other
    two, one female and one male, were from Peru and
    Nicaragua, respectively.

    All of these interviews were conducted at an
    employment and training facility sponsored by a
    local, church-based organization (Hispanic
    Outreach Center). The center has a reputation in
    the community as being supportive of immigrants,
    offering legal and material assistance (e.g.,
    food) to local residents.

    Limitations of Results
    The results of this work are, by definition,
    limited to the Hispanic groups we interviewed.18 We
    also note that it is particularly difficult to
    generalize about respondent perceptions of
    question sensitivity, as such perceptions
    logically depend on many variables. (These would
    include the setting of the interview, the level of
    trust inspired by the particular interviewer, the
    social or political climate prevailing at the time
    a survey is conducted, and possibly other
    factors.) Finally, levels of respondent
    comprehension, acceptance, and sensitivity all
    depend on the specific instruments used, and it is
    possible that future developmental work can yield
    improvement. Moreover, results may vary depending
    on whether the method is used to estimate legal
    status among foreign-born persons or whether it is
    used to estimate a different sensitive item in a
    different population.

    For all these reasons, as stated in our 1998
    report (GAO/GGD-98-164), additional development
    and testing would be needed before using the three-
    card method in a large-scale national survey.

    _______________________________
    1The development and testing was conducted between
    July 1997 and September 1998: initial pretesting
    of instruments (Jul.-Aug., 1997), application in a
    field test with farmworkers (Nov. 1997-Feb. 1998),
    and cognitive interviewing (Sept. 1998).
    2Cognitive interviewing involves examining mental
    processes of respondents as they answer key
    questions; i.e., questioning them about what the
    question meant to them and about how they arrived
    at their answer.
    3Problems clearly observable to the interviewers,
    given the normal question asking-and-answering
    process, were identified in the field test.
    4We did not test the alternative legal status
    cards-i.e., legal status cards 2 and 3, which show
    different categories in Box A. (See figs. 1.2 and
    1.3 in ch. 1.)
    5In the field test with farmworkers, the lead-in
    questionnaire covered work-related health issues,
    such as injuries at work, use of pesticides in the
    fields, drinking water, etc.
    6A copy of the three-card segment of the
    questionnaire used in the field test is shown in
    app. II.
    7Our nonagency reviewers included included Norman
    Bradburn, Senior Vice President for Research,
    National Opinion Research Center, Chicago, IL;
    Robert Groves, Director, Joint Program for Survey
    Methodology, University of Maryland and University
    of Michigan; Professor David M. Heer, Population
    Research Laboratory, University of Southern
    California, Los Angeles, CA; Graham Kalton, Senior
    Vice President, Westat, Inc., Rockville, MD;
    Charles B. Keely, Chairman, Department of
    Demography, Georgetown University, Washington,
    D.C.; Mary Grace Kovar, National Opinion Research
    Center, Washington, D.C.; Demetrios G.
    Papademetriou, Senior Associate and Director,
    International Migration Policy Program, Carnegie
    Endowment for International Peace, Washington,
    D.C.; Jeffrey S. Passel, The Urban Institute,
    Washington, D.C.; Donald B. Rubin, Chairman,
    Department of Statistics, Harvard University,
    Cambridge, MA; Fritz Scheuren, The Urban
    Institute, Washington, D.C.; Michael S.
    Teitelbaum, Co-Chair, U.S. Commission on
    Immigration Reform; and Professor Alan Zaslavsky,
    Department of Health Care Policy, Harvard Medical
    School, Cambridge, MA.
    8One respondent requested to be interviewed in
    English; although Spanish was his native language,
    he wanted to practice his English.
    9Nine forms of this questionnaire were developed
    (see app. II), so that each could be administered
    to nine respondents.
    10Data from our interviews were not included as
    part of the regular NAWS survey data.
    11The Immigration Reform and Control Act was passed
    Nov. 6, 1986, in order to control and deter
    illegal immigration to the United States. Its
    major provisions stipulate legalization of
    undocumented aliens, legalization of certain
    agricultural workers, sanctions for employers who
    knowingly hire undocumented workers, and increased
    enforcement at U.S. borders.
    12We eliminated this card following the field test.
    13We modified this card in the cognitive testing
    phase, which addressed problems with the icons
    that were encountered by the contractor during the
    field test. The most recent version is shown in
    ch. 1.
    14Nine slightly different forms of the three-card
    question series were tested. The contractor tested
    each form using nine different respondents-for a
    total of 81 interviews. Consistent with the NAWS
    procedures, farmworker respondents were paid $10
    cash as an incentive to participate.
    15The source materials upon which we based our work
    include DeMaio, et al., 1993; Polivka and Rothgeb,
    1993; and Willis, n.d.
    16The need for icons that convey the conceptual
    categories without written labels is because not
    all Hispanic adults are literate in Spanish.
    Results of the March 1995 Current Population
    Survey showed that about 11 percent of Hispanics
    age 25 or older had less than a 5th grade
    education.
    17App. IV contains a chart that includes
    demographic characteristics of the Hispanic
    respondents interviewed at the suburban Maryland
    Employment Center between Sept. 1 and 3, 1998.
    18As noted at the outset of this chapter, we did
    not attempt to cover Asian, African, or European
    immigrants. Moreover, the Hispanics we interviewed
    generally did not include professionals or
    semiprofessionals, and they probably
    overrepresented new arrivals and undocumented
    workers.

    Chapter 3
    Results of Preliminary Development and Testing
    Page 46GAO/GGD-00-30 Estimating Sensitive Survey I
    tems
    As explained in the previous chapter, our
    development and testing effort consisted of three
    forms of research-pretests, a field test with
    farmworkers, and subsequent cognitive interviewing
    to correct problems identified in the field test.
    We recognize that these forms of research cannot
    indicate whether respondents provide accurate
    answers. A validity study would be needed for such
    an assessment, and none has been conducted.1
    However, preliminary development and testing has
    provided some relevant information.

    Logically, the three-card method's potential to
    elicit accurate responses on legal status depends
    on whether respondents

    �    comprehend and accept the three-box format
    for answering a question;
    �    generally view the icons as appropriate
    (because we believe the icons are essential aids
    to some respondents' comprehension);
    �    comprehend the various legal status
    categories, as represented by words and icons; and
    �    perceive the three-box question on legal
    status as sensitive-and if so, whether they are
    nevertheless willing to answer the question and,
    indeed, choose a box containing the sensitive
    item.2

    Our preliminary development and testing has
    provided some information on each of these four
    points.

    As detailed below, most respondents appeared to
    comprehend the three-box format for answering; and
    they also appeared to accept this answer format at
    face value. Respondents found the final version of
    the icons to be appropriate for adults. With
    respect to comprehension of the legal status
    categories and related icons, the final version of
    the instruments seemed to communicate
    effectively-although some further improvement of
    specific icons may be possible.

    Over the three phases of research, perceptions of
    the sensitivity of the legal status question were
    mixed, but no respondent refused to answer it, and
    two-thirds picked Box B (which includes the
    sensitive item). In the cognitive interviewing
    phase, five of eight respondents identified the
    legal status question as sensitive but all picked
    Box B. Based on these results, we believe that
    further development and testing efforts are
    justified.

    Results on the Three-Box Answer Format
    In the pretests and the field test with
    farmworkers, the three-box format was used for the
    second introductory card (modes of transportation)
    and for legal status card 1. In the third phase,
    the three-box format was used for both
    introductory cards and for legal status card 1.

    In the initial pretests, most respondents appeared
    to understand the three-box answer format. But two
    needed more specific instructions. One respondent,
    in particular, alerted us to the need for improved
    instructions: When using the transportation card,
    she did not seem to understand the logic of the
    boxes. We therefore revised the instructions for
    the three-box transportation card to more
    specifically describe the logic of the three-box
    format and to emphasize to the respondent that we
    were interested only in which box he or she was
    in-not any specific category. We also added
    assurances, just before asking about legal status
    card 1, that our purpose was not to identify any
    individual person who may be an illegal immigrant.

    Subsequently, in another pretest location, two
    respondents, who were administered the
    questionnaire together, appeared to understand the
    logic of the three-box cards, but falsely claimed
    to be in Box A of legal status card 1. Box A
    contains the green card category, and in
    debriefing, both respondents admitted to being
    illegal aliens. They explained that they had
    picked Box A because they wanted to have a green
    card. We therefore revised the instructions for
    the legal status card by adding the following
    statement: "We are only interested in which BOX
    you are in right now-and not a box you want to be
    in, or have applied to be in, in the future.
    So . . . are you actually in Box A, B, or C right
    now?" The revised wording also emphasized (even
    more strongly than before) that we did not intend
    to identify any particular category in Box B.3

    In the field test, no problems surfaced with
    respect to comprehension of the instructions for
    the three-box answer format (phase 2). That is,
    interviewers did not report problems of this type
    in the debriefing sessions.

    In the cognitive interviewing (phase 3), the first
    introductory card (showing types of houses) used a
    three-box format, and most respondents answered by
    pointing to a specific type of house within Box B.
    However, by the third card (on legal status), one
    respondent appeared not to understand the task; he
    pointed to the specific undocumented category,
    instead of just indicating Box B. But before being
    asked about his legal status, this respondent had
    volunteered to the interviewer that he was
    undocumented.4 Thus, there is some question as to
    whether he misunderstood the task or not. It may
    be that some further testing or revision of the
    instructions is needed to ensure that every
    respondent fully comprehends the task.

    Interviewers in the initial pretests did not
    report any respondent questions about why three
    boxes were being used. Similarly, in the in-depth
    debriefing sessions following the farmworker field
    test, interviewers did not report having to
    provide special explanations about why the three-
    box format was used.5 Because respondents did not
    raise questions about the use of the three-box
    answer format, they did not appear to have a
    problem with it. Therefore, specific questions on
    acceptance of the three-box format were not
    included in the cognitive interviewing. However,
    future cognitive work might be designed to learn
    how respondents perceive the three-box format.

    Results on the Appropriateness of Icons
    The initial pretests did not indicate problems
    with the appropriateness of the icons. However,
    the field test with farmworkers produced mixed
    results. Some interviewers had positive comments
    about the icons,6 but others reported that some
    respondents viewed the icons as childish and
    unnecessary-and even that the cards were
    unnecessary because the respondents felt they had
    understood the verbal instructions and thus did
    not need help.

    Some field-test interviewers indicated that the
    major problem seemed to be the first introductory
    card, which featured appliances. (The appliance
    card is a simple card that was intended to
    familiarize respondents with icons and cards
    before introducing the three-box format. The
    appliance card is shown as fig. 2.1. in ch. 2.)

    To improve the appropriateness of the icons, we
    decided to change the subject matter and design of
    the first introductory card. In preparation for
    the first set of cognitive interviews, we
    developed a new introductory card to replace the
    appliance card. The new card showed different
    housing categories; the housing categories and
    icons were grouped into three boxes. (The final
    version of the new introductory card featuring
    types of homes is shown in fig. 1.4 of ch. 1.) The
    respondent was asked to indicate which box
    contained the type of house he or she lived in at
    age five.7

    Changing the first introductory card meant that
    now both introductory cards would be training
    cards, in which respondents could practice
    answering by selecting a box, rather than by
    indicating individual items or categories.8 The
    transportation card was retained in the same form
    as in the field test (see fig. 1.5 in ch. 1). The
    icons on the legal status card were revised as
    described in the following sections.

    In our debriefing we asked about icons for each
    card, taken as a whole, as follows: "Do you think
    that the icons on this card are appropriate for
    adults?" All considered the icons appropriate for
    use with adults.

    Results on Legal Status Categories
    From the outset, icons were used to aid
    respondents' comprehension of legal status
    categories. In phase 3, to further enhance
    comprehension of these categories, we added a more
    detailed introduction to legal status card 1, in
    which the interviewer gave a verbal summary of the
    written descriptions of the various immigration
    categories represented by the icons. This was to
    ensure that respondents with limited Spanish
    literacy would be provided with the information
    written under the various immigration categories.

    The Green Card Category: Box A, Legal Status Card
    1
    Box A of legal status card 1 contains the category
    "legal permanent resident" with an icon consisting
    of a picture of a green card. The respondent's
    crucial task in picking a box is determining
    whether or not he or she belongs in Box A.
    Problems in the wording of Box A cropped up in the
    field test and were addressed in the cognitive
    interviews.

    Specifically, in the field test with farmworkers,
    most respondents did not appear to be confused
    about which box to choose, but a few respondents
    were uncertain-at least initially. These
    respondents told interviewers that they had fake
    green cards or had borrowed a green card from
    someone-or had "border-crossing cards"9-and that
    they were unsure whether or not Box A might apply
    to them.10 (Fig. 3.1 shows the version of legal
    status card 1 that was used in the cognitive
    interviews. This is the same card that was used in
    the field test.)

    Interviewers had been trained to help respondents,
    when necessary, by restating or paraphrasing
    questions and providing explanations. With help
    from interviewers, all 81 field-test respondents
    were able to pick a box. No respondent was coded
    "not sure." In fact, all chose either Box A or Box
    B. None were coded Box C. Nevertheless, it was
    unclear whether every respondent who selected Box
    A was indicating that he or she actually possessed
    a green card issued in his or her name.

    Figure 3.1: Legal Status Card Used in the
    Cognitive Interviews

    Source: GAO. (The actual size of card is 8-1/2" by
    11.")

    Rewording the Green Card Item
    In the field test instruments, the green card
    category in Box A was labeled (translating from
    the Spanish that appeared on the card): "Legal
    foreign resident with a certificate of permanent
    immigration (also known as the green card)
    obtained officially." This definition allowed some
    ambiguity, and therefore, before conducting the
    cognitive interviews, we reworded it as follows
    (again translating from the Spanish that appeared
    on the card): "Legal foreign resident with a valid
    and official card (also called green card) issued
    for this person by the U. S. Government." The
    cognitive interviewing then tested whether this
    version communicated effectively.

    Testing the Reworded Item
    Specifically, the cognitive interview included a
    vignette designed to provide information on the
    decision-making process used by respondents for
    including an individual in the revised green card
    category in Box A. Respondents were asked their
    opinion about whether a person who possessed an
    official green card that he or she had borrowed
    from a friend or relative belonged in that
    category.

    In the first set of interviews, the wording of the
    one-sentence vignette proved to be awkward to read
    and difficult for the respondents to understand.
    But after the interviewer paraphrased the
    vignette, each of the respondents understood and
    answered the vignette without difficulty. All four
    stated that an immigrant who possessed a borrowed
    official green card did not belong in the legal
    permanent resident category in Box A.

    With respect to the icon for this category
    (picture of a green card), one respondent
    suggested enlarging it somewhat for greater
    visibility. We made this change. We also reworded
    the vignette for the second set of cognitive
    interviews.

    The revised vignette was readily understood. As
    was found in the first day's interviews, all four
    respondents stated that an immigrant with a
    borrowed official green card did not belong in the
    more explicitly defined Box A green card category.

    Other Categories: Box B on Legal Status Card 1
    The farmworker field test yielded criticisms of
    some of the legal status icons in Box B. More
    importantly, some field-test respondents told
    interviewers that the icon for refugees (which
    shows a person running toward the United States)
    appeared to them to represent an undocumented
    worker running away from U.S. Border Patrol or
    other law enforcement officers. The icon for
    student, worker, or tourist visa-a suitcase with
    various stickers-was also viewed as problematic.
    Additionally, one of the interviewers recommended
    replacing the icon for the undocumented category
    with a picture of a green card with a diagonal
    line through the card.

    We changed each of these Box B icons as part of
    the preparation for cognitive interviewing. To
    better represent the idea of someone fleeing
    danger by coming to the United States, we added a
    symbol for an explosion behind the running figure
    and had the figure's arms stretch forward with his
    hands touching the map of the United States. We
    also developed a new icon for the undocumented
    category using the same green card image as
    appears in Box A, but drawing a circle around it
    and a diagonal line through it. In addition, the
    original icon showing a suitcase with stickers was
    replaced by the plainer suitcase icon that is
    routinely used in transportation terminals.

    In the first set of cognitive interviews, the new
    icon for a suitcase was identified as such by the
    respondents, as was the new icon for the
    undocumented category. But only one of the four
    recognized the refugee or asylee category from the
    icon alone. Two respondents perceived the figure
    either as a refugee being pushed away by the map
    of the United States or holding the map up-or as a
    terrorist.11 One respondent recommended that the
    arms of the fleeing figure in the refugee or
    asylee icon extend into the map of the United
    States, thereby dispelling the perception that the
    figure was either holding up the map, or was a
    terrorist being forced away from the country. One
    respondent did not interpret the drawing as
    showing an explosion.

    Before the next set of cognitive interviews, we
    made a number of changes to the cards. The size of
    the green card was increased in the two icons
    using this image. The fleeing figure in the
    refugee or asylee icon was changed to two fleeing
    figures, thereby increasing the suggestion of many
    persons fleeing to safety. The drawing of the
    explosion was enlarged and the word "Boom" added
    to the middle of the image. This style of drawing
    is used in fotonovelas, vividly illustrated
    pictorial storybooks similar to U.S. comic books,
    which are popular among Hispanic adults.

    In the second set of cognitive interviews, all
    four respondents recognized the concept of refugee
    or asylee from the icon alone.

    Cognitive debriefing revealed that, although the
    respondents understood the concepts symbolized by
    the flag (citizen) and suitcase (temporary visa
    status), there was nothing inherent in the flag or
    the suitcase to represent citizenship or temporary
    visa status. While it is true that interviewers
    can explain the icons for respondents who still
    experience some uncertainty, further cognitive
    work might improve the recognizability of the
    icons without the interviewer's help.

    Results on Sensitivity of the Legal Status
    Question
    None of the 116 foreign-born Hispanics interviewed
    across all three phases, refused to answer when
    using the three-box format, and 81 (about 70
    percent) chose the box (Box B) that included a
    sensitive category. However, a number of
    respondents perceived the question as sensitive.

    In the field test with 81 farmworkers, 30 picked
    Box A-that is, claimed to have a valid green card.12
    The remaining 51 respondents (63 percent) selected
    Box B, which contained the undocumented worker
    category. The population of foreign-born
    farmworkers is thought to contain few naturalized
    citizens, few refugees, and very few workers here
    with temporary work visas.13 Thus, we believe that
    most of the farmworkers who selected Box B were
    probably working without legal authorization.

    There were apparent differences in how sensitive
    the legal-status question appeared to various
    respondents.14 Although the field test did not
    include specific questions on sensitivity,
    interviewers reported their impressions, and these
    showed considerable variation, as follows:

    �    At some locations, there were instances where
    respondents voiced hostility, seemed to identify
    the survey with INS, or initially mistook the
    interviewer for a tax collector. Notably, in
    Florida, where the interviewer's impression was
    that all respondents were illegal, respondents
    reportedly appeared fearful of INS and nervous
    about the survey, and some respondents at first
    hesitated to select a box. Nevertheless, all
    Florida respondents picked Box B.
    �    At other locations, respondents appeared to
    be relaxed, and some even volunteered to the
    interviewer that they were here illegally. In
    Arizona, California, and Kentucky, interviewers
    reported that respondents did not appear to find
    the question on legal status intrusive.

    By contrast to the field test, the cognitive
    interviews specifically asked each respondent
    about the sensitivity of the legal status
    question. That is, a two-part question on the
    sensitivity of each three-box card was asked at
    the end of the cognitive interview, as follows:
    "Do you think that anyone would find the question
    for this flashcard to be sensitive? If yes, why is
    that?"

    In the first set of cognitive interviews, two of
    the four respondents stated that the question on
    immigration status was not sensitive because the
    question was asked in an indirect manner. The
    other two indicated that some would find the
    question sensitive. One said this was because
    legal status is confidential information, and some
    people would not want to be asked about it. The
    other respondent said that people do not want to
    tell the truth about their legal status. (Yet this
    respondent had volunteered that he was
    "undocumented" early in the interview.) All four
    respondents in this set of interviews chose Box B,
    which includes the sensitive item.

    In the second set of interviews, one of the four
    respondents did not find the question about
    immigration status sensitive; the other three did
    report sensitivity associated with this question.
    One respondent said that people would be fearful
    of being turned in to the INS. Another said the
    question is sensitive because the majority of
    foreigners he knows are undocumented, and they
    would not want to answer this question. A third
    respondent reported that it is not a bad question,
    but there are some people who would find it
    sensitive.15 Again, however, all four respondents
    chose Box B.

    _______________________________
    1One possibility for a validity study would be to
    survey the clients of legal aid clinics that
    specialize in immigration law, with the
    cooperation of clinic administrators. The
    administrators (who are bound by confidentiality)
    could determine whether each client had chosen the
    correct box. To safeguard client privacy, the
    clinic administrators would disclose the
    percentage of clients who incorrectly chose Box
    A-and also the percentage who incorrectly chose
    Box B (if any)-but no specific individual results.
    2Sensitivity is also of interest from a survey
    planning point of view. For example, sensitive
    questions would probably not be "piggybacked" onto
    a general purpose survey.
    3Also, on six of the nine forms of the field-test
    instrument, we added a follow-up question that
    provided respondents who chose Box A on the legal
    status flashcard with an opportunity to change
    their answers, asking them to "Please recheck Box
    B carefully, because some people pick Box A by
    mistake" (see app. II).
    4If this respondent did not understand, he may not
    be typical because his first language was neither
    Spanish nor English, but was a South American
    Indian language, and the interviewer believed that
    this respondent had not been paying careful
    attention.
    5The debriefing sessions did not include a
    question that specifically asked whether
    respondents had asked about why the three-box
    format was used. However, repeated questions asked
    interviewers to report problems that respondents
    had or other information on respondent reactions
    to the cards.
    6The Florida interviewer in the field test
    reported that some respondents said they found the
    icons (pictures) on the cards useful. The
    California interviewer indicated that respondents
    with less education tended to like the card more
    than those with more education-perhaps reflecting
    reactions to the icons.
    7One respondent pointed out that some respondents
    who, as children, had lived in a "campo" (hut)
    might find the housing question sensitive or
    embarrassing. An alternative would be to ask
    instead about the type of house that the
    respondent believes is most typical for families
    in their home country.
    8The new introductory card, on house types, was
    pretested for acceptability of the icons before
    being used in the formal cognitive interviewing.
    On Aug. 25, 1998, interviews on this card were
    conducted in Spanish with four Hispanic
    respondents, one Nicaraguan and three Salvadoran,
    at a Hispanic Community Center in Silver Spring,
    MD. All four found the icons on this card
    appropriate for adults, and all could recognize
    the type of house from the icon alone. They
    suggested changing the name for the rural dwelling
    category from "casa de compag�a" to "casa de
    campo." Respondents also suggested adding a thatch
    roof to the rustic house. We drew a thatch roof on
    the rural house icon and changed the name of this
    house type from "casa de compag�a" to "casa de
    campo."
    9The term "border crossing identification card"
    means "a document of identity bearing that
    designation issued to an alien who is lawfully
    admitted for permanent residence, or to an alien
    who is a resident in foreign contiguous territory,
    by a consular officer or an immigration officer
    for the purpose of crossing over the borders
    between the United States and foreign contiguous
    territory in accordance with such conditions for
    its issuance and use as may be prescribed by
    regulations." 8 U.S.C. 1101(a)(6).
    10In addition, one farmworker who could not read
    English or Spanish had a difficult time deciding
    in which box he belonged; he repeatedly stated
    that he had married a U.S. citizen and finally
    chose Box A.
    11Two U.S. embassies had been bombed by terrorists
    in the previous week, making headlines in the news
    media.
    12Each field-test respondent who selected Box A was
    able to identify a specific category or program
    under which he or she obtained a green card. Most
    of these (19 out of 30, or about 63 percent) said
    they had obtained their green cards through the
    amnesty under the Immigration Reform and Control
    Act of 1986 (P.L. 99-603), which included
    provisions for Seasonal Agricultural Workers.
    13During 1994-95, only about 3 percent of this
    population claimed to be naturalized citizens
    (Mines et al., 1997), and very few are here on
    legal temporary farmworker visas. For additional
    related work in this area, see GAO/HEHS-98-20,
    Dec. 1997.  About one million farmworkers acquired
    valid green cards through the amnesty under the
    Immigration Reform and Control Act of 1986,
    although many of them have now moved to other
    types of work.
    14We believe the fact that respondents were not
    afraid to pick a box containing the sensitive item
    justifies further development and testing. In
    particular, a validity study would address the
    issue of possible systematic bias in the
    responses.
    15The interviewer's impression is that this
    respondent was a U.S. citizen who believed that
    undocumented persons would find the question
    sensitive.

    Chapter 4
    Estimating Other Sensitive Items With the Three-
    Card Method
    Page 52GAO/GGD-00-30 Estimating Sensitive Survey I
    tems
    The three-card method was originally applied to a
    personal characteristic (immigration status). This
    chapter explores whether the technique might also
    be applied to various sensitive behaviors and
    practices. These include

    �    violence;
    �    sensitive personal choices;
    �    business and tax issues; and possibly,
    �    government workers' noncompliance with
    important reporting requirements or with certain
    other laws and regulations.

    Logically, the three-card method is relevant to
    any sensitive topic where (1) answers can be
    categorized in a set of mutually exclusive and
    exhaustive categories, (2) one of the answer
    categories is sensitive, and (3) the sensitive
    category is not too rare-at least not within a
    defined survey population.

    This chapter provides illustrations of how the
    three-card method might be applied in each of the
    areas listed above. However, we caution the reader
    that these illustrations are, for the most part,
    the result of brainstorming. They have not been
    subjected to empirical development or testing and
    are merely intended to be suggestive of
    possibilities.

    Some of the areas discussed are more sensitive
    than others, and we also note that, logically, the
    more sensitive the topic, the greater the need to
    couple the three-card method with assurances of
    anonymity-such as administering the technique via
    pen-and-pencil "secret ballots" rather than in a
    personal interview situation.

    Violence
    Violent behaviors that the three-card method might
    be used to estimate include road rage and other
    "anger responses," such as spouse abuse.

    Road Rage
    A survey of road rage among young male drivers
    might first ask drivers about incidents (e.g.,
    being cut off) that angered them during a
    specified period (past year, past 5 years) and
    then ask how they reacted the time they expressed
    the most anger.1

    The sensitive answer category would involve
    physical actions by the respondent-such as
    attempting to force the other driver off the road;
    getting out of the car to accost the other driver
    at a stop light; brandishing a weapon; or becoming
    involved in actual physical contact, such as
    shoving. The less sensitive categories might
    include

    �    did nothing, muttered under one's breath, or
    made remarks to passengers in one's car;
    �    honked horn or flashed lights at the other
    car (but did not yell or gesture); and
    �    yelled or gestured angrily at the other
    driver.

    Drivers would specifically be told that we are
    interested in the category that represents their
    most extreme anger response. The categories could
    be arranged on a card with one of the less
    sensitive categories in Box A and the rest of the
    categories in Box B. Box C would represent some
    other response (not included in Box A or Box B).

    Using a multicard, multisample approach, direct
    data would be separately collected on each of the
    less sensitive categories; then an indirect
    estimate of the most sensitive category would be
    obtained by subtraction.

    Other Violent Behaviors
    Spouse abuse and child abuse-as well as police
    brutality and elder abuse in nursing homes-are
    areas where survey questions might be framed in a
    similar fashion to the road rage questions
    described above. For example, a survey of police
    officers might ask them about suspects or
    perpetrators who angered them during the past year
    and ask about their most extreme anger response.
    Less sensitive categories would include ignoring
    the provocation, going by the strictest rules,
    speaking harshly to the suspect, and so on. The
    sensitive category would involve some form of
    physical abuse.

    It also might be possible to use this approach to
    study high school students' responses to other
    students who anger them.

    Sensitive Personal Choices
    Sensitive personal behavior choices-such as
    abortion or drug use-might also be studied using
    the three-card method.

    Abortion
    Though legal, abortion is a sensitive topic for
    many women. Some who have chosen to terminate a
    recent pregnancy might deny ever having been
    pregnant-or might report on the outcome of an
    earlier pregnancy.2 A three-card approach might be
    used to make the question less threatening.

    For example, in asking about the outcome of the
    respondent's most recent pregnancy, one option
    would be to define answer categories such as
    hospital birth, home birth, miscarriage,
    unavoidable termination for medical reasons, and
    abortion (for nonmedical reasons).3 The less
    sensitive categories would appear in Box A of
    alternate cards, and the percent whose most recent
    pregnancy ended with an abortion would be
    calculated indirectly.

    Marijuana Use
    A simple direct question on current (past month)
    marijuana use is potentially sensitive.4 One
    alternative would be to use the three-card method
    to estimate past-month marijuana use-in
    conjunction with estimating use of less sensitive
    substances (such as beer, wine, and hard liquor).

    Here, the less sensitive categories might be

    �    drank beer or wine but no hard liquor and no
    marijuana,
    �    drank hard liquor (with or without beer or
    wine), and
    �    no use of alcohol or marijuana.

    Each of the less sensitive categories would appear
    in Box A on alternative cards and the sensitive
    category-marijuana use (with or without alcohol
    use)-would be estimated indirectly.

    Business and Tax Issues
    There are many illegal, disapproved, or sensitive
    business and tax practices. Examples include
    manufacturers' disposal of hazardous wastes and
    the failure to file tax returns.

    Disposal of Hazardous Wastes
    It might be possible to develop a survey of
    manufacturers in an industry that routinely deals
    with hazardous materials. The sensitive category
    would be the illegal disposal of specific
    hazardous wastes. The less sensitive answer
    categories might include disposal of the specific
    hazardous wastes

    �    in all instances using a contractor;
    �    partly using a contractor and partly self-
    disposal in accordance with EPA guidelines; and
    �    in all instances, self-disposal in accordance
    with EPA guidelines.

    The sensitive category would encompass all direct
    forms of illegal dumping by the manufacturer. An
    actual application of the method would have to be
    developed by working with persons knowledgeable in
    this area.5

    Failure to File Tax Return
    Tax nonfiling is difficult to estimate for self-
    employed persons. To create a number of less
    sensitive categories, the act of filing or not
    filing might be combined with asking about the
    type of return filed and who prepared the return.
    The less sensitive categories might be

    �    my spouse prepared a joint return for us,
    which has been filed;
    �    I prepared a joint return for myself and my
    spouse, and it has been filed;
    �    I prepared my own return, and filed it as an
    individual; and
    �    a tax preparer prepared the return
    (individual or joint), and it has been filed.

    The sensitive category would be: I have not sent
    in a return for last year.

    Government Workers' Noncompliance
    The extent to which certain government
    employees-at federal, state, and local levels-fail
    to comply with reporting requirements or various
    other laws or regulations is of interest from a
    managerial or oversight point of view. For
    example, it would be desirable to monitor whether
    certain types of noncompliance rise above
    threshold levels, thereby signaling the need for
    remedies such as agency-wide ethics training.

    In more serious instances, prevalence data
    might-if available-help define the kinds of
    investigative approaches that would be most
    appropriate. (For example, in defense-related
    work, such data might help determine whether or
    not widespread polygraph tests should be required
    for scientists working at government
    laboratories.)

    In the following sections, two examples explore
    whether noncompliance might be asked about by
    using the three-card method in anonymous surveys
    of government workers.

    Failure to Report Offers of Bribes
    Police officers and certain other government
    workers (for example, at the federal level, e.g.,
    those working for the Customs Service, the Border
    Patrol, or the Postal Service) may be expected or
    required to report offers of bribes or incentives
    that might be offered in hope that the official
    would overlook certain occurrences (such as
    illegal imports or the possession of illegal
    substances). At some agencies or departments,
    regulations or laws require employees to report
    all bribery attempts.6

    It might be that a technique such as the three-
    card method would facilitate asking about
    government workers' compliance or noncompliance
    with these reporting requirements. One possibility
    might be to define the sensitive category as
    having failed to report an offer of an
    inducement-within a specified time period. The
    less sensitive categories might refer to

    �    having been offered inducements both on duty
    and off duty, and reporting all cases;
    �    having experienced one or more such
    occurrences while on official duty only (no
    similar occurrences while off duty), and reporting
    all cases; and
    �    having experienced one or more such
    occurrences while off duty only, and reporting all
    cases.

    Violations of National Security
    Turning to the area of defense-related government
    work, in the fall of 1999, concerns surfaced about
    U.S. scientists' contacts with foreign agents and
    the possibly unauthorized passing of information
    by federal employees. These concerns were aired
    before Congress (Loeb, 1999; and Pincus and Loeb,
    1999).7 Currently, the charged atmosphere might
    prevent any attempt to conduct a survey of
    unauthorized exchanges of information-even if
    perceived as minor or as justified in the
    particular instance. However, in the future, if a
    technique such as the three-card method were used
    to survey these workers, the sensitive category
    might be defined as "gave some unauthorized
    information to a foreign national-not as part of
    an authorized U.S. counterintelligence operation."

    The less sensitive categories might indicate
    whether the employee

    �    had been approached by a foreign national
    seeking unauthorized information, but did not
    provide it;
    �    had been approached by a foreign national,
    and-as part of an authorized operation-gave out
    information provided by U.S. counterintelligence;
    or
    �    had not been approached by a member of a
    foreign intelligence service.

    Again, all categories would be linked to a
    specific time frame (e.g., within the past 5
    years).

    _______________________________
    1For the driving population as a whole, road rage
    expressed in "physical confrontation" within the
    past 5 years is estimated at 4 percent. (This
    statistic is from a 1997 Gallup poll, in which
    respondents who drive were asked if they had
    "honked horn," "shouted," "slowed down," "flashed
    high-beams," or engaged in "verbal exchange" or
    "physical confrontation." See Gallup Poll Monthly,
    No. 383 (Aug. 1997), p. 60). We would anticipate
    that levels of physical confrontation would be
    considerably higher among young male drivers than
    in the driving population as a whole.
    2Researchers have found that underreporting
    abortion "is a persistent problem in studies
    conducted in the United States and elsewhere,
    irrespective of the research design or study
    population" (Udry et al., 1996). These same
    researchers point out that the underreporting of
    abortion also causes biased estimates of
    contraceptive failure, miscarriage, and
    reproductive histories that rely on self-reports.
    3A different set of categories would be birth of a
    girl, birth of a boy, multiple birth, miscarriage,
    unavoidable termination for medical reasons, and
    abortion (for nonmedical reasons).
    4This potential sensitivity may be heightened in
    certain circumstances, such as a workplace survey
    of employees or a survey of military personnel,
    or, students at the service academies.
    5Separately developed cards-with different
    categories-might be used for surveys of trucking
    companies that remove hazardous wastes from
    manufacturers in the relevant industry and
    landfill, incinerator, or other companies that
    receive and dispose of the relevant wastes.
    6For example, U.S. Postal Service employees are
    required to report any bribe or attempted bribe,
    "undue influence or coercion to induce or attempt
    to induce the employee to act or neglect to act in
    regard to his official responsibilities." 39
    C.F.R. 447.61.
    7These issues concerned certain Department of
    Energy national laboratories.

    Chapter 5
    Challenges and Potential Solutions in Using the
    Three-Card Method in a Large-Scale Survey
    Page 59GAO/GGD-00-30 Estimating Sensitive Survey I
    tems
    Often, surveying a new subject or topical area or
    using a new method requires considerable
    development and cost-as well as ingenuity-but the
    payoff can be improved information for the policy-
    making process. Assuming that a specific
    application of the three-card method is fully
    tested, validated, and deemed appropriate for use
    in a large-scale survey, a number of challenges
    would apply. These challenges include

    �    obtaining reasonably low-cost (efficient)
    interviews that include flashcards;
    �    efficiently screening for members of key
    populations (e.g., foreign-born persons, young
    male drivers);
    �    efficiently estimating three-card items
    despite the need to draw three different samples
    and to estimate the sensitive item indirectly;
    �    avoiding nonresponse bias across the three
    samples; and
    �    dealing effectively with issues involved in
    surveying special groups, such as the foreign-
    born.

    Interviews Using Flashcards
    Thus far, all development and testing of the new
    method has been conducted in person, in face-to-
    face interviews. Personal interviews are
    considerably more expensive than telephone
    interviews. But we believe the flashcards are
    essential, and therefore that the three-card
    method is not appropriate for a telephone survey.1

    Adding a three-card format to an ongoing, in-
    person survey could be much less expensive than
    conducting a new in-person survey.2 The three
    samples could be selected for the three-card
    portion of the interview only. But because
    piggybacking onto an existing in-person survey may
    not always be an option,3 other potential low-cost
    alternatives should be considered. Three
    alternative strategies are as follows:

    �    group administration, which has been used to
    survey high school students, for example;4
    �    mail surveys; and
    �    Internet surveys where a computer screen
    displays the equivalent of a three-box card, have
    been conducted with certain populations. Of
    course, many households are not connected to the
    Internet, so for general population surveys a
    feasible option could be a dual-frame or dual-mode
    survey involving the use of personal interviews
    where computers are not available.5

    Applications such as these-which do not involve
    answering aloud-might have an added advantage in
    encouraging more truthful responses to sensitive
    items.6

    It seems likely, however, that many applications
    would require a new in-person survey, because the
    cards are essential and the options outlined above
    may not be feasible. In these instances, the cost
    of a new large-scale in-person survey would be
    considerable.

    Screening for the Relevant Subpopulation
    Some sensitive items are relevant only to a
    specific subpopulation; e.g., legal status is
    relevant only to the foreign-born. Other sensitive
    items may be of most interest in a
    subpopulation-and rare in other groups; for
    example, road rage could be more relevant to young
    male drivers than others-possibly it might be most
    relevant to young males who have been ticketed for
    moving violations in the past 5 years.

    In some cases, the subpopulation of interest might
    be concentrated in one specific location or a
    number of specific locations.7 This could allow
    reasonably efficient direct sampling. In other
    cases, it would be necessary to first screen more
    general populations to identify members of the
    relevant group.8

    A main option here is to use an ongoing survey as
    a "screener" (even if that survey cannot be used
    to ask the three-card questions themselves). It
    may be necessary to "piggyback" screening
    questions onto the existing survey. But a special
    purpose-or even a general purpose-survey may
    already include questions that could serve to
    screen for the subpopulation in question.9 For
    purposes of screening, a telephone survey is often
    as appropriate as an in-person survey. Once
    members of the subpopulation of interest are
    identified, a "linked survey" could be conducted.
    That is, interviewers would revisit identified
    households within 1 to 3 months after the main
    survey to ask a short set of additional questions
    using the three-box format.

    Another option that might be relevant in certain
    circumstances is to use existing records. For
    example, it might be possible to use drivers'
    license records to sample young male drivers (for
    a survey of road rage)-or even to use records of
    moving violations to sample only those with moving
    violations in recent years.

    Efficient Estimation
    The direct estimate of each less sensitive
    category that is featured in Box A is based on
    just one sample (i.e., only those respondents
    answering the card that features that particular
    category in Box A). Obviously, the number of
    respondents in a particular sample is less than
    the total number of respondents in all three
    samples. All else being equal, the fewer
    respondents answering a specific item, the higher
    is the variance-or the margin of error-for the
    resulting estimate.

    Variance is further increased for the indirect
    estimate of the sensitive item (which never
    appears in Box A). This is because the indirect
    estimate is obtained by a linear combination of
    the three direct estimates-and the variance is
    increased by each additional estimate (see app.
    I). Finally, as always, subgroup estimates (e.g.,
    illegal residents who are male or those who are
    female) have a higher variance than total
    estimates (e.g., all illegals) because of the
    reduced number of qualifying respondents.

    The most direct way to reduce the variance of an
    estimate is to increase the sample size; however,
    the dollars needed for additional interviews
    increase costs. Four efficiency-minded
    options-that is, strategies to reduce variance
    without the added cost of increasing the sample
    size-can potentially be used. These strategies,
    some of which can be used in tandem with each
    other, are:

    �    using information about the subpopulation (if
    it is known) to stratify the subpopulation in
    advance;
    �    using a principle of "optimal allocation" to
    determine the relative sizes of the three samples
    (see app. II);
    �    setting sample sizes from every block or
    cluster in proportion to counts of the relevant
    subpopulations in these "blocks" or clusters; and
    �    obtaining some of the three-card information
    by inserting one of the cards in an ongoing
    survey.

    Using a Principle of Optimal Allocation
    The variance can be reduced by setting or choosing
    the relative sizes of the three samples based on
    one of the principles of optimal allocation. As
    background, we note that, given three samples of
    the same size, the variances associated with the
    direct estimates of Box A categories will very
    likely not be the same. (To illustrate this for
    legal status, the percentage of foreign-born who
    possess a green card is probably not the same as
    the percentage who are naturalized citizens.10 This
    is important because, as explained in standard
    statistical texts, a very low or very high
    percentage is associated with lower variance than
    is one near 50 percent.) Depending on the
    population being surveyed, it seems likely that
    with equal sample sizes, some cards would be "high
    variance" cards and others would be "low variance"
    cards.

    To minimize the variance associated with the
    indirect estimate of illegals, which is based on
    three direct estimates, the principle of optimal
    allocation we are using suggests that larger
    number(s) of sample cases be allocated to estimate
    the proportion of the population represented by
    the high variance cards and fewer numbers of
    sample cases be assigned to estimate the
    proportion of the population represented by the
    lower variance cards (see Cochran, 1977, pp.
    96ff).

    Setting Sample Sizes Proportional to Block Sizes
    To ensure that the sampling rates of the relevant
    subpopulation are equal across small clusters
    (i.e., blocks), the sample sizes for the three
    surveys can be allocated proportionally to the
    number of people in the relevant subpopulation in
    each block. For example, suppose that, by using
    the principle of optimal allocation, the sizes of
    the three samples are 500, 300, and 200,
    respectively. Also, assume that the sample area
    has two blocks, the first of which has four times
    as many members of the relevant subpopulation as
    the second. Then the allocation of the three
    samples would be 400, 240, and 160, respectively,
    in the first block, and 100, 60, and 40,
    respectively, in the second block.

    Obtaining Information From Respondents in an
    Ongoing Survey
    Even where it is not possible to use the three-
    card method in an ongoing survey, it may still be
    possible to insert a question on one of the cards
    (e.g., a portion of foreign-born respondents in a
    general purpose survey might be asked whether they
    are U.S. citizens, using a three-box card).11 These
    data could be combined with the full set of three-
    card responses separately obtained in a different
    survey (perhaps a linked follow-back survey as
    discussed above), thus lowering the variance.
    Ideally, there would be tests for possible
    response differences between various survey modes.

    Minimizing the Bias of Nonresponse
    Person (unit) nonresponse could vary across the
    three samples, raising the possibility of
    noncomparability and biased estimates. This might
    be the case, for example, if interviewers
    perceived a particular legal status card as more
    sensitive than the others; that is, they might not
    try as hard to "get the interview" when assigned
    the card perceived as more sensitive. This
    possibility can be avoided by utilizing a "blind"
    randomization procedure, such as a sealed envelope
    that is opened only after contacting the
    respondent or after cooperation has been obtained.

    If a linked "screener" survey is used, the impact
    of person nonresponse might be minimized through
    imputations based on results from the initial
    survey.12 It also seems advisable to sample
    households that were coded as nonresponses in the
    initial survey; screen for target-population
    residents in those households; and conduct three-
    card interviews, as appropriate.

    Turning to item question nonresponse among survey
    participants, we have thus far not encountered any
    instances of item nonresponse to the three-card
    questions-whether in our pretests, the field test,
    or the cognitive interviews. Moreover, although we
    have only tested one card, we believe it is the
    most sensitive of the three because it asks
    whether a respondent has a green card or not.

    If a small amount of item nonresponse were to
    occur, it might be possible to minimize its impact
    by using imputation methods. However, substantial
    amounts of item nonresponse would affect the
    usability of the results.

    Issues in Surveying Special Populations
    There are special challenges in surveying various
    special populations-foreign-born persons, teens,
    the elderly, or other specialized groups. The
    foreign-born population is of special interest in
    this report because the initial application of the
    three-card method applies to that population. The
    problem of combating incomplete coverage of this
    subpopulation was discussed in our 1998 report
    (GAO/GGD-98-164). The following section discusses
    challenges in interviewing members of the foreign-
    born population.

    Interviewing the Foreign-Born
    Efforts to survey foreign-born individuals
    necessarily involve (1) building trust and
    communicating with foreign-born groups and, if
    using the three-card method, (2) selecting icons
    that are effective across diverse groups.

    Building Trust and Communicating With Diverse
    Groups
    Trust and communication are important in all
    surveys, and in all cases, the ability of the
    interviewer to establish rapport is essential.
    However, building trust and communicating
    accurately may be more challenging in surveys that
    focus on a special population-such as teens, the
    elderly, or the foreign-born. In particular, the
    foreign-born population seems to present a unique
    set of challenges because, for example,
    respondents and interviewers may be from very
    different cultures and might not speak the same
    language.

    Logically, matching interviewers to respondents on
    foreign-born status and "home country" is one
    method of maximizing trust and communication. In
    Los Angeles, surveys that targeted residents from
    selected nations-Mexico, El Salvador, and the
    Philippines (DaVanzo et al., 1994; Heer, 1990, pp.
    88-97; Bustamante et al., 1996)-were generally
    able to employ and assign interviewers to match
    respondents' home countries.13 (In our field test
    of one Salvadoran and 80 Mexican farmworkers, the
    interviewers were all fluent in Spanish. Although
    not all were from Mexico, they were experienced in
    interviewing this population.)

    A strategy of "matching on home country" might not
    be feasible for a nationwide survey covering
    immigrant groups from all nations of the globe. A
    national effort to approach the matching condition
    might be possible using three different tactics,
    as follows.

    �    If a linked survey is used,14 and if the
    initial survey identifies the respondent's home
    country, then recontact could begin with an
    explanatory telephone call, and the survey workers
    placing those calls from a central phone bank
    could be immigrants who are matched to each
    respondent on home country. Subsequently, an
    interviewer would make a personal visit. If it is
    known that language assistance will be required,
    the interviewer might carry a cell-phone and
    contact headquarters for help from a survey worker
    from the respondent's home country.
    �    Even if a linked survey is not used, it is
    possible to identify areas according to ethnic
    settlements and to try to provide appropriate
    bilingual interviewers to the extent possible.
    Cards and interview materials can be translated
    and tested in as many languages as possible.15 All
    interviewers could carry cell-phones and contact a
    central phone-bank for help from an interviewer
    who speaks the respondents' home language.
    �    Alternatively, it might be possible to
    develop a self-administered version of the
    immigration status question. That is, instead of
    showing the respondent a card, the interviewer
    would give the respondent an "answer sheet" that
    looks like one of the legal status cards shown at
    the outset of this report. Instructions at the top
    of the answer sheet would tell the respondent, in
    his or her own language, to circle the letter
    beside the box that applies to him or her. The
    instructions would also make clear that if it is
    Box B, we do not want to know which specific
    category applies. This approach might be useful
    for immigrants who are literate in the language of
    their home country.

    Technology can also help overcome communication
    barriers. Audio computer-assisted self-
    interviewing (ACASI) is a technology that allows
    for the administration of questionnaires in
    multiple languages-even when survey field staff
    are monolingual. It has been used successfully to
    interview older Korean immigrants who do not speak
    English. With this technology, "Subjects hear the
    questions through headphones and enter responses
    directly into the computer using the keyboard"
    (Hendershot et al., 1996, p. 165). With the latest
    system, the field interviewer simply plugs
    headphones directly into a port on the laptop
    (Turner et al., 1996, p. 174). Additional benefits
    of ACASI are that visuals can be provided and that
    the respondent's privacy is enhanced vis-�-vis
    interviewers and other members of the household.
    This can further increase the accurate reporting
    of sensitive items (Turner et al., 1998).

    Selection of Icons That Are Effective Across
    Diverse Groups
    Finally, the icons used on the cards deserve
    special attention. The cards and icons shown in
    earlier chapters were pilot-tested with Hispanic
    respondents. It may be necessary to identify icons
    that are more appropriate or effective for other
    cultures (e.g., Asian) or to identify icons that
    transcend cultural barriers. Only cross-cultural
    testing can determine this.

    Challenges Differ Across Targeted Subpopulations
    Because potential applications of the three-card
    method may involve different subpopulations (see
    ch. 4), it can be reasonably expected that there
    may be a variety of different challenges in
    fielding a specific survey. Interviewing a
    nationwide sample of foreign-born persons about
    their legal status, for example, might be
    considerably more difficult than interviewing
    young male drivers about "road rage." The reason
    is that for young male drivers, a language barrier
    may occur in relatively few cases. For this
    reason, generalizations in this area are difficult
    at best.

    _______________________________
    1We believe that the visual display of the boxes
    and the combination of categories in Box B is
    essential. The reason is that this helps convey
    the privacy protection to respondents ("If you're
    in Box B, we do not want to know which category
    applies to you.")
    2Some existing national surveys are conducted
    exclusively face-to-face.  One example is the
    National Health Interview Survey, which involves
    about 100,000 in-person interviews each year.
    Certain other large-scale surveys conduct only a
    portion of their interviews face-to-face, and this
    creates a problem for the use of cards.
    3Sponsors of the ongoing survey might not wish to
    add a somewhat sensitive question series.
    4See "The Monitoring The Future Study, 1975-1997,"
    a survey of drug use among secondary school
    students (Johnston, O'Malley, and Bachman, 1998).
    5Some segments of the foreign-born population may
    be unlikely to own computers with an Internet
    connection, but other segments may be likely to
    have them. The Wall Street Journal recently
    published an article that discusses the pros and
    cons of doing surveys using the Internet,
    including methodological issues (Simons, 1999).
    6Self-administered questionnaires and answer
    sheets have been associated with more accurate
    responses because interviewers need not know the
    sensitive reply. Moreover, if certain kinds of
    interviews are conducted in a household, answering
    aloud may be sensitive because other members of
    the household may overhear the response (see,
    e.g., Gfroerer, 1985, p. 22).
    7To cite one example, a population of farmworkers
    with a large subpopulation of foreign-born
    individuals would probably be located on farms.
    8For a discussion of alternative "sampling
    strategies and sources of recruitment" in
    surveying the foreign-born population, see Loue
    and Bunce (1999).
    9For example, the Current Population Survey asks
    about nativity; this represents a screen for
    foreign-born.
    10For example, it may be that 40 percent of foreign-
    born farmworkers have green cards, whereas less
    than 5 percent are naturalized citizens.
    11Some national surveys currently include a
    question on citizenship. Chip Alexander, Assistant
    Division Chief for Longitudinal and Expenditure
    Surveys Design, Bureau of the Census, pointed out
    a technique of drawing the sample ("double
    sampling") that would minimize the variance.
    Essentially, this would involve piggybacking the
    least sensitive card onto the "screener" survey,
    for a subsample of relevant respondents; a follow-
    on survey, using all three cards in alternate
    samples, would be limited to those not shown a
    card in the screener survey.
    12Suppose, e.g., that of 500 Mexican males who
    participated in the initial survey, 480 responded
    in the follow-up. The responses of the 20 Mexican
    male nonrespondents might be imputed based on (1)
    characteristics reported by all 500 in the initial
    survey, such as occupation, age, and time residing
    in the United States and (2) the three-card
    responses of the 480 Mexican males who did
    participate in the follow-up survey.
    13These surveys apparently had some success in
    asking direct questions about immigration status.
    14That is, if, as discussed above, foreign-born
    respondents are identified in an initial general
    purpose survey and then recontacted for questions
    about immigration.
    15The pilot for the "new immigrant survey" (in
    which new green-card holders from various
    countries are interviewed) included questionnaire
    translations for 6 languages; interviewers
    speaking 11 other languages were hired (Jasso,
    1999).

    Appendix I
    Variance of the Estimates
    Page 64GAO/GGD-00-30 Estimating Sensitive Survey I
    tems
    The statistical expression of the three-card
    estimator of the percent of foreign-born who are
    here illegally, the variance of this estimator,
    and the "technique effect" are shown in figure
    I.1. (See also GAO/GGD-98-164, July 1998.)
    Clearly, there are variance costs associated with
    using an indirect method. First, each estimate of
    legal status is based on one sample, rather than
    all three samples. Second, the indirect estimate
    of illegal immigrants is affected by the variance
    of each of the estimates that is included in its
    calculation. Assuming that the three samples are
    of equal size and that for each card 25 percent of
    respondents belong in Box A, the confidence
    interval for a three-card method estimate of
    illegal immigrants would be three times as large
    as a corresponding direct estimate. Keep in mind,
    however, that variance costs depend heavily upon
    the distribution of immigration status groups in
    the population surveyed and the relative sizes of
    the three samples.

    Two examples are presented below to give the
    reader a flavor of what real-world precision might
    be.

    �    Example 1: In a population of foreign-born
    agricultural workers, the distribution of
    immigration status might be 55 percent illegal; 36
    percent legal permanent residents; 3 percent U.S.
    citizens; and 6 percent temporary workers,
    refugees, or asylees. Assuming this distribution
    and a total sample size of only 1,000, allocated
    with 100 respondents to answer the card with U.S.
    citizen in Box A; 200 to answer the card with
    temporary visas, refugees, or asylees in Box A;
    and 700 to answer the card with legal permanent
    resident in Box A, the 95-percent confidence
    interval for an estimate of 55 percent illegal
    would be 49 to 61 percent.

    �    Example 2: In the residential foreign-born
    population of the United States, taken as a whole,
    the distribution might be 22 percent illegal; 30
    percent U.S. citizens; 38 percent legal permanent
    residents; and 10 percent temporary workers,
    refugees, or asylees (without green cards).
    Assuming this distribution and a total sample size
    of 13,000-the approximate number of foreign-born
    in the Current Population Survey supplement-with
    6,000 respondents allocated to answer the card
    with legal permanent resident in Box A; 5,550
    allocated to answer the card with U.S. citizen in
    Box A; and 1,500 to answer the card with temporary
    workers, refugees, or asylees in Box A, the 95-
    percent confidence interval for an estimate of 22
    percent illegal would be 20 to 24 percent.
    We note that in the foregoing examples, the sizes
    of the three samples were chosen by adapting one
    of the principles of optimal allocation.1 The
    confidence intervals in the examples would have
    been larger if three equal-sized samples had been
    used.

    The formula for calculating the variance of the
    estimates is shown in figure I.1. To estimate the
    percentage of a foreign-born subgroup who are here
    illegally (e.g., percent of foreign-born
    California residents who are here illegally),
    calculations identical to those shown in the
    formula would be performed using only the relevant
    data (e.g., the data for all foreign-born
    residents of California). The variance of such a
    subgroup estimate would depend on the size of the
    sample for that subgroup and the distribution of
    legal status in that group.

    However, to derive an estimate such as the
    percentage of illegal immigrants living in
    California implies a ratio estimate. That is, both
    the numerator and the denominator would involve
    indirect estimates of illegal immigrants (the
    numerator, illegal immigrants in California; the
    denominator, illegal immigrants in the United
    States). Specifically, the numerator would consist
    of the number of foreign-born Californians times
    the estimated percentage of that group who are
    here illegally. The denominator would consist of
    the number of foreign-born residents of the United
    States times the estimated percentage who are here
    illegally. Assuming that the number of foreign-
    born residents is known (e.g., from the census),
    the percentage of illegal immigrants living in
    California reduces to the product of two factors:

    �    a constant, consisting of the number of
    foreign-born living in California divided by the
    number of foreign-born living in the United
    States; and
    �    a ratio estimate-specifically, the ratio of
    the estimate of the percent of foreign-born
    California residents who are here illegally to the
    estimate of the percentage of foreign-born U.S.
    residents who are here illegally.

    Deriving the variance of such an estimate is
    complex and beyond the scope of this paper.
    However, with respect to the first factor, it is
    clear that the constant factor tends to reduce the
    variance for states with relatively few foreign-
    born. With respect to the second factor, a formula
    is available (see Hansen, Hurwitz, and Madow,
    1953, vol. II, p. 107). The overall variance of
    such a ratio may be substantial because the two
    estimates involved in the ratio both carry
    variance costs. Special designs to minimize
    variance costs may be feasible, however, depending
    on the application.

    Figure I.1: Statistical Expression of the Three-
    Card Estimator, Its Variance, and Technique Effect

    Source: GAO/GGD-98-164, July 1998, p. 75.

    _______________________________
    1The key guideline of optimal allocation that we
    followed minimizes the variance of an estimate by
    assigning a somewhat greater proportion of the
    sample where the variance is highest-and a
    somewhat smaller proportion where the variance is
    lowest. (Optimal allocation is usually used to
    determine sample size for strata rather than for
    subsamples.)

    Appendix II
    Different Forms of the Questionnaire Used in the
    Field Test
    Page 78GAO/GGD-00-30 Estimating Sensitive Survey I
    tems
    The 81 interviews were conducted as an add-on to
    the National Agricultural Workers Survey (NAWS).
    We used nine different forms of the three-card
    segment in the field test. The purposes included
    (1) varying certain questions used as structured
    probes to obtain additional information and (2)
    conforming to Office of Management and Budget
    (OMB) policy, which applies to the NAWS survey.
    Specifically, OMB requires a survey form to be
    formally reviewed if it is to be used to survey 10
    or more people. We developed different forms
    because the review time would have exceeded our
    available time.

    The NAWS is a survey that gathers information
    about farmworkers in the United States by asking
    them questions about their basic demographics,
    legal status, education, family size and household
    composition, and wages and working conditions in
    agricultural jobs. Demographic characteristics
    include their ethnic composition, their age and
    gender distribution, and the division among
    immigration and citizenship categories. Our 81
    interviews were conducted using an abridged
    version of the NAWS questionnaire, with our three-
    card series added on.

    Form 1 of the three-card question series has been
    reproduced in figure II.1. The differences among
    the nine forms are explained in table II.1

    Table II.1: Variations in Wording of Survey
    Questions in the Nine Questionnaires We Field-
    Tested
                      The structured probe portion of
                              question GAO-4a
                     Included as   Not    Included with
                     shown on      includ additional
                     form 1 (fig.  ed     explanationb
                     II.1)
    Contains all     Form 1        Form 2 Form 3
    questions
    Omits question   Form 4        Form 5 Form 6
    GAO-1b onlyc
    Omits question   Form 7        Form 8 Form 9
    GAO-3b onlyd
    aSee p. 8, fig. II.1. The multi-paragraph
    structured probe follows the instruction, "If
    respondent chooses Box A, ask: . . . ."
    bInterviewer instructed to provide ad hoc
    explanations if respondent does not understand
    question.
    c As shown on p. 2, fig. II.1, question GAO-1b
    asks about household appliances not present at the
    location where the respondent is living or staying
    now.
    dAs shown on p. 5, fig. II.1, question GAO-3b asks
    if the respondent lives permanently in the United
    States or just works here temporarily.
    Source: GAO.

    Figure II.1:  Form 1 of the Questionnaire

    Appendix III
    Estimates for Demographic Subgroups
    Page 79GAO/GGD-00-30 Estimating Sensitive Survey I
    tems
    Estimates of legal permanent residents (Box A for
    legal status card 1) can be directly obtained for
    demographic subgroups, such as male or female;
    residents of various geographic areas (e.g.,
    California); and age or income groups.

    The only proviso is that demographic information
    must be obtained for all respondents earlier in
    the questionnaire. Subgroup estimates can be
    obtained simply by isolating, for example, male
    respondents in sample 1 and then calculating the
    percentage who chose Box A. Then, perform the same
    procedures separately for female respondents in
    sample 1. Of course, how refined a subgroup can be
    reliably estimated would depend on the size of the
    sample and other aspects of sample design.

    Estimates of foreign-born U.S. citizens in
    different subgroups can be obtained in a similar
    fashion, using data for sample 2 respondents. The
    situation is analogous for the legal statuses that
    appear in Box A of the legal status card shown to
    sample 3.

    The three-card method can also be used to
    indirectly obtain separate subgroup estimates for
    illegal immigrants-at least for sizable subgroups
    of illegals, such as males and females, or
    California residents. This can be done by
    isolating the data for foreign-born respondents in
    the first subgroup of interest (e.g., foreign-born
    males), obtaining separate estimates for Box A in
    each of the three samples, and then subtracting
    percents estimated from each legal status from 100
    percent. The procedure would then be repeated for
    the next subgroup of interest (e.g., isolate data
    for foreign-born females and repeat calculations).
    Such procedures work best for large subgroups
    within the foreign-born population.

         Given a large enough sample, indirect
    estimates of illegals could be obtained for
    several key geographic areas-for example, key
    immigration states like New York and Texas. In
    some cases, it may be desirable to produce
    estimates for smaller areas, such as a county
    (e.g., Los Angeles County) or a group of counties
    (i.e., the counties of Texas that border Mexico).
    Similarly, a separate estimate of illegal
    immigrants might be obtained for a particular
    demographic subgroup (such as women of
    childbearing age or persons below the poverty
    line). Of course, whether a subgroup of illegals
    could be reliably estimated would depend on the
    size of the sample and other aspects of sample
    design.

    Appendix IV
    Data From Respondents Who Participated in the
    Cognitive Interviews
    Page 80GAO/GGD-00-30 Estimating Sensitive Survey I
    tems

    Gende Age Nativit U.S. job   Primar  Understo Found   Respondent comments
    r        y       experience y       od       card 3
                             langua  refugee  sensit
                             ge      icon?    ive?
    Male  38  El      Auto       Spanis  No       No      Use bigger green card icon.
            Salvado factory    h                      It looks like the person in
            r       assembly,                        the refugee icon is holding
                   cleaning,                        up a map, and the explosion
                   bus driver                       looks like a hole in the
                                                   paper. Immigration question
                                                   is good because it is not
                                                   personalized.
    Male  49  Bolivia Lawn care, Quechu  Yes      Yes     Add a fake green card to the
                   carpet     a                      undocumented icon. Some would
                   cleaner,                         find the immigration question
                   electricia                       sensitive because they do not
                   n                                want to tell the truth about
                                                   their status.
    Male  60  Mexico  Constructi Spanis  No       Yes     Immigration question is
                   on,        h                      sensitive because some do not
                   milking                          want to be asked about that.
                   cows                             That is confidential
                                                   information.
    Male  30  Dominic Mowing     Spanis  No       No      Refugee icon is confusing. It
            an      lawns      h                      looks like a terrorist being
            Republi                                 pushed out of the United
            c                                       States.
    Femal 20  Peru    Cleaning   Spanis  Yes      No      The respondent said that she
    e               and child  h                      recognized the icons for
                   care                             "Undocumented,"
                                                   suitcase/visa, and the
                                                   Refugee categories. Change
                                                   diagonal line to crossed
                                                   lines in the " undocumented"
                                                   icon.
    Femal 34  El      Factory    Spanis  Yes      Yes     Use an additional line to
    e        Salvado machine    h       and no          cross out the undocumented
            r       operator,                        icon. Change the refugee
                   cleaning                         symbol to a thatch roof
                                                   without walls; that's the
                                                   symbol for refugee in my
                                                   country. The immigration
                                                   question is not bad, I do not
                                                   find it sensitive, but some
                                                   people would.
    Male  34  Nicarag Painter    Spanis  Yes      Yes     Add visa to suitcase icon.
            ua                h       and no          Add U.S. seal to citizen
                                                   icon. Immigration question is
                                                   sensitive because many people
                                                   are undocumented and would
                                                   not want to answer.
    Male  22  El      Painter,   Spanis  Yes      Yes     Some compesinos (rural
            Salvado carpet     h                      people) wouldn't recognize
            r       installer,                       the undocumented icon. Add
                   commercial                       trees and a "danger" sign to
                   cleaning                         the refugee icon. Some would
                                                   find the immigration question
                                                   sensitive because they would
                                                   fear being turned in to INS.
    Source: GAO compilation of data from cognitive
    interviews.

    Bibliography
    Page 84GAO/GGD-00-30 Estimating Sensitive Survey I
    tems
    Bradburn, Norman M., and Seymour Sudman. Improving
    Interview Method and Questionnaire Design:
    Response Effects to Threatening Questions in
    Survey Research. San Francisco: Jossey-Bass, 1979.

    Bustamente, J.A., et al. "Characteristics of
    Mexican Immigrants, Legal and Undocumented, in Los
    Angeles County: A Pilot Study." Research Report.
    El Colegio de la Frontera Norte and University of
    Southern California, July 1996.

    Cochran, William G. Sampling Techniques. 3rd ed.
    New York: John Wiley & Sons, 1977.

    DaVanzo, Julie, Jennifer Hawes-Dawson, R. Burciaga
    Valdez, and Georges Vernez. Surveying Immigrant
    Communities: Policy Imperatives and Technical
    Challenges, Santa Monica, CA: RAND, 1994.

    DeMaio, Theresa, et al. "Protocol for Pretesting
    Demographic Surveys at the Census Bureau." Report
    of the Pretesting Committee. 1993.

    Droitcour, Judith, et al. "The Item Count
    Technique as a Method of Indirect Questioning: A
    Review of Its Development and a Case Study
    Application. In Measurement Errors in Surveys, by
    Paul P. Biemer et al. New York: John Wiley & Sons,
    Inc., 1991, pp. 185-210.

    Ferguson, George A. Statistical Analysis in
    Psychology and Education. 4th ed. New York: McGraw-
    Hill Book Company, 1976.

    Gallup Poll Monthly, No. 383, August 1997.

    Gfroerer, Joseph. "Influence of Privacy on Self-
    Reported Drug Use by Youths," in Self-Report
    Methods of Estimating Drug Use: Meeting Current
    Challenges to Validity, edited by Beatrice A.
    Rouse, Nicholas J. Kozel, and Louise G. Richards.
    NIDA Research Monograph 57. Rockville, MD:
    National Institute on Drug Abuse, 1985, pp. 22-30.

    Hansen, Morris H., William N. Hurwitz, and William
    G. Madow. Sample Survey Methods and Theory. Volume
    II: Theory. New York: John Wiley & Sons, 1953.

    Heer, David M. Undocumented Mexicans in the United
    States. New York: Cambridge University Press,
    1990.

    Hendershot, Tabitha P., et al. "Multilingual
    ACASI: Using English-Speaking Interviewers to
    Survey Elderly Members of Korean-Speaking
    Households." In Richard Warnecke, Health Survey
    Research Methods Conference Proceedings, DHHS
    Publication No. (PHS) 96-1013. Hyattsville, MD:
    National Center for Health Statistics, 1996, pp.
    165-69.

    Jasso, Guillermina. Department of Sociology, New
    York University. Personal communication, March 16,
    1999.

    Johnston, Lloyd D., Patrick M. O'Malley, and
    Jerald G. Bachman. National Survey Results on Drug
    Use from The Monitoring The Future Study, 1975-
    1997, Vol. 1. Secondary School Students. National
    Institute on Drug Abuse. NIH Publication No. 98-
    4345. Washington, D.C.: U.S. Government Printing
    Office, 1998.

    Kraly, Ellen Percy, and Robert Warren. "Estimates
    of Long-Term Immigration to the United States:
    Moving U.S. Statistics toward United Nations
    Concepts." Demography, 29 (Nov. 1992), pp. 613-26.

    Levine, Daniel B., Kenneth Hill, and Robert Warren
    (eds.). Immigration Statistics: A Story of
    Neglect. Washington, D.C.: National Academy Press,
    1985.

    Locander, W., S. Sudman, and N. Bradburn. "An
    Investigation of Interview Method, Threat and
    Response Distortion." Journal of the American
    Statistical Association,Vol. 71(1976), pp. 269-75.

    Loeb, Vernon. "Senators Challenge Energy Polygraph
    Plan: Nuclear Workers' Reputations Could Be at
    Risk in Unreliable Testing, Officials Say."
    Washington Post, Sept. 26, 1999, p. A20.

    Loue, Sana and Arwen Bunce. "The Assessment of
    Immigration Status in Health Research." Vital and
    Health Statistics. Series 2: Data Evaluation and
    Methods Research No. 127. National Center for
    Health Statistics, Hyattsville, MD, Sept. 1999.
    DHHS Publication No. (PHS) 99-1327. Washington,
    D.C.: U.S. Government Printing Office, 1999.

    Mines, Richard, et al. A Profile of U.S.
    Farmworkers: Demographics, Household Composition,
    Income and Use of Services. Washington, D.C.:
    Department of Labor, Office of the Assistant
    Secretary for Policy, 1997.

    Passel, Jeffrey S., and Rebecca L. Clark.
    Immigrants in New York: Their Legal Status,
    Incomes, and Taxes. Washington, D.C.: The Urban
    Institute, 1998.

    Pincus, Walter, and Vernon Loeb. "Lie Tests Anger
    Lab Scientists: DOE Workers Say Morale
    Threatened." Washington Post, Sept. 23, 1999, pp.
    A3, A14.

    Polivka, Anne E. and Jennifer M. Rothgeb.
    "Overhauling the Current Population Survey:
    Redesigning the Questionnaire." Monthly Labor
    Review 116: 9 (Sept. 1993), pp. 10-28.

    Rossi, Peter H., et al. Handbook of Survey
    Research. Orlando, FL: Academic Press, Inc., 1983.

    Simons, John. "Is a Web Political Poll Reliable?
    Yes? No? Maybe?" Wall Street Journal, April 13,
    1999, p. B1.

    Schryock, Henry S., and Jacob S. Siegel and
    Associates. The Methods and Materials of
    Demography. 4th ed. Washington, D.C.: U.S.
    Government Printing Office, 1980.

    Turner, Charles F., et al. "Impact of ACASI on
    Reporting of Male-Male Sexual Contacts:
    Preliminary Results From the 1995 National Survey
    of Adolescent Males." In Richard Warnecke, Health
    Survey Research Methods Conference Proceedings,
    DHHS Publication No. (PHS) 96-1013. Hyattsville,
    MD: National Center for Health Statistics, 1996,
    pp. 171-76.

    Turner, C. F., et al. "Adolescent Sexual Behavior,
    Drug Use, and Violence: Increased Reporting With
    Computer Survey Technology." Science, Vol. 280
    (1998), pp. 867-73.

    Udry, J. Richard, et al. "A Medical Record Linkage
    Analysis of Abortion Underreporting." Family
    Planning Perspectives, Vol. 28, No. 5,
    September/October 1996.

    U.S. House of Representatives. "Omnibus Oversight
    Hearing on Illegal Immigration." Committee on the
    Judiciary, Subcommittee on Immigration and Claims,
    Mar. 18, 1999.

    U.S. House of Representatives. "Hearing on the H-
    1B Temporary Professional Worker Visa Program and
    Information Technology Workforce Issues."
    Committee on the Judiciary, Subcommittee on
    Immigration and Claims, Aug. 5, 1999.

    U.S. Immigration and Naturalization Service.
    Statistical Yearbook of the Immigration and
    Naturalization Service, 1996. Washington, D.C.:
    U.S. Government Printing Office, 1997.

    U.S. General Accounting Office. Superfund: Half
    the Sites Have All Cleanup Remedies in Place or
    Completed (GAO/RCED-99-245). Washington, D.C.:
    GAO, July 1999.

    U.S. General Accounting Office. Immigration
    Statistics: Information Gaps, Quality Issues Limit
    Utility of Federal Data to Policymakers (GAO/GGD-
    98-164). Washington, D.C.: GAO, July 1998.

    U.S. General Accounting Office. H-2A Agricultural
    Guestworker Program: Changes Could Improve
    Services to Employers and Better Protect Workers
    (GAO/HEHS-98-20). Washington, D.C.: GAO, Dec.
    1997.

    U.S. General Accounting Office. Illegal
    Immigration: INS Overstay Estimation Methods Need
    Improvement (GAO/PEMD-95-20). Washington, D.C.:
    GAO, Sept. 1995.

    U.S. General Accounting Office. Immigration and
    the Labor Market: Nonimmigrant Alien Workers in
    the United States. (GAO/PEMD-92-17). Washington,
    D.C.: GAO, Apr. 1992.

    U.S. General Accounting Office. Immigration
    Reform: Major Changes Likely Under S. 358.
    (GAO/PEMD-90-5). Washington, D.C.: GAO, Nov. 1989.

    U.S. General Accounting Office. Immigration:
    Projected Immigration Under S. 448 and Recent
    Trends in Legal Immigration. (GAO/PEMD-89-12).
    Washington, D.C.: GAO, Apr. 1989.

    U.S. General Accounting Office. Immigration: Data
    Not Sufficient for Proposed Legislation. (GAO/PEMD-
    89-8). Washington, D.C.: GAO, Dec. 1988.

    U.S. General Accounting Office. Immigration: S.
    358 Would Change the Distribution of Immigrant
    Classes. (GAO/T-PEMD-89-1). Washington, D.C.: GAO,
    Mar. 1989.

    U.S. General Accounting Office. Immigration: The
    Future Flow of Legal Immigration to the United
    States. (GAO/PEMD-88-7). Washington, D.C.: GAO,
    Jan. 1988.

    Warner, Stanley. "Randomized Response: A Survey
    Technique for Eliminating Evasive Answer Bias."
    Journal of the American Statistical Association,
    Vol. 60 (1965), pp. 63-69.

    Warren, Robert, and Jeffrey S. Passel. "A Count of
    the Uncountable: Estimates of Undocumented Aliens
    Counted in the 1980 United States Census."
    Demography, Vol. 24:3(1987), pp. 375-93.

    Willis, Gordon B. "Cognitive Interviewing and
    Questionnaire Design: A Training Manual." Office
    of Research and Methodology, National Center for
    Health Statistics, Hyattsville, MD., n.d.

    *** End of Document ***