2000 Census: Best Practices and Lessons Learned for More
Cost-Effective Nonresponse Follow-up (11-FEB-02, GAO-02-196).
Nonresponse follow-up--in which Census Bureau enumerators go
door-to-door to count individuals who have not mailed back their
questionnaires--was the most costly and labor intensive of all
2000 Census operations. According to Bureau data, labor, mileage,
and administrative costs totaled $1.4 billion, or 22 percent of
the $6.5 billion allocated for the 2000 Census. Several practices
were critical to the Bureau's timely competition of nonresponse
follow-up. The Bureau (1) had an aggressive outreach and
promotion campaign, simplified questionnaire, and other efforts
to boost the mail response rate and thus reduce the Bureau's
nonresponse follow-up workload; (2) used a flexible human capital
strategy that enabled it to meet its national recruiting and
hiring goals and position enumerators where they were most
needed; (3) called on local census offices to identify local
enumeration challenges, such as locked apartment buildings and
gated communities, and to develop action plans to address them;
and (4) applied ambitious interim "stretch" goals that encouraged
local census offices to finish 80 percent of their nonresponse
follow-up workload within the first four weeks and be completely
finished by the end of the eighth week, as opposed to the
ten-week time frame specified in the Bureau's master schedule.
Although these initiatives were key to meeting tight time frames
for nonresponse follow-ups, the Bureau's experience in
implementing them highlights challenges for the next census in
2010. First, maintaining the response rate is becoming
increasingly expensive. Second, public participation in the
census remains problematic. Third, the address lists used for
nonresponse follow-up did not always contain the latest available
information because the Bureau found it was infeasible to remove
many late-responding households. Fourth, the Bureau's stretch
goals appeared to produce mixed results. Finally, there are
questions about how reinterview procedures aimed at detecting
enumerator fraud and other quality problems were implemented.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-02-196
ACCNO: A02725
TITLE: 2000 Census: Best Practices and Lessons Learned for More
Cost-Effective Nonresponse Follow-up
DATE: 02/11/2002
SUBJECT: Administrative costs
Agency missions
Best practices
Census
Cost control
Data collection
Information systems
Labor costs
Strategic planning
Transportation costs
2000 Decennial Census
2010 Decennial Census
Census Bureau Accuracy and Coverage
Evaluation Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-02-196
United States General Accounting Office
GAO Report to Congressional Committees
February 2002
2000 CENSUS
Best Practices and Lessons Learned for More Cost-Effective Nonresponse
Follow-up
a
GAO-02-196
Contents
Letter
Results in Brief
Background
Scope and Methodology
The Bureau Used an Aggressive Outreach and Promotion Campaign and Other
Strategies to Boost the Mail Response Rate but Public Cooperation Remains
Problematic
Flexible Human Capital Strategies Helped the Bureau Meet Its Recruitment
Goals
Local Census Offices Planned in Advance for Specific Enumeration Challenges
The Bureau's Stretch Goals to Complete Nonresponse Follow-up May Have
Produced Mixed Results
Questions Surround Whether Certain Reinterview Procedures Were Implemented
as Intended
Conclusions
Recommendations for Executive Action
Agency Comments and Our Evaluation 1 2 5 7
10
14
18
23
29 32 33 33
Appendixes
Appendix I: Local Census Offices Included in This Review 38 Appendix II:
Comments from the Secretary of Commerce 41 Appendix III: GAO Contacts and
Staff Acknowledgments 47
Related GAO Products on the Results of the 2000 Census and Lessons Learned
for a More Cost-Effective Census in 2010
Figures Figure 1: Local Census Offices Generally Completed Nonresponse
Follow-up Ahead of Schedule 6 Figure 2: Nonresponse Follow-up Workload
Completion Rates for the 1990 and 2000 Censuses 7
Contents
Figure 3: Public Cooperation with the Census Has Steadily
Declined 12
Figure 4: 2000 Census Return Rates Declined in Most States
Compared to 1990 13
Figure 5: Local Managers' Perceptions of Recruiting and Hiring 17
Figure 6: Local Managers' Perceptions of the Accuracy of
Nonresponse Follow-up Address Lists 20
Figure 7: Local Managers' Perceptions of the Accuracy of Maps 21
Figure 8: Local Managers' Views on the Impact of Scheduling
Pressures on the Quality of Nonresponse Follow-up 24
Figure 9: Collection of Partial Interview and Closeout Data
Remained Relatively Constant Throughout Nonresponse
Follow-up 27
Figure 10: Percentage of Local Census Offices Collecting Less
Complete Data 28
A
United States General Accounting Office Washington, D.C. 20548
February 11, 2002
The Honorable Joseph I. Lieberman
Chairman
The Honorable Fred Thompson
Ranking Minority Member
Committee on Governmental Affairs
United States Senate
The Honorable Dan Burton
Chairman
The Honorable Henry A. Waxman
Ranking Minority Member
Committee on Government Reform
House of Representatives
The Honorable Dave Weldon
Chairman
The Honorable Danny K. Davis
Ranking Minority Member
Subcommittee on Civil Service and Agency Organization
Committee on Government Reform
House of Representatives
Nonresponse follow-up-where enumerators from the Bureau of the
Census went door-to-door to count those individuals who did not mail back
their questionnaires-was the most costly and labor intensive of all 2000
Census operations. According to bureau data, labor, mileage, and certain
administrative costs alone amounted to about $1.4 billion, or about 22
percent of the total $6.5 billion allocated for the 2000 Census from fiscal
year 1991 through fiscal year 2003. In terms of employment, the bureau
hired about a half a million enumerators, which temporarily made it one of
the nation's largest employers, surpassed by only a handful of big
organizations like Wal-Mart and the U.S. Postal Service. Moreover, the
workload and schedule of nonresponse follow-up-the need to collect data
from about 42 million nonresponding households within a 10-week time
frame-made the conduct of this operation extraordinarily difficult and
complex.
In our prior work we noted that the success of nonresponse follow-up
would depend in large part on the bureau's ability to maintain data quality
while completing the operation on schedule, before error rates increased
as people moved or had trouble recalling who was living at their homes on
Census Day-April 1. Timeliness was also important for keeping subsequent
census operations on-track. In particular, this included the Accuracy and
Coverage Evaluation (A.C.E.), which was a separate sample survey designed to
assess the quality of the population data collected in the 2000 Census. For
methodological reasons, the bureau needed to complete its field data
collection workload for nonresponse follow-up before A.C.E. field data
collection could begin.
To its credit, the bureau generally completed nonresponse follow-up
consistent with its operational plan. Nationwide, according to bureau data,
the 511 local census offices located in the 50 states generally completed
nonresponse follow-up in slightly less time than the bureau's planned
10-week schedule. This was a noteworthy accomplishment given the operational
uncertainties the bureau faced, and stands in sharp contrast to the bureau's
1990 experience when nonresponse follow-up was hampered by unanticipated
workload and staffing problems and was completed 6 weeks behind schedule.
This report is the latest in our series of reviews that examine the results
of key census-taking operations and highlight opportunities for reform (see
the last page of this report for a list of products issued to date). Our
objectives were to identify (1) practices that contributed to the timely
completion of nonresponse follow-up and (2) lessons learned in implementing
these practices that the bureau may want to consider as it plans for
nonresponse follow-up during the next census in 2010.
Results in Brief Several practices were critical to the bureau's timely
completion of nonresponse follow-up. The bureau
* had an aggressive outreach and promotion campaign, simplified
questionnaire, and other efforts to boost the mail response rate and thus
reduce the bureau's nonresponse follow-up workload;
* used a flexible human capital strategy that enabled it to meet its
national recruiting and hiring goals and position enumerators where they
were most needed;
* called on local census offices to identify local enumeration challenges,
such as locked apartment buildings and gated communities, and to develop
action plans to address them; and
* applied ambitious interim "stretch" goals that encouraged local census
offices to finish 80 percent of their nonresponse follow-up workload
within the first 4 weeks and be completely finished by the end of the 8th
week, as opposed to the 10-week time frame specified in the bureau's master
schedule.
Although these initiatives were key to meeting nonresponse follow-up's tight
time frames, the bureau's experience in implementing them highlights several
significant challenges that lie ahead for the next census in 2010. First,
maintaining the response rate is becoming increasingly expensive. While the
bureau achieved similar response rates in 1990 and 2000 (65 percent in 1990
and 64 percent in 2000), the bureau spent far more money on outreach and
promotion in 2000: about $3.19 per household in 2000 compared to $0.88 in
1990 (in constant fiscal year 2000 dollars). Moreover, given a variety of
social, demographic, and attitudinal trends, such as changes in household
makeup and stability, concerns over privacy, and an increasing
non-English-speaking population, achieving comparable results in 2010 will
likely require an even larger investment of bureau resources.
Second, public participation in the census remains problematic. Indeed,
preliminary data on the mail return rate-a more precise indicator of public
cooperation with the census than the mail response rate-declined from 74
percent to 72 percent from 1990 to 2000.1 Also, there still appears to be a
large gap between the relatively large number of people who were aware of
the 2000 Census and those that actually responded. Bridging this gap has
been a longstanding difficulty for the bureau.
Third, the address lists used for nonresponse follow-up did not always
contain the latest available information, in part because the bureau found
it was infeasible to remove many late-responding households. As a result,
enumerators needed to visit over 773,000 households that had already mailed
back their questionnaires-an effort that approached $22 million in
additional costs for nonresponse follow-up, based on our estimate, and
confused respondents. An additional challenge was that some of the maps
enumerators used to help them find addresses during nonresponse follow-up
contained inaccuracies.
1 The initial mail response rate is calculated as a percentage of all forms
in the mail-back universe from which the bureau received a questionnaire. It
factors in housing units that are discovered to be nonexistent or unoccupied
during nonresponse follow-up. The bureau uses this percentage as an
indicator of its nonresponse follow-up workload. This differs from the mail
return rate which the bureau uses as a measure of public cooperation. It is
the percentage of forms the bureau receives from occupied housing units in
the mail-back universe and is calculated after the bureau completes the
enumeration process.
Fourth, the bureau's stretch goals appeared to produce mixed results. On the
one hand, on the basis of our survey of local census office managers, we
estimate that about 41 percent of managers believed scheduling pressures had
little or no impact on the quality of the nonresponse follow-up operation.
Another 17 percent of managers believed that such pressure had a positive or
significantly positive impact. On the other hand, about 40 percent of the
local census office managers believed that scheduling pressures during
nonresponse follow-up had a negative or significantly negative impact on the
quality of the operation. A common concern appeared to be that scheduling
pressures created a culture that emphasized quantity over quality.
One indicator of the quality of nonresponse follow-up is the completeness of
the data collected by enumerators. During nonresponse follow-up, a small
number of local census offices-in some highly publicized
incidents-improperly collected less complete data and took other short-cuts
(which the bureau took steps to rectify). Nationally, however, our analysis
of bureau data found that those offices that completed their follow-up
workloads faster than the others did not collect larger quantities of less
complete data, such as partial interviews.
Finally, questions surround the extent to which certain reinterview
procedures aimed at detecting enumerator fraud and other quality problems
were implemented throughout the entire nonresponse follow-up operation as
intended. The decision to subject enumerators' work to these procedures was
at the discretion of local census personnel. Fifty-two local census offices
(about 10 percent of all local offices) did not conduct any reinterviews
after a random check of enumerators' initial work. A senior bureau quality
assurance official expressed concerns about the adequacy of quality
assurance coverage toward the end of nonresponse follow-up at these offices
because once random reinterviews were completed at those offices, there were
no additional checks specifically designed to detect fabricated data.
In light of these challenges, as the bureau plans for the next national head
count in 2010, we recommend that the Secretary of Commerce ensure that the
bureau
* develop and refine the lessons learned from the nonresponse follow-up
effort and apply them to the planning efforts for 2010;
* assess, to the extent practicable, why people who were aware of the census
did not participate, and develop appropriate marketing strategies;
* develop and test options that could generate more current nonresponse
follow-up address lists and maps;
* ensure that the bureau's procedures and incentives for the timely
completion of nonresponse follow-up emphasize the collection of quality data
and proper enumeration techniques as much as speed; and
* ensure that the bureau's reinterview procedures, as implemented, are
sufficient for consistently and reliably detecting potential quality
problems throughout the full duration of enumerators' employment on
nonresponse follow-up.
The Secretary of Commerce forwarded written comments from the Bureau of the
Census on a draft of this report. The bureau concurred with all five of our
recommendations. The bureau also clarified several key points and provided
additional information and perspective, which we incorporated in our report
as appropriate.
Background In conducting nonresponse follow-up, the bureau has historically
faced the twin challenge of (1) collecting quality data (by obtaining
complete and accurate information directly from household members) while (2)
finishing the operation on schedule, before error rates can increase as
people move or have trouble recalling who was living at their homes on
Census Day (April 1), as well as keeping subsequent operations on-track.
Nonresponse follow-up was scheduled to begin on April 27, 2000, and end 10
weeks later, on July 7, 2000.
Local census offices generally finished their nonresponse follow-up
workloads ahead of the bureau's 10-week schedule.2 As shown in figure 1, of
the bureau's 511 local offices in the 50 states, 463 (91 percent) finished
nonresponse follow-up by the end of the eighth week of the operation,
consistent with the bureau's internal stretch goals. Moreover, nine local
offices completed their workloads in as little as 5 weeks or less.
2 The completion time excludes certain follow-up activities conducted after
the bureau finished its initial workload.
Figure 1: Local Census Offices Generally Completed Nonresponse Follow-up
Ahead of Schedule
250 Number of local census offices
225
200
175
150
125
100
75
50
25
0
45 6789
Week that nonresponse follow-up was finished
Source: GAO analysis of Census Bureau data.
The timely completion of nonresponse follow-up in 2000 stands in sharp
contrast to the bureau's experience during the 1990 Census. As shown in
figure 2, at the end of the 6-week scheduled time frame for nonresponse
follow-up during the 1990 Census, the bureau had not completed the
operation. In fact, as of two days prior to the scheduled end date, just two
local census offices had completed the operation and the bureau had only
completed about 72 percent of its 34 million household follow-up workload.
It took the bureau a total of 14 weeks to complete the entire operation. By
comparison, as noted above, the bureau completed nonresponse follow-up in
less than 10 weeks during the 2000 Census.
Figure 2 also highlights the drop-off in production that occurs during the
later weeks of nonresponse follow-up. According to the bureau, the decline
occurs because unresolved cases at the end of nonresponse follow-up are
typically the most difficult to reach, either because they are uncooperative
or are rarely at home and are unknown to neighbors.
Figure 2: Nonresponse Follow-up Workload Completion Rates for the 1990 and
2000 Censuses
Percentage of nonresponse follow-up workload completed 100 90
80
70
60
50
40
30
20
10 0
To meet our objectives, we used a combination of approaches and methods to
examine the conduct of nonresponse follow-up. These included statistical
analyses; interviews with key bureau headquarters officials, regional census
center officials, and local census office managers and staff; observations
of local census offices' nonresponse follow-up operations; and reviews of
relevant documentation.
To examine the factors that contributed to the timely completion of
nonresponse follow-up, we interviewed local census office managers and other
supervisory staff at 60 local census offices we visited across the country.
These offices generally faced specific enumeration challenges when
nonresponse follow-up began in late April, and were thus prone to
operational problems that could affect data quality (see app. I for a
123 4567891011121314
Scheduled end of nonreponse follow-up 1990 Scheduled end of nonreponse
follow-up 2000
Week of nonresponse follow-up
2000 completion rate 1990 completion rate
Source: GAO analysis of Census Bureau data.
Scope and Methodology
complete list of the offices we visited). Specifically, these offices had
(1) a larger nonresponse follow-up workload than initially planned; (2)
multiple areas that were relatively hard-to-enumerate, such as
non-English-speaking groups; and (3) difficulties meeting their enumerator
recruiting goals. During these visits, which took place in June and July
2000, we also observed office operations to see how office staff were
processing questionnaires; at 12 of these offices we attended enumerator
training; and at 31 offices we reviewed key reinterview documents in a given
week during nonresponse follow-up. The local census offices we visited
represent a mix of urban, suburban, and rural locations. However, because
they were judgmentally selected, our findings from these visits cannot be
projected to the universe of local census offices.
To obtain a broader perspective of the conduct of nonresponse follow-up, we
used the results of our survey of a stratified random sample of managers at
250 local census offices. The survey-which asked these managers about the
implementation of a number of key field operations- is generalizable to the
511 local census offices located in the 50 states. 3 We obtained responses
from managers at 236 local census offices (about a 94 percent overall
response rate). All reported percentages are estimates based on the sample
and are subject to some sampling error as well as nonsampling error. In
general, percentage estimates in this report for the entire sample have
confidence intervals ranging from about � 4 to � 5 percentage points at the
95 percent confidence interval. In other words, if all managers in our local
census office population had been surveyed, the chances are 95 out of 100
that the result obtained would not differ from our sample estimate in the
more extreme cases by more than � 5 percent.
3 Our analysis did not include nine local census offices located in Puerto
Rico.
To examine whether the pace of nonresponse follow-up was associated with the
collection of less complete data, in addition to the efforts described
above, we analyzed bureau data on the weekly progress of nonresponse
follow-up. Specific measures we analyzed included the time it took local
census offices to finish nonresponse follow-up and the proportion of their
cases completed by (1) "close-out" interviews, where questionnaires only
contain basic information on the status of the housing unit (e.g., whether
it was occupied), or (2) "partial" interviews, which contain more
information than a close-out interview but are still less than complete. The
completeness of the data collected by enumerators is one measure of the
quality of nonresponse follow-up, and these two measures were the best
indicators of completeness available from the database. We included data
from the 511 offices located in the 50 states and controlled for enumeration
difficulty using an index measure developed by the bureau.4 We did not
include any outliers that the bureau identified as erroneous (for example,
outliers resulting from coding errors).5
We did our audit work at the local census offices identified in appendix I
and their respective regional census centers; bureau headquarters in
Suitland, Maryland; and Washington, DC, from March 2000 through September
2001. Our work was done in accordance with generally accepted government
auditing standards.
We requested comments on a draft of this report from the Secretary of
Commerce. On January 10, 2002, the Secretary forwarded the bureau's written
comments on the draft (see app. II) which we address at the end of this
report.
4 The index measure, or "hard-to-count score," was based on variables
contained in the 1990 Data for Census 2000 Planning Database, such as the
percent of households with no adult who speaks English well.
5 Of the 511 local offices, 3 were not included in the analysis of partial
interviews and 12 were not included in the analysis of closeout interviews
because the bureau identified their values for these variables as erroneous
due to coding errors.
The Bureau Used an Aggressive Outreach and Promotion Campaign and Other
Strategies to Boost the Mail Response Rate but Public Cooperation Remains
Problematic
Key to the bureau's timely completion of nonresponse follow-up in 2000 was a
higher than expected initial mail response rate that decreased the bureau's
follow-up workload. In addition to reducing the staff, time, and money
required to complete the census count, the bureau's past experience and
evaluations suggest that the quality of data obtained from questionnaires
returned by mail is better than the data collected by enumerators.
To help raise the mail response rate, the bureau (1) hired a consortium of
private-sector advertising agencies, led by Young & Rubicam, to develop a
national, multimedia paid advertising program, and (2) partnered with local
governments, community groups, businesses, nongovernmental organizations,
and other entities to promote the census on a grassroots basis (we discuss
the bureau's partnership program in more detail in our August 2001 report).6
The outreach and promotion campaign encouraged people to complete their
census questionnaires by conveying the message that census participation
helped their communities. The bureau also helped boost the mail response
rate by using simplified questionnaires, which was consistent with our past
suggestions,7 and by developing more ways to respond to the census, such as
using the Internet.
The bureau achieved an initial mail response rate of about 64 percent, which
was about 3 percentage points higher than the 61 percent response rate the
bureau expected when planning for nonresponse follow-up.8 This, in turn,
resulted in a nonresponse follow-up workload of about 42 million housing
units, which was about 4 million fewer housing units than the bureau would
have faced under its planning assumption of a 61 percent mail response rate.
6 2000 Census: Review of Partnership Program Highlights Best Practices for
Future Operations (GAO-01-579, Aug. 20, 2001).
7 See for example, Decennial Census: 1990 Results Show Need for Fundamental
Reform (GAO/GGD-92-94, June 9, 1992).
8 For the 2000 Census, the bureau used what it refers to as an "initial
response rate" to provide a measure of the scope of the nonresponse
follow-up operation. This initial response rate is defined as the percentage
of all questionnaires that are completed and returned by April 18, 2000. The
rate includes the number of questionnaires that are mailed back, transmitted
via the Internet, or completed over the telephone through the bureau's
Telephone Questionnaire Assistance program. It also includes Be Counted
Forms that have census identification numbers. On September 19, 2000, the
bureau announced that it had achieved a final mail-back response rate of 67
percent.
In addition to surpassing its national response rate goals, the bureau
exceeded its own expectations at the local level. Of the 511 local census
offices, 378 (74 percent) met or exceeded the bureau's expected response
rate. In so doing, these offices reduced their nonresponse follow-up
workloads from the expected levels by between 54 and 58,329 housing units.
The remaining 133 offices (26 percent) did not meet their expected response
rate and the workload at these offices increased from their expected levels
by between 279 and 33,402 housing units.
Securing Public Participation While Controlling Costs Remains a Considerable
Challenge for the 2010 Census
The bureau's success in surpassing its response rate goals was noteworthy
given the formidable societal challenges it faced. These challenges included
attitudinal factors such as concerns over privacy, and demographic trends
such as more complex living arrangements. However, as the bureau plans for
the next census in 2010, it faces the difficulty of boosting public
participation while keeping costs manageable.
As we noted in our December 2001 report, although the bureau achieved
similar response rates in 1990 and 2000 (65 percent in 1990 and 64 percent
in 2000), the bureau spent far more money on outreach and promotion in 2000:
about $3.19 per household in 2000 compared to $0.88 in 1990 (in constant
fiscal year 2000 dollars), an increase of 260 percent.9 Moreover, the
societal challenges the bureau encountered in 1990 and 2000 will probably be
more complex in 2010, and simply staying on par with the 2000 response rate
will likely require an even greater investment of bureau resources.
Further, while the mail response rate provides a direct indication of the
nonresponse workload, it is an imperfect measure of public cooperation with
the census as it is calculated as a percentage of all forms in the mail-back
universe from which the bureau received a questionnaire. Because the
mail-back universe includes housing units that the bureau determines are
nonexistent or vacant during nonresponse follow-up, a more precise measure
of public cooperation is the mail return rate, which excludes vacant and
nonexistent housing units. According to preliminary bureau data, the mail
return rate for the 2000 Census was 72 percent, a decline of 2 percentage
points from the 74 percent mail return rate the bureau achieved in 1990. As
shown in figure 3, in 2000, the bureau reduced, but did not
9 2000 Census: Significant Increase in Cost Per Housing Unit Compared to
1990 Census (GAO-02-31, Dec. 11, 2001).
reverse, the steady decline in public cooperation that has occurred with
each decennial census since the bureau first initiated a national
mail-out/mail-back approach in 1970. Bureau officials said they would
further examine the reasons for the decline in the return rate as part of
its Census 2000 evaluations.
Figure 3: Public Cooperation with the Census Has Steadily Declined
100 Mail Return Rate
87
80
60
40
20
0
1970 1980 1990 2000 Decennial Year
Source: GAO analysis of Census Bureau data.
In addition, as shown in figure 4, the results to date show that just three
states increased their mail return rates compared to the 1990 Census.
Overall, preliminary bureau data shows the change in mail return rates from
1990 through 2000 ranged from an increase of about 1 percentage point in
Massachusetts and California to a decline of about 9 percentage points in
Kentucky.
The bureau's outreach and promotion efforts will also face the historical
hurdle of bridging the gap that exists between the public's awareness of the
census on the one hand, and its motivation to respond on the other. Various
polls conducted for the 2000 Census suggested that the public's awareness of
the census was over 90 percent; and yet, as noted earlier, the actual return
rate was much lower-72 percent of the nation's households. The bureau faced
a similar issue in 1990 when 93 percent of the public reported being aware
of the census, but the return rate was 74 percent. In our previous work, we
noted that closing this gap would be a significant challenge for the bureau,
and as the bureau plans for the 2010 Census, it will be important for it to
explore approaches that more effectively convert the public's awareness of
the census into a willingness to respond.10
Flexible Human Capital Strategies Helped the Bureau Meet Its Recruitment
Goals
A second factor that was instrumental to the operational success of
nonresponse follow-up was an ample and sufficiently skilled enumerator
workforce. Based on anticipated turnover and the expected workload to carry
out its four largest field data collection operations-of which nonresponse
follow-up was the largest-the bureau set a recruitment goal of 2.4 million
qualified applicants.11 In addition to the sheer volume of recruits needed,
the bureau's efforts were complicated by the fact that it was competing for
employees in a historically tight national labor market. Nevertheless, when
nonresponse follow-up began on April 27, the bureau had recruited over 2.5
million qualified applicants.
The bureau surmounted its human capital challenge with an aggressive
recruitment strategy that helped make the bureau a more attractive employer
to prospective candidates and ensured a steady stream of applicants. Key
ingredients of the bureau's recruitment efforts included the following:
1. A geographic pay scale with wages set at 65 to 75 percent of local
prevailing wages (from about $8.25 to $18.50 per hour for enumerators). The
bureau also used its flexibility to raise pay rates for those census offices
that were encountering recruitment difficulties.
10 2000 Census: Preparations for Dress Rehearsal Leave Many Unanswered
Questions (GAO/GGD-98-74, Mar. 26, 1998).
11 The bureau later adjusted its qualified applicant goal to 2.1 million
based on the actual nonresponse follow-up workload.
For example, a manager at one of the Charlotte region's local census offices
told us that the office was having difficulty obtaining needed staff in part
because census wages were uncompetitive. According to this manager, the
region approved a pay increase for the office's enumerators and office
clerks, which helped the office obtain staff. In all, when nonresponse
follow-up began, the bureau raised pay rates for field staff at eight local
offices to address those offices' recruiting challenges.
2. Partnerships with state, local, and tribal governments, community groups,
and other organizations to help recruit employees and provide free
facilities to test applicants. For example, Clergy United, an organization
representing churches in the Detroit metropolitan area, provided space for
testing census job applicants in December 1998. The organization even
conducted pre-tests several days before each bureau-administered test so
those applicants could familiarize themselves with the testing format.
3. A recruitment advertising campaign, which totaled over $2.3 million, that
variously emphasized the ability to earn good pay, work flexible hours,
learn new skills, and do something important for one's community. Moreover,
the advertisements were in a variety of languages to attract different
ethnic groups, and were also targeted to different races, senior citizens,
retirees, and people seeking part-time employment. The bureau advertised
using traditional outlets such as newspaper classified sections, as well as
more novel media including Internet banners and messages on utility and
credit card bills.
4. Obtaining exemptions from the majority of state governments so that
individuals receiving Temporary Assistance for Needy Families, Medicaid, and
selected other types of public assistance would not have their benefits
reduced when earning census income, thus making census jobs more attractive.
At the start of nonresponse follow-up, 44 states and the Virgin Islands had
granted an exemption for one or more of these programs.
5. Encouraging local offices to continue their recruiting efforts throughout
nonresponse follow-up, regardless of whether offices had met their
recruiting goals, to ensure a steady stream of available applicants.
The bureau matched these initiatives with an ongoing monitoring effort that
enabled bureau officials to rapidly respond to recruiting difficulties. For
example, during the last 2 weeks of April, the bureau mailed over 5 million
recruiting postcards to Boston, Charlotte, and other locations where it
found recruitment efforts were lagging.
Based on the results of our local census office visits, it is clear that the
bureau's human capital strategy had positive outcomes. Of the 60 local
census offices we visited, officials at 59 offices provided useable
responses to our question about whether their offices had the type of staff
they needed to conduct nonresponse follow-up, including staff with
particular language skills to enumerate in targeted areas.12 Officials at 54
of the 59 offices said they had the type of staff they needed to conduct
nonresponse follow-up. For example, officials in the Boston North office
said they hired enumerators who spoke Japanese, Vietnamese, Portuguese,
Spanish, French, Russian, and Chinese, while Pittsburgh office officials
said they had enumerators that knew sign language to communicate with deaf
residents.
Managers at local census offices we surveyed provided additional perspective
on recruiting needed field staff. As shown in figure 5, 30 percent of the
respondents believed that the bureau's ability to recruit and hire
high-quality field staff needed no improvements. While managers at 52
percent of the local offices commented that some improvement to the
recruiting and hiring process was needed and another 17 percent commented
that a significant amount of improvement was needed, their suggestions
varied. Managers' suggestions generally related to various hiring practices,
such as a greater use of face-to-face interviews to select managers at local
census offices and earlier recruitment advertising.
12 At one of the local census offices we visited, we were unable to obtain a
useable response to this question generally because the local census
office's managers were unavailable during the time of our review.
Figure 5: Local Managers' Perceptions of Recruiting and Hiring
100 Percentage of local census offices
52
10 0
e
eent
judgto
No basis
Noentvem
entneeded vemo
pr
needed
needed
ignificantSim
omSvem
o
o
prim
primExtent of improvement needed
Source: GAO survey of local census office managers.
Once nonresponse follow-up began, bureau officials tracked production rates
as the primary measure of whether local offices had met their staffing
goals. For example, bureau officials said that both bureau headquarters and
regional census center staff monitored local census offices' production
daily. If an office was not meeting its production goals, bureau
headquarters officials said they worked with regional census personnel, who
in turn worked with the local census office manager, to determine the
reasons for the shortfall and the actions necessary to increase production.
Possible actions included bringing in enumerators from neighboring local
census offices.
Overall, preliminary bureau data shows that about 500,000 enumerators worked
on nonresponse follow-up. Nationally, the bureau established a hiring goal
of 292,000 enumerator positions for nonresponse follow-up,
which represented two people working approximately 25 hours per week for
each position and assumed 100 percent turnover, according to bureau
officials. The bureau has not yet analyzed how many enumerators charged at
least 25 hours per week during nonresponse follow-up. Moreover, according to
a senior bureau official, the bureau has not decided whether it will do such
an analysis for 2010 planning purposes. According to this official, because
the bureau hired about 500,000 enumerators and completed the operation a
week ahead of schedule, they believe the bureau generally met its hiring
goal.
Local Census Offices Planned in Advance for Specific Enumeration Challenges
A third factor that contributed to the timely completion of nonresponse
follow-up was preparing in advance for probable enumeration challenges. To
do this, the bureau called on local census offices and their respective
regional census centers to develop action plans that, among other things,
identified hard-to-enumerate areas within their jurisdictions, such as
immigrant neighborhoods, and propose strategies for dealing with those
challenges. These strategies included such methods as paired/team
enumeration for high-crime areas, and hiring bilingual enumerators. While
this early planning effort helped local census offices react to a variety of
enumeration challenges, the currency and accuracy of the nonresponse
follow-up address lists and maps remained problematic for a number of local
census offices.
Most Local Offices Used Action Plans to Address Enumeration Challenges
Of the 60 local census offices we visited, officials at 55 offices provided
useable responses to our question about how, if at all, their offices used
their action plan for hard-to-enumerate areas during nonresponse
follow-up.13 Officials at 51 of 55 offices said their offices used the
strategies in their action plan to address the enumeration challenges they
faced.
At the offices we visited, a frequently cited enumeration challenge was
gaining access to gated communities or secure apartment buildings. Officials
at 42 of the 60 offices we visited identified this as a problem. To address
it, officials said they developed partnerships with building management and
community leaders, among other strategies. In an Atlanta
13 At five of the local census offices we visited, we were unable to obtain
a useable response to this question generally because local census office
managers were either unavailable or did not know.
office, for example, local officials said they sent letters to managers of
gated communities that stressed the importance of the census. Similarly,
officials in a Chicago office said they personally phoned managers of secure
apartment buildings. When enumerators from a Milwaukee local census office
encountered problems accessing locked apartment buildings, local census
officials told us that the City of Milwaukee sent aldermen to visit the
building managers and encourage them to participate in the census.
Another common enumeration challenge appeared to be obtaining cooperation
from residents-cited as a difficulty by officials at 34 of the 60 offices we
visited. One problem they noted was obtaining responses to the long-form
questionnaire-either in its entirety or to specific items, such as
income-related questions--which, according to local census officials, some
residents found to be intrusive.
Enumerators also encountered residents who were unwilling to participate in
the census because of language and cultural differences, or their fears of
government. The bureau's standardized training for enumerators included
procedures for handling refusals. Local census officials encouraged public
participation with a variety of approaches as well. For example, census
officials in Cleveland and Cincinnati said they provided additional training
for enumerators on how to handle refusals and practiced what was taught in
mock interviews. Officials in other census offices said they partnered with
local community leaders who subsequently helped reach out to
hard-to-enumerate groups, hired people who were bilingual or otherwise
trusted and known by residents, and held media campaigns. Overall, according
to bureau data, close to 470,000 households of the approximately 42 million
making up the nonresponse follow-up workload (about 1 percent), refused to
participate in the census.
The Accuracy and Currency of Nonresponse Follow-up Address Lists and Maps
Appeared to Be Problematic
Of the 60 local census offices we visited, officials at 52 offices provided
useable responses to our question about whether their offices' nonresponse
follow-up address list reflected the most accurate and current
information.14 Officials at 21 of the 52 offices said that their lists
generally were not accurate and current. Nationwide, as shown in figure 6,
based on
14 At eight local census offices we visited, we were unable to obtain a
useable response to this question generally because local census office
managers were either unavailable or did not know.
our survey of local census office managers, we estimate that managers at
approximately 50 percent of local census offices believed that some
improvement was needed in the accuracy of address lists for nonresponse
follow-up. We estimated that managers at about 22 percent of local census
offices believed that a significant amount of improvement was needed.
Figure 6: Local Managers ' Perceptions of the Accuracy of Nonresponse
Follow-up Address Lists
100 Percentage of local census offices
50
Noent
neededvem
eent
entneeded o neededvempr
basisNoto
ejudg
ignificantSim
omSvem
o
o
prim
prim
Source: GAO survey of local census office managers.
Among the more frequent problems managers cited were duplicate addresses and
changes not being made from prior operations. For example, at a local census
office in the Seattle region, managers said that some addresses were
residences or businesses that had been gone for 10-15 years and should have
been deleted in previous census operations but were not.
Local census officials we visited cited problems with the accuracy of the
census maps as well. Of the 60 local census offices we visited, officials at
58 offices provided useable responses to our question about whether the most
accurate and current information was reflected on the nonresponse follow-up
maps. 15 Officials at about a third of local census offices-21 of 58
offices-said the nonresponse follow-up maps did not reflect the most
accurate and current information.
Further, as shown in figure 7, based on our survey of local census office
managers, at about 41 percent of the offices, managers believed that some
improvement was needed in maps for nonresponse follow-up. At about 23
percent of the offices, managers believed that a significant amount of
improvement was needed in these maps.
Figure 7: Local Managers' Perceptions of the Accuracy of Maps 100 Percentage
of local census offices
41
Noentneededvem
eSomentvem
basisto
e
entneeded vemopr
ignificantSim
needed judgNo
o
o
prim
primExtent of improvement needed
Source: GAO survey of local census office managers.
15 At two of the local census offices we visited, we were unable to obtain a
useable response to this question generally because local census office
managers were either unavailable or did not know.
Managers who commented that improvements were needed to the nonresponse
follow-up maps said the maps were difficult to use, not updated from prior
operations, and contained errors. For example, an official at a local census
office in the Atlanta region said that some roads on the map did not exist
or were not oriented correctly on the census maps. To address this
difficulty, local office staff purchased commercial maps or used the
Internet to help them locate some housing units.
The bureau developed its master address list and maps using a series of
operations that made incremental updates designed to continuously improve
the completeness and accuracy of the master address file and maps. A number
of these updates occurred during nonresponse follow-up when enumerators
encountered, for example, nonexistent or duplicate housing units, or units
that needed to be added to the address list. As a result, the bureau was
expecting some discrepancies between the nonresponse follow-up address list
and what enumerators found in the field when they went door-to-door, which
could account for some of the local census officials' perceptions.
Another factor that affected the currency of the nonresponse follow-up
address list was the cut-off date for mail-back responses. The bureau set
April 11, 2000, as the deadline for mail-back responses for purposes of
generating the address list for nonresponse follow-up. In a subsequent late
mail return operation, the bureau updated its field follow-up workload by
removing those households for which questionnaires were received from April
11 through April 18. However, according to bureau officials, the bureau
continued to receive questionnaires, in part because of an unexpected boost
from its outreach and promotion campaign. For example, by April 30-less than
2 weeks after the bureau removed the late mail returns that it had
checked-in as of April 18--the bureau received 773,784 additional
questionnaires. Bureau headquarters officials told us it was infeasible to
remove the late returns from the nonresponse follow-up address lists and
thus, enumerators needed to visit these households.
The cost of these visits approached $22 million, based on our earlier
estimate that a 1-percentage point increase in workload could add at least
$34 million in direct salary, benefits, and travel costs to the price tag of
nonresponse follow-up.16 In addition, the bureau's data processing centers
then had to reconcile the duplicate questionnaires. According to officials
16 GAO/GGD-00-6, December 14, 1999.
at some local offices we visited, the visits to households that had already
responded confused residents who questioned why enumerators came to collect
information from them after they had mailed back their census forms.
The Bureau's Stretch To help ensure that local census offices completed
nonresponse follow-up
on schedule, the bureau developed ambitious interim stretch goals.
TheseGoals to Complete goals called on local census offices to finish 80
percent of their Nonresponse Follow-nonresponse follow-up workload within
the first 4 weeks of the operation up May Have Produced and be completely
finished by the end of the eighth week. Under the
bureau's master schedule, local census offices had 10 weeks to completeMixed
Results the operation.
Local Census Office Managers Cited Both Positive and Negative Effects of the
Nonresponse Follow-up Schedule on the Quality of the Operation
Our survey of local census office managers asked what impact, if any,
scheduling pressures to complete nonresponse follow-up had on the quality of
the operation. On the one hand, as shown in figure 8, about 41 percent of
the local census office managers believed that scheduling pressures had
little or no impact on the quality of the operation, while about 17 percent
believed that such pressure had a positive or significantly positive impact.
At a local census office in the New York region, for example, the local
census office manager stated that, "pressuring people a little gave them the
motivation to produce." Managers in local census offices located in the
Dallas region commented that the schedule "kept people on their toes and
caused them to put forth their best effort" and that it "had a positive
impact, particularly on quality."
On the other hand, managers at a substantial number of local census offices
had the opposite view. As shown in figure 8, about 40 percent of the
respondents believed that scheduling pressure during nonresponse follow-up
had a negative or significantly negative impact on the quality of the
operation.
Figure 8: Local Managers' Views on the Impact of Scheduling Pressures on the
Quality of Nonresponse Follow-up
100 Percentage of local census offices
41 40
ositive to
Psignificantl
ve
basisNoto
Negative tonegatisignificantl
y
ve
Little or no e
y
judg
positi Type of impact
Note: Due to rounding, percentages do not equal 100 percent. Source: GAO
survey of local census office managers.
Of those managers who believed that the pressure to complete nonresponse
follow-up adversely affected the quality of the operation, a common
perception appeared to be that production was emphasized more than accuracy
and that the schedule required local census offices to curtail procedures
that could have improved data quality. For example, managers at some local
census offices told us that the bureau's regional census centers encouraged
competition between local census offices by, among other actions, ranking
local census offices by their progress and distributing the results to local
managers. Managers at some local census offices believed that such
competition fostered a culture where quantity was more important than
quality. As one manager told us, the bureau's
ambitious nonresponse follow-up schedule led the manager "to put enormous
pressure on people in the field to complete the operation quickly, and this
affected the quality of data." However, none of the managers we surveyed
cited specific examples of where corners were cut or quality was
compromised.
The Pace of Nonresponse Follow-up Was Not Associated with the Collection of
Less Complete Data
One measure of the quality of nonresponse follow-up is the completeness of
the data collected by enumerators. The bureau went to great lengths to
obtain complete data directly from household members. Bureau procedures
generally called for enumerators to make up to three personal visits and
three telephone calls to each household on different days of the week at
different times until they obtained needed information on that household.
However, in cases where household members could not be contacted or refused
to answer all or part of the census questionnaire, enumerators were
permitted to obtain data via proxy (a neighbor, building manager, or other
nonhousehold member presumed to know about its residents), or collect less
complete data than called for by the census questionnaire. Such data include
(1) "closeout" interviews, where questionnaires only contain the information
on the status of the housing unit (e.g., whether or not it was occupied),
and the number of residents and (2) "partial" interviews, which contain more
information than a closeout interview but less than a completed
questionnaire.
There were several well-publicized breakdowns in these enumeration
procedures at a small number of local census offices that took short cuts to
complete their work (which the bureau later took steps to rectify).
Nationally, however, our analysis of bureau data found no statistically
significant association between the week individual local census offices
finished their nonresponse follow-up workload and the percentage of
partial17 or closeout18 interviews they reported, after controlling for the
enumeration difficulty level of each local office's area19 (at the time of
our
17 Results of regression: t = -1.65; p = 0.10. 18 Results of regression: t =
-0.44; p = 0.66. 19 We used an index measure (hard-to-count score) developed
by the bureau.
review, the bureau did not have information on data collected via proxy
interviews).
Neither did we find a statistically significant relationship between the
week that local census offices finished their nonresponse follow-up workload
and the amount of residual workload,20 they had, if any. The residual
workload consisted of households that were part of the original follow-up
workload, but from which the bureau did not receive a questionnaire from the
local census offices, and thus had not been processed through data capture.
According to bureau data, 519 local offices had to conduct residual
nonresponse follow-up on 121,792 households.
Similarly, we did not find an association between week-to-week "spikes" in
local census offices' production and the percentage of either partial or
closeout interview data reported. Spikes or surges in production could
indicate that local census offices were cutting corners to complete their
workloads by a specific deadline. Nationally, we found no relationship
between the number of questionnaires finished each week and either the
percentage of those finished that were closeout interviews21 or partial
interviews.22
Overall, as shown in figure 9, as nonresponse follow-up progressed, the
proportion of closeout and partial interview data collected relative to the
amount of questionnaires finished remained relatively constant.
20 Results of regression: t = -.04; p = 0.97. 21 Results of correlation: r =
-.08. 22 Results of correlation: r = -.15.
Figure 9: Collection of Partial Interview and Closeout Data Remained
Relatively Constant Throughout Nonresponse Follow-up
45 Millions of Respondants
100 100
40 35 30 25 20 15 10 5 0
2 3 456 789 Ending week that nonresponse follow-up completed
Cumulative nonresponse follow-up workload finished Cumulative partial
interview data collected
Note: There were no bureau data available for weeks 1 and 10. Comparable
data for 1990 were not available for comparison to 2000 results. Percentage
of workload finished is out of the total workload; percentages of partial
interviews and closeouts are out of the workload completed.
Source: GAO analysis of Census Bureau data.
Moreover, only a small percentage of most local census offices' nonresponse
follow-up workload was finished using closeout and partial interviews. As
shown in figure 10, of the 499 local offices where reliable closeout data
were available,23 413 (83 percent) reported that less than 2
23 We excluded data for those local census offices that, according to the
bureau, were not reliable because of various anomalies, such as inaccurate
coding of questionnaires by local office staff.
percent of their questionnaires were finished in this manner, while 19
offices (4 percent) reported 5 percent or more of their finished nonresponse
follow-up work as closeout interviews. For partial interviews, of the 508
offices where reliable data were available, 267 (53 percent) reported
collecting less than 2 percent of such data, while 47 offices (9 percent)
reported 5 percent or more of their finished work as partial interviews. The
median percentages of closeout and partial interviews were .8 percent and
1.9 percent, respectively.
90
83
80 70 60
53
50
40
30
20
10
0 Partial interviews Closeout interviews Percentage of nonresponse follow-up
workload
Less than 2 percent
2 percent to less than 5 percent
5 percent or more
Note: Comparable data for 1990 were not available for comparison to 2000
results. Source: GAO analysis of Census Bureau data.
At those local census offices that had substantially higher levels of
closeout and partial interview data than other offices, the bureau said that
some of this was understandable given the enumeration challenges that these
census offices faced. For example, according to the bureau, the relatively
high partial interview rate at a New York local office (3.8 percent of that
office's finished nonresponse follow-up workload) was in line with the
regional average of 2.2 percent, partly due to the difficulty that staff had
in gaining access to apartment buildings. Once building managers gave
enumerators access and they were able to obtain information from proxies,
the number of refusals may have decreased, but the number of partial
interviews increased because the proxies could not provide complete
information.
Still, as noted above, some local census offices inappropriately used
certain enumeration techniques. For example, the Hialeah, Florida, office
reported finishing its nonresponse follow-up workload in 5 weeks-well ahead
of the 8-week stretch goals and 10 weeks allotted for the operation. The
Homestead, Florida, office-where Hialeah-trained enumerators were later
transferred to help complete nonresponse follow-up-reported finishing its
workload in 7 weeks. The Commerce Department's Office of the Inspector
General later found that Hialeah-trained enumerators did not make the
required number of visits and telephone calls before contacting a proxy for
information, and did not properly implement quality control procedures
designed to detect data falsification.24 The bureau responded to these
findings by, among other actions, reworking over 64,000 questionnaires from
the Hialeah and Homestead offices.
Questions Surround Whether Certain Reinterview Procedures Were Implemented
as Intended
To help ensure that enumerators followed proper enumeration procedures and
were not falsifying data, the bureau "reinterviewed" households under
certain circumstances to check enumerators' work. As such, reinterviews were
a critical component of the bureau's quality assurance program for
nonresponse follow-up. If falsification was detected during a reinterview,
the local office was to terminate the enumerator and redo all of the
enumerator's work. Enumerators making inadvertent errors were to correct
their mistakes and be retrained. The bureau conducted three types of
reinterviews:
1. Random reinterviews were to be performed on a sample of enumerators' work
during the early weeks of their employment. Seven randomly selected
questionnaires from each enumerator's first 70 cases were to have been
reinterviewed.
24 For more information on this incident, see U.S. Department of Commerce,
Office of Inspector General, Bureau of the Census: Re-enumeration at Three
Local Census Offices in Florida: Hialeah, Broward South, and Homestead
(ESD-13215-0-0001, Sept. 29, 2000).
2. Administrative reinterviews checked the work of enumerators whose
performance in certain dimensions (e.g., the number of partial interviews
conducted) differed significantly from that of other enumerators employed in
the same area-and there was no justification for the difference. In such
cases, enumerators could be fabricating data. According to the bureau,
administrative tests were designed to identify enumerators who were making
errors that were more likely to occur toward the end of the operation, after
the random check of enumerators' initial work. They were conducted at the
discretion of local census officials.
3. Supplemental reinterviews were to be conducted at the discretion of local
census officials when they had some basis for concern about the quality of
an enumerator's work.
On the basis of our work and that of the bureau, we found that local census
office officials often used their discretion to not conduct administrative
and supplemental reinterviews and thus, a number of local offices did not
conduct such reinterviews. At those offices, once the random check of
enumerators' initial work was completed, there were no additional checks
specifically designed to catch enumerators suspected of falsifying data.
This raises questions about the reinterview program's ability to ensure the
quality of enumerators' work over the full duration of their employment on
nonresponse follow-up.
Local Managers Often Decided Against Conducting Administrative Reinterviews
Of the 520 local census offices, 52 offices (10 percent) conducted no
administrative and no supplemental reinterviews, according to bureau data.25
An additional 14 offices (3 percent) conducted no administrative
reinterviews, and an additional 231 offices (44 percent) conducted no
supplemental reinterviews.
A chief in the bureau's Quality Assurance Office expressed concern about the
adequacy of quality assurance coverage toward the end of nonresponse
follow-up for offices that did not conduct administrative and supplemental
reinterviews. According to this official, this meant that once random
reinterviews were completed at those offices, there were no additional
checks specifically designed to detect fabricated data. Although
25 In addition to the 511 local census offices located in the United States,
there were 9 offices in Puerto Rico.
enumerators' immediate supervisors were to check enumerators' work daily,
these reviews were generally designed to identify enumerators who were
completing questionnaires incorrectly (e.g., not following the proper
question sequence and writing illegibly), whereas administrative and
supplemental reinterviews were aimed at identifying enumerators who were
intentionally falsifying data.
Bureau officials said that at those local census offices that did not
conduct any administrative reinterviews, local census office managers could
conduct supplemental reinterviews if warranted. However, managers employed
this option infrequently. Of the 66 local offices that did not conduct any
administrative reinterviews, just 14 conducted supplemental reinterviews.
Reasons that local census managers could use-as specified by the bureau-for
not conducting an administrative reinterview included (1) the enumerator no
longer worked in the area for which the administrative test was conducted;
(2) the enumerator's work was characteristic with the area (e.g., the
enumerator reported a large number of vacant housing units and the area had
a large number of seasonal housing units); or (3) other reason, with an
accompanying explanation. Managers were to document their decision on the
bureau's administrative reinterview trouble reports listing the suspect
enumerators.
Our analysis of a week's worth of administrative reinterview trouble reports
at 31 local census offices found that while a number of enumerators were
flagged for administrative reinterviews, local census office officials
typically decided against conducting them. Specifically, of the 3,784
enumerators identified for possible reinterview, local officials subjected
the work of 154 enumerators (4 percent) to reinterviews, and passed on 3,392
enumerators (90 percent). For 306 of the 3,874 enumerators (8 percent)
listed on the administrative trouble reports we reviewed, there was no
indication of a final decision on whether or not to subject the future work
of these enumerators to administrative reinterview.
Overall, local census offices conducted far fewer administrative
reinterviews than the bureau had anticipated. Local census offices conducted
276,832 administrative reinterviews-146,993 (35 percent) fewer than the
423,825 administrative reinterviews the bureau had expected based on a
number of factors, including the number of cases completed per hour during
the 1990 Census, and the estimated workload in 2000. Whether this was due to
better quality work on the part of
enumerators, or local managers deciding against subjecting enumerators' work
to reinterviews, is unknown. However, as administrative reinterviews were
designed to detect fabrication and other quality problems more likely to
occur toward the end of nonresponse follow-up after the random check of
enumerators' initial work, it will be important for the bureau to examine
whether local census offices properly conducted administrative reinterviews,
and thus ensure the quality of nonresponse follow-up data throughout the
duration of the operation.
Conclusions Although nonresponse follow-up was fraught with extraordinary
managerial and logistical challenges, the bureau generally completed
nonresponse follow-up consistent with its operational plan-a remarkable
accomplishment given the scope and complexity of the effort. Our review
highlighted several strategies that were key to the bureau's success
including (1) an aggressive outreach and promotion campaign and other
efforts aimed at boosting the mail response rate and lowering the bureau's
nonresponse follow-up workload; (2) a flexible recruiting strategy that made
the bureau a competitive employer in a tight national labor market; (3)
advance planning for addressing location-specific enumeration challenges;
and (4) ambitious stretch goals that encouraged local managers to accelerate
the pace of the operation. It will be important for the bureau to document
the lessons learned from these initiatives and use them to help inform
planning efforts for the next decennial census in 2010.
It will also be important for the bureau to address the continuing
significant challenges that were revealed by the conduct of nonresponse
follow-up in 2000, including
* achieving an acceptable response rate (and thus lowering the bureau's
follow-up workload) while controlling costs;
* reversing the downward trend in public participation in the census, in
part by converting the relatively large number of people who are aware of
the census into census respondents;
* keeping the address list and maps used for nonresponse follow-up accurate
and up-to-date;
* finding the right mix of incentives to motivate local census offices to
complete nonresponse follow-up on schedule without compromising data
quality; and
* ensuring that reinterview procedures provide sufficient quality assurance
coverage through the full duration of enumerators' employment on nonresponse
follow-up.
Recommendations for Executive Action
As the bureau plans for the next national head count in 2010, we recommend
that the Secretary of Commerce ensure that the bureau take the following
actions to help ensure that nonresponse follow-up is conducted as cost
effectively as possible:
* Identify and refine lessons learned from the 2000 nonresponse follow-up
operation and apply them to the bureau's plans for the 2010 Census.
* Assess to the extent practicable, why people who were aware of the census
did not return their census questionnaires and develop appropriate marketing
countermeasures to bridge the gap between their awareness of the census on
the one hand, and their motivation to respond on the other.
* Develop and test procedural and technological options that have the
potential to generate a more accurate and up-to-date address list and set of
maps for nonresponse follow-up. As part of this effort, the bureau should
explore how to refresh the nonresponse follow-up address list more
frequently, even as nonresponse follow-up is underway, so that enumerators
would not have to make costly visits to late-responding households. The
bureau also needs to examine the methods it uses in activities that precede
nonresponse follow-up to develop and update the nonresponse address list and
associated maps. Specifically, the bureau should determine the extent to
which updates that should have been made were properly reflected in the
nonresponse follow-up list and maps, and take appropriate corrective actions
to address any problems it identifies.
* Ensure that the bureau's procedures and incentives for the timely
completion of nonresponse follow-up emphasize the collection of quality data
and proper enumeration techniques as much as speed.
* Examine the bureau's reinterview procedures-particularly as they relate to
the discretion given to local census officials-to help ensure that the
procedures are sufficient for consistently and reliably detecting potential
problems throughout the duration of enumerators' employment on nonresponse
follow-up.
Agency Comments and The Secretary of Commerce forwarded written comments
from the Bureau of the Census on a draft of this report. The bureau
concurred with all five
Our Evaluation of our recommendations and had no specific comments on them.
The bureau also clarified several key points and provided additional
information and perspective, which we incorporated in our report as
appropriate.
The bureau noted that, in addition to the locked apartment buildings that we
cited in the Results in Brief section of our report, gated communities were
also an enumeration challenge. While the body of the report already
contained this information, we added it to the Results in Brief section as
well.
Our draft report stated: "One reason for the errors in the nonresponse
follow-up address lists was that the bureau found it was infeasible to
remove late-responding households. As a result, enumerators needed to visit
over 773,000 households that had already mailed back their questionnaires. .
. ." The bureau commented that it made a conscious decision to conduct these
visits based on logistical concerns and, as a result, the bureau believes
that our use of the terms "errors" and "needlessly" do not take this into
consideration and are misleading.
Because the bureau could not refresh its nonresponse follow-up address list
to reflect households that responded after April 18, the bureau had no
choice but to send enumerators to those households and collect the
information in-person. However, the term "needed to" better characterizes
the bureau's lack of options and we revised the text accordingly. We also
deleted the term "errors."
In response to our finding that 52 local census offices did not conduct any
reinterviews after an initial random check of enumerators' work, the bureau
commented that the initial random check was not a minimal activity in that
it involved reinterviewing up to seven cases per enumerator. The bureau also
noted that there were no operational requirements to conduct a specific
number of administrative or supplemental reinterviews. We agree with the
bureau's comments. Indeed, the draft report already included information on
the number of initial random reinterviews the bureau conducted and the
discretionary nature of administrative and supplemental reinterviews.
Nevertheless, it is also true, as we note in our report, that once those 52
local census offices completed the seven random reinterviews, there were no
additional checks specifically designed to catch enumerators suspected of
falsifying data. Moreover, we reported that nationwide, local census offices
conducted far fewer administrative reinterviews than the bureau had
expected. As we note in the report, whether this was due to the quality of
enumerators' work or local managers using their discretion and opting not to
subject enumerators' work to reinterviews, is unknown.
With respect to the bureau's monitoring of local census office's
productivity, the bureau noted that headquarters officials did not work
directly with local census office staff as noted in the draft; rather,
headquarters personnel worked with the bureau's regional census centers, and
they in turn worked with the local offices. We revised the text to reflect
this information.
With respect to our observation that several local census offices had to
quickly respond to unanticipated challenges, such as working with
nonresponse follow-up address lists and maps that were not accurate or
current, the bureau commented that there were standard procedures in the
nonresponse follow-up enumerator manual on how to deal with map/register
discrepancies. We verified this and revised the text accordingly.
In describing the steps that local census officials took to encourage public
participation in the census, we noted that census officials in Cleveland and
Cincinnati said they provided additional training for enumerators on how to
handle refusals. The bureau noted that standardized training was provided,
across the nation, on options for handling refusals, and information was
also provided in the nonresponse follow-up enumerator manual. We verified
this information and added it to the report.
The bureau commented that the address list and map difficulties that
enumerators encountered were not nonresponse problems because, as we note in
the report, and the bureau agrees, they should have been dealt with in
earlier census operations. Nevertheless, the problems did not surface until
nonresponse follow-up when enumerators encountered duplicate and nonexistent
addresses, and were less productive as a result. For this reason, the report
recommends that the bureau examine the methods it uses in activities that
precede nonresponse follow-up to ensure the address lists and maps used for
nonresponse follow-up are accurate and up-to-date.
In response to our statement that nonresponse follow-up was to help verify
changes to the address list from earlier address list development
operations, the bureau commented that nonresponse follow-up was conducted to
enumerate households from which it did not receive a completed
questionnaire; map and address updates were incidental. We agree with the
bureau on the primary purpose of nonresponse follow-up and revised the text
to better reflect this point. However, the bureau's program master plan for
the master address file includes nonresponse follow-up as one of a number of
address list development and maintenance
operations, and the bureau expected enumerators to update maps and address
registers as needed as part of their field visits.
The bureau said it could not confirm data in our draft report on the number
of vacant and deleted units identified during nonresponse follow-up and
suggested removing this information. Although we obtained the data directly
from the bureau, given the bureau's concerns, we deleted the section.
In commenting on the fact that we did not find a statistically significant
relationship between the week that local census offices finished their
follow-up workload and the amount of their residual workload, the bureau
stated that the report needed to reflect the fact that residual nonresponse
consisted of housing units for which completed questionnaires had not been
processed through data capture. We revised the draft accordingly.
The bureau noted that assistant managers for field operations, among other
local census officials, could request supplemental reinterviews, and not
just field operations supervisors as we stated in our report. We revised our
draft to include this information.
With respect to our findings concerning the reinterview program's ability to
detect problems, particularly at the end of nonresponse follow-up, the
bureau commented that there was turnover in the enumerator workforce;
consequently, with new hires, random reinterviews were conducted during all
stages of the operation. As we note in the report, 52 local census offices
(about 10 percent of all local offices), did not conduct any administrative
and supplemental reinterviews. Thus, once these offices completed the random
reinterviews on the initial work of newly hired enumerators, there were no
additional checks specifically designed to catch enumerators suspected of
falsifying data. We added language to better clarify this point.
The bureau said that it was uncertain as to the methodology and
documentation used for deriving figures on the number of reinterviews the
bureau conducted. We obtained the data from the bureau's cost and progress
system.
The bureau stated that there was no evidence that data quality was
compromised to motivate on-time completion of nonresponse follow-up. Our
research suggests that the impact of the bureau's incentives to motivate
timeliness was less clear-cut given the fact that, as we note in our report,
(1) about 40 percent of the local census office managers believed
that scheduling pressures had a negative or significantly negative impact on
the quality of nonresponse follow-up, and (2) a small number of local census
offices took short-cuts to complete their work (which the bureau later took
steps to rectify). Thus, while we agree with the bureau that maintaining
data quality should be a given in determining motivational elements, the
extent to which the bureau accomplished this goal for nonresponse follow-up
appeared to have had mixed results.
In commenting on our conclusion that it will be important for the bureau to
ensure that reinterview procedures provide sufficient quality assurance
through the full duration of nonresponse follow-up, the bureau noted that
the reinterview operation must be designed to provide sufficient quality
assurance coverage. We revised the text accordingly
We are sending copies of this report to the Honorable Dan Miller and Carolyn
B. Maloney, House of Representatives, and those in other interested
congressional committees; the Secretary of Commerce; and the Acting Director
of the Bureau of the Census. Copies will be made available to others on
request. Major contributors to this report are included in appendix III. If
you have any questions concerning this report, please call me on (202)
512-6806.
Patricia A. Dalton Director Strategic Issues
Appendix I
Local Census Offices Included in This Review
Local Census Offices in the Atlanta East Census Bureau's Atlanta Bradenton
Region Fort Myers
Local Census Offices in the Boston North Census Bureau's Boston Burlington
Region Hartford
Providence
Local Census Offices in the Ashland-Hanover
Census Bureau's Charlotte Beaufort
Region Conway Greenville, North Carolina, East
Greenville, North Carolina, West
Wilmington
Local Census Offices in the Census Bureau's Chicago Region
Chicago Central
Chicago Far North
Chicago Near North
Chicago Near South
Chicago Near Southwest
Chicago West
Indianapolis
Midland
Milwaukee
Superior
Local Census Offices in the Corpus Christi
Census Bureau's Dallas Dallas Central Region Greenville, Mississippi
Harris County, Northeast
Laredo
McAllen
New Orleans Central
Orleans Parish
Appendix I
Local Census Offices Included in This Review
Local Census Offices in the Flagstaff
Census Bureau's Denver Las Cruces Region Las Vegas
Phoenix South Santa Fe Yuma
Local Census Offices in the Cincinnati Census Bureau's Detroit Cleveland
Region Marquette
Local Census Offices in the Kansas City Census Bureau's Kansas Moorhead City
Region St. Louis City
Local Census Offices in the Baltimore West
Census Bureau's Philadelphia North
Philadelphia Region Philadelphia South Pittsburgh
Local Census Offices in the Hollywood/Mid-Wilshire Census Bureau 's Los Los
Angeles Downtown Angeles Region Santa Monica
Local Census Offices in the Bronx Northeast
Census Bureau's New York Brooklyn Central Region Brooklyn East
Brooklyn Northeast
New York East
New York North
New York Northeast
Appendix I
Local Census Offices Included in This Review
Local Census Offices in the Portland
Census Bureau's Seattle San Francisco Northeast
Region San Francisco Southeast
Appendix II
Comments from the Secretary of Commerce
Appendix II
Comments from the Secretary of Commerce
Appendix II
Comments from the Secretary of Commerce
Appendix II
Comments from the Secretary of Commerce
Appendix II
Comments from the Secretary of Commerce
Appendix II
Comments from the Secretary of Commerce
Appendix III
GAO Contacts and Staff Acknowledgments
GAO Contacts Patricia A. Dalton, (202) 512-6806 Robert Goldenkoff, (202)
512-2757
Acknowledgments In addition to those named above, the following headquarters
staff made key contributions to this report: Wendy Ahmed; Tom Beall; James
Fields; Rich Hung; Lily Kim; J. Christopher Mihm; Victoria E. Miller; Vicky
L. Miller; Ty Mitchell; Anne Rhodes-Kline; Lynn Wasielewski; Susan Wallace.
The following staff from the Western Regional Office also contributed to
this report: James Bancroft; Robert Bresky; Arthur Davis; Julian Fogle;
Araceli Hutsell; RoJeanne Liu; Elizabeth Dolan; Thomas Schulz; Nico Sloss;
Cornelius Williams.
The following staff from the Central Regional Office also contributed to
this report: Richard Burrell; Michael De La Garza; Maria Durant; Donald
Ficklin; Ron Haun; Arturo Holguin, Jr.; Reid Jones; Stefani Jonkman; Roger
Kolar; Tom Laetz; Miquel Salas; Enemencio Sanchez; Jeremy Schupbach; Melvin
Thomas; Richard Tsuhara; Theresa Wagner; Patrick Ward; Linda Kay Willard;
Cleofas Zapata, Jr.
The following staff from the Eastern Regional Office also contributed to
this report: Cammillia Campbell; Lara Carreon; Betty Clark; Johnetta
Gatlin-Brown; Marshall Hamlett; Carlean Jones; Janet Keller; Cameron
Killough; Jean Lee; Christopher Miller; S. Monty Peters; Sharon Reid;
Matthew Smith.
Related GAO Products on the Results of the 2000 Census and Lessons Learned
for a More Cost-Effective Census in 2010
2000 Census: Coverage Evaluation Interviewing Overcame Challenges, but
Further Research Needed (GAO-02-26, December 31, 2001).
2000 Census: Analysis of Fiscal Year 2000 Budget and Internal Control
Weaknesses at the U.S. Census Bureau (GAO-02-30, December 28, 2001).
2000 Census: Significant Increase in Cost Per Housing Unit Compared to 1990
Census (GAO-02-31, December 11, 2001).
2000 Census: Better Productivity Data Needed for Future Planning and
Budgeting (GAO-02-4, October 4, 2001).
2000 Census: Review of Partnership Program Highlights Best Practices for
Future Operations (GAO-01-579, August 20, 2001).
Decennial Censuses: Historical Data on Enumerator Productivity Are Limited
(GAO-01-208R, January 5, 2001).
2000 Census: Information on Short-and Long-Form Response Rates
(GAO/GGD-00-127R, June 7, 2000).
GAO's Mission The General Accounting Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability of
the federal government for the American people. GAO examines the use of
public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to good
government is reflected in its core values of accountability, integrity, and
reliability.
Obtaining Copies of GAO Reports and Testimony
The fastest and easiest way to obtain copies of GAO documents is through the
Internet. GAO's Web site (www.gao.gov) contains abstracts and full-text
files of current reports and testimony and an expanding archive of older
products. The Web site features a search engine to help you locate documents
using key words and phrases. You can print these documents in their
entirety, including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its Web
site daily. The list contains links to the full-text document files. To have
GAO E-mail this list to you every afternoon, go to www.gao.gov and select
"Subscribe to daily e-mail alert for newly released products" under the GAO
Reports heading.
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out to
the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:
U.S. General Accounting Office
P.O. Box 37050
Washington, D.C. 20013
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
Visit GAO's Document GAO Building
Room 1100, 700 4th Street, NW (corner of 4th and G Streets, NW)Distribution
Center Washington, D.C. 20013
To Report Fraud, Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm,Waste, and Abuse in E-mail:
[email protected], orFederal Programs 1-800-424-5454 or (202) 512-7470
(automated answering system).
Public Affairs Jeff Nelligan, Managing Director, [email protected] (202)
512-4800 U.S. General Accounting Office, 441 G. Street NW, Room 7149,
Washington, D.C. 20548
Presorted Standard
Postage & Fees Paid
GAO
Permit No. GI00
United States
General Accounting Office
Washington, D.C. 20548-0001
Official Business
Penalty for Private Use $300
Address Correction Requested
*** End of document. ***