Decennial Census: Methods for Collecting and Reporting Data on
the Homeless and Others without Conventional Housing Need
Refinement (17-JAN-03, GAO-03-227).
The Bureau of the Census partnered with local governments,
advocacy groups, and other organizations to help it enumerate
people without conventional housing. Counting this
population--which includes shelter residents and the
homeless--has been a longstanding challenge for the Bureau. A
number of organizations put substantial resources into an
operation the Bureau called Service-Based Enumeration. In return,
some expected the Bureau to provide data that would help them
plan and deliver employment, health, and other services. However,
the Bureau did not release the data as planned, which raised
questions about the Bureau's decision-making on data quality
issues. In response to a congressional request, GAO examined the
Bureau's decision-making process behind its change in plans.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-03-227
ACCNO: A05883
TITLE: Decennial Census: Methods for Collecting and Reporting
Data on the Homeless and Others without Conventional Housing Need
Refinement
DATE: 01/17/2003
SUBJECT: Census
Data collection
Decision making
Evaluation methods
Homelessness
1990 Decennial Census
2000 Decennial Census
2010 Decennial Census
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-03-227
A
Report to Congressional Requesters
January 2003 DECENNIAL CENSUS Methods for Collecting and Reporting Data on
the Homeless and Others without Conventional Housing Need Refinement
GAO- 03- 227
Letter
January 17, 2003 The Honorable Henry A. Waxman Ranking Minority Member
Committee on Government Reform House of Representatives The Honorable
Danny K. Davis Ranking Minority Member Subcommittee on Civil Service,
Census and Agency Organization Committee on Government Reform House of
Representatives The Honorable Carolyn B. Maloney The Honorable Dennis J.
Kucinich The Honorable Wm. Lacy Clay House of Representatives For the 2000
Census, the Bureau of the Census employed several initiatives to help
ensure a complete and accurate count of people without conventional
housing. Enumerating this segment of the population, which contains, among
others, people referred to as *homeless,* has been an ongoing problem for
the Bureau. In one initiative, known as Service- Based Enumeration, census
enumerators attempted to count these individuals at emergency and
transitional shelters, soup kitchens, regularly scheduled mobile food
vans, as well as at what the Bureau calls *targeted nonsheltered outdoor
locations* (TNSOL) for people living on the street at targeted locations
who do not use services. To help locate and count people, the Bureau
partnered with organizations providing services to the homeless and local
governments, some of which put substantial resources into their efforts.
In return, some of these organizations expected the Bureau to provide data
that would help them plan and deliver health, employment, and other
services directed toward this population.
However, in its review of the emergency and transitional shelter data, the
Bureau identified serious concerns with the quality of the data and
concluded that the data should not be released without explanation of
their extensive limitations and caveats. As a result, the Bureau decided
not to separately report the emergency and transitional shelter data in
the initial
release of summary files as originally planned. Instead, the Bureau
combined the emergency and transitional shelter data with a category
called *other non- institutional group quarters.* At the census tract
level (small statistical subdivisions of counties) this category included
soup kitchens, regularly scheduled mobile food vans, shelters for victims
of
domestic violence, residential care facilities providing protective
oversight, staff quarters including those for nurses and interns at
military and general hospitals, and living quarters for victims of natural
disasters. At the block level, in addition to the categories listed above,
*other non- institutional group quarters* included group homes, religious
group quarters, other nonhousehold living situations, and workers*
dormitories. Aggregating the shelter numbers with these other data raised
concerns among some data users that the Bureau was suppressing the
results.
You asked us to examine the Bureau*s decision- making process behind its
change in plans. As agreed with your offices, this report examines (1) the
Bureau*s plans for reporting the results of Service- Based Enumeration and
its reasons for changing those plans and (2) the Bureau*s protocols for
releasing data.
Members of the Congress also raised concerns about the quality of Hispanic
subgroup data and asked us to review the Bureau*s decisionmaking process
for collecting and reporting ethnicity information. The results of that
study are included in a companion report. 1 Both reports are part of our
ongoing series on the lessons learned from the 2000 Census that can help
inform the planning effort for 2010. (See the Related GAO Products section
for a list of reports issued to date on census issues.)
Results in Brief The Bureau*s original plan for disseminating Service-
Based Enumeration data was outlined in an April 1999 internal memorandum
that called for the
separate release of data on *emergency and transitional shelters,* but did
not specify why the Bureau was not separately releasing data on the other
locations enumerated during the Service- Based Enumeration* soup kitchens,
regularly scheduled mobile food vans, shelters for victims of domestic
violence, and targeted nonsheltered outdoor locations. The Bureau*s plan
reflected its experience during the 1990 Census when it released separate
counts of people found at emergency shelters, street locations, and
similar locations. However, those counts proved to be incomplete. The
Bureau indicated from the beginning that these 1990 1 U. S. General
Accounting Office, Decennial Census: Methods for Collecting and Reporting
Hispanic Subgroup Data Need Refinement, GAO- 03- 228 (Washington, D. C.:
Jan. 17, 2003).
counts could not be added together to produce a count of the homeless
population. Despite the Bureau*s warnings to the contrary, the data were
often misinterpreted as a *homeless* count. Thus, in developing its data
release plans, the Bureau took steps to ensure that the Service- Based
Enumeration figures could not be added together and used as a homeless
count. In January 2001, the Bureau changed its earlier decision to include
the data on emergency and transitional shelters in one of its early data
releases because a procedure used to refine the Service- Based Enumeration
data proved to be unreliable. Although the Bureau had tested the procedure
earlier in the decade, because of methodological limitations the test did
not reveal any flaws. However, because problems with the procedure
surfaced during the review of 2000 Census operations, the Bureau decided
to
combine the emergency and transitional shelter data with the *other
noninstitutional group quarters* category that also includes data on
people enumerated in the other categories of the Service- Based
Enumeration and in several other group locations, such as facilities for
victims of natural disasters. As a result of this decision, the Bureau did
not separately report any data from the Service- Based Enumeration in its
initial release of Census 2000 data. These were the only data with
separate reporting categories that the Bureau decided to collapse into
another category.
In the fall of 2001, the Bureau produced a special report on the emergency
and transitional shelter data* including most of the same data that the
Bureau earlier stated it could not release because of quality concerns.
This report did not include data on targeted nonsheltered outdoor
locations, or on soup kitchens and mobile food vans. The Bureau added a
lengthy discussion of the limitations of the data and emphasized that they
should not be interpreted as a count of the homeless population. Although
the Bureau worked closely with a number of government entities,
advocacy groups, and other organizations to conduct Service- Based
Enumeration, reconciling its often competing data needs proved
challenging. Compounding the Bureau*s difficulties, expectation gaps
developed between these entities and the Bureau because the Bureau did not
always clearly and consistently communicate its plans.
A key cause of the Bureau*s shifting position on reporting the
ServiceBased Enumeration data appears to be its lack of clear, documented,
and consistently applied guidelines governing the release of data from the
2000 Census. Had these guidelines been in place at the time of the census,
they
could have helped Bureau managers decide whether to release the
ServiceBased Enumeration data and how to characterize these data.
Additionally, the Bureau could use the guidelines to defend its decisions
once they were made, thus helping to ensure that the Bureau*s decisions
both are, and appear to be, completely objective.
To ensure that the 2010 Census will provide data users with more complete,
accurate, and useful information on people without conventional housing,
we recommend that the Secretary of Commerce direct the Bureau of the
Census to ensure that the procedures for enumerating and estimating
segments of the population without conventional housing are properly
tested and evaluated under conditions as similar to the census as
possible. In addition, the Bureau should develop clearly documented,
transparent, and consistently applied agencywide guidelines for releasing
all census data to the public and ensure that plans for releasing data are
clearly and consistently communicated to the public.
The Secretary of Commerce forwarded written comments from the Bureau of
the Census on a draft of this report (see app. I). The Bureau agreed with
our recommendations and is taking steps to implement them, but took
exception to our findings concerning the adequacy of its data quality
guidelines and communication with the public.
Background The procedures the Bureau used during the 1990 Census to count
people without conventional housing had limitations that resulted in
incomplete
data. 2 To address these limitations and help improve the quality of the
data, the Bureau used a procedure for the 2000 Census called Service-
Based Enumeration that attempted to count people where they receive
services such as emergency shelters, soup kitchens, and regularly
scheduled mobile
food vans. Service- Based Enumeration also counted people in targeted
nonsheltered outdoor locations such as encampments beneath bridges. The
operation occurred from March 27 through March 29, 2000. 3 2 For further
information see, U. S. General Accounting Office, 1990 Census: Limitations
in
Methods and Procedures to Include the Homeless, GAO/ GGD- 92- 1
(Washington, D. C.: Dec. 30, 1991). 3 For information on the conduct of
Service- Based Enumeration see, U. S. General Accounting Office, 2000
Census: Progress Report on the Mail Response Rate and Key Operations, GAO/
T- GGD/ AIMD- 00- 136 (Washington, D. C.: Apr. 5, 2000).
According to Bureau officials, Service- Based Enumeration was not
designed, and was never intended, to provide a specific count of homeless
persons. Instead, the operation was part of a larger effort to count
people without conventional housing, including people in *institutional
group quarters* such as correctional facilities, nursing homes, and mental
hospitals, and *non- institutional group quarters* such as college
dormitories, military quarters, and group homes. Service- Based
Enumeration counted people in specific categories of noninstitutional
group quarters.
To help ensure a complete count of people without conventional housing,
the Bureau partnered with local governments and community advocacy groups
to obtain lists of service locations and to assist with the
enumeration. 4 In some cases, the Bureau hired clients of the advocacy
groups and other people trusted by the homeless to conduct Service- Based
Enumeration. For example in Atlanta, an advocacy group for homeless
veterans helped the Bureau employ homeless veterans to improve the count
of this population. Local governments helped the Bureau as well, often
investing considerable resources. For example, Los Angeles paid to keep
its city- run shelters open on the night they were enumerated so that
people using their services could be counted.
Scope and To address your concerns about the Bureau*s dissemination of
data on Methodology
persons without conventional housing, we agreed to examine (1) the
Bureau*s plans for reporting the results of Service- Based Enumeration and
its reasons for changing those plans and (2) the Bureau*s protocols for
releasing data. To accomplish these objectives, we interviewed key Bureau
officials and reviewed relevant Bureau documents and data such as
operational plans, decision memorandums, and the Bureau*s partnership
program evaluation.
In order to obtain the perspective of data users, partners, and
stakeholders, we conducted in- person and telephone interviews with
homeless advocates, local government officials, and representatives of
public service agencies in New York City, Los Angeles, Cleveland, Atlanta,
and Washington, D. C. These cities had large numbers of people without
4 For additional information see, U. S. General Accounting Office, 2000
Census: Review of Partnership Program Highlights Best Practices for Future
Operations, GAO- 01- 579 (Washington, D. C.: Aug. 20, 2001).
conventional housing and they were actively involved with the Bureau
during the 2000 Census. The organizations we contacted also provided
relevant documentation, such as comprehensive file documents relating to
partnership activities.
In addition to the above locations, we did our audit work at Bureau
headquarters in Suitland, Maryland. Our audit work was conducted from
April 2002 through September 2002 in accordance with generally accepted
government auditing standards. We requested comments on a draft of this
report from the Secretary of
Commerce. On November 21, 2002, the Secretary forwarded the Bureau*s
written comments on the draft (see app. I). We address these comments at
the end of this report.
The Bureau of the Under the Bureau*s original plan for releasing Service-
Based Enumeration
data in Summary File- 1 (SF- 1), 5 the emergency and transitional shelter
Census Twice Changed
count was one of several categories of noninstitutional group quarters
data Plans for Reporting
that were to be reported separately. Other people counted in the
ServiceBased Service- Based
Enumeration, including people counted at targeted nonsheltered Enumeration
Data outdoor locations, soup kitchens, and regularly scheduled mobile food
vans, were to be combined and reported under the category *other
noninstitutional group quarters.* This category also included residential
care facilities providing protective oversight, shelters against domestic
violence, staff dormitories for nurses and interns at military and general
hospitals, and living quarters for victims of natural disasters. 5 The SF-
1 is the summary file in which the Bureau presents population and housing
data for
the total population. Other than the Census 2000 Redistricting Data
Summary File, the SF- 1 file is the first product released after the
census.
This decision was documented in an April 1999 internal memorandum from the
Bureau*s Assistant Division Chief for Special Population Statistics to the
Assistant Division Chief for Census Programs. The Service- Based
Enumeration operation took place a year later, in March 2000. The April
1999 plan was in large part a reaction to the challenges the Bureau faced
counting the emergency shelter and street population during the 1990
Census. Although the Bureau disseminated separate counts of people found
at emergency shelters, preidentified street locations, and similar sites,
the counts proved to be incomplete. 6 Moreover, the Bureau stated in its
October 2001 report that despite its warnings to the contrary, the data
were sometimes misinterpreted as a *homeless* count. The October report
does not offer an example of this, but the
misinterpretation clearly played a role in a lawsuit against the Bureau. 7
As a result, when designing the 2000 census, the Bureau attempted to both
improve the count and take precautions to ensure that the Service- Based
Enumeration count would not be misconstrued as a count of the homeless.
The Bureau*s data dissemination plans took into account the
recommendations of the Commerce Secretary*s 2000 Census Advisory
Committee, a panel that included representatives of advocacy and other
groups (including representatives from organizations that represent local
governments) that met periodically to review the Bureau*s plans. The
homeless population was represented by the National Coalition for the
Homeless* an advocacy group that coordinates a network of 300 state and
local housing and homeless organizations. In its January 1999 final
report, the Census 2000 Advisory Committee recommended that special
attention be paid to tabulating the results of Service- Based Enumeration
and targeted outdoor enumerations so that they could not be aggregated and
used as a homeless count. 6 For more information on the 1990 count see, U.
S. General Accounting Office, Counting the Homeless: Limitations of 1990
Census Results and Methodology, GAO/ T- GGD- 91- 29 (Washington, D. C.:
May 9, 1991) and 1990 Census: Limitations in Methods and Procedures to
Include the Homeless, GAO/ GGD- 92- 1 (Washington, D. C.: Dec. 30, 1991).
7 One example was the National Law Center On Homelessness and Poverty, et.
al., v. Brown,
where plaintiffs alleged that the Department of Commerce effectively
excluded the nation*s homeless population from the 1990 decennial census
in violation of the Constitution and the Administrative Procedure Act. The
District Court denied the plaintiff*s claim (1994 WL 521334) (D. D. C.
Sept. 15, 1994) and the Court of Appeals dismissed the lawsuit for lack of
standing. 91 F. 3d 178 (D. C. Cir. 1996).
The Bureau Changed Its In January 2001, 5 months before the SF- 1 release,
the Bureau reversed its
Dissemination Plans April 1999 decision to release emergency and
transitional shelter data Because of Data Quality
separately because of *data quality concerns.* Instead, as shown in figure
Concerns
1, the Bureau planned to combine the emergency and transitional
Figure 1: The Bureau Changed Its Original Plan to Release Emergency and
Transitional Shelter Data and Combined Them with Other Noninstitutional
Group Quarters Data
Original plan (April 1999) Revised plan (January 2001)
Group quarters Group quarters
Institutionalized population Non- Noninstitutionalized Institutionalized
Population
population reporting categories
reporting categories
Correctional institutions
College dorms
Noninstitutionalized population
Nursing homes
Military quarters
reporting categories
Hospitals
Group homes
College dorms
Juvenile institutions
Religious group quarters
Military quarters
Worker dormitories
Group homes
Crews of maritime vessels
Religious group quarters
Emergency and transitional shelters
Worker dormitories
Crews of maritime vessels
Other noninstitutional group quarters
Other noninstitutional group quarters
The other noninstitutional group quarters categories are:
Soup kitchens and regularly scheduled food vans
The other noninstitutional group quarters categories are:
Targeted nonsheltered outdoor locations Emergency and transitional
shelters
Residential facilities "providing protective oversight" Living quarters
for victims of natural disasters
Soup kitchens and regularly scheduled food vans Staff residents of
institutions
Targeted nonsheltered outdoor locations Shelters for victims of domestic
violence
Residential facilities "providing protective oversight" Living quarters
for victims of natural disasters Staff residents of institutions
Source: GAO depiction based on Bureau of the Census data. Shelters for
victims of domestic violence
shelter data with the *other non- institutional group quarters.* This
category contained data on a variety of living arrangements including
facilities for natural disaster victims. The Bureau*s decision was
contained in an internal Bureau memorandum from the Chief of the
Population Division to the Chief of the Decennial Systems and Contracts
Management
Office. Bureau officials told us that the decision to exclude a separate
emergency and transitional shelter count in SF- 1 was made between
December 2000 and January 2001, by the Director of the Decennial Census
with input from the Associate Director Decennial Census, the Population
Division, the Associate Director for Demographic Programs, the Decennial
Management Division, and the Decennial Statistical Studies Division. 8
According to Bureau officials, their concerns focused on the accuracy of a
new statistical procedure called *multiplicity estimation* that adjusted
the number counted to better reflect the number of actual shelter users.
Because Service- Based Enumeration only counted people who were at these
facilities on the day of enumeration, the Bureau intended to use
multiplicity estimation to calculate the number of people who used these
facilities but were not present during Service- Based Enumeration. The
multiplicity estimation procedure was based on information from those who
were counted and on the number of times they used the service facilities
in the prior week. An estimate of people not counted on the day of
enumeration was added to the count of people who were. According to the
Bureau, the multiplicity estimates tested well during the 1998 dress
rehearsal for the 2000 Census possibly because the three rehearsal sites
did not offer large enough sample sizes of the appropriate populations to
adequately test this procedure. 9 However, during the 2000 Census the
Bureau found that a census question pertaining to facility usage upon
which the multiplicity estimates were based had a low response rate.
Moreover, the Bureau found that respondents, particularly in shelters, did
not answer the question accurately. Due to data quality concerns, the
Bureau decided not to use multiplicity estimation to adjust the data and
consequently decided not to report the data separately.
8 Although the Bureau changed its plan in January 2001, the technical
documentation for SF- 1, released at the same time, still indicated that
emergency and transitional shelter data would be separately reported. 9
The dress rehearsal for the 2000 Census was conducted in Sacramento,
California, City of Columbia, South Carolina, and Menominee County,
Wisconsin, including the Menominee
Indian Reservation. The dress rehearsal was designed to test the overall
design of the 2000 Census.
Bureau officials said they did not announce the change in plans because
they were still evaluating the problems with the data. It was not until
June 2000 that the Bureau began recalculating the data and making a final
decision on which categories to aggregate. Ultimately, the Bureau did not
report any of the Service- Based Enumeration data separately in SF- 1.
Emergency and transitional shelter data were the only data that were to be
released in SF- 1 under separate reporting categories that the Bureau
decided to combine with another category.
The Bureau Produced a The release of the SF- 1 data in June 2001 produced
public discussion in the
Special Report in October press, among census partners, and in the
Congress about the Bureau*s 2001 on the Emergency and
decision to not separately release Service- Based Enumeration data. In a
Transitional Shelter briefing for staff of the House Committee on
Governmental Affairs, the Associate Director of the Decennial Census
announced that the Bureau
Population planned to produce a separate report on the emergency and
transitional
shelter data. In October 2001, the Bureau issued a special report,
entitled Emergency and Transitional Shelter Population: 2000. This report
separately identified emergency and transitional shelter data for various
levels of geography down to the census tract level with 100 or more people
in emergency and transitional shelters. The report did not include data
for the populations in targeted nonsheltered outdoor locations, soup
kitchens, regularly scheduled mobile food vans, and shelters for domestic
violence.
The 17- page report contains an extensive discussion on the limitations of
the data. For example, the Bureau noted that the data in the report should
not be construed as a count of people without conventional housing.
Moreover, the emergency and transitional shelter data at the census tract
level are not in the hard copy, but rather in the Internet version of the
report. 10 The Bureau stated that all Census 2000 data at the tract level
are available on the Internet and are not available in printed reports.
The October report contains most of the same data that were to be released
under the April 1999 dissemination plan for SF- 1. The Bureau asserted
that the data quality concerns with the emergency and transitional shelter
data (cited when it changed the plan to release these data in SF- 1)
required that
the data be presented in a manner that allowed the Bureau to clearly 10 U.
S. Bureau of the Census, Population in Emergency and Transitional Shelters
(PHC- T- 12) (Washington, D. C.: Oct. 30, 2001). http:// www. census. gov/
population/ www/ cen2000/ phc- t12. html
outline the data*s limitations. The October 2001 report contained an
extended discussion of these limitations. The October 2001 report also
identified reasons the Bureau did not (and never planned to) separately
release data on people counted at targeted nonsheltered outdoor locations,
soup kitchens, regularly scheduled mobile
food vans, and shelters for victims of domestic violence, including the
following.
People without conventional housing who were at outside locations other
than the targeted nonsheltered outdoor locations identified for the census
were not included in the TNSOL operation. For the purposes of the TNSOL
operation, the definition of *outdoor*
excluded both mobile and transient locations used by people experiencing
homelessness as well as abandoned buildings.
The option was given to the individuals found at soup kitchens and
regularly scheduled mobile food vans to select *usual home elsewhere.* For
example, if an individual enumerated at a soup kitchen listed a usual home
elsewhere, then that person was tabulated at their usual residence and not
at the service location. Therefore, the data on this population would not
reflect a true count of the individuals using these services. Prior to
publicly releasing the October report, the Bureau asked two
representatives from the National Coalition for the Homeless to review a
draft of the portion of the report that described the limitations of the
data. The National Coalition for the Homeless commented extensively on the
section containing the caveats and limitations in order to strengthen the
report. A member of the Board of Directors for the National Coalition for
the Homeless told us that he provided this feedback both as an academician
and a stakeholder. Bureau officials stated that because of its position on
the Bureau*s Census Advisory Committee, the National Coalition for the
Homeless was the only advocacy group that reviewed any
portion of the October 2001 report prior to its publication. Meeting Data
Users* Needs
The controversy surrounding the release of the combined Service- Based
Proved Challenging
Enumeration data highlights the challenges the Bureau faced in 2000 trying
to meet the needs of various data users and the work the Bureau still
needs to do when planning for the 2010 Census to better reconcile those
needs. For example, several organizations we contacted favored the
separate
release of the Service- Based Enumeration data categories. Indeed, local
government officials we talked to in New York City believed that the data
would help with grant applications, projections about future service
needs, and determining their success in getting people off the streets and
into shelters. The Executive Director of the Northeast Ohio Coalition for
the
Homeless stated that the city of Cleveland does not do its own count of
this population and, therefore, the Bureau numbers are the only ones
available on this segment of the population. Los Angeles city officials
wanted the Service- Based Enumeration data so they could better target
their services and, like Cleveland, Los Angeles did not have its own data.
Several of these
entities stated that the potential misuse of data was not a valid reason
for not separately releasing data.
In addition, the majority of the organizations we contacted partnered with
the Bureau anticipating that they would be able to use the Service- Based
Enumeration data to evaluate whether improvements were made in enumerating
local populations without conventional housing in 2000
compared to 1990. The Assistant City Attorney of Los Angeles estimated
that Los Angeles spent about $300,000 on the effort to improve the count
of Los Angeles*s people without conventional housing. For example, as part
of an extensive effort to help the Bureau develop a list of targeted
nonsheltered outdoor locations, the city provided senior Bureau staff with
a helicopter tour over some outdoor locations where people without
conventional housing lived. The Assistant City Attorney of Los Angeles
stated that she believed the city would get the targeted nonsheltered
outdoor locations data that they helped collect and wanted to review. In
addition, because of the Bureau*s focus on counting people at shelters,
the city kept shelters open on the night of the enumeration at its own
expense even though shelters in Los Angeles typically do not have many
people during warm weather. Los Angeles expected to have detailed data to
use to evaluate the effectiveness of its resource allocation. However, the
National Coalition for the Homeless and other advocates of
the homeless opposed the separate release of any of the Service- Based
Enumeration data. They were concerned that these data could be misused as
a count of the homeless population and lead to flawed decision- making by
policymakers. Ultimately, the Bureau left a number of data users
unsatisfied. Those who
wanted the Service- Based Enumeration categories released separately did
not feel the Bureau met their expectations with the data released in SF- 1
or with the release of the October report. Users who opposed the separate
release of the data and were pleased that SF- 1 combined the Service-
Based Enumeration components with other data were displeased that the
October 2001 report was released.
The difficulties the Bureau experienced trying to reconcile the competing
needs and interests of data users illustrates the importance of effective
communication between the Bureau and its key data users and partners to
ensure no expectation gaps develop. More than just a good business
practice, federal internal control standards require agencies to have
effective external communications with groups that can have a serious
impact on programs, projects, operations, and other activities. 11
However, our conversations with several Bureau partners and our review of
Bureau documents suggest that communications were sometimes vague and
insufficient. For example, although the April 1999 memorandum that
outlined the Bureau*s initial data dissemination plans was written a year
before the 2000 Census, this information may not have been effectively
communicated to the Bureau*s partners. Indeed, at a Capitol Hill briefing
on this topic in June 2001, Bureau officials themselves acknowledged that
they did not do a good job of communicating on this issue. Some of the
partners we spoke to indicated that had they known earlier about the
Bureau*s plans to limit the release of Service- Based Enumeration data
they might have focused their resources on different census operations.
Further, our review of Bureau documents indicated that the information on
the *official plan* for the release of the different Service- Based
Enumeration categories of data was limited and inconsistent. Some partners
stated that they did not know that the Bureau never intended to report the
targeted nonsheltered outdoor location data. Although the Bureau made
numerous presentations on Service- Based
Enumeration that emphasized there would be no count of the homeless, the
Bureau provided little detail on how components of Service- Based
Enumeration would actually be presented. In the absence of clear
communication from the Bureau, partners developed their own expectations
of what would be released. Several of the local officials and advocates
that we spoke to expected that the data would be released in the 11 U. S.
General Accounting Office, Internal Control Standards, Internal Control
Management and Evaluation Tool (August 2001) and U. S. General Accounting
Office, Internal Control Standards for Internal Control in the Federal
Government (November 1999).
same detail as it was in 1990, because they were not told otherwise. For
example, a Los Angeles government official said that the Bureau stated it
would not provide a homeless count in 1990, but it still released the
street count data separately. By focusing resources on counting specific
categories of the population, the Bureau may have created expectations
that there would be a count of that population. Census Bureau Had
A cause of the Bureau*s shifting position on reporting the components of
Few Documented
Service- Based Enumeration appears to be its lack of documented, clear,
transparent, and consistently applied guidelines governing the release of
Guidelines Governing
data from the 2000 Census. Except for some guidance aimed at protecting
the Release of Census
the confidentiality of census records, the Bureau had few written Data
guidelines on the level of quality needed to release data to the public.
Had these guidelines been in place during the decennial census, they could
have informed the Bureau*s decision on whether to release the Service-
Based Enumeration data, how to characterize these data, and help defend
the decision after it was made. Such guidelines could also provide a basis
ahead of time for expectations about the conditions under which data will
or will not be released.
Although Bureau officials emphasized that the Bureau has a long tradition
of high standards and procedures that yield quality data (to its credit,
the Bureau*s quality assurance practices identified the problem with the
multiplicity estimator), the officials acknowledged that these standards
were primarily part of the agency*s institutional knowledge. The written
guidance that did exist appeared to be vague and insufficient for making
consistent decisions on the quality thresholds needed for releasing data
to
the public, and the circumstances under which it might be appropriate to
suppress certain data.
According to the Bureau*s Associate Director for Methodology and
Standards, a technical paper issued in 1974 and revised in 1987 contained
the Bureau*s only written guidelines for discussion and presentation of
errors in data. This paper noted that, *[ e] stimates for individual cells
of a published table should not be suppressed solely because they are
subject to large sampling errors or large nonsampling variances, provided
users are given adequate caution of the lack of reliability of the data.
On the other hand, data known to have very serious bias may be
suppressed.* 12 A Newly Created Bureau
In 2000, the Bureau initiated a new quality assurance program to document
Program Could Provide
Bureau- wide protocols designed to ensure the quality of data collected
and Guidelines for Releasing
disseminated by the Bureau. The Bureau*s Methodology and Standards Data
Council is charged with setting statistical and survey quality standards
and guidelines for Bureau surveys and censuses. In support of this role,
the council has established a quality framework in which the demographic,
economic, and decennial areas can share and support common principles,
standards, and guidelines. The quality framework covers eight unique
areas, one of which is dissemination.
Because this Bureau program is in its initial stages, we could not
evaluate it. However, Bureau officials believe that the program is a
significant first step in addressing the lack of agencywide written
guidelines for releasing data. The initiative appears to be consistent
with Office of Management and Budget guidelines issued in February 2002
requiring federal agencies to issue their own guidance for ensuring and
maximizing the quality, objectivity, utility, and integrity of information
disseminated by the
agency. 13 As the Bureau develops its guidelines, it will be important
that they be well documented, transparent, clearly defined, and
consistently applied.
Conclusions Although Service- Based Enumeration was designed to address
the challenges the Bureau encountered during the 1990 Census in obtaining
a
complete count of people without conventional housing, the Bureau*s
experience during the 2000 Census suggests that tallying this population
12 U. S. Department of Commerce, Technical Paper 32, Standards for
Discussion and Presentation of Errors in Data (March 1974), p. 3. 13
Issued at 67 Fed. Reg. 8452 (Feb. 22, 2002).
group remains problematic. Moreover, the Bureau*s difficulties were
compounded by its shifting position on how to report the data once they
were collected. A number of government, community, and advocacy
organizations helped the Bureau enumerate this population group. However,
the Bureau, by first planning to release the data one way, then
changing the decision, and ultimately releasing the data anyway* all for
reasons that were not clearly articulated to the Bureau*s stakeholders*
raised questions about the Bureau*s decision- making on data quality
issues. As noted at the beginning of this report, related questions have
also been raised about how the Bureau collected and reported data on
Hispanic subgroups. To the extent similar incidents occur in the future,
they could undermine public confidence in the accuracy and credibility of
Bureau data.
Thus, as the Bureau plans for the 2010 Census, it will be important for it
to refine its methods for enumerating people living in unconventional
housing and reporting the resulting data, in part by properly testing and
evaluating those methods. As noted earlier, the Bureau could not properly
test a key statistical technique during the census dress rehearsal because
the sample size was too small.
Moreover, while addressing the competing needs and desires of data users
will likely remain a considerable challenge, it will be important for the
Bureau to more effectively articulate its plans to avoid the expectation
gaps that occurred during 2000. The Bureau*s plans for collecting data on
persons without conventional housing need to specify how the Bureau
plans to separately report these data. Bureau- wide guidelines on the
level of quality needed to release data to the public, on how and when to
document data limitations, and on the circumstances under which it is
acceptable to suppress data, could help the Bureau be more accountable and
consistent in its dealings with data users and stakeholders, and help
ensure that the Bureau*s decisions both are, and appear to be, totally
objective. Recommendations for
To ensure that the 2010 Census will provide public data users with more
Executive Action
complete, accurate, and useful information on the segment of the
population without conventional housing, we recommend that the Secretary
of Commerce direct the Director of the Bureau of the Census to do the
following.
1. Ensure that all procedures for enumerating and estimating segments of
the population without conventional housing are properly tested and
evaluated under conditions as similar to the census as possible.
2. Develop agencywide guidelines for Bureau decisions on the level of
quality needed to release data to the public, how to characterize any
limitations in the data, and when it is acceptable to suppress the data
for reasons other than protecting the confidentiality of respondents.
Ensure that these guidelines are documented, transparent, clearly
defined, and consistently applied. 3. Ensure that the Bureau*s plans for
releasing data are clearly and
consistently communicated with the public. Agency Comments and The
Secretary of Commerce forwarded written comments from the Bureau Our
Evaluation
of the Census on a draft of this report (see app. I). The Bureau agreed
with each of our recommendations and, as indicated in the letter, is
taking steps to implement them. However, it expressed several general
concerns about our findings. The Bureau*s principal concerns and our
response are presented below. The Bureau also suggested minor wording
changes to provide additional context and clarification. We accepted the
Bureau*s suggestions and made changes to the text as appropriate. The
Bureau took exception to our findings concerning the adequacy of its
data quality guidelines, noting that the Bureau*s decisions regarding the
release and characterization of emergency and transitional shelter data
were based on established guidelines for data quality. However, the Bureau
did not cite any written guidelines to support its position. As noted in
our report, Bureau officials, including the Associate Director for
Methodology and Standards, told us that the Bureau had few written
guidelines, standards, or procedures related to the quality of data
released to the public. In this report we acknowledge the Bureau*s
tradition of high standards and procedures that yield quality data.
However, according to the Bureau, these standards are generally
undocumented and part of the agency*s institutional knowledge. To provide
a basis for consistent decision- making and clear communication within the
Bureau and to the
public, guidelines on the quality of data released to the public must be
fully documented, transparent, clearly defined, and consistently applied.
Additionally, the Bureau said that when data do not meet an acceptable
level of quality, it considers various options for modifying its
dissemination
plans. The Bureau*s decision to delay the release of the emergency and
transitional shelter data may have been entirely appropriate. Our concern
is not that the Bureau changed its plans, but that it could not provide us
its guidelines for determining an acceptable level of quality or clearly
indicate how it determined that the data did not meet minimal quality
standards for
release in SF- 1. The Bureau also commented that its decisions regarding
the distribution of data from SF- 1 were well publicized and that the only
change in Bureau plans for the release of Service- Based Enumeration data
was the decision
to delay release of the emergency and transitional shelter data. This
report focused on the changing plans for the release of the emergency and
transitional shelter data and noted that the Bureau never intended to
release any other data from the Service- Based Enumeration. However, we
found that the Bureau did not effectively communicate its decisions with
its partners or the public. Decisions on the release of the emergency and
transitional shelter data were contained in internal decision memoranda.
We found that these decisions were not always reflected in new releases of
the SF- 1 documentation. Although Bureau officials told us that they
always intended to produce a separate report on emergency and transitional
shelter data, they did not make this intention public when the SF- 1 data
were released. Some stakeholders did not realize that the Bureau was not
releasing emergency and transitional shelter data with SF- 1 until they
examined the SF- 1 data. As we stated in our report, these communication
problems can undermine stakeholder and public confidence in the Bureau
and its products. As agreed with your office, unless you publicly announce
its contents earlier, we plan no further distribution of this report until
30 days from its issue date. At that time, we will send copies of this
report to the Chairman of the House Committee on Government Reform, the
Chairman of the Subcommittee on Civil Service, Census and Agency
Organization, the Secretary of Commerce, and the Director of the Bureau of
the Census. Copies will be made available to others on request. This
report will also be available at no charge on GAO*s home page at http://
www. gao. gov.
Please contact me on (202) 512- 6806 or by e- mail at daltonp@ gao. gov if
you have any questions. Other key contributors to this report were Robert
Goldenkoff, Timothy Wexler, Elizabeth Powell, Chris Miller, James
Whitcomb, Ty Mitchell, Robert Parker, and Michael Volpe.
Patricia A. Dalton Director Strategic Issues
Appendi Appendi xes x I Comments from the Secretary of Commerce
Related GAO Products on the Results of the 2000 Census and Lessons Learned
for a More Effective Census in 2010
2000 Census: Refinements to Full Count Review Program Could Improve Future
Data Quality. GAO- 02- 562. Washington, D. C.: July 3, 2002. 2000 Census:
Coverage Evaluation Matching Implemented As Planned,
but Census Bureau Should Evaluate Lessons Learned. GAO- 02- 297.
Washington, D. C.: March 14, 2002.
2000 Census: Best Practices and Lessons Learned for a More CostEffective
Nonresponse Follow- Up. GAO- 02- 196. Washington, D. C.: February 11,
2002.
2000 Census: Coverage Evaluation Interviewing Overcame Challenges, but
Further Research Needed. GAO- 02- 26. Washington, D. C.: December 31,
2001.
2000 Census: Analysis of Fiscal Year 2000 Budget and Internal Control
Weaknesses at the U. S. Census Bureau. GAO- 02- 30. Washington, D. C.:
December 28, 2001.
2000 Census: Significant Increase in Cost Per Housing Unit Compared to
1990 Census. GAO- 02- 31. Washington, D. C.: December 11, 2001.
2000 Census: Better Productivity Data Needed for Future Planning and
Budgeting. GAO- 02- 4. Washington, D. C.: October 4, 2001.
2000 Census: Review of Partnership Program Highlights Best Practices for
Future Operations. GAO- 01- 579. Washington, D. C.: August 20, 2001.
Decennial Censuses: Historical Data on Enumerator Productivity Are
Limited. GAO- 01- 208R. Washington, D. C.: January 5, 2001.
2000 Census: Information on Short- and Long- Form Response Rates. GAO/
GGD- 00- 127R. Washington, D. C.: June 7, 2000.
(450102)
GAO*s Mission The General Accounting Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve
the performance and accountability of the federal government for the
American people. GAO examines the use of public funds; evaluates federal
programs and policies; and provides analyses, recommendations, and other
assistance to help Congress make informed oversight, policy, and funding
decisions. GAO*s commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of The fastest and easiest way to obtain copies of GAO
documents at no cost is
through the Internet. GAO*s Web site (www. gao. gov) contains abstracts
and fulltext GAO Reports and
files of current reports and testimony and an expanding archive of older
Testimony
products. The Web site features a search engine to help you locate
documents using key words and phrases. You can print these documents in
their entirety, including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as *Today*s Reports,* on its
Web site daily. The list contains links to the full- text document files.
To have GAO e- mail this
list to you every afternoon, go to www. gao. gov and select *Subscribe to
GAO Mailing Lists* under *Order GAO Products* heading.
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO
also accepts VISA and Mastercard. Orders for 100 or more copies mailed to
a single address are discounted 25 percent. Orders should be sent to:
U. S. General Accounting Office 441 G Street NW, Room LM Washington, D. C.
20548 To order by Phone: Voice: (202) 512- 6000 TDD: (202) 512- 2537 Fax:
(202) 512- 6061
To Report Fraud, Contact:
Waste, and Abuse in Web site: www. gao. gov/ fraudnet/ fraudnet. htm E-
mail: fraudnet@ gao. gov Federal Programs
Automated answering system: (800) 424- 5454 or (202) 512- 7470 Public
Affairs Jeff Nelligan, Managing Director, NelliganJ@ gao. gov (202) 512-
4800 U. S. General Accounting Office, 441 G Street NW, Room 7149
Washington, D. C. 20548
a
GAO United States General Accounting Office
The Bureau's original plan for releasing Service- Based Enumeration data
was outlined in an April 1999 internal memorandum that called for the
separate release of data on people counted at "emergency and transitional
shelters." The Bureau planned to combine other components of ServiceBased
Enumeration, including people counted at soup kitchens, regularly
scheduled mobile food vans, and certain outdoor locations, into a single
category. Driving the Bureau's decision was its experience during the 1990
Census when it released separate counts of people found at shelters, on
the
street, and similar locations that proved to be incomplete. The Bureau
also tried to ensure that the Service- Based Enumeration figures could not
be used as a "homeless" count, because it was not designed to provide a
specific count of the homeless. Instead, the operation was part of a
larger effort to count people without conventional housing.
The Homeless Are Hard to Enumerate
Source: U. S. Bureau of the Census.
In January 2001, the Bureau changed its earlier decision because a
statistical procedure used to refine the emergency and transitional
shelter data proved to be unreliable, which lowered the quality of the
data. In response, the Bureau combined the shelter data with a category
called "other noninstitutional group quarters," a category that also
includes data on people enumerated in several other group locations such
as facilities for victims of natural disasters. In the fall of 2001, the
Bureau produced a heavily qualified special report on the shelter data. A
key cause of the Bureau's shifting position on reporting these data
appears to be its lack of well documented, transparent, clearly defined,
and consistently applied guidelines on the minimum quality necessary for
releasing data. Had these guidelines been in place at the time of the
census, the Bureau could have been better positioned to make an objective
decision on releasing these figures. Additionally, the Bureau could have
used the guidance to explain to data users the reasons for the decision,
eliminating any appearance of censorship and arbitrariness. Because the
Bureau did not always adequately communicate its plans for releasing the
data, expectation gaps developed between the Bureau and entities that
helped with Service- Based Enumeration.
DECENNIAL CENSUS
Methods for Collecting and Reporting Data on the Homeless and Others
Without Conventional Housing Need Refinement
www. gao. gov/ cgi- bin/ getrpt? GAO- 03- 227. To view the full report,
including the scope and methodology, click on the link above. For more
information, contact Particia A. Dalton at (202) 512- 6806. Highlights of
GAO- 03- 227, a report to
Congressional Requesters
January 2003
The Bureau of the Census partnered with local governments, advocacy
groups, and other organizations to help it enumerate people without
conventional housing. Counting this population* which includes shelter
residents and the homeless* has been a longstanding challenge for
the Bureau. A number of organizations put substantial resources into an
operation the Bureau called Service- Based Enumeration. In return, some
expected the Bureau to provide data that would help them plan and
deliver employment, health, and other services. However, the Bureau did
not release the data as planned, which raised questions
about the Bureau*s decisionmaking on data quality issues. In response to a
congressional request, GAO examined the Bureau*s decision- making process
behind its change in plans. The Secretary of Commerce should direct the
Bureau to (1) properly test and evaluate procedures for counting people
without conventional housing; (2) develop guidelines for decisions on the
level of quality needed to release data to the public, how to characterize
any limitations, and when it is acceptable to suppress data; and (3)
ensure that plans for releasing data are clearly communicated to data
users.
The Bureau agreed with GAO*s recommendations, but took issue with our
findings on the adequacy of its data quality guidelines.
Page i GAO- 03- 227 Decennial Census
Contents Letter 1
Results In Brief 2 Background 4 Scope and Methodology 5 The Bureau of the
Census Twice Changed Plans for Reporting
Service- Based Enumeration Data 6 Census Bureau Had Few Documented
Guidelines Governing the
Release of Census Data 14 Conclusions 15 Recommendations for Executive
Action 16 Agency Comments and Our Evaluation 17
Appendix
Appendix I: Comments from the Secretary of Commerce 20 Related GAO
Products on the Results of the 2000 Census and Lessons Learned for a More
Effective Census in 2010
30 Figure Figure 1: The Bureau Changed Its Original Plan to Release
Emergency and Transitional Shelter Data and Combined
Them with Other Noninstitutional Group Quarters Data 8
This is a work of the U. S. Government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. It may contain
copyrighted graphics, images or other materials. Permission from the
copyright holder may be necessary should you wish to reproduce copyrighted
materials separately from GAO*s product.
Page 1 GAO- 03- 227 Decennial Census United States General Accounting
Office Washington, D. C. 20548
Page 1 GAO- 03- 227 Decennial Census
A
Page 2 GAO- 03- 227 Decennial Census
Page 3 GAO- 03- 227 Decennial Census
Page 4 GAO- 03- 227 Decennial Census
Page 5 GAO- 03- 227 Decennial Census
Page 6 GAO- 03- 227 Decennial Census
Page 7 GAO- 03- 227 Decennial Census
Page 8 GAO- 03- 227 Decennial Census
Page 9 GAO- 03- 227 Decennial Census
Page 10 GAO- 03- 227 Decennial Census
Page 11 GAO- 03- 227 Decennial Census
Page 12 GAO- 03- 227 Decennial Census
Page 13 GAO- 03- 227 Decennial Census
Page 14 GAO- 03- 227 Decennial Census
Page 15 GAO- 03- 227 Decennial Census
Page 16 GAO- 03- 227 Decennial Census
Page 17 GAO- 03- 227 Decennial Census
Page 18 GAO- 03- 227 Decennial Census
Page 19 GAO- 03- 227 Decennial Census
Page 20 GAO- 03- 227 Decennial Census
Appendix I
Appendix I Comments from the Secretary of Commerce Page 21 GAO- 03- 227
Decennial Census
Appendix I Comments from the Secretary of Commerce Page 22 GAO- 03- 227
Decennial Census
Appendix I Comments from the Secretary of Commerce Page 23 GAO- 03- 227
Decennial Census
Appendix I Comments from the Secretary of Commerce Page 24 GAO- 03- 227
Decennial Census
Appendix I Comments from the Secretary of Commerce Page 25 GAO- 03- 227
Decennial Census
Appendix I Comments from the Secretary of Commerce Page 26 GAO- 03- 227
Decennial Census
Appendix I Comments from the Secretary of Commerce Page 27 GAO- 03- 227
Decennial Census
Appendix I Comments from the Secretary of Commerce Page 28 GAO- 03- 227
Decennial Census
Appendix I Comments from the Secretary of Commerce Page 29 GAO- 03- 227
Decennial Census
Page 30 GAO- 03- 227 Decennial Census
United States General Accounting Office Washington, D. C. 20548- 0001
Official Business Penalty for Private Use $300 Address Service Requested
Presorted Standard Postage & Fees Paid
GAO Permit No. GI00
*** End of document. ***