American Community Survey: Key Unresolved Issues (07-OCT-04,
GAO-05-82).
The Congress asked GAO to review operational and programmatic
aspects of the Census Bureau's ACS that will affect the
reliability of small geographic area data. The ACS will be a mail
survey of about 3 million households annually, whose results will
be cumulated over 5 years to produce estimates that will replace
information previously provided by the Decennial Census long
form. In addition, annual data will be published for geographic
areas with 65,000+ populations and as 3-year averages for areas
with populations of 20,000 to 65,000. Annual data will be
published beginning in 2006 with data for 2005. The 5-year
averages for 2008-12 will provide data for small geographic
areas.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-05-82
ACCNO: A12973
TITLE: American Community Survey: Key Unresolved Issues
DATE: 10/07/2004
SUBJECT: Census
Evaluation methods
Operational testing
Population statistics
Program evaluation
Statistical data
Surveys
Data collection
Data integrity
Rural areas
2010 Decennial Census
Census Bureau American Community Survey
2000 Decennial Census
1960 Decennial Census
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-05-82
United States Government Accountability Office
GAO Congressional Requesters
October 2004
AMERICAN COMMUNITY SURVEY
Key Unresolved Issues
GAO-05-82
Highlights of GAO-05-82, a report to congressional requesters
The Congress asked GAO to review operational and programmatic aspects of
the Census Bureau's ACS that will affect the reliability of small
geographic area data. The ACS will be a mail survey of about 3 million
households annually, whose results will be cumulated over 5 years to
produce estimates that will replace information previously provided by the
Decennial Census long form. In addition, annual data will be published for
geographic areas with 65,000+ populations and as 3-year averages for areas
with populations of 20,000 to 65,000. Annual data will be published
beginning in 2006 with data for 2005. The 5-year averages for 2008-12 will
provide data for small geographic areas.
The Secretary of Commerce should direct the Census Bureau to revise the
ACS evaluation and testing plan, focusing on issues GAO identifies; give
stakeholders meaningful input on related decisions; and make the
underlying information public. The Secretary should direct the Bureau to
set a schedule for incorporating operational and programmatic changes into
the 5-year averages for 2008-12.
In commenting on a draft of this report, the Secretary stated that
Commerce has already addressed most of the key issues we identified in
this report. We believe, however, that the matters are not being fully
addressed and need further attention by Commerce.
www.gao.gov/cgi-bin/getrpt?GAO-05-82. To view the full product, including
the scope and methodology, click on the link above. For more information,
contact Robert P. Parker (202) 512-9750 or [email protected].
October 2004
AMERICAN COMMUNITY SURVEY
Key Unresolved Issues
The Census Bureau's development of the American Community Survey goes back
several decades and has included intensive research and field testing
programs, as well as substantial outreach efforts, in particular through
the reports and workshops at the National Academy of Sciences (NAS).
However, if the ACS is to be an adequate replacement for the Decennial
Census long form as the major source of data on small geographic areas and
if it is to provide similar annual data for larger areas, the Census
Bureau will need to
o incorporate in a timely manner the resolution of issues it has already
identified in the ACS testing and 2000 Decennial Census evaluation
programs, such as the residence concept, group quarters, and questions on
disability;
o complete the ACS testing plan as originally planned, such as the
comparison and evaluation of long form-ACS supplementary survey data at
the state level, to identify other unresolved issues and to provide
information for users of 2000 Decennial Census long-form data that will be
necessary for the transition to the full ACS;
o evaluate and consult with stakeholders and users on the resolution of
issues identified in this report, such as the methodology for deriving
population and housing controls, guidance for users on the impact of the
characteristics of multiyear averages for small geographic areas, and the
presentation of dollar-denominated values;
o coordinate the results of the testing program for the 2010 Decennial
Census short form with the ACS implementation schedule; and
o resolve all issues so that the ACS estimates beginning with 2008 are
consistent with the ACS estimates for 2009-12 and with the 2010 Census
short form.
Although the Census Bureau has solicited advice from external stakeholders
and users and has supported research by its own staff on most of the
issues identified in this report, there is no indication that the Census
Bureau has yet followed this advice or implemented plans for consultation
on resolving these issues. In addition, it has been more than a year since
the Census Bureau announced that it was looking into establishing an ACS
partnership program that would involve advisory groups and expert panels
to improve the program, but no such program has been established.
Another issue related to the proposed ACS is how the Census Bureau might
provide more timely and reliable small geographic area data. This goal
could be accomplished, but it would require additional funding. The most
direct approach would be to increase the sample size for 2009-11. This
increase would enable the Bureau to provide small geographic area data
that would be the replacement for the 2010 Census long form 1 year
earlier.
Contents
Letter
Results in Brief
Background
Outstanding Issues Jeopardize ACS's Replacement of the Long
Form
Improving Timeliness and Quality of Small Geographic Area Data
Would Increase Costs
Resolving Outstanding Issues Needs a Time Schedule
Conclusions
Recommendations for Executive Action
Agency Comments and Our Evaluation
1
3 6
10
21 22 25 26 26
Appendix I Scope and Methodology
Appendix II Recent NAS Findings on Continuous Measurement
and the ACS 35
1998 NAS Workshop on the ACS 36
2000 Interim Report 39
2001 Letter Report 41
2003 Interim Report 42
The 2000 Census: Counting under Adversity 45
Appendix III The Decennial Census Long Form and the Evolution
of the ACS Plan 47
Decennial Census Long Form 47
Evolution of ACS Plan 47
Appendix IV Continuous Measurement ACS Testing and Development Program
Appendix V Current Status of Unresolved Issues 62
Independent Controls for Population and Housing Characteristics 62
Operational Issues 70
Valuation and Presentation of Dollar-Denominated Data Items 73
Evaluations of ACS, Long-Form, and CPS Data Comparisons 76
Information on Key Properties of Multiyear Averages 81
External Consultation 86
Appendix VI Comments from the Department of Commerce 89
Appendix VII GAO Contacts and Staff Acknowledgments 104
GAO Contacts 104
Staff Acknowledgments 104
Bibliography
History of the Long Form and Mid-Decade Census
Census Bureau ACS Reports
Census Bureau Advisory Committee Presentations
Census Bureau Continuous Measurement Series
Census Bureau Internal Reports
Census 2000 Evaluation Reports
Census Bureau 2003 JSM Staff Papers
Other Census Bureau Staff Research Papers
Association of Public Data Users Papers
Congressional Hearings and Testimony
Other Reports and Papers
105
105 105 107 108 108 109 109 110 111 112 112
Related GAO Products
Tables
Table 1: ACS Milestone Events and Unresolved Issues, 2004-13 23 Table 2:
Continuous Measurement and ACS Funding, Fiscal Years 1995-2005 49 Table 3:
The 2000 Census Long Form and ACS Use of Independent Controls for
Population and Housing Characteristics 63 Table 4: Population Comparison
for Counties in 2000 from ICPE and 2000 Census by County Size 67
Abbreviations
ACS American Community Survey
ASA American Statistical Association
BEA Bureau of Economic Analysis
BLS Bureau of Labor Statistics
C2SS Census 2000 Supplementary Survey
CM Continuous Measurement
CPI Consumer Price Index
CPS Current Population Survey
HUD Department of Housing and Urban Development
ICPE Intercensal Population Estimates
NAS National Academy of Sciences
OMB Office of Management and Budget
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
United States Government Accountability Office Washington, DC 20548
October 8, 2004
The Honorable Tom Davis, Chairman
The Honorable Henry Waxman, Ranking Minority Member
Committee on Government Reform
House of Representatives
The Honorable William Lacy Clay, Ranking Minority Member
Subcommittee on Technology, Information Policy, Intergovernmental
Relations, and the Census Committee on Government Reform House of
Representatives
The Honorable Adam Putnam House of Representatives
The Census Bureau has designed the 2010 Decennial Census around three new
operations. One will replace the Census long-form questionnaire with the
American Community Survey (ACS).1 Testing the ACS began in 1996 and full
implementation will begin in 2005 and continue as long as the program
receives annual funding.
A separate long-form questionnaire has been mailed to a sample of
households once a decade to collect detailed information on demographic,
housing, social, and economic characteristics since the 1960 Decennial
Census. This information has been the main source of information for small
geographic areas, including tracts and block groups; it has been used
extensively by federal agencies for program implementation and by state
and local governments for programmatic and planning purposes. In the 2000
Decennial Census, the long form was mailed to a sample of about 19 million
housing units.
The ACS will contain the same questions as the long form but will be
mailed monthly to an annual sample of 3 million housing units. With the
smaller sample, the ACS is designed to provide the same information at
1We discuss the other operations, which relate to the address list and the
short-form census, in full in GAO, 2010 Census: Cost and Design Issues
Need to Be Addressed Soon, GAO-04-37 (Washington, D.C.: Jan. 15, 2004).
the same level of geographic detail as the long form by means of a
continuous measurement methodology in which survey responses will be
accumulated over time. The Census Bureau has determined that in order to
produce reliable estimates at the same geographic level of detail as the
long form, ACS results will be cumulated over 5 years. It also has
determined that the ACS will provide reliable estimates for geographic
areas with populations of 20,000 to 65,000 by cumulating ACS responses
over 3 years and for geographic areas with populations of more than 65,000
by cumulating ACS results for 1 year but that these estimates will be less
reliable than the corresponding long-form estimates.
According to the plan the Congress approved, the first annual ACS data for
geographic areas with populations larger than 65,000 will be published
beginning in 2006 with data for 2005; 3-year averages for geographic areas
with populations between 20,000 and 65,000 will begin in 2008; and 5-year
averages for geographic areas with populations smaller than 20,000,
including tracts and block groups, will begin in 2010. The 5-year averages
for 2008-12 to be published in 2013 will replace the 2010 Decennial Census
long form for small geographic areas, as they will be centered on 2010 and
closely reflect the population and housing characteristics data from the
2010 Decennial Census short form. In replacing the long form, the ACS will
provide the same long-form data items at the same level of geographic area
detail but in a more timely way. Whereas the long form provided small
geographic detail once a decade, the ACS will provide annual estimates for
large geographic areas and estimates for smaller areas in terms of 3-year
or 5-year averages.
You asked us to examine issues about replacing the long form with the ACS
related to the reliability of data for small geographic areas. As agreed
with your offices, our objectives for this report were to (1) review the
Census Bureau's testing program on operational and programmatic aspects
that will affect the reliability of small geographic area data and (2)
determine whether alternatives to the proposed ACS would provide more
frequent and more reliable data for small geographic areas.
To address these topics, we reviewed ACS-related Census Bureau documents,
congressional testimony, National Academy of Sciences (NAS) reports, and
consultants' reports prepared for the Census Bureau, Bureau of Labor
Statistics (BLS), and Department of Housing and Urban Development (HUD).
We also interviewed small-area data experts on the latest NAS report on
the ACS and reviewed the Census Bureau's responses to recommendations on
the ACS in our earlier reports.
Results in Brief
Key Unresolved Issues
We conducted our work between April 2003 and August 2004 in accordance
with generally accepted government auditing standards. We describe our
scope and methodology in more detail in appendix I.
If the ACS is to be an adequate replacement for the Decennial Census long
form as the major source of data on small geographic areas and if it is to
provide similar annual data for larger areas, we believe that the Census
Bureau will need to (1) incorporate in a timely manner the resolution of
issues it has already identified in testing the ACS, (2) complete the ACS
evaluation and testing plan to identify other issues and provide
information for users that will be necessary for the transition to the
full ACS, (3) evaluate issues identified in this report and consult with
stakeholders and users on their resolution, (4) coordinate the results of
the testing program for the 2010 Decennial Census short form with the ACS
implementation schedule, and (5) resolve all issues so that the ACS
estimates beginning with 2008 are consistent with the ACS estimates for
2009-12 and with the 2010 Census short form.
Unresolved issues that might affect the reliability of ACS small
geographic area data include (1) the introduction of a new concept of
residence, (2) the uncertainty about the new methodology for deriving
independent controls for population and housing characteristics, (3) the
lack of guidance for users from the Census Bureau on the characteristics
of multiyear averages for small geographic areas, and (4) operational
procedures, such as questionnaire design and the adjustment to
dollar denominated values, and to the consistency between ACS and 2000
Census long-form data.
The Census Bureau has announced that it will adopt a concept of "current
residence" for determining the geographic location of seasonal residents
for the full ACS. The concept will differ from "usual residence," used for
decennial censuses and the ACS testing programs. Under the usual residence
concept, people who spend their winter in Florida and the rest of the year
in New Hampshire, for example, are recorded as residents of New Hampshire;
college students living away from home in dormitories are recorded as
residents of the college. Under the current residence concept, people have
only one residence at any point in time, but their place of residence does
not have to be the same throughout the year. Although the Census Bureau
plans to change this concept for the ACS, it has reported that sufficient
research has not been conducted to make a final set of rules for
determining current residence. In addition, it found
User Guidance on Multiyear Averages
ACS Implementation Schedule
problems with the residence questions used in 2000 but does not plan to
incorporate improved questions until 2010.
To determine independent controls for population and housing
characteristics, which will be used to adjust ACS sample results, the ACS
will use the characteristics derived from decennial censuses for the
census year and for other years from the Census Bureau's Intercensal
Population Estimates (ICPE) program. The Census Bureau has not developed a
methodology for using ICPE for the full ACS to derive controls consistent
with the ACS residence concept and ACS reference period or at the same
level of geography used for the 2000 Census long form.
Before data for 2005 on places with populations of 65,000 or more can be
released in 2006, a methodology is needed to provide controls that reflect
changes in the residence concept and reference period. A methodology for
controls for places with populations of more than 20,000 that incorporates
ICPE revisions is needed before the first multiyear averages are released
in 2008. In addition, if the averages for 2008-12 are to replace the 2010
Census long form, the methodology for incorporating 2010 Census data and
the related revisions to ICPE data will be needed in 2009.
ACS data for geographic areas with populations smaller than 65,000 will be
presented only in terms of multiyear averages. Because of the statistical
properties of these averages and users' unfamiliarity with them, we found
that it is critical for the Census Bureau to provide users with guidance
on topics such as the reliability of multiyear averages for areas with
rapidly changing populations, the reliability of trends calculated from
annual changes in multiyear averages, and the use of multiple estimates
from the ACS data for geographic areas with populations larger than
20,000. Census Bureau officials told us that they agreed with the need for
such guidance but had no plans for its contents.
We found that the latest schedule for the 2010 Decennial Census does not
provide adequate time for the Census Bureau to incorporate into the full
ACS program changes necessary for the ACS data for 2008-12 to be reliable
enough to replace the 2010 Census long form. We identified issues that
need to be resolved before the 2006 release of the 2005 ACS and other
issues that need to be resolved before the release of the first 3-year
averages in 2008. The most important issues to be resolved are those that
need to be in place by 2008, when the collection of data for calculating
the 5-year averages (for 2008-12) that will replace the 2010 Census long
form will begin. Prompt resolution of the other issues would improve
Alternatives to Improve Small Geographic Area Data
consistency between the 2005-07 ACS data and the ACS data beginning with
2008.
Besides the key unresolved issues discussed above, we also identified an
alternative to the proposed ACS that would provide more timely and
reliable small geographic area data. This alternative would require
additional funding to support a larger sample. Under an alternative,
patterned after the Census Bureau's initial plan to replace the 2000
Census long form, the sample size for 2009-11 would be increased to 4.8
million housing units and then reduced to 3.0 million housing units for
subsequent years. The larger sample would provide small geographic area
data that would be the replacement for the 2010 Census long form from
3-year averages (for 2009-11). These averages would be as reliable as the
proposed 5-year averages (for 2008-12) and would provide the replacement
for the long form data 1 year earlier. The larger sample could also be
used permanently after 2011 and would provide continuous 3-year averages
for small geographic areas.
In written comments on a draft of this report, the Secretary of Commerce
addressed three of the four recommendations we addressed to him. Regarding
the first recommendation, the Secretary stated that the current ACS
testing and evaluation plan already included the issues we have identified
in the report. In following up to the Secretary's response, we learned
that there is not yet a written plan, but only a rough outline of the
types of work planned. Therefore, we believe our recommendation remains
valid. Regarding the second recommendation, suggesting that the Census
Bureau provide key stakeholders more direct and timely input into
decisions on these issues, the Secretary stated that he believes that the
present consultation process is adequate. We disagree, because as noted in
appendix II of our report, the Census Bureau has not been responsive to
recommendations from several National Academy of Sciences reports relating
to the ACS. The Secretary agreed with our third recommendation that the
Census Bureau provide public documentation for key decisions on issues we
have identified in this report. The Secretary did not respond directly to
our recommendation that he direct the Census Bureau to prepare a schedule
for the 2010 Census that ensures that all necessary changes are made in
time for the 2008 ACS so the 5-year ACS averages for 2008-2012 will be an
adequate replacement for the 2010 long form for small geographic areas.
The comments from the Secretary also include a list of detailed technical
comments from the Census Bureau. We reviewed each of these comments and
revised the report where appropriate.
Background
Now that the Census Bureau has congressional approval to begin the full
ACS, data collection will begin in November 2004. The ACS test survey of a
sample of 800,000 housing units, which has been conducted since 2000, will
end in December 2004. The Bureau has been using this survey, known as the
ACS Supplementary Survey, to test procedures and to produce annual data
for geographic areas with populations of 250,000 or more. As one part of
the test program, the supplementary survey data for 2000 have been
compared with corresponding data from the 2000 Census long form to
evaluate the quality of the ACS data and to provide users with information
to make the transition from the long-form data to the full ACS data.
According to the plan the Congress approved, the first annual ACS data for
geographic areas with populations larger than 65,000 will be published
beginning in 2006 with data for 2005; 3-year averages for geographic areas
with populations between 20,000 and 65,000 will begin in 2008; and 5-year
averages for geographic areas with populations smaller than 20,000,
including Census tracts and block groups, will begin in 2010. The 5-year
averages for 2008-12 will replace the 2010 Decennial Census long form for
small geographic areas; they will be published in 2013 and will
incorporate population and housing characteristics data from the 2010
Decennial Census short form.
In replacing the long form, the ACS will provide the same long-form data
items at the same level of geographic area detail but in a more timely
way. Whereas the long form provided small geographic detail once a decade,
the ACS will provide annual estimates for large geographic areas and
estimates for smaller areas in terms of 3-year or 5-year averages; the
5-year averages will provide data at the same geographic area level as the
long form. According to the Census Bureau, these 5-year averages will be
about as accurate as the long-form data; the annual and 3-year averages
will be significantly less reliable than the long-form data but more
reliable than existing annual household surveys the Census Bureau
conducts.2
In the remainder of the Background section of this report, we briefly
describe the major differences between the ACS and the Decennial Census
long form. We also discuss the Census Bureau's outreach program, designed
to involve stakeholders and users in shaping the ACS. Appendix III
provides additional background information on the evolution of the
2We discuss the relative quality of the ACS and the long form in GAO, The
American Community Survey: Accuracy and Timeliness Issues, GAO-02-956R
(Washington, D.C.: Sept. 30, 2002), pp. 8-13.
ACS plan, appendix IV on the ACS testing and measurement program. Appendix
II describes recent NAS findings on Continuous Measurement (CM) and the
ACS.
Multiyear Averages
The 2000 Census long form used a decennial sample of about 19 million
housing units; the full ACS will use an annual sample of 3 million housing
units. In order to provide reliable estimates for geographic areas with
populations of 65,000 or less, monthly ACS responses will be cumulated
over several years-3 years for places with populations of 20,000 to 65,000
and 5 years for places with populations smaller than 20,000. Because of
the statistical properties of these averages and users' unfamiliarity with
them, the Census Bureau has long recognized the need to provide guidance
on such topics as the reliability of the averages for areas with rapidly
changing population and the use of multiple estimates for states and
other, larger geographic areas.
The Concept of Residence
For the 2000 Decennial Census, the ACS test programs, and federal
household surveys, including the Current Population Survey (CPS), seasonal
residents are recorded in a geographic area according to a concept of
usual residence. As we noted above, under this concept, people who spend
their winter in Florida and the rest of the year in New Hampshire, for
example, are recorded as residents of New Hampshire; college students
living away from home in dormitories are recorded as residents of the
college.
For the full ACS, the Census Bureau has announced its decision to change
the concept to current residence. According to the Census Bureau, although
each concept requires that a person have only one residence at any point
in time, current residence recognizes that the place of residence does not
have to be the same throughout a year, allowing ACS data to more closely
reflect the actual characteristics of each area. The Census Bureau plans
to use current residence because the ACS is conducted every month and
produces annual averages rather than point-in-time estimates, unlike the
Decennial Census. Current residence is uniquely suited to the ACS, because
it continuously collects information from independent monthly samples
throughout all months of all years. Because the ACS is designed to produce
a continuous measure of the characteristics of states, counties, and other
places every year, the new residence rules were needed for seasonal and
migratory individuals.
Reference Period The underlying population and housing characteristics
data for the 2000 Census long form were for April 1, 2000. For the ACS
test program, the underlying population and housing characteristics
varied. For all years except 2000, they were for July 1; for 2000, they
were for April 1. For the full ACS, because the data are collected
monthly, the reference period will be the average for the year, and the
Census Bureau will assume this average is equivalent to data for July 1.
Independent Controls for Population and Housing Characteristics
The ACS will use population characteristics (age, sex, race, and
ethnicity) and housing characteristics (occupied and vacant units) derived
from an independent source and not from the results collected in the
survey. Using independent controls for these characteristics is standard
practice to correct sample survey results for the effects of nonresponse
and undercoverage. Population and housing characteristics from the 2000
Census short form were used as independent controls for the 2000 Census
long form, down to the tract level. For the ACS supplementary surveys,
independent controls were from ICPE, which uses Decennial Census
short-form data as benchmarks and administrative record data to
interpolate between and extrapolate from the census benchmarks. ICPE
develops and disseminates annual estimates of the total population and the
distribution by age, sex, race, and Hispanic origin for the nation, state,
counties, and functioning government units. ICPE provides annual estimates
of population and housing characteristics at the county level, and for
some subcounty levels, as of July 1, using the usual residence concept for
seasonal residents.
Dollar-Denominated Data Items
According to current Census Bureau plans, annual estimates of
dollar denominated data items, such as income, rent, and housing-related
expenses, will be presented after adjustment for inflation in order to
facilitate comparisons over time. As in the ACS test programs, only annual
estimates with this adjustment will be presented. The Census Bureau also
has decided to continue to adjust annual data collected each month in the
ACS to a calendar year basis. It will be using the Consumer Price Index
(CPI) for the annual and monthly adjustments for all geographic areas.3
3The CPI is a national-level price index that BLS compiles. It also
compiles separate price indexes for selected geographic areas, but these
indexes do not measure differences in the level of prices among areas.
Operational Differences The long form and ACS will also differ in how
operations are conducted, such as nonresponse follow-up and data capture.
For the 2000 Census long form, nonresponse follow-up was conducted for all
nonrespondents. For the ACS supplementary surveys and for the full ACS,
nonresponse follow up will be conducted for a sample of one-third of all
nonrespondents. For the 2000 Census long form, all data items were entered
using automated optical character recognition procedure; data from the ACS
will be manually keyed.
Group Quarters
The ACS supplementary surveys excluded persons living in group quarters.
Group quarters-which include nursing homes, prisons, college dormitories,
military barracks, institutions for juveniles, and emergency and
transitional shelters for the homeless-accounted for roughly 2.8 percent
of the population in 2000. The Census Bureau decided not to cover these
persons in the supplementary surveys, to avoid duplication with the 2000
Census, and because it lacked funding to cover them in subsequent years.
Procedures for including in the ACS persons living in group quarters
beginning with 2005 are discussed in the Census Bureau's ACS Operations
Plan, issued in March 2003.4 In addition, it has announced that it intends
to continue testing procedures to improve the mailing list for group
quarters to be used for the 2010 Decennial Census.
Outreach
The Census Bureau has long recognized the need to seek input from
stakeholders and users in making decisions for all its programs. The
Census Bureau sponsors technical reports that NAS prepares. (In appendix
II, we summarize recent NAS reports on the ACS and related decennial
censuses.) The Census Bureau has also held conferences on the ACS and has
contracted with Westat Inc. to organize two conferences of experts on
specific aspects of the ACS. Additionally, the Census Advisory Committees,
which are Census Bureau-appointed advisory committees whose members
represent professional associations such as the American Statistical
Association (ASA) and the American Marketing Association, meet twice a
year. The Census Bureau and other federal statistical agencies also
participate in the quarterly meetings of the Council of Professional
Associations on Federal Statistics, whose members include
4U.S. Census Bureau, American Community Survey Operations Plan, Release 1
(Washington, D.C.: March 2003), pp. 52-53.
Outstanding Issues Jeopardize ACS's Replacement of the Long Form
professional associations, businesses, research institutes, and others
5
interested in federal statistics.
To obtain input from other federal agencies, the Office of Management and
Budget (OMB) established an interagency advisory committee for the ACS in
2000. The committee's major purpose was to coordinate the review of
questions to be included in the ACS. Because of the committee's limited
focus, the Census Bureau established the ACS Federal Agency Information
Program in 2003, responding to a recommendation we made.6 This program is
designed to assist each federal agency that has a current or potential use
for ACS data to achieve a smooth transition to using the ACS.
From its beginnings in the mid-1990s, the Census Bureau's development plan
for the ACS was designed to ensure that the ACS would satisfactorily
replace the Decennial Census long form as the major source of small
geographic area data. In our review of the plan, we found that the Census
Bureau, as well as key ACS stakeholders, had for many years identified the
key issues that needed to be resolved if the ACS were to reach this goal.
We have identified the following unresolved issues from our research
(described in appendix I):
o the methodology to be used for deriving independent controls for
population and housing characteristics with ACS definitions of place of
residence and reference date,
o improvements needed to operational procedures,
o methods for valuation and presentation of dollar-denominated data
items,
o comprehensive analysis of the comparability between new ACS data and
corresponding data from the 2000 Census long-form and 2004 supplementary
survey, and
o the provision of user guidance on multiyear averages.
5This group's objective is to provide an open dialogue between its members
and federal statistical agencies. See Council of Professional Associations
on Federal Statistics, http://www.copafs.org (May 10, 2004).
6See our recommendations in GAO-02-956R, pp. 25-26. For information on the
Federal Agency Information Program, see Census Bureau, American Community
Survey, http://www.census.gov/acs/www (May 10, 2004).
Despite the Census Bureau's early identification of issues critical to the
successful replacement of the 2010 Decennial Census long form as the new
source of small geographic area data, we found that its plans to resolve
these issues have been only partially completed. Furthermore, we found
that despite recent changes to the ACS implementation schedule, it is not
fully synchronized with the Census Bureau's time schedule for implementing
the testing program for the 2010 Decennial Census. Consequently, if these
issues are not resolved in a timely manner, the Census Bureau's plan to
replace the 2010 Decennial Census long form with the 2008-12 ACS averages
for detailed geographic areas will be jeopardized.
A Methodology for Independent Controls for Population and Housing
Characteristics Is Lacking
It is standard practice to use independent controls for population and
housing characteristics to correct the results of sample surveys for the
effects of nonresponse and undercoverage. For the 2000 Census long form,
characteristics from the 2000 Census short form were used as independent
controls down to the tract level. For the annual ACS supplementary
surveys, characteristics from ICPE were used as the independent
7
controls.
Independent controls for the full ACS will require a new methodology.
Short-form data are available only once every 10 years, and the annual
ICPE estimates do not provide data for the detailed geographic areas
needed to prepare long-form detail and do not use the ACS residence
concept or reference period. The new methodology is critical to the
reliability of the ACS estimates of small geographic areas that ICPE does
not provide and of areas that have large numbers of seasonal residents.
Census Bureau staff have long recognized the need for the new methodology.
For example, a 1995 paper by Love, Dalzell, and Alexander expressed
concern about population controls and residence rules as well
7ICPE develops and disseminates annual "official" estimates of the total
population and the distribution by age, sex, race, and Hispanic origin for
the nation, state, counties, and functioning government units. The program
is authorized by 13 U.S.C. S:181, which requires the production of
"current data on total population and population characteristics." The
estimates of population and housing characteristics are as of July 1 of
each year, using the usual resident concept for seasonal residents. For
details on subcounty estimates, see U.S. Census Bureau, "Estimates and
Projections Area Documentation: Subcounty Total Population Estimates,"
http://www.census.gov.
as the need for consultation with users on these topics.8 They reported
that the Census Bureau was planning to conduct research using data from
the 1996 test sites to produce controls at the census tract and block
group levels. They also noted that the Census Bureau would need to conduct
research on the residence rule. A 2000 paper by Alexander and Wetrogan
also discussed the issue of population controls.9 They reviewed possible
methods for using ICPE to develop controls for the ACS and noted the need
to consult with users on how to present information on the differences in
ACS controls and ICPE in ACS publications.
Key stakeholders, including experts on the ACS we interviewed in August
2003 (listed in appendix I), expressed similar concerns about the
methodology.
It appears that no progress had been made on a new methodology until the
Census Bureau reported in October 2003 to its advisory committees on the
status of a new methodology to derive controls. It announced that when
full ACS collection starts in November 2004, (1) interim procedures would
be used and (2) a final methodology would not be determined until after
the necessary research was completed. The Census Bureau did not provide a
date when the methodology would be incorporated.
In our review of Census Bureau presentations about the new methodology
(described in detail in appendix V), we found that it had no plans to
maintain time-series consistency of the population and housing controls by
routinely incorporating the regular revisions to ICPE estimates into the
ACS. Without such revisions, there could be a significant lack of
comparability in the ACS data being averaged, and the reliability of
multiyear estimates would be reduced. For example, without such revisions,
the 2008-12 averages that are to replace the 2010 Decennial Census long
form would be based on controls extrapolated from the 2000 Census for
2008-09 and controls from the 2010 Census for 2010-12. In addition,
time-series consistency in the annual ACS data would be reduced,
especially in the data for 2010 and previous years. Census
8Susan Love, Donald Dalzell, and Charles Alexander, "Constructing a Major
Survey: Operational Plans and Issues for Continuous Measurement,"
presented at the annual American Statistical Association meeting, Orlando,
Florida, August 1995.
9Charles H. Alexander and Signe Wetrogan, "Integrating the American
Community Survey and the Intercensal Demographic Estimates Program,"
presented at the Joint Statistical Meetings, Indianapolis, Indiana, August
14, 2000.
Bureau officials told us that they were not planning any such revisions,
unless the inconsistencies between 2010 ICPE and 2010 Census
characteristics were significant, even though there were significant
inconsistencies between the 2000 ICPE estimates and the 2000 Census
data, especially for small geographic areas.
We found that regularly incorporating all revisions to ICPE into the ACS
would improve ACS reliability and that planning would give users advance
notice on the Census Bureau's revision practice. The need for such
planning is critical, as evidenced by the failure that occurred in January
2004, when a revised set of ICPE data was incorporated into the
calculation of monthly CPS data on employment. Initially, the revised
employment estimates were released without a revision of the
pre-2004 data, resulting in a significant discontinuity between December
2003 and January 2004. As a result of users' dissatisfaction about the
discontinuity, a consistent set of employment estimates was released.10
Finally, failure to adequately involve stakeholders in the decision
process
may contribute to significant misunderstanding about the use of the ACS
estimates and corresponding estimates from the Decennial Census. In past
decennial censuses, except for the very smallest geographic areas, the
population and housing characteristics data published as part of the long
form detail were the same as the official data based on data collected on
the short form. Because of differences in the residence and reference
period concepts and the use of multiyear averages for small geographic
areas, there will be less consistency between the ACS averages for
2008-12 and the 2010 Census data.
Operational Issues Have Not Been Addressed
The Census Bureau has identified operational issues with the ACS test
programs, primarily from its evaluation studies on the 2000 Decennial
Census and Census Bureau staff research papers on comparisons between data
collected in the ACS 2000 Supplementary Survey and the 2000 Decennial
Census long form. These issues (described in detail in appendix
V) include problems with questionnaire design, nonresponse followup, and
data capture, as well as coverage of persons living in group quarters.
10See Bureau of Labor Statistics, "Labor Force and Employment Estimates
Smoothed for Population Adjustments, 1990-2003," Washington, D.C., March
3, 2004. U.S. Department of Labor, Bureau of Labor Statistics,
Demographics, Demographic Characteristics of the Labor Force (Current
Population Survey), http://www.bls.gov/cps/cpspopsm.pdf (May 10, 2004).
For example, the Census Bureau conducted a study to evaluate the design of
the ACS questions that are needed to implement the residence concept and
reference period for the ACS.11 The study suggested that additional
testing was needed for the questions about multiple residences and noted
"that asking these questions on a person basis may produce different and
probably better data than asking them on a household basis."12 Similarly,
the authors found potential problems with the identification of seasonal
residents. We were not able to identify in the Census Bureau's plans
whether these issues would be addressed before implementation of the full
ACS.
We also found, for the implementation of the full ACS for 2005, that the
Census Bureau had addressed only the inclusion of group quarters and that
it may not resolve the issue of questionnaire design until 2010. In
addition, even for group quarters, it is planning for improvements that
may not be included until 2010. Furthermore, not all problems have been
identified because of the delays in the Census Bureau's completing the
evaluation studies of comparisons of long-form and ACS data items.
Moreover, the Census Bureau's plans do not provide for external
consultations on key decisions about resolving issues.
Although the Census Bureau has acknowledged the importance of the timing
of incorporating changes to resolve the various issues, any delay in
implementing solutions to 2010 would not meet the needs of the ACS
collection and production schedule. For example, in its March 2003 ACS
operations plan, the Census Bureau recognized the need for maintaining
questionnaire continuity to calculate consistent multiyear averages. It
also has reported that it needs to incorporate changes in the ACS
questionnaire no later than 2008 because changes introduced after 2008 and
before 2013 would create inconsistencies in calculating the 5-year
averages that are to replace the 2010 Decennial Census long form.
Nevertheless, we found that the Census Bureau's current time schedule does
not call for resolving issues such as questionnaire design before 2008.
Incorporating changes into the ACS beginning with 2008 will help maintain
the reliability of the 5-year averages for small geographic areas; failing
to
11Theresa J. DeMaio and Kristen A. Hughes, "Report of Cognitive Research
on the Residence Rules and Seasonality Questions on the American Community
Survey," U.S. Census Bureau, Statistical Research Division, Washington,
D.C., July 2003.
12DeMaio and Hughes, pp. 9-10.
incorporate them beginning with 2005 will reduce the reliability of the
annual changes in the ACS data.
With regard to external consultation, we found that the Census Bureau's
plans do not include time for consulting with stakeholders and users,
despite NAS, BLS, and Census Advisory Committee suggestions and
recommendations. For example, in a February 15, 2001, report to the Census
Bureau, the NAS Panel on Research on Future Census Methods recommended
that it conduct evaluation studies on "the effectiveness of operations
used to designate special places and enumerate the group quarters and
homeless populations."13 Members of the Census Advisory Committee had
raised similar concerns. In a 2003 report prepared for BLS, their
consultant had made a number of recommendations about the questions on
employment.
We found that the Census Bureau needs to develop a time schedule so that
changes can be introduced to minimize inconsistencies between the 2005 and
subsequent ACS data and to ensure that all necessary changes are made so
that the ACS data for 2008-12 that will replace the 2010 Decennial Census
long form will be collected consistently. In addition, the prompt
completion of the ACS-long-form comparison studies and related evaluations
will provide sufficient time for the Census Bureau to consult with
stakeholders and to provide users with the information they need to
understand the effect of making changes to the ACS questionnaires or
procedures between 2005 and 2008.
Plans for Valuation and When the Census Bureau began releasing data from
the ACS test Presentation of Dollar-programs, all dollar-denominated items
such as incomes, housing values, Denominated Data Items rents, and
housing-related expenditures were adjusted for inflation. As in
the ACS test programs, only annual estimates with this adjustment will be
Are Questionable presented, and when the Census Bureau releases ACS data
for each new year, it revises all dollar-denominated data for prior years.
It makes a similar inflation adjustment for the annual income data
collected in the
13Benjamin F. King, Chair, Panel on Research on Future Census Methods,
National Academy of Sciences, letter to William Barron, Acting Director,
U.S. Bureau of the Census, Washington, D.C., February 15, 2001, pp. 3-4.
The National Academies, National Academies Press, 2010 Census Panel Letter
Report (2001),
http://books.nap.edu/html/2010_census_panel/letterreport.pdf (May 10,
2004).
CPS, but it releases the unadjusted estimates.14 The Census Bureau also
has decided to continue to adjust annual data collected each month in the
ACS to a calendar year basis. It will be using the CPI for the annual and
monthly adjustments for all geographic areas.
The treatment of dollar-denominated data items is critical to all users of
these data. It is particularly critical for federal agencies that will be
using the ACS instead of the long form for many government programs to
determine the allocation of funds or program eligibility. It is also
critical to users of dollar-denominated items for small geographic areas
because the inflation adjustments under the current procedure are based on
a national average index.
In our review of the development and implementation of the ACS, we
identified questions on the appropriateness of the methodology for the
adjustment and the suppression of the unadjusted annual values. A report
prepared for HUD found problems with the calculation of the adjustment and
the use of the adjustment for income measures used for HUD programs. The
report also noted that the lack of the unadjusted annual data would
severely limit HUD's use of calculations appropriate to its program needs.
Research by Census Bureau staff questioned the adjustment for incomes when
they found that it was a probable source of difference between income data
from the supplementary survey and corresponding data from the CPS and the
2000 Census long form.15 (We discuss these findings in detail in appendix
V.)
Our statisticians reviewed these findings and found a similar problem with
the calculation of the adjustment because of the lack of trending
adjustment. We found that the Census Bureau could estimate calendar year
values using a combination of past trends in related series, information
from other ACS respondents, or known information such as changes in
cost-of-living adjustments for various transfer payment programs and
changes in wage rates. We also found that converting ACS
14BLS makes a similar adjustment to the average weekly earnings data from
the monthly establishment survey.
15For the HUD report, see ORC Macro, The American Community Survey:
Challenges and Opportunities for HUD (Calverton, Md.: Sept. 27, 2002). For
a complete discussion of the role of the inflation adjustment in
differences between the ACS and CPS measures of income, see Kirby G.
Posey, Edward Welniak, and Charles Nelson, "Income in the ACS: Comparisons
to Census 2000," presented at the Joint Statistical Meetings, San
Francisco, California, August 7, 2003.
data from monthly to calendar year data is similar to conversion issues
faced by other agencies that collect annual statistics compiled on a
fiscal year basis and that the procedures these agencies use could be
adapted for the ACS.16 With regard to the use of a national cost-of-living
adjustment, we have previously reported that for purposes such as
allocating federal funds to states using income and poverty data, the CPI,
a national measure of inflation, does not reflect variations in geographic
areas.17 Census Bureau staff have reported similar findings.18
The HUD and Census Bureau findings and our review raise serious questions
about the inflation adjustments. We found no documentation explaining the
rationale for the adjustment for either the ACS or the CPS, where its use
is limited to income data. Bureau officials informed us that alternative
procedures had not been examined and that stakeholders or users had not
been consulted on the adjustment.
Evaluations of Comparisons Are Incomplete and Users Lack Information on ACS
Time-Series Consistency
We noted above that one of the Census Bureau's major justifications for
the ACS test programs has been its comparing data collected in these
programs, and corresponding data from the 2000 Decennial Census short and
long forms, to identify operational problems. Another major justification
for the ACS test programs has been the use of these comparisons, and
comparisons with corresponding data from the CPS, to inform users in
making the transition from the 2000 long form to the ACS.
16For the procedures the Internal Revenue Service and the Department of
Labor use, see SOI 2000: Corporation Income Tax Returns (Washington, D.C.:
September 2003). Internal Revenue Service, Tax Statistics, Statistics of
Income, SOI Products and Services, Corporation Tax Statistics-Complete
Report Publications,
http://www.irs.gov/taxstats/article/0,,id=112834,00.html (May 10, 2004),
and U.S. Department of Labor, Pension and Welfare Benefits Administration,
Private Pension Plan Bulletin: Abstract of 1998 Form 5500 Annual Reports,
no. 11 (Washington, D.C.: winter 2001-02).
http://www.dol.gov/ebsa/PDF/1998pensionplanbulletin.pdf (May 10, 2004).
17For example, in GAO, Poverty Measurement: Adjusting for Geographic
Cost-of-Living Difference, GAO/GGD-95-64 (Washington, D.C.: Mar. 9, 1995),
we noted that experts generally agreed that it is appropriate to adjust
state-level poverty counts for cost-of-living differences but that they
differed on the most appropriate method of making such adjustments. In
Medicaid Formula: Differences in Funding Ability among States Often Are
Widened, GAO-03-620 (Washington, D.C.: July 10, 2003), we showed that
using different cost-of-living adjustments at the state level
significantly affected the amount of federal funding.
18Charles Nelson and Kathleen Short, "The Distributional Implications of
Geographic Adjustment of Poverty Thresholds," U.S. Census Bureau, Housing
and Household Economics and Statistics Division, Washington, D.C.,
December 2003.
Census Bureau Director Kenneth Prewitt emphasized the importance of
transition needs in testimony to the Congress in 2000 when he reported the
following about the ACS test programs:
"These data will also contribute to a comparison with data from Census
2000 that is
necessary because there are differences in methods and definitions between
the census
and the ACS. Moreover, decision makers will want to compare an area's data
to those from
Census 2000. Comparisons using data from the operational test and from the
31 sites are
essential to determine how much measured change between Census 2000 and
future years
of the ACS is real and how much is due to operational differences between
the ACS and the census."19
Despite acknowledging the importance of these comparisons, the Census
Bureau's publication of evaluations of the comparisons has been delayed,
and their scope has been reduced in terms of levels, data items, and time
period. The lack of information will create problems for ACS users who
will be comparing the annual ACS data for 2005 (to be released in
mid 2006) with 2000 Decennial Census data or comparing annual ACS
supplementary survey data beginning with 2000.
In addition to delaying the release of the evaluation studies, the Census
Bureau has reduced their scope. For the evaluations of ACS test site data,
local experts did not participate in the evaluation of the comparisons for
27 of the 31 test sites. For the 4 test sites that were studied by local
experts, they did not cover subcounty local government units. For
evaluations of ACS supplementary survey data, the Census Bureau has
eliminated the analyses of comparisons between (1) the 2000 supplementary
survey and the 2000 long form for geographic areas with populations of
250,000 or more and (2) the supplementary surveys for 2000-02 to
corresponding data from the CPS. It has reduced the scope of its
evaluation studies by also eliminating comparisons of single-year
19Kenneth Prewitt, Director, U.S. Bureau of the Census, before the U.S.
House of Representatives, Committee on Government Reform, Subcommittee on
the Census, Summary of House Hearing on ACS July 20, 2000, The American
Community Survey: A Replacement for the Census Long Form? Serial 106-246
(Washington, D.C.: July 20, 2000).
estimates for most subnational areas and comparisons of data items such as
financial characteristics of housing.20
NAS found that the Census Bureau has not placed sufficient priority on
completing the necessary evaluation studies.21 Furthermore, we found that
the Census Bureau does not have a plan that includes the timely completion
of all the studies. Once the studies are complete, it will need to
incorporate the findings into ACS operations, consult with stakeholders,
and provide users with the information they need to make the transition
from the long form to the ACS. The plan will be needed to ensure that as
many changes as possible can be introduced before the first annual ACS
estimates are published in 2006 and that all necessary changes are
implemented before 2008.
We found that the delays in completing the evaluations and their reduction
in scope are likely to affect the use of the ACS in improving the small
geographic area estimates of unemployment and poverty. For example, Labor
uses the unemployment data extensively to administer a variety of federal
programs. Several other departments use the poverty rates for similar
purposes.22
Users Are Not Informed on Key Properties of Multiyear Averages
One of the major differences between the ACS and the long form is that the
ACS will provide data for geographic areas with populations smaller than
65,000 in terms of multiyear averages. Experts outside and inside the
Census Bureau have identified serious issues regarding the statistical
properties of multiyear averages and have recommended that the Census
Bureau provide guidance to federal agencies and others on their use. We
found that stakeholders have urged the Census Bureau for many years to
provide guidance on the strengths and weaknesses of these averages. The
most recent request for guidance on using multiyear averages came in the
20Census Bureau officials indicated that some of the delay in completing
the planned evaluation studies may have resulted from the Census Bureau's
need to devote additional resources to completing the evaluation of the
2000 Census Accuracy, Coverage, and Evaluation program and to a survey to
test the effect of conducting the ACS as a voluntary survey.
21See Daniel L. Cork, Michael L. Cohen, and Benjamin F. King, eds.,
Planning the 2010 Census: Second Interim Report (Washington, D.C.:
National Academies Press, 2003), p. 99.
22In Statistical Policy Directive 14, OMB designated the CPS as the
official source of statistical measures of poverty. The U.S. Department of
Health and Human Services also designated the CPS as the source of poverty
measures for its programs in "Annual Update of the HHS Poverty
Guidelines," 67 Fed. Reg. 6931 (Feb. 14, 2002).
July 2003 report by the NAS Panel on Research on Future Census Methods:
"The Census Bureau should issue a user's guide that details the
statistical implications of the difference between point-in-time and
moving average estimates for various uses."23 In the report's executive
summary, the panel also stated that "The Census Bureau must do significant
work in informing data users and stakeholders of the features and the
problems of working with moving average-based estimates."24 It also
expressed particular concern about the use of the multiyear (or moving)
averages in fund allocation formulas.
Stakeholders have requested guidance on topics such as (1) the reliability
of multiyear averages for areas with rapidly changing populations, (2) the
reliability of trends calculated from annual changes in multiyear
averages, and (3) the selection of ACS data for geographic areas with
populations larger than 20,000 for which there will be multiple estimates.
The Census Bureau has recognized the need for such guidance but has not
announced any information about its contents or when it might be
available, even though the guidance is needed well in advance of the
release of the first multiyear averages in 2008.
We also found that plans for research to evaluate the statistical
properties of multiyear averages are limited. The contracts to evaluate
3-year averages for the ACS test sites cover only averages for 1999-2001,
with no comparisons with averages for 2000-02, 2001-03, or 1999-2003. In
addition, the evaluation studies discussed earlier lack any time-series
dimension, such as comparisons of the supplementary surveys with annual
data from the CPS. Thus, it appears that the Census Bureau has missed the
opportunity to test (1) distortion and stability in multiyear averages,
(2) differences between multiple estimates for the same geographic areas,
and (3) the use of annual ACS data for small geographic areas.
Meaningful External We found that in recent years, the Census Bureau has
used its outreach Consultation on Key Issues efforts with stakeholders and
users primarily to gain support for the ACS. Is Needed Although it also
has solicited advice from NAS panels, advisory committee
members, and experts at workshops and conferences on some of the
issues we have identified in this report, there is no indication that the
Census Bureau will be following this advice. (For additional information,
23Cork, Cohen, and King, pp. 99-100. 24Cork, Cohen, and King, p. 6.
Improving Timeliness and Quality of Small Geographic Area Data Would Increase
Costs
see appendix V.) Likewise, it has not yet followed similar advice from us,
other government agencies, or even its own staff.
It has been more than a year since the Census Bureau announced, in March
2003, that it was looking into establishing an ACS partnership program
that would involve advisory groups and expert panels to help it improve
the program. We found that no such program has been established yet. Given
that many key issues remain unresolved and that the Census Bureau has no
plans to seek advice on resolving them, key aspects of the ACS will
receive little or no input unless the Census Bureau revises its plans.
In 1994, the Congress began to fund testing of the survey to replace the
2000 Decennial Census long form, beginning with the 2000 Census. In
reviewing the development of the ACS, we found that the Census Bureau was
planning to replace the 2000 long form by starting the ACS program with an
annual sample of 4.8 million housing units for 1999, 2000, and 2001 and
reducing the sample for subsequent years to 3 million.25 The larger sample
would have provided 3-year averages for all small geographic areas for
2000 and would have provided data for the smallest geographic areas of the
same quality as the traditional long form. In fiscal year 1998, plans to
introduce the ACS to replace the census long form were delayed until after
the 2000 Census was completed.
When the Census Bureau submitted its plans in 1998 to replace the long
form for the 2010 Decennial Census, a similar increase in sample size for
2009-11 was not proposed. Thus, compared with the plans for 2000, data for
small geographic areas for 2010 would be delayed by a year and would be
based on 5-year averages. When we reviewed the previous plan and other
alternatives to the proposed ACS that would provide more timely and
reliable data for small geographic areas, we determined that the only
viable alternative to the current plans would be to expand the sample size
for 2009-11, as proposed earlier.
25See, for example, Charles Alexander, "A Prototype Continuous Measurement
System for the U.S. Census of Population and Housing," CM-17, presented at
the annual meeting of the Population Association of America, Miami,
Florida, May 5, 1994, and Harry A. Scarr, Deputy Director, Census Bureau,
"Continuous Measurement," Association of Public Data Users,
Fredericksburg, Virginia, October 16, 1994.
Resolving Outstanding Issues Needs a Time Schedule
This expansion would allow the Census Bureau to publish data for
geographic areas with populations smaller than 20,000 a year earlier, and
it would provide more reliable small-area data than under the currently
planned 5-year averages. In addition, if the Congress were to provide the
additional funds for this alternative, an additional year would be
available for the Census Bureau to resolve issues we have identified in
this report by giving it until the collection of data for 2009 rather than
for 2008. According to Census Bureau estimates, increasing the sample size
for the 3 years would add about $250 million to the estimated $500 million
cost for the 3 years, using the smaller sample.
The most recent Census Bureau schedule for implementing the ACS over the
complete cycle of the 2010 Decennial Census was prepared in December 2003.
Except for the completion of the questionnaire for the 2008 ACS, the
milestones do not cover the resolution of issues that it has already
identified and issues we identify in this report. (See table 1.) Ideally,
all these issues should be resolved before the first annual results of the
full ACS sample are released. However, the Census Bureau has already
announced that final plans for calculating independent population and
housing controls with ACS residence and reference concepts will not be
available for several years, the 2004 test plans for the 2010 Decennial
Census will cover group quarters and residence rules, reports from the
2004 tests will not be completed until 2005, and the 2006 test plans for
2010 also cover group quarters.
Table 1: ACS Milestone Events and Unresolved Issues, 2004-13
Fiscal
quarter ACS milestone event Unresolved issue
2004 2005
Q4 Expand the ACS sample to 250,000 (1) Residence rule to be used
addresses per
month. (2) Changes to operational procedures,
questionnaire
design, etc., based on analyses of
differences between
earlier ACS estimates and other
sources such as 2000
Census or on evaluation of 2000 Census
data
Q1 Submit proposed topics for 2008 ACS to Congress (1) Changes based on
analyses of differences between earlier ACS estimates and other sources
such as 2000 Census or on evaluation of 2000 Census data
(2) Consultation with stakeholders and users
Q4 Publish 2004 ACS single-year (1) Information on degree of
results for all states and stability of year-to-year
most areas with population changes in 2000-04 ACS based on
250,000+ comparisons with
corresponding data from CPS and other
surveys
(2) Release of dollar-denominated
data items without
adjustments for inflation and
adjustment methodology
2006
Q1 Submit actual questions for 2008 ACS to Congress Changes to questions
to reflect results of analysis of differences between ACS test data and
2000 Census longform data, evaluation of reporting in 2000 Census, and
results of 2004 Census test
Q4 Publish 2005 ACS single-year results for all geographic areas and
population groups of 65,000+
(1) Methodology for calculating independent controls for population
characteristics and housing units based on ACS definition of residence and
reference period
(2) Source of independent controls for geographic areas not covered by
ICPE
(3) Level of geographic detail to be released-for example, counties with
population of less than 65,000 or incorporated places other than counties
with population of 65,000 or more
(4) Information on consistency between 2004 and 2005 results
2007
Q1 Determine final content for the 2008 ACS (1) Changes to questions to
reflect results of analysis of differences between ACS test data and 2000
Census longform data, evaluation of reporting in 2000 Census, and results
of 2004 and 2006 Census tests
(2) Consultation with stakeholders and users
Fiscal
quarter ACS milestone event Unresolved issue
Q4 Publish 2006 ACS single-year (1) See 2006 Q4
results for all geographic
areas with population groups of (2) Guidance for users on statistical
65,000+ properties of multiyear
averages to be released in 2008 and on
use of single-year
results and multiyear accumulations for
same geographic
area
Q1 Implement content or methodology changes for 2008 (1) Consultation with
stakeholders and users ACS data collection (first year of 5-year ACS (2)
Final decisions on 2010 Census short form accumulation to replace 2010
long form)
Q4 Publish 2007 ACS single-year results for all geographic (1) See 2006 Q4
areas with population 65,000+, publish 3-year (2005-07) (2) Plans and
procedures for 3-year ACS accumulation-
accumulation for all areas with population of 20,000+ for example,
revision to independent controls for previous years
Q1 Complete 2008 ACS data collection Changes to operational procedures,
such as sampling rate for nonresponse followup
Q4 Publish 2008 ACS single-year results for all geographic Updated
guidance for users on statistical properties of areas with population
65,000+, publish 3-year (2006-08) multiyear averages to be released in
2009 and on use of accumulation for areas with population 20,000+
single-year results and multiyear accumulations for same geographic area
Q4 Publish 2009 ACS single-year results for all geographic (1) See 2008 Q4
areas with population 65,000+, publish 3-year (2007-09) (2) Incorporation
of revisions to independent controls for accumulation for areas with
population 20,000+, publish 2005-08
5-year (2005-09) accumulation for all areas
Q4 Publish 2010 ACS single-year results for all geographic (1) Incorporate
revisions to independent controls beginning areas with population 65,000+,
publish 3-year (2008-10) with 2005 for benchmarking to 2010 Census
accumulation for areas with population 20,000+, publish (2) Methodology
for April 1 reference date for independent5-year (2006-10) accumulation
for all areas controls for 2010 Census
Q4 Publish 2011 ACS single-year results for all geographic (1) See 2011 Q4
areas with population 65,000+, publish 3-year (2009-11) (2) Reconcile
differences between ACS and 2010 Census accumulation for areas with
population 20,000+, publish short form5-year (2007-11) accumulation for
all areas
2013
Q4 Publish 2012 ACS single-year results for all geographic (1) See 2010 Q4
areas with population 65,000+, publish 3-year (2010-12) (2) Level of
geographic detail from the 2010 Census to be accumulation for areas with
population 20,000+, publish used for independent controls similar to that
used for the5-year (2008-12) accumulation for all areas 2000 Decennial
Census
Source: U.S. Census Bureau and GAO analysis.
Conclusions
In addition, the Census Bureau has announced that comparisons of 2000 ACS
and 2000 Census long-form data critical to the transition to the full ACS
will be limited. Nevertheless, users who need the evaluation of these
comparisons to compare data from the 2000 Decennial Census long form with
data from the new ACS data or from the ACS supplementary surveys would
benefit from the early resolution of other issues. For example, resolving
issues before the release of the first 3-year averages (2005-07) would
improve the consistency between these averages and the subsequent ACS
data. Resolving all issues for the 2008 ACS is critical if these data are
to be fully consistent with the ACS data for 2009-12 and the 2008-12
averages are to be fully consistent with the 2010 Decennial Census
short-form data. As we noted above, the Census Bureau's schedule does call
for timely completion of the 2008 questionnaire. However, if questions to
be included in the 2010 Census short form are changed during the
congressional and OMB approval processes, currently scheduled for 2008 and
2009, data collected on the 2010 Census short form will be inconsistent
with the ACS data.
The Census Bureau's development of the ACS goes back several decades and
has included intensive research and field testing programs, as well as
substantial outreach efforts, in particular through the reports and
workshops at NAS. However, its current plan to begin full implementation
of the ACS for 2005 has several critical deficiencies. The Census Bureau
has not completed its testing program, and it has not acted to resolve key
issues already identified by the ACS test program, by evaluation studies
of the 2000 Decennial Census, by Census Bureau research studies, and by
stakeholders and users, including us, NAS, and other federal agencies.
Furthermore, the ACS implementation plan and the 2010 Decennial Census
test programs are not synchronized, and there is no comprehensive program
for external consultation on the resolution of these issues. Without
prompt resolution of issues such as those relating to the calculation of
independent controls for small geographic areas and the consistency of
data used to calculate multiyear averages, the ACS will not be an adequate
replacement for the long form in the 2010 Decennial Census. If the Census
Bureau is not able to use the ACS to replace the long form, the Congress
and other stakeholders need to be advised in 2005 in order to allow for
the Census Bureau time to reinstate the long form for the 2010 Census.26
26GAO-04-37, pp. 11-12.
Recommendations for Executive Action
Agency Comments and Our Evaluation
To ensure that the ACS is an adequate replacement for the Decennial Census
long form, we recommend that the Secretary of Commerce direct the Census
Bureau to (1) revise the ACS evaluation and testing plan and focus on the
issues we have identified in this report; (2) provide key stakeholders,
such as the National Academy of Sciences, with meaningful and timely input
on decisions relating to these issues; and (3) make public the information
underlying the Census Bureau's decisions on these issues when it makes the
decisions. We also recommend that the Secretary direct the Census Bureau
to prepare a time schedule for the 2010 Decennial Census that provides for
resolving these issues by incorporating all operational and programmatic
changes into the 2008 ACS so that the 5-year averages for 2008-12 will
adequately replace the 2010 Decennial Census long-form data for small
geographic areas. These revisions should be reflected in the single,
comprehensive project plan for the 2010 Census, as we have previously
recommended.
In written comments on a draft of this report, the Secretary of Commerce
provided comments on our recommendations. (The Secretary's comments are
reprinted in appendix VI.) He disagreed with our recommendation that the
ACS evaluation and testing plan needed to be revised to focus on issues we
have identified in this report, stating that the current ACS testing and
evaluation plan already included these issues. In following up on the
Secretary's response, we learned that there is not yet a written plan, but
only a rough outline of the types of work planned. Therefore, we believe
our recommendation remains valid. The Secretary did not accept our
recommendation to provide key stakeholders more direct and timely input
into decisions on these issues because he believes that the present
consultation process is adequate. We disagree, because as noted in
Appendix II of our report, the Census Bureau has not been responsive to
recommendations from several National Academy of Sciences reports relating
to the ACS. The Secretary agreed with the recommendation that the Census
Bureau provide public documentation for key decisions on issues we have
identified in this report. The Secretary did not respond directly to our
recommendation that he direct the Census Bureau to prepare a schedule for
the 2010 Census that ensures that all necessary changes are made in time
for the 2008 ACS so the 5-year ACS averages for 2008-2012 will be an
adequate replacement for the 2010 long form for small geographic areas.
The Secretary provided comments on the five major outstanding issues that,
in our view, jeopardize the ACS as a replacement of the long form: lack of
methodology for independent controls, operational issues not
addressed, questionable plans for dollar-denominated items, incomplete
evaluations and lack of information on ACS time-series consistency, and
lack of information about multiyear averages.
The Secretary disagreed with our findings about the lack of a methodology
for independent population and housing controls. He stated that a
methodology for the ACS was already in place. On the issue that changes to
that methodology are needed to account for the difference in the ACS
residence concept, the Secretary agreed that a change was needed but
stated that it could be delayed for several more years. On the issue of
independent controls for subcounty areas, he stated that the Census Bureau
had no plans to develop such controls, which we found were used for the
2000 Census long form, but that it might develop such controls using data
from the ACS or administrative records. However, he did not respond to our
findings about the use of existing subcounty area data from the ICPE or
from the 2010 Census short form. The Secretary stated that the Census
Bureau also had no plans to revise the ICPE. On the issue of the ACS
reference period, the Secretary reported that the Census Bureau had
recently decided to assume that July 1 would be used as the reference
period. The Secretary did not comment on GAO's findings about the lack of
plans to incorporate into the ACS 2010 Census data and related revisions
to the ICPE estimates for previous years.
We disagree with the Secretary's comments about the independent subcounty
population and housing controls and believe that their use in the ACS is
needed for the ACS to be an adequate replacement for the 2010 Census long
form for small geographical areas. We found that independent controls from
the 2000 Census short form were used for detailed geographic areas for the
2000 Census long form and that differences in counts of population and
housing (occupied and vacant) between the long form and the short form
were limited to the smallest geographic areas. The similar use of 2010
Census short-form counts in the ACS also would minimize differences in
these counts from the ACS and the 2010 Census. Consequently, we disagree
with the Bureau's plan not to commit to the development of subcounty
controls and its plans not to base these controls on ICPE total population
and housing estimates, which are prepared annually for all general
government units, and the more detailed and reliable data from the 2010
Census short form.
We also disagree with the Secretary that the implementation of a new
methodology for independent controls with subcounty controls and the new
residence concept can wait until 2008. As we noted in our report, we found
that controls for subcounty areas with population of more than
65,000 will be needed before the 2005 ACS estimates are released for these
areas in 2006 and that controls for subcounty areas with populations of
more than 20,000 will be needed before the first multiyear averages are
released in 2008. (For the 2000 Census long form, controls for most areas
of this size were from the 2000 Census short form.) With regard to the new
residence concept, a decision to delay introducing a new methodology until
2008 would create time-series inconsistencies between 2000-2007 and 2008
and subsequent years. These inconsistencies could be very significant for
geographic areas with a large population of seasonal residents.
The Secretary did not comment on our findings about the need for a
methodology to revisions relating to the ICPE into the ACS. We found that
this methodology, which is important to both the time-series consistency
of the annual ACS estimates and to the multiyear averages, is not covered
by the current ACS methodology, but that it will be needed when the 2010
Decennial Census short form data become available. We found that it has
been the Census Bureau's practice for the ICPE, whose estimates are used
as the independent controls for the ACS, to be benchmarked to the
decennial census short-form data and that it uses similar practices for
many other Census Bureau programs. For the ICPE, the Bureau will replace
the 2010 ICPE estimates with the 2010 Census data, and use the differences
in these estimates to revise the ICPE estimates back to the previous
benchmark year, which for 2010 will be 2001. (Table 4 of our report shows
the impact of benchmarking on county population data for 2000.) It should
be noted that we found that this practice is not followed in all Census
Bureau programs. For example, for the Annual Economic and Social
Supplement to the CPS, the Census Bureau introduced the benchmark
information from the 2000 Decennial Census into the 2001 estimates and
presented the data on both the old and the revised basis. This approach,
to present estimates on an old and new basis for a single year, may be
appropriate for an annual survey. However, GAO found that because of the
use of multiyear averages in the ACS, it is imperative that the ACS
estimates for all years beginning with 2001 be revised. Without such a
revision program, ACS estimates for 2010, which we assume will not be
released until the 2010 Census short-form data have been incorporated,
will be inconsistent with the 2009 estimates. In addition, the ACS
estimates for 2008 and 2009 used to calculate the 5-year averages that
will replace the 2010 Census long form will be based on controls that are
inconsistent with those for 2010-12. Based on the revisions for 2000 shown
in our report, there could be many significant inconsistencies, especially
for small geographic areas.
Although the Secretary did not comment on the issue of revision, in its
technical comments on our draft report, the Census Bureau reported
(comment 22) that with regard to incorporating 2010 Census data, it has
decided "to make appropriate changes to the [ACS] population controls when
necessary, including the possibility of reweighting the data around the
2010 time period and for all multiyear estimates." We disagree with the
Census Bureau's approach primarily because it is not consistent with the
practices used by the Census Bureau to incorporate census data into
surveys and programs such as the ICPE and monthly retail sales that are
controlled or benchmarked to a census or similar data set. For these
surveys, it revises all previously published data on a predetermined
schedule using a transparent statistical procedure. Most important, these
procedures do not depend on the size of revisions, which can only be
determined after a benchmark is completed. Regardless of the benchmarking
procedures adopted for the ACS, we believe that the Census Bureau needs to
have extensive consultation with external stakeholders to make its
decision. In addition, because of the complexity of most benchmarking
procedures, the Census Bureau needs to begin this consultation as soon as
possible.
With regard to the recent Census Bureau decision about the reference
period for the ACS, we are pleased that a decision has been made because
any delay in this decision would have resulted in additional time-series
inconsistencies in the ACS. We have changed our report to reflect this
decision. Unfortunately, we have no documentation on the research
underlying the decision and, as has been the case in other key decisions,
we do not believe that there was any public discussion of this decision.
The second issue identified in our report related to the operational
aspects of the ACS, including questionnaire design and the collection of
data for persons living in group quarters. On these issues, the Secretary
limited his comments to the questionnaires and addressed our findings that
improvements identified as part of the 2000 Census cognitive testing
research and research based on comparisons of ACS and 2000 Census
long-form data would not be completed until 2008. The Secretary noted that
the Census Bureau has resolved the issue of finalizing the ACS questions,
including the questions to be asked on the 2010 Census short form before
2008. Although this recent decision appears to have resolved the
scheduling issue, we believe that uncertainties remain as to whether this
schedule can be met. For example, the ACS milestones in the latest
available schedule call for final approval of the questions by the
Congress and by OMB in 2008 and 2009, respectively, so that any changes
made as a result of these steps would not be incorporated into the 2008
questions.
As the Census Bureau has recognized, failure to maintain consistency in
the questions for the 2008-2012 ACS will result in inconsistencies in the
5year averages centered on 2010, which are the averages designed to
provide the small geographic area data that would have been collected on
the 2010 Census long form. In addition, the recently released ACS
evaluation reports identify issues on which new research is necessary,
including the issues with the questions on disability identified in our
report, but the Census Bureau has not indicated its plan to complete this
additional research or to consult with stakeholders about decisions
related to the research. Although the Secretary did not comment on our
findings with regard to group quarters, we remain concerned that the work
on group quarters being conducted as part of the 2004, 2006, and 2008
tests for the 2010 Census will not be reflected in the ACS beginning with
2008.
Our report also identified as unresolved issues the two inflation
adjustments that the Census Bureau is applying on all dollar-denominated
ACS items. The first adjustment is used to convert annual data collected
each month in the ACS to a calendar year basis. This adjustment recognizes
that the annual data collected in the ACS are for different periods
because the data are collected monthly and cover the previous 12 months.
The second adjustment is used to present dollar-denominated items in
dollars of the most recent calendar year. This adjustment eliminates the
impact of inflation when comparing data across years. The index used for
both adjustments is the national-level CPI. The Secretary correctly
observed that the CPI is a generally accepted measure of inflation and
that most federal programs that allocate funds do not use regional
measures of inflation. However, these observations did not directly
address GAO's findings about the adjustments or the concerns raised by HUD
in its report on future use of the ACS, which are discussed in appendix V
of our report. For example, the Secretary did not address our finding
about a lack of a rationale for adjusting items other than incomes for
changes in overall inflation rather than adjusting with indexes, such as
wage rates or rent, that are directly related to the item being adjusted.
He did indicate that the Census Bureau would reconsider its present policy
of showing only the inflation-adjusted annual estimates and multiyear
averages. We believe our findings about the need for the Census Bureau to
provide a comprehensive rationale for the two adjustments still apply.
The Secretary disagreed with the issue we identified on completeness of
the Census Bureau's comparison and evaluation reports. He noted that after
our draft report was completed, the Census Bureau released seven
additional comparison reports and that it planned to prepare additional
reports to evaluate issues we identified on the time-series consistency of
the annual ACS estimates. However, despite earlier statements by the
Census Bureau to compare and evaluate differences between state-level
estimates from the Census 2000 Supplementary Survey (C2SS) and the 2000
Census long form, these reports did not include any reference to the
preparation of such comparisons, and the Secretary did not indicate they
would be prepared. Because the focus of the long form and the ACS is on
data from small geographic areas, we believe that reports on states and on
other areas with population of 250,000 or more should be prepared.
The last issue we identified was the need to provide users with guidance
on the interpretation of key properties of multiyear averages. The
Secretary agreed about the need but noted that guidance is not needed in
2005. He reported on a newly created NAS panel that will be studying many
of the key issues identified in our report. However, we believe that the
Census Bureau should begin to release guidance on the averages before the
first multiyear averages are released in 2008. One area in which such
guidance will be needed is the interpretation and use of the multiple ACS
estimates. When the 2005-07 averages are released in 2008, users will have
annual estimates for some of these areas for 2006 as well as the 3year
averages, which will be centered on 2006. In 2010, when the first 5year
averages are released (2005-09), users will have three sets of ACS
estimates for places with populations larger than 20,000. For example, for
each state, there will be an annual estimate for 2007 as well as 3-year
and 5-year averages centered on 2007.
The comments from the Secretary also include a list of detailed technical
comments from the Census Bureau. We reviewed each of these comments and
revised the report where appropriate.
As agreed with your offices, unless you release the report's contents
earlier, we plan no further distribution of it until 30 days from its
issue date. We will then send copies to the Secretary of Commerce, the
Director of the U.S. Census Bureau, and others who are interested. Copies
will be made available to others on request. This report will also be
available at no charge on GAO's Web site at http://www.gao.gov.
If you or your staff have any questions concerning this report, please
contact me at (202) 512-9750. Other staff who made major contributions to
this report are listed in appendix VII.
Robert P. Parker Chief Statistician
Appendix I: Scope and Methodology
We used a combination of approaches and methods to examine the Census
Bureau's plans to develop, test, and implement the American Community
Survey (ACS). We reviewed published and unpublished ACS-related Census
Bureau reports, papers, presentations, budget documents, and congressional
testimony; National Academy of Sciences (NAS) reports; congressional
testimony delivered by outside experts; and consultants' reports prepared
for the Census Bureau, the Bureau of Labor Statistics (BLS), and the
Department of Housing and Urban Development (HUD).
We reviewed an extensive set of internal planning documents prepared
between 1992 and 1995 that the Census Bureau provided, relevant papers
Census Bureau staff presented at professional association meetings and
similar symposiums from 1995 on, and evaluation reports based on the 2000
Census. We also reviewed official Census Bureau presentations in special
reports, congressional testimony, and recent advisory committee meetings.
We reviewed similar materials NAS and consultants prepared for the Census
Bureau and other federal agencies, as well as materials we prepared. The
most important documents we reviewed are listed in the bibliography,
organized by document type, at the end of this report.
In addition, we conducted independent research and analysis. To assess the
evaluations the Census Bureau conducted to assist users in making the
transition from the 2000 Census long form to the ACS, we obtained data
from the 2000 Census and 2000 ACS (the Census 2000 Supplementary Survey)
and prepared comparisons of key detailed data items at the state level. To
determine the potential effect of replacing independent population and
housing characteristics controls from the 2000 Census with corresponding
data from the 2010 Census, we compared county-level intercensal estimates
for April 1, 2000, based on the 1990 Census, with 2000 Census counts. We
also analyzed the Census Bureau's use of independent controls for
estimates of population and housing characteristics for previous decennial
censuses and its plans for the ACS.
To assess alternatives to the ACS, we spoke to current Census Bureau
officials and individuals familiar with early efforts to provide more
frequent long-form type data, including the Mid-Decade Census. We also
reviewed congressional hearings on these developments and Census Bureau
documents prepared in the 1990s on the Continuous Measurement program and
on implementing the ACS to replace the 2000 Census long form. We did not
independently verify the cost information the Census Bureau provided for
the alternative we discuss. We also interviewed staff of the NAS Committee
on National Statistics and outside small-area data experts. The outside
experts we interviewed were
Appendix I: Scope and Methodology
Constance Citro, Committee on National Statistics
Michael Cohen, Committee on National Statistics
Linda Gage, California State Department of Finance
Edwin Goldfield, Committee on National Statistics
Ken Hodges, Claritas Inc.
Graham Kalton, Westat Inc.
Terri Ann Lowenthal, Consultant
Joseph Salvo, New York City Planning Department
Edward J. Spar, Council of Professional Associations for Federal
Statistics
Paul Voss, University of Wisconsin
Appendix II: Recent NAS Findings on Continuous Measurement and the ACS
In Modernizing the U.S. Census, a 1995 report, the NAS Panel on Census
Requirements in the Year 2000 and Beyond stated that
"Although we believe that the proposed continuous measurement system
deserves serious
evaluation, we conclude that much work remains to develop credible
estimates of its net
costs and to answer many other fundamental questions about data quality,
the use of small
area estimates based on cumulated data, how continuous measurement could
be integrated
with existing household surveys, and its advantages compared with other
means of
providing more frequent small-area estimates. In our judgment, it will not
be possible to
complete this work in time to consider the use of continuous measurement
in place of the long form for the 2000 census."1
The panel concluded that
"With regard to proposals to drop the long form in the next decennial
census and substitute
a continuous monthly survey to obtain relevant data, substantial further
research and
preparatory work are required to thoroughly evaluate the likely effect and
costs of these
proposals. Continuous measurement deserves serious consideration as a
means of
providing more frequent small-area data; however, the necessary research
and evaluation cannot be completed in time for the 2000 census." 2
Although 1994 saw the first proposals to implement the continuous
measurement methodology as a replacement for the 2000 Census long form,
the Census Bureau changed its plans in 1998, shifting to implementation to
replace the long form in 2010.
Since 1995, NAS has produced several reports that relate totally or in
part to the ACS, including a summary of a September 13, 1998, Committee on
National Statistics workshop at NAS, two interim reports, a letter report,
and a final report, by the Panel on Research on Future Census Methods, and
a report released in early 2004 by the Panel to Review the 2000
1See Barry Edmonston and Charles Schultze, eds., Modernizing the U.S.
Census (Washington, D.C.: National Academies Press, 1995), p. 9. The panel
was mandated by Public Law 102-125 and funded by the Census Bureau.
2Edmonston and Schultze, p. 3.
Appendix II: Recent NAS Findings on Continuous Measurement and the ACS
Census.3 (In this appendix, we do not discuss NAS reports after 1995 in
which the ACS was discussed as a potential data source for federal
programs.4)
With few exceptions, the members of these two NAS panels and the workshop
participants reported findings that cover most of what we have identified
as unresolved issues and summarize in this appendix.5 The NAS reports and
ours differ somewhat in emphasis. We have focused on the production and
use of ACS data, whereas NAS focused more on data collection and
processing methodologies. These differences may reflect the fact that NAS
panel members are very sophisticated users who are more likely to use ACS
microdata files and make their own adjustments for methodological issues;
they make little use of the regular ACS publications.
1998 NAS Workshop NAS sponsored a 1-day workshop in September 1998 to
discuss methodological issues related to the ACS. Experts prepared
"thought
on the ACS pieces" on issues NAS staff selected, with input from Census
Bureau staff. The workshop's specific discussion topics were combinations
of information across areas and across time, funding formula, weighting
and imputation, sample and questionnaire design, and calibration of the
output from this survey with that from the long form. The thought pieces
and
3See National Research Council, Committee on National Statistics, The
American Community Survey: Summary of a Workshop (Washington, D.C.:
National Academy Press, 2001); Michael L. Cohen and Benjamin F. King,
eds., Designing the 2010 Census: First Interim Report (Washington, D.C.:
National Academy Press, 2000); Benjamin F. King, National Academy of
Sciences, to William Barron, U.S. Bureau of the Census, February 15, 2001;
Daniel L. Cork, Michael L. Cohen, and Benjamin F. King, eds., Planning the
2010 Census: Second Interim Report (Washington, D.C.: National Academies
Press, 2003); Daniel L. Cork, Michael L. Cohen, and Benjamin F. King,
eds., Reengineering the 2010 Census: Risks and Challenges (Washington,
D.C.: National Academies Press, 2004); and Constance F. Citro, Daniel L.
Cork, and Janet L. Norwood, eds., The 2000 Census: Counting under
Adversity (Washington, D.C.: National Academies Press, 2004).
4See, for example, Constance F. Citro and Graham Kalton, eds., Small-Area
Income and Poverty Estimates: Priorities for 2000 and Beyond (Washington,
D.C.: National Academy Press, 2000), and Thomas B. Jabine, Thomas A.
Louis, and Allen L. Schirm, eds., Choosing the Right Formula: Initial
Report (Washington, D.C.: National Academy Press, 2001).
5The exceptions are the Census Bureau's inflation adjustments for
dollar-denominated data items and the specific use of 2010 Census
population and housing controls.
Appendix II: Recent NAS Findings on Continuous Measurement and the ACS
comments on them prepared Census Bureau staff for the discussions at the
workshop.6
NAS noted in the report on the conference that its six focus issues
reflected only a partial list of key ACS topics; the report's conclusions
identified other key issues.7 Stating that the workshop's purpose was "to
assist the Census Bureau in developing a research agenda to address these
and other methodological issues," the report pointed out that the Census
Bureau's past focus on the ACS
"has been on refining data collection, leaving the final answers to the
difficult analysis
questions for later. Thus, procedures for nonresponse and undercoverage
adjustment were
modeled, to the extent possible, after current procedures used for the
census long form.
Now that data collection has matured as the ACS demonstration phase is
well under way,
the Census Bureau is developing a research plan and initiating research to
address all
issues relating to ACS methodology. Fall 1998 therefore seemed an
opportune moment for
a workshop to assist the Census Bureau in developing a research agenda to
deal with many of these challenging issues." 8
The report contained no specific recommendations but identified areas
where additional research was needed, including issues we have expressed
concern about, such as the availability of multiple ACS estimates for
geographic areas with populations larger than 20,000 and the likelihood of
differences between ACS estimates and estimates from a Decennial Census
short form. From our perspective, the most relevant of the workshop's
specific issues were (1) combining information across time, (2) weighting
and imputation, and (3) calibrating the output from this survey with that
from the long form.
Technical papers in the workshop's agenda book contained considerable
discussion of time-series issues. The discussion in this section of the
workshop focused on replacing moving averages with time-series modeling
and using current household survey data to develop models. Speaking for
the Census Bureau, Alexander stated that "Our current plan
6Charles Alexander, of the Census Bureau's Demographic Statistical Methods
Division, who had directed most of its research on the ACS, prepared
comments for the Census Bureau. See National Research Council, Committee
on National Statistics, The American Community Survey: Summary of a
Workshop, p. 5.
7National Research Council, pp. 48-49.
8National Research Council, pp. 1 and 3.
Appendix II: Recent NAS Findings on Continuous Measurement and the ACS
is to release annual data for even very small areas and let users perform
their own time series analyses. We welcome ideas about what the Bureau's
role should be . . ."9
On the evaluation of comparisons between the ACS test data, the workshop
report noted that the objective of the comparison using the national
sample data was
"to make comparisons between the long form and ACS for all states, large
metropolitan areas, large substate areas, and population groups.
"The objective of the 1999-2001 comparison is to understand the factors
associated with
the differences between the 1999-2001 ACS and the 2000 long form in the 31
areas, using
the second comparison study to develop a calibration model to adjust the
2000 long-form estimates to roughly represent what the full ACS would have
yielded in 2000."10
Chapter 7 of the report was devoted to a discussion of calibration. The
report stated that the model would "determine the effects that would be
expected when switching from the long-form estimates to those from the ACS
on various applications of long-form data." Once adjusted, the calibrated
long-form data for 2000 can be compared with ACS data that are collected
following full field implementation in 2003, "in order to understand the
dynamics over time of such characteristics as poverty and employment."11
The technical papers in the workshop's agenda book also noted other
comparisons of ACS data. For example, Alexander discussed comparisons with
CPS data, reporting that
"We very much like the idea of viewing information from an ongoing
comparison of ACS to CPS and other surveys as a way to help understand how
the ACS `error profile' might be
9Charles Alexander, Technical Paper, prepared for Workshop on the American
Community Survey, Committee on National Statistics, National Academy of
Sciences, Washington, D.C., September 1998, p. 3-2.
10National Research Council, p. 4.
11National Research Council, pp. 4 and 5. The Census Bureau provided
funding for the development of this model in fiscal year 1999, but there
is no report that the model was completed.
Appendix II: Recent NAS Findings on Continuous Measurement and the ACS
changing over time and using this to help interpret ACS data in the
context of the long-term time series of census estimates."12
The use of independent controls for population and housing characteristics
was also discussed at the workshop, but very generally, because the Census
Bureau had not yet developed proposals for the controls. For example, the
report's chapter 5 discussed improving the existing population controls.
The Census Bureau reported discomfort with the quality of the existing
county-level controls (from ICPE) and agreed that the ACS could be used to
improve these estimates.13 The Census Bureau also acknowledged that
differences in residence rules and reference period would complicate the
calculation of population weights. However, no discussion was reported of
how the population counts from the 2010 Census would be used.
The report referred to moving averages in the conclusions chapter as one
of the methodological problems noted at the workshop:
"the development of estimates that (a) sum to estimates at higher levels
of geographic
aggregation and (b) more closely approximate direct estimates at higher
levels of
aggregation . . . in the event that aggregate estimates are not
constrained to
(approximately) equal direct estimates (and also the release of direct
estimates at lower levels of aggregation for analysis purposes) . . . ."14
2000 Interim Report
The Panel on Research on Future Census Methods, sponsored by the Census
Bureau, was formed to examine alternative designs for the 2010 Census and
to assist the Census Bureau in planning tests and analyses to help assess
and compare their advantages and disadvantages. In addition to the first
interim report, Designing the 2010 Census, released in 2000, a letter
report was issued in 2001, and a second interim report was issued in 2003
(both discussed below). The panel issued a final report in 2004.
The panel's first interim report identified information from 2000 Census
data useful in assessing designs for the 2010 Census. In the executive
12Alexander, Technical Paper, pp. 3-7. 13National Research Council, p. 26.
14National Research Council, p. 48.
Appendix II: Recent NAS Findings on Continuous Measurement and the ACS
summary, the panel made four specific recommendations and proposed other
changes. One of the recommendations, relating to evaluation studies, is
directly relevant to our report:
"The Census Bureau should develop a detailed plan for each evaluation
study on how to
analyze the data collected and how to use the results in decision making
concerning 2010
census design. The Census Bureau should then use these plans to identify
the benefits and
resources required for each evaluation study, set priorities among them,
and allocate
sufficient resources for the careful completion of all or, at least, the
highest priority evaluations."15
In addition, the report proposed three changes for the 2010 Census and
ACS. The first proposed change to the direction and nature of the
evaluation program, was that the Census Bureau use the "ACS as a census
testing platform":
"The American Community Survey is a proposed national, continuous,
mailout-mailback
survey of 250,000 households per month, with field follow-up that makes
use of techniques
closely related to those used in the census. Therefore, rather than rely
exclusively on the
two or three large-scale census tests, which are always at least slightly
limited in their
generalizability by the specific locations selected, the Census Bureau
could use the ACS as
a platform for testing possible changes in the census. This work could
serve as preliminary testing to the larger mid-decade tests for the census
design."16
The second proposed change called for "a match study of the census short
form and the ACS." This proposal, which could provide information on the
effect of a change in the residence rule in the 2010 census, stated:
"The decennial census makes use of one residence rule definition, the ACS
uses a second, and a third approach is being tested in the alternative
questionnaire study. As the Census Bureau is well aware (based on the
allocation of an experiment to this issue), confusion over residence rules
is a source of possibly substantial error in the census. . . . The Census
Bureau needs to find the residence rule (within the set of rules
satisfying legal and other restrictions) that results in the most accurate
estimates. To learn more about this issue, the
15Michael L. Cohen and Benjamin F. King, eds., Designing the 2010 Census:
First Interim Report (Washington, D.C.: National Academy Press, 2000), p.
2. The three other recommendations covered the master trace sample
database, the 2000 Census administrative records research program, and the
activities of local organizations that helped with the census count.
16Cohen and King, p. 34.
Appendix II: Recent NAS Findings on Continuous Measurement and the ACS
2001 Letter Report
panel proposes an ACS-short-form match study in 2000 to examine this and
other shortform measurement error issues."17
The third proposed change was the recommendation that the Census Bureau
"form an ACS advisory group" to improve its efforts to consult with
stakeholders. The panel stated:
"The development of the ACS raises a number of issues related to the
quality of and
planning for the 2010 census. There are also many other important
technical issues raised
by the introduction of the ACS into the federal statistical system.
Formation of a technical working group could help to address many of these
issues."18
The 2001 letter report-addressed from Benjamin King, Chair of the Panel on
Research on Future Census Methods, to William Barron Jr., Acting Director
of the Census Bureau-was prepared in response to a December 7, 2000,
presentation by Census Bureau staff on the major elements of the Census
Bureau's strategy for the 2010 Census. The panel recommended that the
Census Bureau produce a "business plan" for the 2010 Census that would
provide an overall framework for development. It recommended that this
plan include (1) a statement of objectives, (2) a timeline for completing
tasks, (3) a cost-benefit analysis, and (4) more complete information on
coordinating tasks within the Census Bureau.19 The panel also recommended
the preparation of specific types of evaluation studies.
On the evaluation studies, the panel reported,
"The Bureau is currently conducting a wide array of evaluation studies and
experiments
designed to assess the quality of the 2000 census and inform approaches to
the 2010
census. As noted above, the panel applauds the scope of these evaluation
studies. However,
the panel is concerned that the Bureau has not sufficiently focused its
evaluation program and has instead labeled most of its evaluation
categories as high priority."20
17Cohen and King, pp. 34-35.
18Cohen and King, eds., p. 38. The Census Bureau has established the
Federal Agency Information Program on the ACS in response to a
recommendation we made in GAO-02-956R, p. 25. Information is at the Census
Bureau's Web site at http://www.census.gov/acs/www/.
19These recommendations were similar to recommendations we made in
GAO-04-37, pp. 33-34.
20King, to Barron, p. 3.
Appendix II: Recent NAS Findings on Continuous Measurement and the ACS
In the letter report's conclusions, the panel recommended that the Census
Bureau give the highest priority to studies and data analysis in seven
specific areas, most of which related to the ACS. The panel's list of
studies and analyses included the following recommendations, which we have
also discussed:
"comparison of estimates from the ACS and 2000 census long-form data, in
sites where
both are available; coverage of the population, disaggregated by
demographic and
geographic subgroups; the effectiveness of major automated systems for
data collection,
capture, and processing; the quality and completeness of long-form data
collection; and the
effectiveness of operations used to designate special places and enumerate
the group quarters and homeless populations."21
In making these recommendations, the panel noted the need for the Census
Bureau to maintain a strategy that would provide for "a smooth transition"
from the long form to the ACS. The panel urged the Census Bureau
"to broaden its justification for the ACS, detailing the need for and use
of long-form data
and how those data needs will be addressed through the ACS, perhaps in
conjunction with
the CPS and other demographic surveys. Accordingly, the Bureau should
expedite ongoing
evaluations that assess the quality of ACS data relative to the quality
associated with the traditional census long form."22
In the second interim report, Planning the 2010 Census, issued in 2003,
the panel identified four areas of primary interest: reengineering the
census, geographic coding, the ACS, and testing for the 2010 Census. With
regard to the ACS, the panel reported in the executive summary that
"The most basic question the panel faces regarding the ACS is whether it
is a satisfactory
replacement for the census long form. We recognize that significant
estimation and
weighting challenges must be addressed and that more research is needed on
the relative quality of ACS and long-form estimates."23
21King, to Barron, p. 4. The other studies related to the 2000 Census
mailing list and the effect of local partnerships on the 2000 Census
collection process.
22King, to Barron, p. 2.
23Cork, Cohen, and King, p. 6.
2003 Interim Report
Appendix II: Recent NAS Findings on Continuous Measurement and the ACS
The panel found that the Census Bureau needed to complete evaluations of
differences between 2000 Census long-form data and data from the ACS test
sites and from the 2000-02 supplementary surveys. It also found that the
Census Bureau needed to undertake a major effort to inform data users and
stakeholders of the results of these evaluations and the features and
problems of working with multiyear averages. One of the four main topics
of this report was a separate chapter on the ACS, in which the panel
discussed the following recommendations:
"The Census Bureau should carry out more research to understand the
differences between and relative quality of ACS estimates and long-form
estimates, with particular attention to measurement error and error from
nonresponse and imputation. The Census Bureau must work on ways to
effectively communicate and articulate those findings to interested
stakeholders, particularly potential end users of the data.
The Census Bureau should make ACS data available (protecting
confidentiality) to analysts in the 31 ACS test sites to facilitate the
comparison of ACS and census long-form estimates as a means of assessing
the quality of ACS data as a replacement for census long-form data. Again,
with appropriate safeguards, the Census Bureau should release ACS data to
the broader research community for evaluation purposes.
The Census Bureau should issue a user's guide that details the statistical
implications of the difference between point-in-time and moving average
estimates for various uses.
The Census Bureau should identify the costs and benefits of various
approaches to
collecting characteristics information should support for the full ACS not
be forthcoming.
These costs and benefits should be presented for review so that decisions
on the ACS and its alternatives can be fully informed."24
With regard to the first recommendation, the panel stated that
"The fact that the Census Bureau has not done more in comparing the data
collected from the 31 test sites, the C2SS, and the 2001 and 2002
Supplementary Surveys with the data collected by the 2000 census long form
is disappointing. Such analyses would help assess the quality of ACS data
and would be helpful in making the argument for transition from the long
form to the ACS. This deficiency is probably due to limited analytic
resources at
24Cork, Cohen, and King, pp. 99-102. As we noted above, we discussed
issues related to planning for the 2010 Census in GAO-04-37.
Appendix II: Recent NAS Findings on Continuous Measurement and the ACS
the Census Bureau and creates an argument for `farming out' this analysis
to outside researchers."25
On the recommendation about the need for more information on multiyear or
moving averages, the panel discussed several technical issues. The panel
commented that
"The ramifications of this basic concept emerge when moving average
estimates are
entered into sensitive allocation formulas or compared against strict
eligibility cutoffs. A
smoothed estimate may mask or smooth over an individual year drop in level
of need, thus
keeping the locality eligible for benefits; conversely, it may also mask
individual-year
spikes in activity and thus disqualify an area from benefits. It is clear
that the use of
smoothed estimates is neither uniformly advantageous nor disadvantageous
to a locality; what is not clear is how often major discrepancies may
occur in practice."26
On using moving-average data to measure year-to-year changes, the panel
commented:
"It is incorrect to use annual estimates based on moving averages over
several years when
assessing change since some of the data are from overlapping time periods
and hence
identical. At the least, the results will yield incorrect estimates of the
variance of the
estimates of change. Therefore, users should be cautioned about this
aspect of the use of moving averages."27
In both recommendations on evaluations and moving averages, the panel
called for the Census Bureau to engage in a greatly expanded effort to
inform users and stakeholders. It also suggested that the Census Bureau
farm out some of the research efforts.
In summarizing the results of its efforts, the panel noted the 1995 NAS
report, as follows:
"Eight years later, faced with the task of offering advice on making the
vision of continuous measurement a reality in the 2010 census, the
similarity between the arguments then and now is uncanny. Similar, too,
are the points of concern; the current panel is hard-pressed to improve
upon the basic summary of concerns outlined by our predecessors. We are,
25Cork, Cohen, and King, p. 99. 26Cork, Cohen, and King, p. 86. 27Cork,
Cohen, and King, p. 87.
Appendix II: Recent NAS Findings on Continuous Measurement and the ACS
The 2000 Census: Counting under Adversity
however, much more sanguine that a compelling case can be made for the ACS
and that it is a viable long-form replacement in the 2010 census."28
However, while the panel was identifying its concerns, it also supported
full funding of the ACS, believing that existing "flaws" in the plan could
be resolved.
In 2004, the Panel to Review the 2000 Census, sponsored by the Census
Bureau, issued a report entitled The 2000 Census: Counting under
Adversity. The findings were based primarily on the panel's review of
information from the 2000 Census. The panel's charge had been to "review
the statistical methods of the 2000 Census, particularly the use of the
Accuracy and Coverage Evaluation Program and dual-systems estimation, and
other census procedures that may affect the completeness and quality of
the data."29 Thus, although the report focused on the 2000 Census, it made
some recommendations for improving the 2010 Census. Its major
recommendation was that
"the Census Bureau, the administration, and Congress agree on the basic
design for the
2010 census no later than 2006 in order to permit an appropriate,
well-planned dress
rehearsal in 2008. In particular, this agreement should specify the role
of the new American
Community Survey (ACS). Further delay will undercut the ability of the ACS
to provide, by
2010, small-area data of the type traditionally collected on the census
long-form sample and will jeopardize 2010 planning, which currently
assumes a short-form-only census."30
In its discussion of the 2010 Census, the report included several
recommendations on ACS operations and evaluations. The panel recommended
that the Census Bureau develop estimates of the effect on estimates from
the 2000 long form resulting from imputation as well as sampling
variability and nonresponse. More specifically,
"the Bureau should also study the effects of imputation on the
distributions of characteristics and the relationships among them and
conduct research on improved
28Cork, Cohen, and King, p. 98.
29Citro, Cork, and Norwood, eds., The 2000 Census: Counting under
Adversity, p. 1.
30Citro, Cork, and Norwood, pp. 2-3.
Appendix II: Recent NAS Findings on Continuous Measurement and the ACS
imputation methods for use in the American Community Survey (or the 2010
census if it includes a long-form sample)."31
Finally, the panel recommended that the Census Bureau's plans for the 2010
Census "include research on the trade-offs in costs and accuracy between
imputation and additional fieldwork for missing data (Recommendation
4.2)."32 The panel also recommended that the Census Bureau
"publish distributions of characteristics and item imputation rates, for
the 2010 census and
the American Community Survey (when it includes group quarters residents),
that
distinguish household residents from the group quarters population (at
least the
institutionalized component). Such separation would make it easier for
data users to
compare census and ACS estimates with household surveys and would
facilitate
comparative assessments of data quality for these two populations by the
Census Bureau and others."33
The panel's findings were similar to our findings, with one major
difference. The panel's findings imply that some research on the ACS can
be conducted after the results of the 2010 Census short form become
available. In contrast, we see that such research is needed in order to
improve the ACS by 2008, the first year in which ACS data will enter into
the calculation of the 5-year average estimates (2008-12) that will
replace the long form.
31Citro, Cork, and Norwood, pp. 10-11.
32Citro, Cork, and Norwood, p. 11.
33Citro, Cork, and Norwood, p. 301. See recommendations 7.1 and 7.3.
Appendix III: The Decennial Census Long Form and the Evolution of the ACS Plan
Decennial Census In the decennial census for 1940 and for 1950, the
Census Bureau used a single form to collect, from all households,
population and key
Long Form characteristics such as age and gender and, from a sample of
households, detailed demographic, economic, and housing items. In the 1940
Census, the Census Bureau used a sample of 5 percent of the population to
collect data on questions on income, internal migration, and Social
Security status, as well as on more refined questions on unemployment. In
addition, the Congress authorized a new set of questions about the types
of plumbing, heating, and appliances in dwellings.
Beginning with the 1960 Census, the first conducted by mail, it became
necessary to use separate forms-a short form to collect population data
from all households and a long form to collect the detailed items from a
sample of households. In the 2000 Census, for example, the Census Bureau
conducted a sample of 17 percent of the population and asked 45 questions
on the long form. Since 1960, the long form has evolved into a
cost-efficient way to collect data federal agencies need that minimizes
respondent burden. For 2000, for example, the long form consisted of 45
questions that the Census Bureau developed working through OMB and with
the consent of the Congress.1 Each question provided information required
by statute. Thus, the 2000 long form provided all federal departments and
agencies with critical data, and it was estimated that these data were
used to allocate more than $200 billion in federal funds.2
Evolution of ACS Plan In the 1950s, Census Bureau officials and users of
Decennial Census data had begun to develop a program to provide
intercensal data on population characteristics. The first major proposal
to provide intercensal data called for a mid-decade census that would
provide information every 5 years. In 1976, the Congress enacted
legislation to require a mid-decade census beginning with 1985, but did
not fully fund the program.
In the late 1980s, the Census Bureau shifted efforts to provide
intercensal estimates to a program based on CM methodology, or Continuous
1The long form is also discussed in Joseph Salvo and Arun Peter Lobo, "The
American Community Survey: Quality of Response by Mode of Data Collection
in the Bronx Test Site," presented at 2002 Joint Statistical Meetings, New
York, August 14, 2002; Margo Anderson, ed., Encyclopedia of the U.S.
Census (Washington, D.C.: CQ Press, 2000); and GAO, Decennial Census:
Overview of Historical Census Issues, GAO/GGD-98-103 (Washington, D.C.:
May 1, 1998).
2See GAO-02-956R.
Appendix III: The Decennial Census Long Form and the Evolution of the ACS
Plan
Measurement. This approach would provide for more timely population data
as well as the detailed demographic, economic, and housing data collected
every 10 years by the Decennial Census long form.3 The program would
integrate a new sample survey, existing surveys, administrative records,
and statistical modeling. After a thorough analysis of alternatives based
on this methodology, the Census Bureau developed a plan similar to the
current ACS to replace the 2000 Census long form.
Initial $2.6 million funding for the CM program was included in the 2000
Decennial Census budget for fiscal year 1995. These funds were to develop,
test, and evaluate a CM program to replace the Decennial Census long form
and to provide more timely long-form type data. In the program description
in the budget documents, the Census Bureau reported that it planned to
develop the new program that would integrate a new sample survey, existing
surveys, administrative records, and statistical modeling. Table 2 shows
that about $330 million has been provided to fund the CM program since
1995, with funding provided separately until 2003 and additional funding
from both the 2000 and 2010 Decennial Census programs. Beginning with
2003, all funding has been provided as part of the 2010 Census program.
The Census Bureau requested $165 million for fiscal year 2005.
3Data items similar or identical to those collected on the long form are
collected by the Annual Demographic Survey (CPS's March supplement),
Annual Housing Survey, and other surveys. However, these surveys' samples
limit the data they provide to the national level and selected states and
metropolitan areas. Annual data for all small geographic areas are
available from (1) administrative records on unemployment insurance and
wages and federal income tax and Medicare records; (2) statistical series
the Bureau of Economic Analysis (BEA) prepares, such as local area
personal income data; and (3) model-based series such as the Census
Bureau's Small Area Income and Poverty Estimates program.
Appendix III: The Decennial Census Long Form and the Evolution of the ACS
Plan
Table 2: Continuous Measurement and ACS Funding, Fiscal Years 1995-2005
Fiscal year Budgeta Activity Other information
1995 $2.6 Develop and test a continuous measurement (CM) system to None
replace 2000 Census long form; study integration of
administrative records, existing current surveys, and
statistical
modeling with a new survey for CM programs
1996 10.0 Develop and test a CM system to Continue evaluating and
replace 2000 Census long developing an
form and provide annual data integrated CM program
1997 16.6 Rename test survey American Develop methods for
Community Survey (ACS); integrating
complete data collection and administrative records and
processing for 1996 survey at information from
4 test sites; develop list to household survey data into CM
cover group quarters program
1998 16.6 Continue testing and processing Make long-run plans for
1997 test site survey data replacing 2010 long
form with full implementation
of ACS in
2003
1999 20.0 Publish 1998 test site data and Develop statistical models to
expand test site surveys to evaluate
31 sites; prepare for comparisons of long form to
supplementary surveys from supplementary
national
sample to begin with 2000 to surveys; continue testing to
compare with 2000 Census long- integrate
information from
form data administrative records
2000 47.0 Provide for 2000 supplementary survey; continue comparison
studies of differences for test site areas; develop plans for comparison
studies with supplementary survey and using multiyear averages
None 2001 45.2 Provide for continuation of supplementary surveys; continue
Continue processing data from test sites for comparison studies
testing and comparison studies with data from test sites and national
survey; begin testing in Puerto Rico
2002 56.1 Prepare for full implementation in Continue evaluating
2003 comparison studies for
test-site and supplementary
survey data
2003 57.1 Prepare to begin full Complete testing for group
implementation for 2005 quarters and
Puerto Rico
2004 64.8 Prepare to begin full None
implementation for 2005
2005 165.0 Full implementation None
Sources: Budget of the United States, House of Representatives Report
108-401, and Census Bureau budget documents.
aDollars in millions. Fiscal year 1995 funding provided as part of the
2000 Decennial Census program. Fiscal year 1996-99 funding provided by CM
program. Fiscal year 2000 and 2001 funding provided as both the CM program
and part of the 2000 Census program. Fiscal year 2002 funding provided as
both the CM program and part of the 2010 Census program. Beginning with
fiscal year 2003, funding provided as part of the 2010 Census program.
Fiscal year 2005 figure is the budget request.
In 1996 and 1997, funding was provided to field-test what became the ACS,
to replace the 2000 Census long form. The ACS was to begin in 1999 with an
annual sample of 4.8 million housing units for 1999, 2000, and 2001 and 3
million housing units for subsequent years. Under this plan, a 3-year
Appendix III: The Decennial Census Long Form and the Evolution of the ACS
Plan
average of ACS data for 1999-2001 was to replace the 2000 Census long
form.4 It would provide the same detailed items and same level of
geographic detail as the traditional long form with about the same
quality. Annual ACS data would subsequently be provided for geographic
areas with populations of 65,000 or more, 3-year averages would provide
ACS data for geographic areas with populations larger than 20,000, and
5-year averages would provide ACS data for small geographic areas, such as
census tracts, small towns, and rural areas. The 5-year average for 2010,
2020, and beyond would replace future Decennial Census long forms.
In the 1998 budget request, the Census Bureau shifted the timing for
replacing the long form from the 2000 Census to the 2010 Census. As a
result, it was funded to conduct annual supplementary surveys of 750,000
households beginning with 2000, in addition to the ACS testing at four
test sites (or counties). The Census 2000 Supplementary Survey, known as
C2SS, and the surveys for subsequent years were to be used to test the
feasibility of collecting long-form data at the same time as, but in a
separate process from, the Decennial Census. Data from C2SS and the
supplementary surveys were also to be used to test ACS data usability and
reliability and to evaluate operational and programmatic issues associated
with implementing the ACS. Also, the number of test sites was increased to
31 by 1999. Funding to compare and evaluate differences between data
collected from the 2000 Census long form and the ACS testing programs
began in 1999, to develop data to expand coverage to group quarters and
Puerto Rico in 2001. Plans to integrate existing surveys, administrative
records, and statistical modeling into the new program were dropped in
2001.
The 1998 budget request also reported that the Census Bureau would proceed
with plans to replace the 2010 Census long form with an ACS based on an
annual sample of 3 million housing units, as with the previous plan.
Unlike that plan, the sample size for 2009-11 would not increase to
provide 3-year averages for 2010. This revised plan called for full
implementation of the ACS in 2003. Full ACS data for 2003 to 2007 would
have made 5-year averages available in 2008, 4 years before the long-form
sample statistics from the 2010 Census would become available. However,
4Harry A. Scarr presented this proposal as Deputy Director of the Census
Bureau at an Association of Public Data Users conference in
Fredericksburg, Virginia, on October 16, 1994.
Appendix III: The Decennial Census Long Form and the Evolution of the ACS
Plan
budget decisions by the Congress delayed full implementation until the
fourth quarter of fiscal year 2004.
Appendix IV: Continuous Measurement ACS Testing and Development Program
The Congress initially provided funds for testing the CM methodology in
1994. As we have noted, the Census Bureau had begun formal testing of the
CM program in 1996 with an operational test of the ACS in four counties;
this test was expanded to 31 test sites by 1999. A second testing program,
the Supplementary Survey program, began in 2000 as a part of the 2000
Decennial Census. The Census Bureau designed C2SS to test the feasibility
of collecting long-form data at the same time as, but in a separate
process from, the 2000 Decennial Census. Data from C2SS and the same
supplementary surveys, beginning with 2001, were also to be used to test
ACS data usability and reliability. According to the Census Bureau, these
surveys were to be used to examine technical, statistical, and operational
issues associated with implementing the ACS and to document the key
results in a series of reports.1
Before field testing began, the Census Bureau had conducted an extensive
research program to identify the issues related to using the CM
methodology and to replacing the long form. The research program resulted
in a series of 20 reports, known as the Continuous Measurement Series,
between 1992 and 1995.2 These reports, most of which were prepared by
Charles Alexander, addressed a wide range of topics such as replacing the
2000 Census long form, collecting intercensal population data, and
integrating the ACS with existing household surveys. The reports on
replacing the long form identified the key issues that needed testing, and
they served as the primary input to the Census Bureau's ACS test program.
These issues included those subsequently tested by the Census Bureau as
well as the unresolved issues we identify in this report.
Following the CM reports, Census Bureau staff presented papers from 1995
through 2001 on ACS testing at various professional association and
similar meetings, as well as at a 1998 symposium on the ACS sponsored by
the Census Bureau.3 For example, the 1995 paper by Love, Dalzell, and
Alexander discussed issues related to the evaluation of the 1996 test site
1The Census Bureau also prepared a number of internal papers that
evaluated the results of the 2000 Census long form and recommended changes
that applied to the ACS. We discuss some of these papers in this report.
2Most of the papers in this series are not available to the public (a few
are on the Census Bureau's ACS Web site at
http://www.census.gov/acs/www/), but the Census Bureau provided us with a
complete set.
3Some of the papers from this March 25, 1998, symposium are available on
the Census Bureau's ACS Web site; several presented information from the
1996 ACS testing.
Appendix IV: Continuous Measurement ACS Testing and Development Program
results, expressing concern about population controls and residence rules
as well as the need for consultation with users.4 They reported that the
Census Bureau was planning to conduct research using data from the 1996
test sites to produce controls at the census tract and block group level.
They also noted that the Census Bureau would need to conduct research on
the residence rule.
Alexander and Wetrogan also discussed the issue of population controls in
their 2000 paper.5 They reviewed possible methods for using ICPE to
develop controls for the ACS and discussed using ACS estimates on the
foreign-born U.S. population to improve the Census Bureau's
foreignmigration component of the intercensal estimates. (They reported
that this effort would be part of what the Census Bureau had previously
referred to as the Program of Integrated Estimates.) They also noted the
need to consult with users on how to present information on the
differences in ACS controls and ICPE in ACS publications.
Several papers have focused on the key role of evaluating differences
among the ACS test data, census long-form data, and CPS data. Alexander,
Dahl, and Weidman reported in 1997 that during the demonstration period,
they would be working closely with experts familiar with specific test
sites to learn about the quality of the ACS estimates.6 For example, they
reported that the Census Bureau would be looking into sources of
differences between the 1999-2001 ACS test-site average estimates and the
2000 Census long-form results and using the results of differences between
the 2000-02 national sample and the 2000 long form to generate modelbased
estimates for small geographic areas. The authors noted that these
model-based estimates, based largely on information from test sites, would
be used to interpret changes between 2000 and future ACS estimates.
4Susan Love, Donald Dalzell, and Charles Alexander, "Constructing a Major
Survey: Operational Plans and Issues for Continuous Measurement,"
presented at the annual American Statistical Association meeting, Orlando,
Florida, August 16, 1995.
5Charles H. Alexander and Signe Wetrogan, "Integrating the American
Community Survey and the Intercensal Demographic Estimates Program,"
presented at the Joint Statistical Meetings, Indianapolis, Indiana, August
14, 2000.
6Charles H. Alexander, Scot Dahl, and Lynn Weidman, "Making Estimates from
the American Community Survey," presented at the Annual American
Statistical Association Meeting, Anaheim, California, August 13, 1997.
Appendix IV: Continuous Measurement ACS Testing and Development Program
In another 1997 paper, Davis and Alexander reported the Census Bureau's
action plan for evaluation studies.7 They called for evaluating the
results of all test sites and releasing the expert review of the analyses
of the differences between the 1999-2001 ACS and the 2000 Census long
form. The schedule called for releasing this information before beginning
the implementation of the full ACS. Alexander's 1998 paper on completed
research, research in progress, and planned research included among the
four items for planned research a "close study of differences between
1999-2001 ACS and 2000 long form in comparison areas."8
The quality of the ACS measures of income was the subject of the paper
Posey and Welniak presented at the Census Bureau's 1998 symposium on the
ACS.9 They compared income reported in the 1996 ACS and 1990 Decennial
Census in an effort to evaluate the quality of the ACS income data. One of
the adjustments they made to compare the two series was for the effect of
inflation between 1990 and 1996. They noted that the results of the
comparisons indicated a potential problem that may relate to the ACS
inflation adjustment. (They described the calculation of the adjustment,
which is based on the CPI, but did not provide a rationale for using the
adjustment in the ongoing ACS data.)
Alexander and two BLS staff reported in 1999 on the potential for using
the ACS to improve labor force data from the CPS for state and smaller
geographic levels.10 They stressed that to develop procedures for making
7Mary Ellen Davis and Charles H. Alexander, "The American Community
Survey: The Census Bureau's Plan to Provide Timely 21st Century Data,"
Delaware Dataline, Summer 1997. U.S. Census Bureau, American Community
Survey, Advanced Methodology, Papers and Presentations,
http://www.census.gov/acs/www (June 3, 2004).
8Charles Alexander, "Recent Developments in the American Community
Survey," presented at the American Statistical Association Meeting,
Dallas, Texas, August 12, 1998. U.S. Census Bureau, American Community
Survey, Advanced Methodology, Papers and Presentations,
http://www.census.gov/acs/www (June 3, 2004).
9Posey and Welniak, "Income in the ACS: Comparisons to the 1990 Census."
10Shail Butani, Charles Alexander, and James Esposito, "Using the American
Community Survey to Enhance the Current Population Survey: Opportunities
and Issues," presented at the 1999 Federal Committee on Statistical
Methodology Research Conference, Arlington, Virginia, November 15-17,
1999.
Appendix IV: Continuous Measurement ACS Testing and Development Program
these improvements, much research would be needed to evaluate differences
between the ACS and CPS.11
The last research paper in this period was Alexander's 2001 paper focusing
on the origins of the CM methodology and its developers.12 He discussed
the ACS in the context of the methodology, noting several important
differences related to the nature of the ACS. He included a review of the
Census Bureau's testing and evaluation program, noting that the ACS
testsite program had been expanded and that national sample supplementary
surveys had been added. He said that these test data would be compared
with the 2000 Census long-form data and that in 2001 and 2002, the
Supplementary Survey would be used as part of the transition to the ACS.
He also pointed to unresolved issues relating to the residence rule and
the multiyear averages, because they would provide users with multiple
estimates for geographic areas with populations larger than 20,000.
Between 2001 and 2003, the Census Bureau has issued three official reports
and one internal report on the status of the ACS testing and development
program. In Demonstrating Operational Feasibility, published in July 2001,
the Census Bureau gave a brief history of the ACS development program,
which by 2001 was focused on preparing for full implementation in 2003
(although the Census Bureau later revised this to 2004) but on its
operational feasibility, using data from C2SS.13 On the basis of the
Census Bureau's analysis of the results of its tests of operation
feasibility, it reported the tests a success. However, it recognized that
more evaluation on measures of data quality was necessary, as well as on
differences between ACS and 2000 Census long-form data. The Census Bureau
announced that over the next 2 years it would issue reports comparing data
from the 2000 Census long form at the national, state, and
11To assist in this research, BLS wrote a contract with Wayne Vroman of
the Urban Institute for his 2003 report, "Comparing Labor Market
Indicators from the CPS and ACS."
12Charles H. Alexander, "Still Rolling: Leslie Kish's `Rolling Samples'
and the American Community Survey," in Proceedings of Statistics Canada
Symposium 2001: October 16-19 (Ottawa, Canada: Statistics Canada, 2002).
13U.S. Census Bureau, Meeting 21st Century Demographic Data Needs, Report
1, Demonstrating Operational Feasibility (Washington, D.C.: July 2001).
C2SS, conducted as part of the 2000 Census, was a national survey of about
700,000 households and designed to test the operational feasibility of
collecting long-form data at the same time as, but separately from, the
Decennial Census. Its questionnaire was essentially the same as the long
form. The survey has been conducted annually since 2000.
Appendix IV: Continuous Measurement ACS Testing and Development Program
smaller geographic areas with data from the C2SS and the ACS development
program.
Demonstrating Survey Quality, published in May 2002, focused on measures
of C2SS survey quality, summarizing sampling and nonsampling error levels
in both C2SS and the 31 ACS test sites.14 The Census Bureau used
available, generally accepted measures of quality.15 On the basis of its
analysis of the results of these quality tests, the Census Bureau reported
the tests a success. This conclusion rested on test results that showed
the C2SS program capable of providing reliable long-form data.
As in the July 2001 report, the Census Bureau recognized that more
evaluation was necessary on measures of data quality as well as on
differences between ACS and 2000 Census long-form data and the detailed
estimates produced from C2SS. The Census Bureau repeated its commitment
that over the next year and a half, it would release other reports to (1)
analyze in detail basic demographic characteristics (relationship, race,
tenure) produced from the C2SS at the national and state levels, including
comparisons between C2SS and Census 2000; (2) describe the data release
plan and products for the ACS and the usability and accessibility of
estimates resulting from ACS methods; and (3) give several detailed
analyses of selected social, economic, and housing characteristics
(education, income, commuting patterns), including comparisons between
C2SS and Census 2000 at the national and some subnational levels.
In June 2002, shortly after Demonstrating Survey Quality was released, a
team of Census Bureau specialists who had been working on the ACS for
several years prepared an internal report on testing. They presented a
revised program development plan and identified key questions to be
answered in testing the adequacy of the ACS in replacing the Decennial
Census long form. Their plan included the preparation of a series of nine
14U.S. Census Bureau, Meeting 21st Century Demographic Data Needs, Report
2, Demonstrating Survey Quality (Washington, D.C.: May 2002).
15U.S. Office of Management and Budget, Measuring and Reporting Sources of
Errors in Surveys, Statistical Policy Working Paper 31 (Washington D.C.:
July 2001).
Appendix IV: Continuous Measurement ACS Testing and Development Program
evaluation reports over 2 years.16 The reports that evaluated differences
between the 2000 Census short-form data (100 percent reported) and
corresponding C2SS items were included in Demonstrating Survey Quality.
Three reports to be completed between October 2002 and January 2003 would
evaluate differences between the detailed housing, social, and economic
characteristics between C2SS and the 2000 Census long form, as described
in Demonstrating Survey Quality. (Although this schedule was later
extended to the end of 2003, these three reports still had not been
released when we prepared our final draft of this report.)
Finally, the team's plan included a report that would focus on the
comparisons of 3-year averages for the basic demographic, housing, social,
and economic characteristics from the C2SS and ACS test sites and
comparable estimates in the 2000 Census long form. The last report in the
plan would compare data for 2001 and 2002 with measures shown in
Demonstrating Operational Feasibility. The plan did not provide completion
dates for these reports.
American Community Survey Operations Plan, Release 1, published in March
2003, identified research projects to be completed in preparation for full
implementation of the ACS.17 Two projects were on "weighting and
estimation," which covered the methodology for using independent
population and housing controls, and on "program of integrated estimates,"
which covered the calculation of these controls from the Census Bureau's
intercensal population estimates program. The operations plan also
reported on the schedule for completing several comparison and evaluation
projects with ACS and 2000 Census long-form data discussed in
Demonstrating Survey Quality. It discussed the need to evaluate multiyear
estimates from the supplementary surveys to demonstrate the usability,
reliability, and stability of ACS estimates over
16In 2000, the Census Bureau established an ACS Research and Evaluation
Steering Committee to develop a series of reports on key results from the
ACS development program. A team was to manage the program and identify key
questions whose answers would demonstrate the adequacy of the ACS as a
replacement for the Decennial Census long form. See U.S. Census Bureau,
American Community Survey Development Report Series Program Plan
(Washington, D.C.: rev. June 12, 2002), p. 5.
17U.S. Census Bureau, American Community Survey Operations Plan, Release
1.
Appendix IV: Continuous Measurement ACS Testing and Development Program
time, and it stated that a report comparing 3-year ACS data with data from
the 2000 Census long form would be released in mid-2003.18
The Census Bureau reported that the results of these research projects
would not be available in 2004. Instead, it said, it would use interim
procedures, taking "extensive long-term investigation and experimentation"
to develop final procedures.19 For the ACS weighting and estimation
project, the Census Bureau reported that it would be using an interim
adjustment to adjust the intercensal population and housing
characteristics estimates to the ACS residence concept. The Census Bureau
reported that ACS estimates of occupied housing units, households, and
householders should agree at all geographic levels.
For the program of integrated estimates project, the operations plan
discussed the need for more research to introduce improvements to the
estimates from ICPE. (The ACS estimates are weighted to a population
benchmark, either the most recent Decennial Census results or the most
recent ICPE estimates.) The Census Bureau reported that because the
accuracy of the intercensal estimates is important to overall ACS
accuracy, it is important to use ACS data wherever appropriate to improve
the intercensal estimates. The plan for the program on integrated
estimates will use information from the 2000 Census, more current ACS
distributions of population characteristics, and administrative records to
produce improved population and housing unit estimates for all areas,
including small areas. The plan also discussed improving housing
characteristics by incorporating ACS distributions of local area vacancy
rates and household characteristics into statistical models to better
estimate subcounty populations. No time schedule for completing the
research was provided.
Finally, the March 2003 American Community Survey Operations Plan, Release
1 discussed a plan in the ACS to cover group quarters. Persons living in
group quarters live in places that the Census Bureau does not classify as
housing units-for example, nursing homes, prisons, college dormitories,
military barracks, institutions for juveniles, and emergency and
transitional shelters for the homeless. Such residences accounted for
18In April 2003, the Census Bureau let a contract with local experts to
study and evaluate selected differences between 1999-2001 averages from
four test sites and corresponding 2000 long-form data.
19U.S. Census Bureau, American Community Survey Operations Plan, Release
1, p. 36.
Appendix IV: Continuous Measurement ACS Testing and Development Program
roughly 2.8 percent of the population in 2000. Although data on group
quarters were collected at the test sites beginning with 1999, data were
not collected in C2SS or subsequent supplementary surveys. The operations
plan discussed the use of an updated Census 2000 Special Places file for
the sampling frame for the full ACS. In this case, the plan noted,
training field representatives on collecting data from this population is
to begin in October 2004, so that full data collection production can
begin in January 2005.
Census Bureau staff made a presentation on comparison and evaluation
reports at the April 2003 meetings of the Census Advisory Committee. The
paper's author reported that work was under way on the comparison reports
noted in the March 2003 operations plan, and she described the methodology
to be used to evaluate differences between the 2000 long form and C2SS.
She also reported that the results of the comparisons would be used to
identify how the ACS should be improved but that additional research would
be needed to address consistency over time between the 2000 Census and the
full ACS. She stressed the importance of evaluating consistency "in
educating users on the transition from the decennial census sample
estimates to the ACS estimates."20
With regard to the comparison report of selected demographic, housing,
social, and economic characteristics of 3-year estimates from the ACS test
sites to the 2000 Census, the Census Bureau let four contracts with local
experts to conduct comparisons of 3-year averages of ACS data for
1999-2001 for selected test sites with selected 2000 Census long-form data
as well as 2000 Census population and housing unit characteristics. The
comparisons, prepared at the county and census tract levels, would be made
for measures of data quality (self-response rates, sample unit nonresponse
rates, item nonresponse rates, and sample completeness ratios), as well as
for data levels (counts, percentages, means, and medians) for demographic,
social, economic, and housing characteristics.
In summer 2003, Census Bureau staff presented a number of research papers
on the ACS at the annual Joint Statistical Meetings. Papers evaluated
differences between long-form and C2SS data items, such as persons with
disabilities, educational attainments, and income. Most of the
20Deborah H. Griffin, "Comparing Characteristics from the American
Community Survey and Census 2000: Methodology," presented at Census
Advisory Committee of Professional Associations Meetings, Washington,
D.C., April 10-11, 2003, p. 2.
Appendix IV: Continuous Measurement ACS Testing and Development Program
papers that provided comparisons with long-form data indicated whether
differences were statistically significant for every comparison.
Comparisons were presented at a variety of geographic levels (national,
state, and test site levels). Some papers cited operational differences as
possible explanatory factors, but information was not presented using a
standard set of factors.
The Census Bureau published ACS-2010 Census Consistency Review Plan, an
internal document, at the beginning of October 2003. Its purpose was to
identify methods for major operations used in the ACS and for the 2010
Census that were likely to lead to inconsistent results and to recommend
ways to address these inconsistencies.21 Papers prepared on these
operations were to discuss how an issue might result in inconsistencies
between the ACS and 2010 Census results and to set forth options for
dealing with consistency issues, including a research process. The plan
identified residence rules and group quarters as two topics. It did not
discuss completing the work in time to incorporate it into the full ACS in
the next several years.
Also in October 2003, the Census Bureau made two public announcements
related to the ACS development plan at the Census Advisory Committee
meetings. Two papers related directly to projects described in American
Community Survey Operations Plan, Release 1. In "Enhancing the Intercensal
Population Estimates Program with ACS Data: Summary of Research Projects,"
Weidman and Wetrogan reported on research to improve the intercensal
estimates by using ACS data for two "high priority" areas-international
migration and internal migration. This work was being conducted within the
Program of Integrated Estimates.22 The second paper described options for
determining population control weights for ACS implementation in fall 2004
but did not indicate that research was under way to determine the effect
of the options.23
21U.S. Census Bureau, ACS-2010 Consistency Review Plan (Washington, D.C.:
October 1, 2003).
22Lynn Weidman and Signe Wetrogan, "Enhancing the Intercensal Population
Estimates Program with ACS Data: Summary of Research Projects," Census
Advisory Committee of Professional Associations meeting, Washington, D.C.,
October 23, 2003.
23Navarro, "American Community Survey: Use of Population Estimates as
Controls in the ACS Weighting," presented at Census Bureau Advisory
Committee of Professional Associations meeting, Washington, D.C., October
23, 2003.
Appendix IV: Continuous Measurement ACS Testing and Development Program
Another source of information related to ACS development was the various
reports prepared as part of the Census 2000 Testing, Experimentation, and
Evaluation Program. Schneider's January 2004 report compared employment,
income, and poverty estimates from the 2000 Census long form and the
CPS.24 From this comparison, the author concluded that this work should be
continued in an effort to use the results of the comparisons to improve
consistency between data collected in the CPS and data in the ACS; the ACS
uses the same questions as the 2000 long form. The author also identified
for additional research long-form questions that performed badly, based on
a reinterview survey.
From May to July 2004, the Census Bureau released seven ACS evaluation
reports. Four reports compared data from the 2000 Census long form and the
C2SS at the national level. Two reports compared these long-form data with
1999-2001 data from the ACS test sites for selected counties and one of
these compared these data at the tract level. The other report reviewed
operational data from the 2001 and 2002 supplementary surveys. In most of
the reports comparing long-form and ACS data, the Census Bureau identified
additional work that was needed to improve the quality of the ACS
estimates or to help explain differences between the two sets of data for
2000. As noted earlier, these comparisons were limited to the national
level. (The seven new reports are listed in the bibliography.)
24Paula Schneider, Content and Data Quality in Census 2000, Census 2000
Testing, Experimentation, and Evaluation Program Topic Report No. 12
(Washington, D.C.: U.S. Census Bureau, January 22, 2004).
Appendix V: Current Status of Unresolved Issues
Independent Controls According to the Census Bureau's plans, the
calculation of independent controls for population characteristics (age,
sex, race, and ethnicity) and
for Population and housing characteristics for the full ACS will require a
significantly
Housing different methodology from that used for the ACS supplementary
surveys. Controls will be needed at the same level of geographic area
detail as Characteristics those that were used for the 2000 Census long
form and will need to reflect the new concepts of residence and reference
period underlying the ACS.
For the annual ACS supplementary surveys, these characteristics were used
from ICPE as the independent controls. ICPE uses Decennial Census
short-form data as benchmarks and administrative record data to
interpolate between and extrapolate from the census benchmarks.1 The
program provides "official" annual estimates of population and housing
characteristics at the county level, and for some subcounty levels, as of
July 1 of each year, using the usual residence concept for seasonal
residents. The program also provides annual estimates of total population
and housing units for all areas of general-purpose government, such as
cities, villages, towns, and townships.2 Table 3 shows information on the
calculation of the independent controls used for the 2000 Census long
form, the ACS supplementary series, and the fully implemented ACS through
2012.
1ICPE develops and disseminates annual estimates of the total population
and the distribution by age, sex, race, and Hispanic origin for the
nation, state, counties, and functioning government units. The program is
authorized by 13 U.S.C. S:181, which requires the production of "current
data on total population and population characteristics."
2For additional details, see U.S. Census Bureau, "Estimates and
Projections Area Documentation: Subcounty Total Population Estimates,"
http://www.census.gov.
Appendix V: Current Status of Unresolved Issues
Table 3: The 2000 Census Long Form and ACS Use of Independent Controls for
Population and Housing Characteristics
Survey Source of Residence
and Reference
date controlsa Weighting Items weighed concept Comments
area period
2000 2000 Census About 65,000 Population: age Usual Apr.
Census areas group 1,
long form (13), sex, race 2000
(6),
Hispanic origin
(2).
Housing: occupied
or
vacant, owner or
renter
ACS test site
1999 ICPE County bPopulation: age group, Usual July 1,
benchmarked to sex, race (3), Hispanic 1999
1990 Census origin (2). Housing: no
direct use of housing
weights
2000 2000 Census County bPopulation: age group, Usual Apr. 1,
sex, race/ Hispanic 2000
origin (6), Housing: total
number of units
2001-04c ICPE County bPopulation: age group, Usual July 1,
benchmarked to sex, race/ Hispanic 2001-04
2000 Census origin (6). Housing: total
units
ACS supplementary survey
2000 2000 Census County or county bPopulation: age Usual Apr. 1,
group,
combinations sex, race/ Hispanic 2000
origin (6).
Housing: total
units
2001-04c ICPE bPopulation: age Usual
County or county group, July 1,
benchmarked to combinations (14), sex, race/ 2001-04
Hispanic
2000 Census origin (6).
Housing: total
units
Appendix V: Current Status of Unresolved Issues
Survey and Source of Residence Reference date controlsa Weighting area
Items weighed concept period Comments Full ACS
2005-09,d ICPE benchmarked to 2000 Census adjusted to ACS residence
concept
(adjustment methodology not announced)
Three options:e Not announced Current Not
(1) Intercensal announcedf
estimates (usual
residence) for large
areas and ACS
estimates (current
residence) for small
areas
(2) Option 1 but model
based estimates to
modify intercensal
estimates for large
areas to current
residence
(3) Develop methods to
generate current
residence estimates for
all small areas
Other: Same areas as
2000 Census long form
using intercensal
estimates and detail
from 2000 census
Residence concept changes; weighting area options part of research program
to determine weighting areas for use with 3-and 5year averages for 2010
2010e 2010 Census Not Not Current Not 2010
announcedg announced short-form
data
announced replace
adjusted to ACS ICPE
residence estimates
concept
(adjustment
methodology not
announced)
ICPE Not announcedg Not announced Current Not
benchmarked to announced
2010 Census
adjusted to ACS
residence concept
(adjustment
methodology not
announced)
2012 ICPE Not announcedg Not announced Current Not
benchmarked to announced
2010 Census
adjusted to ACS
residence concept
(adjustment
methodology not
announced)
Source: GAO analysis of Census Bureau documents.
Appendix V: Current Status of Unresolved Issues
aThe Intercensal Population Estimates Program (ICPE) develops and
disseminates annual estimates of the total population and the distribution
by age, sex, race, and Hispanic origin for the nation, state, counties,
and total population for subcounty functioning government units. ICPE is
authorized by 13 U.S.C. S:181, which requires the production of "current
data on total population and population characteristics." ICPE estimates,
benchmarked to the latest Decennial Census counts, are compiled using
administrative record data on births, deaths, and migration. Because they
are benchmarked to the census, they reflect the usual residence concept.
They are adjusted to reflect the counts as of July each year.
bNot available.
cAssumes that test program and supplementary surveys end after 2004
dWhen 2010 Census estimates become available, ICPE estimates beginning
with 2001 will be revised to reflect the new benchmark. There is no
announced use of revised ICPE estimates to revise previously published ACS
estimates.
eThe three options are from Alfredo Navarro, "American Community Survey:
Use of Population Estimates as Controls in the ACS Weighting," presented
at Census Bureau Advisory Committee of Professional Associations meeting,
Washington, D.C., October 23, 2003.
fOne of the Department of Commerce comments on our draft report stated
that the Census Bureau would be using July 1 as the reference period for a
given year's ACS annual average.
gThere is no announced use of the tract or block group data from the 2010
census.
Using ICPE for the ACS supplementary surveys, the Census Bureau prepared
controls for counties, or combinations of counties. As shown in table 3,
for the residence concept, controls from the 2000 Census and ICPE, which
were based on the usual residence concept, were used. The reference period
for the ACS test program for all years except 2000 was for July 1; for
2000, it was for April 1. (Controls for the 2000 Census long form also
were for April 1.)
For the full ACS, the Census Bureau will use controls based on the current
residence concept. According to the Census Bureau, the current residence
concept recognizes that the place of residence does not have to be the
same throughout a year, so that the current residence concept allows the
ACS data to more closely reflect the actual characteristics of each area.
The Census Bureau will use the current residence concept because the ACS
is conducted every month and produces annual averages rather than
point-in-time estimates, as the Decennial Census does. Also, because the
ACS data are collected monthly, it will be necessary to use independent
controls that define the reference period as the average for the year
using a July 1 reference period.
To produce ACS estimates for the full sample, the Census Bureau will need
new methodologies for calculating independent controls. For the first
annual estimates, for 2005, a methodology will be needed to provide
ACS-defined controls for all places with population of 65,000 or more,
including those for which intercensal population estimates are not
Appendix V: Current Status of Unresolved Issues
available. For the 2005-07 estimates, which will be used to calculate the
first multiyear averages, a methodology for controls for geographic areas
with populations between 20,000 and 65,000 will be needed. For the 2008-12
estimates, a methodology for controls down to the geographic levels used
for the 2000 Census long form will be needed. Finally, when the population
and housing characteristics data from the 2010 Census short form become
available and are incorporated into the ICPE estimates, another new
methodology will be needed to revise the ACS controls for 2010.3
The Census Bureau also has reported that it is not planning to revise
earlier years' ACS data for consistency with revised 2010 estimates unless
the inconsistencies between the 2010 ICPE and 2010 Census characteristics
were significant. Table 4 shows the differences between population
estimates at the county level for 2000 using ICPE based on the 1990 Census
and the corresponding data from the 2000 Census. In 2000, the population
estimates for almost 20 percent of the counties differed by more than 5
percent. For counties whose population was smaller than 20,000, almost 25
percent had similar differences.
3The initial ACS estimates for 2010 are to be released before the 2010
Census-based ICPE estimates are available.
Appendix V: Current Status of Unresolved Issues
Table 4: Population Comparison for Counties in 2000 from ICPE and 2000 Census by
County Size
Number of counties with ratio of
Percentage of
counties with ratio
Less 0.90- 0.95- 1.00- 1.05- 1.10 less than 0.95 or
County than or
population a Total 0.90 0.949 0.999 1.049 1.099 more more than 1.05
All counties 3,141 118 384 1,722 809 88 20
Less than 1,348 67 174 672 355 63 17
20,000
20,000 to less
than 65,000 1,046 41 112 576 294 20 3
65,000 to less 516 9 62 315 125 5 0
than 250,000
250,000 or
more 231 1 36 159 35 0 0
Sources: Census Bureau reports and GAO analysis.
Note: Initial intercensal estimates for 2000 were benchmarked to the 1990
Census; counties include county equivalents, such as parishes in
Louisiana.
aPopulation classes reflect level of geographic area detail to be
calculated from ACS. For example, geographic areas with populations
smaller than 20,000 will be available using 5-year averages.
Census Bureau staff had long recognized the need for new methodologies to
develop independent controls for the ACS. For example, a 1995 paper by
Love, Dalzell, and Alexander, discussing issues related to evaluating the
1996 test site results, expressed concern about independent controls and
residence rules, as well as the need for consultation with users.4 In
1998, the Census Bureau sponsored a conference on the quality of ACS data
for rural data users. In the final report on this conference, the Westat
authors concluded that the Census Bureau needed to continue and expand its
contacts with stakeholders and to conduct additional research on several
issues, including independent controls.5 Alexander and Wetrogan also
discussed this issue at the 2000 Joint Statistical Meetings when they
4Susan Love, Donald Dalzell, and Charles Alexander, "Constructing a Major
Survey: Operational Plans and Issues for Continuous Measurement,"
presented at the annual American Statistical Association meeting, Orlando,
Florida, August 16, 1995.
5This May 14-15, 1998, conference was held in response to U.S. Senate
Appropriations Committee Report 105-48, 105th Cong., 1st sess. (July 16,
1997), title II, p. 64, which had stated that "The outside evaluator
should review the ACS to determine whether there is an antirural bias in
its design." Graham Kalton and others prepared the conference report for
the Census Bureau: The American Community Survey: The Quality of Rural
Data (Rockville, Md.: Westat, June 29, 1998).
Appendix V: Current Status of Unresolved Issues
reviewed possible methods for using ICPE estimates.6 They also noted the
need to consult with users on how to present information on the
differences in ACS controls and ICPE in ACS publications.
Census Bureau staff also recognized that the new ACS would create
differences between (1) ACS population and housing characteristics data
and the corresponding "official" data from the Decennial Census and (2)
ACS population and housing characteristics data and the "official" ICPE
population estimates, which are benchmarked to Decennial Census data. They
also recognized that the creation of new controls for the ACS would result
in inconsistencies between ACS data and data from federal household
surveys, such as the CPS, whose population and housing characteristics are
also based on the Decennial Census and ICPE estimates. Such differences
might hinder the use of ACS data to expand and improve small geographic
area estimates based on the other surveys. (CPS provides official national
estimates of labor force information, such as the unemployment rate and
income estimates used to calculate the number of persons in poverty.)
In March 2003, the Census Bureau announced that it did not have a final
methodology and that such methodologies would not be established for
several years. In March 2003 in American Community Survey Operations Plan,
Release 1, the Census Bureau identified research projects to be completed
in preparation for full implementation of the ACS. One of these projects,
"weighting and estimation," covered the methodology for calculating the
independent controls for the ACS; a second, "program of integrated
estimates," covers the calculation of these controls from the ICPE. This
plan also reported that the results of these research projects would not
be available in 2004 to begin implementing them with the start of the full
ACS. Instead, the Census Bureau said it would use interim procedures and
that it would take "extensive long-term investigation and experimentation"
to develop final procedures.
For the weighting and estimation project, the Census Bureau reported that
it would be using an interim adjustment to adjust the intercensal
population and housing characteristics estimates to the ACS residence and
reference period concepts. This project would include research to examine
the need to achieve agreement between the estimates of occupied
6Alexander and Wetrogan, "Integrating the American Community Survey and
the Intercensal Demographic Estimates Program."
Appendix V: Current Status of Unresolved Issues
housing units, households, and householders at all geographic levels. The
Census Bureau reported that work on the project to revise and simplify the
weighting methodology began in early 2003, that preliminary papers
documenting the revisions might be available by summer 2004, and that
research would continue for several years.
For the program of integrated estimates project, the operations plan
discussed the need for more research to introduce improvements to the ICPE
estimates using information from the 2000 Census, more current ACS
distributions of population characteristics, and administrative records to
produce improved population and housing unit estimates for all areas,
including small geographic areas. The plan also discussed improving the
housing characteristics. ACS distributions of local area vacancy rates and
household characteristics can be incorporated into statistical models that
use distributions of housing unit characteristics to better estimate
subcounty populations. No time schedule was provided for completing the
research.7
In October 2003, Census Bureau staff presented a paper at the Census
Advisory Committee meetings that described the options being considered to
convert the ICPE estimate to the current residence concept.8 The paper
described options for determining controls for ACS implementation in fall
2004 but did not indicate that research was under way to determine the
options' effects. A second paper at the same meetings reported on research
to improve the intercensal estimates by using ACS data for two "high
priority" areas-international migration and internal migration. This work
was being conducted as part of the Program of Integrated Estimates.9
Although the latest NAS report on the ACS does not specifically note
issues relating to independent controls, we asked experts who had
participated in preparing NAS reports, as well as other experts in small
area data, the following question about ACS weighting:
7ACS data on foreign-born persons were used to estimate the national
levels of international migration that were incorporated into the
intercensal estimates for 2003, based on the 2000, 2001, and 2002 ACS.
8Navarro, "American Community Survey."
9Weidman and Wetrogan, "Enhancing the Intercensal Population Estimates
Program with ACS Data."
Appendix V: Current Status of Unresolved Issues
"Given the newly benchmarked intercensal estimates, the following question
arises regarding the use of the 2010 Census data in the ACS: Should ACS
estimates continue to be controlled to 2010 Census data at the county or
county group level and differences between the ACS and census population
counts and characteristics allocated proportionately to the tract or block
group levels? Or should ACS estimates be controlled to 2010 Census data at
the tract and block group level, as would have been the case with a long
form?"
All the experts agreed that the ACS should be controlled to the decennial
census, but several noted that they had not thought about the issue and
had not heard anything from the Census Bureau on the issue. (The experts
are listed in app. I.)
Operational Issues
The Census Bureau has identified operational issues with the ACS test
programs, primarily from information from evaluation studies on the 2000
Decennial Census and Census Bureau staff research papers on comparisons
between data collected in the ACS 2000 supplementary survey and the 2000
Decennial Census long form. These issues include problems with
questionnaire design, nonresponse followup, and data capture, as well as
coverage of persons living in group quarters.
In January 2004, the Census Bureau released the results of a key
evaluation study of 2000 Decennial Census long-form data, using a
reinterview survey.10 The study identified problems with long-form
questions, which are the same as those used for the ACS, and proposed
several research efforts based on a statistical evaluation of the quality
of the responses to each question. For questions identified as having
significant quality problems, the study recommended research on the design
of the form and placement of the questions and suggested using cognitive
experts in testing revised questions. The study also recommended that the
Census Bureau and BLS work on the ACS employment and unemployment
questions to ensure that they would complement the BLS local area
unemployment statistics program.
The Census Bureau also conducted a study to evaluate the design of the ACS
questions that are needed to implement the residence concept and reference
period for the ACS.11 The study suggested that additional testing
10Schneider, Content and Data Quality in Census 2000.
11Theresa J. DeMaio and Kristen A. Hughes, "Report of Cognitive Research
on the Residence Rules and Seasonality Questions on the American Community
Survey," U.S. Census Bureau, Statistical Research Division, Washington,
D.C., July 2003.
Appendix V: Current Status of Unresolved Issues
was needed for the questions about multiple residences (currently, the
last set of questions in the housing section). It noted "that asking these
questions on a person basis may produce different and probably better data
than asking them on a household basis."12 The study was limited in scope
and did not assess how accurately ACS respondents assign persons
associated with the household to a current residence.
In the ACS, the Bureau uses "In the past 12 months . . ." whereas the
Census Bureau used "In 1999 . . ." for the long form. Because the
reference date is not fixed, it is important for a respondent to supply
the date that the ACS questionnaire filled out. Otherwise, it cannot be
determined whether there is an inconsistency in an ACS questionnaire
received in late April 2004 that lists a resident aged 10 with a birthdate
of April 15, 1993.13
Census Bureau staff also discussed operational issues in research papers,
based on evaluations of comparisons between 2000 Decennial Census
long-form and ACS 2000 supplementary survey data for selected items
presented at the 2003 Joint Statistical Meetings. A paper on income data
identified the new question on the reference period as a potential source
of problems, even though an additional instruction had been added to the
ACS questionnaire in 1999.14 The authors expressed concern that some ACS
respondents may misinterpret the question on "income in the past 12
months" as a request for monthly income instead of income during the
previous year. The paper also included recommendations for additional
research on the effect of the data capture methods. For the 2000 long
form, all data items were entered with an automated optical character
recognition procedure; data from the ACS will be manually keyed.
Another paper, presented at the same 2003 meetings, that evaluated
differences in the data on disabled persons found large and significant
differences at the national level and also recommended that new questions
12DeMaio and Hughes, "Report of Cognitive Research on the Residence Rules
and Seasonality Questions," pp. 9-10.
13In contrast, a 2000 Census questionnaire received in late April 2000
that listed a resident aged 11 with an April 15, 1989, birthdate would be
considered inconsistent because the person was aged 10 on census day.
14Kirby G. Posey, Edward Welniak, and Charles Nelson, "Income in the
American Community Survey: Comparisons to Census 2000," presented at the
Joint Statistical Meetings, San Francisco, California, August 7, 2003.
Appendix V: Current Status of Unresolved Issues
be tested.15 Additional areas were identified for further research, based
on evaluations of questions such as educational enrollment, ancestry, and
grandparents caring for grandchildren. These areas included specific
facets of the mailout-mailback system and nonresponse followup. For
example, nonresponse follow-up for the 2000 long form was conducted for
all nonrespondents, but for the ACS test program and for the full ACS,
nonresponse follow-up will be conducted for a sample of one-third on all
nonrespondents.
The Census Bureau also has discussed issues with the expansion of ACS
coverage to include persons living in group quarters-for example, nursing
homes, prisons, college dormitories, military barracks, institutions for
juveniles, homeless shelters.16 In October 2002, it informed its advisory
committee members of the formation of a special planning team to address
issues on the definition of group quarters and duplication in the address
file. From the minutes of this meeting, it appears that this team will
focus on group quarters in the context of the 2010 Census short form. In
the ACS March 2003 operations plan, the Census Bureau reported on a new
project to cover group quarters in the full ACS.17 The Census Bureau
reported that the special project was needed because of the special
challenges of developing an updated address list; in the past, such a list
had been updated only once a decade. According to the Census Bureau, tests
on the new list were to be completed in time for use in the full ACS in
January 2005. In addition, an internal planning document issued in October
2003 identified group quarters (and residence rules) as special problems
and instructed staff to provide recommendations on the collection of data
on them in January 2004.18 Usually, the Census Bureau tests new questions.
According to recent Census Bureau decisions, those
15Sharon M. Stern, "Counting People with Disabilities: How Survey
Methodology Influences Estimates in Census 2000 and the Census 2000
Supplementary Survey," presented at the Joint Statistical Meetings, San
Francisco, California, August 7, 2003.
16Although data on group quarters were collected at the ACS test sites in
1999 and 2001, data on them were not collected in the ACS supplementary
surveys, which began in 2000. The Census Bureau made this decision to
avoid duplication with the 2000 Decennial Census and because it lacked
funding to cover them in subsequent years.
17Census Bureau, American Community Survey Operations Plan, Release 1.
18U.S. Census Bureau, ACS-2010 Consistency Review Plan.
Appendix V: Current Status of Unresolved Issues
Valuation and Presentation of Dollar-Denominated Data Items
tests would have to be completed so that new questions could be
incorporated into the 2008 ACS questionnaire.19
The Census Bureau has adjusted all dollar-denominated items from the ACS
testing programs, such as incomes, housing values, rents, and
housing-related expenditures, for inflation. For example, ACS data for
2001 and 2002 released in September 2003 for median household income are
expressed in 2002 dollars. This practice means that when each added year
of ACS data is released, all dollar-denominated items for prior years will
be revised. The Census Bureau makes a similar adjustment for the annual
income data collected in the CPS. Unlike the ACS, the Census Bureau
releases annual CPS data without the adjustment. In addition, the annual
values collected in the ACS were adjusted to the calendar year. It will be
using the CPI for the annual and monthly adjustments for all geographic
areas.
A report prepared for HUD found problems with the adjustment, including
(1) the lack of a "trending" adjustment in the calculation of annual
averages, (2) the use of the adjustment for multiyear averages, (3) the
adjustment for cost of living for data items other than income, and (4)
the lack of the unadjusted annual data that would enable HUD to use
alternative methodologies. In addition, research by Census Bureau staff
questioned the adjustment for incomes when they found that it was a
probable source of difference between income data from the supplementary
survey and corresponding data from the CPS and the 2000 Census long form.
20
The report prepared for HUD provided a detailed review of HUD's use of the
ACS for program applications. On the methodology for the inflation
adjustment, the first step should be a trending adjustment that would
convert the reported monthly data to a calendar year basis. Discussing
this omission, the report stated,
19 U.S. Census Bureau, 2010 Decision Memorandum Series No. 5. (Washington,
D.C.: June 3, 2004) and 2010 Planning Memorandum Series No. 24
(Washington, D.C.: June 9, 2004).
20For a complete discussion, see ORC Macro, The American Community Survey:
Challenges and Opportunities for HUD (Calverton, Md.: Sept. 27, 2002). For
a complete discussion of the role of the inflation adjustment in
differences between the ACS and CPS measures of income, see Posey,
Welniak, and Nelson, "Income in the American Community Survey."
Appendix V: Current Status of Unresolved Issues
"Making an inflation adjustment is not the same as trending. The cost of
living adjustment
assumes that the purchasing power measured at any point in the data
collection period
remains constant throughout the period. For example, assume that the cost
of living rises
by 3 percent a year. If a household reports an annual income of $50,000 in
January, a cost
of living adjustment to the end of the year would increase this income to
$51,500, the
amount needed in December to equal the purchasing power of $50,000 in
January. A
trending adjustment makes no assumption about purchasing power. It
attempts to track
movements in dollar income. Assume that dollar income is growing at 5
percent a year.
Then a trending adjustment to the end of the year would increase the
$50,000 reported in January to $52,500 in December."21
HUD's second concern was that the methodology the Census Bureau used to
calculate the adjustment was not appropriate for multiyear averages. The
HUD report stated,
"The Census Bureau plans to report income in constant dollars. Income
information
collected in the various months will be adjusted for inflation so that all
collected income
will be expressed in dollars with the same purchasing power, presumably
the purchasing
power of dollars in December of the survey year. For moving average
tabulations, all
income information will be adjusted for changes in purchasing power over
the period used
to calculate the moving average. In other words, income reported by a
respondent in the
first month of a five-year moving average will be adjusted for almost five
years of inflation."22
To illustrate this problem, the HUD report gave the following example:
"The standard Census Bureau tables for areas over 65,000 will tabulate the
rents reported
by respondents over the twelve months during which data were collected. A
unit reporting
a contract rent of $800 in January might actually be paying $850 in
December. The standard
table would record this unit as having a rent of $800. The standard Census
Bureau tables
for areas under 20,000 will tabulate rents reported by respondents over a
sixty-month
period. A unit reporting a contract rent of $800 in the January of the
first year might
actually be paying $1,070 in December of the fifth year. The standard
table would record this unit as having a rent of $800."23
21ORC Macro, p. 16. 22ORC Macro, p. 44. 23ORC Macro, pp. 16-17.
Appendix V: Current Status of Unresolved Issues
Such changes would not be captured with an adjustment based on the
allitems CPI.
The HUD report also noted that the inflation index the Census Bureau
proposed related to income and not to the other dollar value data, such as
rent, utility costs, or home value, where a purchasing power concept did
not meet HUD's needs. The report concluded that to overcome this problem,
before HUD could use dollar-denominated data from the ACS, it would first
have to eliminate the inflation adjustment from the published data. The
report stated: "For applications that involve trending income, HUD users
will have to center the ACS information at an appropriate point in the
collection period and remove the inflation adjustment before applying a
trending factor."24 In addition, it noted that
"The ACS will generate income distributions comparable to those from the
decennial
census, but the distributions will have a feature that will complicate the
use of income data
from the ACS in APP [HUD's Annual Performance Plan] measures. Whereas the
decennial long form measures money income, the ACS reports average
purchasing power."25
The report thus recommended that HUD use the unadjusted data-data that the
Census Bureau had not planned to publish-in order to make the changes
needed for HUD.
The validity of the Census Bureau's inflation adjustment was also
questioned in research Census Bureau staff conducted to evaluate
differences between the data reported in the ACS supplementary surveys. In
a paper presented at the 2003 Joint Statistical Meetings, staff evaluated
differences between income data from C2SS and the 2000 Census long form,
as well as the CPS.26 The paper summarized the major differences in the
income data from these sources in terms of data collection, capture, and
processing and provided preliminary assessments of their contributions to
these differences. The authors noted the need for further research on the
effect of the difference in reference period and the inflation adjustment,
as well as operational aspects such as data capture. With regard to the
inflation adjustment, they reported:
24ORC Macro, p. 44.
25ORC Macro, p. 208.
26See Posey, Welniak, and Nelson.
Appendix V: Current Status of Unresolved Issues
"If no CPI adjustment had been made to the dollars reported on either
Census 2000 or
C2SS/ACS, the difference between medians at the U.S. level would have been
smaller than
the 4.6 percent shown in Table 3 [omitted]. Instead, the difference would
have been 2.5
percent. Since adjustment clearly played a role in determining the size of
the difference
between Census 2000 and C2SS/ACS estimates, it would be worthwhile to
examine the
costs and benefits of adjusting C2SS/ACS incomes as well as the choice of
factors used to adjust them."27
The authors summarized their findings by concluding that "it is clear that
we are just at the beginning stages of understanding why Census 2000 and
C2SS income figures differ."28 They noted that the income comparisons are
most critical because these Census Bureau data are used in the calculation
of the number of people in poverty.
In a December 2003 research paper, Census Bureau staff examined concerns
about the absence from the official poverty measures of adjustment for
geographic differences in cost of living. Like the ACS, for which the
Census Bureau is assuming that the cost of living is the same throughout
all geographic areas, the poverty measures are based on the same
assumption. The authors concluded that the use of a poverty measure that
takes into account geographic differences in housing costs, would
significantly change the poverty measures in many states.29
One of the Census Bureau's major justifications for the ACS test programs
has been its comparing data collected in these programs, and corresponding
data from the 2000 Decennial Census short and long forms, to identify
operational problems. Another major justification for the ACS test
programs has been the use of these comparisons, and comparisons with
corresponding data from the CPS, to inform users in making the transition
from the 2000 long form to the ACS.
The Census Bureau's 1999 request to OMB for approval of the forms for the
ACS test programs stated that
Evaluations of ACS, Long-Form, and CPS Data Comparisons
27Posey, Welniak, and Nelson, p. 14.
28Posey, Welniak, and Nelson, p. 15.
29Charles Nelson and Kathleen Short, "The Distributional Implications of
Geographic Adjustment of Poverty Thresholds," U.S. Census Bureau, Housing
and Household Economics and Statistics Division, Washington, D.C.,
December 2003.
Appendix V: Current Status of Unresolved Issues
"to make a transition from the Census 2000 long form to collecting
long-form data
throughout the decade, we will begin ACS data collection in 1,203
counties. This data
collection will allow for comparison of estimates from Census 2000 with
estimates from the
ACS for all states, large cities, and population subgroups, and will help
data users and the
Census Bureau understand the differences between estimates from the ACS
and the Census 2000 long form."30
In testimony to the Congress a year later, Kenneth Prewitt, the Census
Bureau's Director, referred to the ongoing ACS test programs:
"These data will also contribute to a comparison with data from Census
2000 that is
necessary because there are differences in methods and definitions between
the census
and the ACS. Moreover, decision makers will want to compare an area's data
to those from
Census 2000. Comparisons using data from the operational test and from the
31 sites are
essential to determine how much measured change between Census 2000 and
future years
of the ACS is real and how much is due to operational differences between
the ACS and the census."31
When the Census Bureau began in 2001 to report on full implementation of
the ACS, its first report focused on the operational feasibility of
conducting the ACS.32 Its second report in 2002 focused on differences in
operational characteristics of the ACS and the census long form, such as
response rates and the extent of imputations.33
The 2002 report stated that three reports evaluating differences between
the ACS and census long form would be published at the end of 2003.34 The
Census Bureau repeated this schedule in March 2003 when it released
3064 Fed. Reg. 64 48759-48760 (Sept. 8, 1999). The Census Bureau made
similar statements about the importance of comparisons in its request to
OMB to extend approval of the supplementary survey forms in 67 Fed. Reg.
67 21629 (May 1, 2002).
31Kenneth Prewitt, Director, U.S. Bureau of the Census, before the U.S.
House of Representatives, Committee on Government Reform, Subcommittee on
the Census, Summary of House Hearing on ACS July 20, 2000, The American
Community Survey: A Replacement for the Census Long Form? Serial 106-246
(Washington, D.C.: July 20, 2000).
32U.S. Census Bureau, Meeting 21st Century Demographic Data Needs:
Implementing the American Community Survey, Report 1, Demonstrating
Operational Feasibility.
33U.S. Census Bureau, Meeting 21st Century Demographic Data Needs:
Implementing the American Community Survey, Report 2, Demonstrating Survey
Quality.
34U.S. Census Bureau, American Community Survey Development Report Series
Program Plan, an internal report prepared a month later, had called for
completing one of these reports by the end of 2002.
Appendix V: Current Status of Unresolved Issues
another official report on ACS plans.35 In September, we were told by one
of the ACS experts that consultants had been hired to conduct evaluations
for 4 of the 31 test sites. The reports on comparisons with long-form
items and for the test sites were published in May, June, and July 2004.
The results of these comparisons are similar to comparisons and
evaluations of long-form data items previously prepared by Census Bureau
staff, BLS, and GAO.
In September 2002, we prepared national and state comparisons between the
2000 ACS supplementary survey and the 2000 Decennial Census long form for
about 10 items and between the 2000 ACS supplementary survey and the 2000
CPS for the poverty and unemployment rates. From the longform comparisons,
we reported that
"These comparisons showed large national differences for key items that
did not appear to
be accounted for by coverage differences between the two surveys. For
example, at the
national level, the largest differences were for these items: (1) for the
number of housing
units lacking complete plumbing facilities, with the long-form estimate 27
percent higher
than the estimate from the supplementary survey, and (2) for the number of
unpaid family
workers, with the long-form estimate 59 percent lower. . . . We also found
a great degree of variation in the state differences between the long form
and the supplementary survey."36
From the CPS comparisons, we reported that
"We found that at the national and state levels, there were small
differences for the
unemployment rate and for the poverty rate for all individuals. In
contrast, comparisons of
these rates for the CPS with these two surveys showed larger differences.
The national
unemployment rate, according to the CPS, was 4.0 percent, compared with
5.8 percent for
the long form and 5.4 percent for the supplementary survey. The national
rate for
individuals in poverty for the CPS was 11.3 percent, compared with 12.4
percent for the long form and 12.5 percent for the supplementary
survey."37
Given these results, we recommended that the Census Bureau expand the
scope of evaluation studies to develop supplementary survey estimates for
states and large places consistent with the 2000 long form and that it
35U.S. Census Bureau, American Community Survey Operations Plan, Release
1.
36GAO-02-956R, p. 11.
37GAO-02-956R, p. 12.
Appendix V: Current Status of Unresolved Issues
include in its evaluations comparisons of year-to-year changes for 2001
and 2002, using data from the supplementary surveys and the CPS at the
national and state levels for key economic and housing items.
In September 2003, BLS received a report from a consultant who had been
hired to evaluate differences between labor force data, such as the
unemployment rate, reported in the ACS test programs and the CPS.38 The
evaluation's purpose was to provide BLS with information on whether and
how to incorporate ACS data into its measures of unemployment and the
labor force. The consultant compared several labor market indicators from
the CPS and ACS for 2000-02 at the national and state levels:
"Relative to the CPS, the ACS consistently generates lower estimates of
the labor force and
employment but higher estimates of unemployment. These patterns are
present in each of the years 2000, 2001, and 2002. They are repeated in
nearly all state-level data as well."39
He made a series of recommendations for additional research, some
requiring additional information from the Census Bureau.
The need for such research was also reported in a January 2004 Census
Bureau report that examined differences between labor force data from the
CPS and the 2000 Decennial Census long form.40 Other findings and
recommendations for further research similar to ours and those of BLS were
also reported in research papers Census Bureau staff presented at the 2003
Joint Statistical Meetings. One paper on comparisons of income data for
2000 from the 2000 Decennial Census long form and the 2000 ACS
Supplementary Survey reported that it
"provided a summary of the major differences between the two income data
sources, in
terms of data collection, capture, and processing, and provided very
preliminary assessments of the possible role these differences may have
played."41
The authors reported that additional work was needed to understand the
differences and offered recommendations for further research.
38Wayne Vroman, Comparing Labor Market Indicators from the CPS and ACS
(Washington, D.C.: Urban Institute, September 2003).
39Vroman, p. 23.
40Schneider.
41Posey, Welniak, and Nelson, "Income in the American Community Survey,"
p. 14.
Appendix V: Current Status of Unresolved Issues
Another paper presented at the same meetings examined differences between
the national estimates for people aged 5 or older with a disability-48.9
million was the 2000 Census long-form estimate, 39.7 million the C2SS
estimate. The author did not determine which estimate was more reliable
but did find that the wording of some questions might explain the overall
difference. In addition, the author reported that more work, such as
additional analysis of currently available data and testing of new
questions, was needed to clearly identify the reasons for the
difference.42 The differences in disability data were also the subject of
a National Council on Disability position paper, which recommended changes
to the questions on disability.43
In addition to results from these comparison studies, the NAS Panel on
Research on Future Census Methods found in July 2003 that the Census
Bureau needed to complete evaluations of differences between 2000 Census
long-form data and data from the ACS test sites and the 2000-02
Supplementary Surveys. Specifically, the panel stated that
"The Census Bureau should carry out more research to understand the
differences between
and relative quality of ACS estimates and long-form estimates, with
particular attention to
measurement error and error from nonresponse and imputation. The Census
Bureau must
work on ways to effectively communicate and articulate those findings to
interested stakeholders, particularly potential end users of the data."44
The panel also stated that, to facilitate this effort,
"The Census Bureau should make ACS data available (protecting
confidentiality) to
analysts in the 31 ACS test sites to facilitate the comparison of ACS and
census long-form
estimates as a means of assessing the quality of ACS data as a replacement
for census long
form data. Again, with appropriate safeguards, the Census Bureau should
release ACS data to the broader research community for evaluation
purposes."45
42Stern, "Counting People with Disabilities."
43Lex Frieden, Chair, National Council on Disability, "Improving Federal
Disability Data," Washington, D.C., January 8, 2004. National Council on
Disability, Newsroom, Publications, 2004,
http://www.ncd.gov/newsroom/publications/2004/publications.htm (May 11,
2004).
44Cork, Cohen, and King, p. 99.
45Cork, Cohen, and King, p. 99.
Appendix V: Current Status of Unresolved Issues
Information on Key Properties of Multiyear Averages
One of the major differences between the ACS and the long form it will
replace is that the ACS will provide data for geographic areas with
populations smaller than 65,000 in terms of multiyear averages. Because of
the statistical properties of these averages and users' unfamiliarity with
them, we and many other stakeholders have identified these averages as a
major challenge for users, including federal agencies. The Census Bureau
has recognized the need for such guidance on the averages but has not made
public plans for the topics to be discussed or when the guidance will be
published.
From the 1998 conference that the Census Bureau had asked Westat to
conduct on the quality of ACS data for rural data users, the report's
authors concluded that "On the basis of the full exchange between the
Bureau and the participants, they saw no evidence of an antirural bias in
the design of the ACS."46 Nevertheless, they also concluded that the
Census Bureau needed to conduct and expand its contacts with stakeholders
and to conduct additional research on several issues we discussed in our
report, including population controls, operational aspects of nonresponse
followup, and multiyear averages. For these averages, the conference
report noted that there would be issues with small geographic areas and
the interpretation of changes in these averages:
"In discussing this issue, a number of the participants thought that
averages were
particularly problematic for those areas in which change is irregular. For
example, the
question was raised as to the meaning of `average poverty' over a 5-year
period in which
poverty rose and fell from one year to the next and, thus, the average
would have no obvious meaning."47
The report made similar comments with regard to such characteristics as
unemployment and income. Although the conference participants had
generally agreed with these concerns, the report pointed out that annually
updating the 5-year averages "will provide some insight into trends,
although turning points will be difficult to discern precisely, as will
shortterm trends."48
46Graham Kalton and others, The American Community Survey: The Quality of
Rural Data, A Report of a Conference (Rockville, Md.: Westat, June 29,
1998), p. 3.
47Kalton and others, p. 12.
48Kalton and others, p. 13.
Appendix V: Current Status of Unresolved Issues
About a year later, the Census Bureau had Westat convene another
conference, this one focusing on the use of multiyear averages. The 1999
report concluded:
"Although a 5-year moving average will generally provide reasonably
reliable cross-section
statistics for all areas, including very small communities, some care will
have to be
exercised in choosing time periods for which changes in population or
their characteristics
are measured. With 5-year averages, four-fifths of the data in a pair of
neighboring years
will be identical. The change being measured will then be one-fifth of the
difference
between the most recent year and the first year of the earlier time
period. The sampling
errors of the differences will thus be based on annual sample sizes, not
5-year averages,
and will generally be too large to make useful inferences for small areas.
The two 5-year
averages that are being compared should generally be discrete and
non-overlapping
periods, e.g., 2003-2007 and 2008-2012. These comparisons will have about
the same
reliability as changes between two censuses using data collected in the
Census long form."49
Census Bureau staff have been well aware of the difficulties of using the
new multiyear averages. The Census Bureau's Charles Alexander presented a
paper at a 2001 Statistics Canada conference on statistical methodology in
which he recognized that the multiplicity of estimates for the same
geographic area would be an issue for users (and for the Bureau). He said
that the Census Bureau's presentation would
"encourage analysts to use the same length of cumulation when comparing
areas of
different sizes . . . . For example, we would use one year for comparing
states, but would recommend 5 years for all the counties in a table
comparing large and small counties."50
Alexander noted that this approach differed from that of Kish, the
developer of the concept of a "rolling sample," who would "let us use
tables of counties with one-year estimates for large counties, 3-year
averages for medium-sized ones, and 5-year averages for small ones." He
concluded this section of the paper by saying, "It will be interesting to
see what practices data users will adopt in this regard."51
49Westat Inc., The American Community Survey: A Report on the Use of
Multi-Year Averages (Rockville, Md.: April 30, 1999), p. 12.
50Alexander, p. 6.
51Alexander, p. 6.
Appendix V: Current Status of Unresolved Issues
At the fall 2002 Census Advisory Committee Meetings, Navarro presented a
paper that Alexander had written. Focusing primarily on the quality of the
5-year averages, the paper noted advantages and shortcomings, including
that multiyear averages are not useful in all situations. For example,
"If there is little change in the population over the time covered by the
average, the
interpretation is about the same as that of a point-in-time estimate with
the advantage that
the ACS estimate is more current than the historical decennial census
long-form estimate."52
The paper provided examples with "naive" assumptions about how users
extrapolate between censuses to show that multiyear averages "work." By
implication, under other conditions, users will need guidance on when
multiyear averages can be used. The paper also did not discuss the
interpretation of changes in the multiyear averages, as in the 1999 Westat
conference report or multiple estimates, which Alexander had discussed in
his paper for the 2001 Statistics Canada conference.
In September 2002, two reports focused on issues related to the
statistical properties of multiyear averages. We published a report on
several aspects of the ACS, including the selection of questions and the
feasibility of conducting the ACS as a voluntary survey, and HUD released
a report prepared for its staff on the use of the ACS for HUD programs.53
We stated in our report that the Census Bureau evaluation would not
discuss "measures of stability of annual ACS data and ACS multiyear
averages." We recommended that, as a first step, the Census Bureau
"Analyze and report on differences between year-to-year changes for 2001
and 2002, using
the data-from ACS special supplements and the CPS at the national and
state levels-for
key economic and housing characteristics, such as the unemployment and
poverty rates, to determine the stability of the annual ACS data."54
52Charles Alexander, "A Discussion of the Quality of Estimates from the
American Community Survey for Small Population Groups," written August 26,
2002, for the Census Advisory Committee of Professional Associations
meeting, Washington, D.C., October 2-3, 2002, p. 3.
53GAO-02-956R and ORC Macro.
54GAO-02-956R, p. 25.
Appendix V: Current Status of Unresolved Issues
We also discussed the need for additional information on the
characteristics of the multiyear averages to help federal agencies make
the transition to the ACS. We specifically noted the need for information
on the selection of ACS data for geographic areas with populations larger
than 20,000 for which there will be multiple estimates. On this issue, we
stated that, "In addition, we found that the ACS development program did
not cover information about different ways to integrate the annual data
for states and large counties and the 3-and 5-year averages for smaller
counties."55
For example, federal agencies that need state data can choose to use the
annual data, multiyear averages of the annual data, or 3-year or 5-year
ACS averages. Federal agencies that also need county data can choose to
use the most recent annual data for large counties and adjust the averages
of the smaller counties to agree with annual data. Alternatively, they can
choose to use various combinations of multiyear averages. As many federal
agencies, as well as state and local governments, will be using the ACS
data for allocating funds, Census Bureau guidance would reduce the
inconsistent use of the multiple estimates.
HUD is a major user of Decennial Census long-form data for various program
applications. Its contract with ORC Macro to review how the ACS will
affect HUD programs that previously relied on the Decennial Census long
form for geographic area data resulted in a report that made two points
about the multiyear averages, in addition to raising the previously
discussed issues on the inflation adjustment to income. One of these
issues related to interpretations of changes in the multiyear averages and
their stability; the other related to the availability of multiple
estimates for the same area.
The ORC Macro report noted that year-to-year stability is important and
needs to be addressed. It warned that the "differences in the precision of
estimates or year-to-year changes in estimates can create problems for HUD
applications."56 The report used eligibility and level of benefits as an
example of what could vary because of the effect of sampling variability
on these changes. ORC Macro also stated: "The ACS will report data using
different reporting periods for different sized areas. Inconsistent or
55GAO-02-956R, p. 15. 56ORC Macro, p. vi.
Appendix V: Current Status of Unresolved Issues
multiple reporting periods can create problems for HUD applications."57
ACS data for many geographic areas will be available in terms of annual
estimates and 3- and 5-year averages, and the annual and 3-year averages
(for larger areas) will be available before estimates for smaller areas.
As a result, HUD will have to choose from multiple measures for some
geographic areas. The study noted that HUD might decide to (1) continue to
use 2000 long-form data until 2008, when the first 5-year average data
will be available for all levels of geography, or (2) use the most
recently
58
available data in all cases.
ORC Macro's report also expressed concern about the amount of annual ACS
data that the Census Bureau will release for areas with populations
smaller than 65,000, whose accuracy the Census Bureau has found does not
meet publication standards. According to the study, the Census Bureau
informed HUD that beginning in 2008, it would provide researchers and
planners a "research file" containing annual ACS data for areas of all
sizes, including census tracts. ORC Macro recommended that if the Census
Bureau does release these data, HUD consider using these "unofficial"
research file results in some of its applications. The study noted,
however, that if HUD decided to use these unofficial data but other
agencies decided not to use them, there would be no standardization across
government programs in funding allocation where the same ACS items were
used.
The most recent request for the Census Bureau to provide users with
guidance on using multiyear averages came in the July 2003 report by the
NAS Panel on Research on Future Census Methods. The panel stated that "The
Census Bureau should issue a user's guide that details the statistical
implications of the difference between point-in-time and moving average
estimates for various uses."59 In the report's executive summary, the
panel stated that "The Census Bureau must do significant work in informing
data users and stakeholders of the features and the problems of working
with moving average-based estimates."60 It also expressed particular
concern about the use of the multiyear (moving) averages in fund
allocation formulas, noting that a multiyear average
57ORC Macro, p. vi.
58Since ORC Macro's study was issued, full implementation of the ACS has
been delayed; the first 5-year averages will not be available until 2010.
59Cork, Cohen, and King, pp. 99-100.
60Cork, Cohen, and King, p. 6.
Appendix V: Current Status of Unresolved Issues
"is a smoothed estimate; by averaging a particular time period's data
observation with
those within a particular time window, the resulting estimate is meant to
follow the general
trend of the series but not be as extreme as any of the individual points.
The ramifications
of this basic concept emerge when moving average estimates are entered
into sensitive
allocation formulas or compared against strict eligibility cutoffs. A
smoothed estimate may
mask or smooth over an individual year drop in level of need, thus keeping
the locality
eligible for benefits; conversely, it may also mask individual-year spikes
in activity and thus
disqualify an area from benefits. It is clear that the use of smoothed
estimates is neither
uniformly advantageous nor disadvantageous to a locality; what is not
clear is how often major discrepancies may occur in practice."61
On another issue with multiyear averages, the panel noted, as the Westat
report had done, the issue of interpreting year-to-year changes, stating,
"It is incorrect to use annual estimates based on moving averages over
several years when
assessing change since some of the data are from overlapping time periods
and hence
identical. At the least, the results will yield incorrect estimates of the
variance of the
estimates of change. Therefore, users should be cautioned about this
aspect of the use of moving averages."62
During the past decade's development of the ACS, the Census Bureau has had
many opportunities to consult with and take account of input from
stakeholders and users in making key decision on the programs. It has (1)
sponsored NAS panels, (2) held user conferences, (3) hired consultants to
organize two conferences, (4) met regularly with its advisory committees
and other user groups, and (5) encouraged its staff to present reports at
ASA meetings and meetings of similar professional organizations. In the
past several years, we and other federal agencies have reported on the ACS
and provided recommendations to the Census Bureau. It established the ACS
Federal Agency Information Program in 2003, responding to a recommendation
we had made.63 It also announced
External Consultation
61Cork, Cohen, and King, p. 86.
62Cork, Cohen, and King, p. 87.
63GAO-02-956R, pp. 25-26. Information on the Federal Agency Information
Program is at Census Bureau, American Community Survey,
http://www.census.gov/acs/www (May 11, 2004).
Appendix V: Current Status of Unresolved Issues
last year that it was looking into establishing a partnership with the
Congress and its oversight entities.64
Despite these opportunities, many stakeholders have observed that these
consultations have not been successful. NAS noted the Census Bureau's lack
of response to recommendations in last year's report on the 2010 Census.
The Panel on Research on Future Census methods offered the following
comment by referring to a 1995 NAS report:
"Eight years later, faced with the task of offering advice on making the
vision of continuous
measurement a reality in the 2010 census, the similarity between the
arguments then and
now is uncanny. Similar, too, are the points of concern; the current panel
is hard-pressed to
improve upon the basic summary of concerns outlined by our predecessors.
We are,
however, much more sanguine that a compelling case can be made for the ACS
and that it is a viable long-form replacement in the 2010 census."65
The Census Bureau has neither responded to the panel's first interim
report in 2000 nor indicated that the recommendations were being adopted.
The Census Bureau also has not responded to recommendations and issues
raised by HUD and BLS. For example, it has not responded to HUD's
recommendations on the ACS adjustments to dollar-denominated items or to
BLS's recommendations on the ACS labor force questions. (On the issue of
dollar-denominated items, we found no indication that the Census Bureau
had ever consulted outside experts about either the methodology or the
implementation.)
Census Bureau summaries of discussion about the ACS at its Advisory
Committee meetings from October 2000 to April 2003 also indicate a lack of
responsiveness.66 During this period, committee members raised concerns
about the ACS. In particular, they made recommendations about many of the
issues we have discussed in this report, including the evaluations of ACS
and long-form comparisons, the new residence rules, independent controls,
ICPE, group quarters, and Spanish-language questionnaires. At the April
2003 meeting, ASA committee members also requested a change in the
structure of the Advisory Committee meetings,
64Census Bureau, American Community Survey Operations Plan, Release 1, p.
56.
65Cork, Cohen, and King, p. 98.
66The record of committee recommendations for the October 2003 meetings
was not available.
Appendix V: Current Status of Unresolved Issues
asking the Census Bureau to spend less time on update sessions and more
time on sessions devoted to gathering more detailed input, commentary, and
recommendations on topics the Census Bureau needs and wants advice on.
Although the Census Bureau has addressed issues related to ICPE and
Spanish-language questionnaires, the meeting summaries do not report that
it followed recommendations in other areas.
Appendix VI: Comments from the Department of Commerce
Appendix VI: Comments from the Department of Commerce
Appendix VI: Comments from the Department of Commerce
Appendix VI: Comments from the Department of Commerce
Appendix VI: Comments from the Department of Commerce
Appendix VI: Comments from the Department of Commerce
Appendix VI: Comments from the Department of Commerce
Appendix VI: Comments from the Department of Commerce
Appendix VI: Comments from the Department of Commerce
Appendix VI: Comments from the Department of Commerce
Appendix VI: Comments from the Department of Commerce
Appendix VI: Comments from the Department of Commerce
Appendix VI: Comments from the Department of Commerce
Appendix VI: Comments from the Department of Commerce
Appendix VI: Comments from the Department of Commerce
Appendix VII: GAO Contacts and Staff Acknowledgments
GAO Contacts Robert P. Parker (202) 512-9750, [email protected].
Christopher Moriarity (202) 512-5420, [email protected]
Staff Additional staff who made major contributions to this report were
Heather Von Behren, Penny Pickett, Mitchell Karpman, Michael Volpe,
Acknowledgments Andrea Levine, Patricia Dalton, and Robert Goldenkoff.
Bibliography
History of the Long Form and Mid-Decade Census
Census Bureau ACS Reports
The first section in this bibliography lists documents on the history of
the long form and mid-decade census. The remaining works are divided
between numerous types of Census Bureau reports and papers, Association of
Public Data Users papers, congressional hearings and testimony, and other
reports and papers. Recent reports from the National Academy of Sciences
are discussed in appendix II. Related GAO Products are listed in a
separate section at the end of this report.
Alexander, Charles H. "Still Rolling: Leslie Kish's `Rolling Samples' and
the American Community Survey." In Proceedings of Statistics Canada
Symposium 2001: October 16-19. Ottawa: Statistics Canada, 2002.
Anderson, Margo J., ed. Encyclopedia of the U.S. Census. Washington, D.C.:
CQ Press, 2000.
House of Representatives, Committee on Post Office and Civil Service,
Subcommittee on Census and Population. Review of Major Alternatives for
the Census in the Year 2000. Serial 102-25. Washington, D.C.: August 1,
1991.
House of Representatives, Committee on Post Office and Civil Service.
Census Confidentiality/Mid-Decade Sample Survey Bill. Report 93-246.
Washington, D.C.: June 4, 1973.
House of Representatives, Committee on Post Office and Civil Service.
Mid-Decade Censuses of Population, Unemployment, and Housing.
Report 780. Washington, D.C.: August 12, 1965.
Salvo, Joseph, and Arun Peter Lobo. The American Community Survey: Quality
of Response by Mode of Data Collection in the Bronx Test Site.
Presented at 2002 Joint Statistical Meetings, New York City, August 14,
2002.
American Community Survey Operations Plan, Release 1. Washington, D.C.:
March 2003.
Meeting 21st Century Demographic Data Needs: Implementing the American
Community Survey. Report 1. Demonstrating Operational Feasibility.
Washington, D.C.: July 2001.
Bibliography
Meeting 21st Century Demographic Data Needs: Implementing the American
Community Survey. Report 2. Demonstrating Survey Quality. Washington,
D.C.: May 2002.
Meeting 21st Century Demographic Data Needs: Implementing the American
Community Survey. Report 3. Testing the Use of Voluntary Methods.
Washington, D.C.: December 2003.
Meeting 21st Century Demographic Data Needs: Implementing the American
Community Survey. Report 4. Comparing General Demographic and Housing
Characteristics With Census 2000. Washington, D.C.: May 2004.
Meeting 21st Century Demographic Data Needs: Implementing the American
Community Survey. Report 5. Comparing Economic Characteristics With Census
2000. Washington, D.C.: May 2004.
Meeting 21st Century Demographic Data Needs: Implementing the American
Community Survey. Report 6. The 2001-2002 Operational Feasibility Report
of the American Community Survey. Washington, D.C.: May 2004.
Meeting 21st Century Demographic Data Needs: Implementing the American
Community Survey. Report 7. Comparing Quality Measures: The American
Community Survey's Three-Year Averages and Census 2000's Long Form Sample
Estimates. Washington, D.C.: June 2004.
Meeting 21st Century Demographic Data Needs: Implementing the American
Community Survey. Report 8. Comparison of the American Community Survey
Three-Year Averages and the Census Sample for a Sample of Counties and
Tracts. Washington, D.C.: June 2004.
Meeting 21st Century Demographic Data Needs: Implementing the American
Community Survey. Report 9. Comparing Social Characteristics With Census
2000. Washington, D.C.: June 2004.
Meeting 21st Century Demographic Data Needs: Implementing the American
Community Survey. Report 10. Comparing Housing Characteristics With Census
2000. Washington, D.C.: July 2004.
Bibliography
Census Bureau Advisory Committee Presentations
The presentations in this section were made by the Census Bureau's
Decennial Census Advisory Committee, Professional Association Advisory
committees, and Race and Ethnic Advisory Committees.
The ACS: Data Products to Meet User Needs. Race and Ethnic Advisory
Committee meeting, Washington, D.C., March 14, 2001.
Alexander, Charles, Alfredo Navarro, and Deborah Griffin. Update on ACS
Evaluations. Decennial Census Advisory Committee meeting, Washington,
D.C., November 5, 2001.
Gordon, Nancy. The American Community Survey. Joint Meeting of the Census
Bureau Advisory Committees, Washington, D.C., July 28, 2000.
Gordon, Nancy. The American Community Survey. Decennial Census Advisory
Committee meeting, Washington, D.C., September 21-22, 2000.
Gordon, Nancy. The American Community Survey. Race and Ethnic Advisory
Committees meeting, Washington, D.C., November 2, 2000.
Gordon, Nancy. American Community Survey Update. Decennial Census Advisory
Committee meeting, Washington, D.C., May 2, 2002.
Griffin, Deborah. An Overview of the Research and Evaluation Program for
the American Community Survey. Decennial Census Advisory Committee
meeting, Alexandria, Virginia, October 2-4, 2002.
Griffin, Deborah H. Comparing Characteristics from the American Community
Survey and Census 2000: Methodology. Census Advisory Committee of
Professional Associations meeting, Washington, D.C., April 10-11, 2003.
Navarro, Alfredo. American Community Survey: Use of Population Estimates
as Controls in the ACS Weighting. Census Advisory Committee of
Professional Associations meeting, Washington, D.C., October 23, 2003.
Navarro, Alfredo. A Discussion of the Quality of Estimates from the
American Community Survey for Small Population Groups. Census Advisory
Committee of Professional Associations meeting, Washington, D.C., October
2-3, 2002.
Weidman, Lynn, and Signe Wetrogan. Enhancing the Intercensal Population
Estimates Program with ACS Data: Summary of Research
Bibliography
Census Bureau Continuous Measurement Series
Projects. Census Advisory Committee of Professional Associations meeting,
Washington, D.C., October 23, 2003.
The memorandums listed here from the 20 in the Continuous Measurement
series are those most directly related to topics we review in this report.
Alexander, Charles H. A Continuous Measurement Alternative for the U.S.
Census. CM-10, October 28, 1993. CM-11 summarized this paper at the 1993
annual meeting of the American Statistical Association, San Francisco,
California, August 10, 1993.
Alexander, Charles H. Further Exploration of Issues Raised at the CNSTAT
Requirements Panel Meeting. CM-13. Internal Census Bureau memorandum,
Washington, D.C., January 31, 1994.
Alexander, Charles H. A Prototype Continuous Measurement System for the
U.S. Census of Population and Housing. CM-17. Presented at the annual
meeting of the Population Association of America, Miami, Florida, May 5,
1994.
Alexander, Charles H. Some Ideas for Integrating the Continuous
Measurement System into the Nation's System of Household Surveys.
CM-19A. Internal Census Bureau memorandum, Washington, D.C., January 6,
1995.
Census Bureau Internal Reports
2004 Census Test Operational Plan. Washington, D.C.: September 29, 2003.
2010 Census Decision Memorandum Series No. 5, Finalizing Content for the
100 Percent Items in the 2010 Census and the American Community Survey.
Washington, D.C.: June 3, 2004.
2010 Census Planning Memorandum Series No. 24, Action Plan: 2010 Research
and Development Planning Group on Race and Ethnic Data Collection,
Tabulation, and Editing. Washington, D.C.: June 9, 2004.
2010 Census Planning Memorandum Series No. 26, Action Plan: 2010 Research
and Development Planning Group on Special Places/Group Quarters
Development and Testing. Washington, D.C.: March 8, 2004.
Bibliography
Census 2000 Evaluation Reports
Census Bureau 2003 JSM Staff Papers
ACS-2010 Consistency Review Plan. Washington, D.C.: October 1, 2003.
American Community Survey Development Report Series Program Plan.
Washington, D.C.: rev. June 12, 2002.
Abramson, Florence. Special Place/Group Quarters Enumeration. Census 2000
Testing, Experimentation, and Evaluation Program, Topic Report No. 5. U.S.
Census Bureau, Washington, D.C., February 2004.
Adlakha, Arjun, J. Gregory Robinson, Kirsten West, and Antonio Bruce.
Assessment of Consistency of Census Data with Demographic Benchmarks at
the Subnational Level. Census 2000 Evaluation O.20. U.S. Census Bureau,
Washington, D.C., August 18, 2003.
Clarke, Sandra, John Iceland, Thomas Palumbo, Kirby Posey, and Mai
Weismantle. Comparing Employment, Income, and Poverty: Census 2000 and the
Current Population Survey. Census 2000 Auxiliary Evaluation. U.S. Census
Bureau, Washington, D.C., September 2003.
Palumbo, Thomas and Paul Siegel. Accuracy of Data for Employment Status as
Measured by the CPS-Census 2000 Match. Census 2000 Evaluation B.7. U.S.
Census Bureau, Washington, D.C., May 4, 2004.
Schneider, Paula. Content and Data Quality in Census 2000. Census 2000
Testing, Experimentation, and Evaluation Program, Topic Report No.
12. U.S. Census Bureau, Washington, D.C., January 22, 2004.
Bureau staff presented many ACS-related papers at the August 2003 Joint
Statistical Meetings in San Francisco, California. We reviewed the papers
in this section in detail because they were related to comparisons between
ACS estimates and 2000 Census results.
Boggess, Scott, and Nikki L. Graf. Measuring Education: A Comparison of
the Decennial Census and the American Community Survey.
Presented at Joint Statistical Meetings, San Francisco, California, August
7, 2003.
Dye, Jane Lawler. Grandparents Living with and Providing Care for
Grandchildren: A Comparison of Data from Census 2000 and 2000 American
Community Survey. Presented at Joint Statistical Meetings, San Francisco,
California, August 7, 2003.
Bibliography
Other Census Bureau
Staff Research Papers
Love, Susan, and Deborah Griffin. A Closer Look at the Quality of Small
Area Estimates from the American Community Survey. Presented at Joint
Statistical Meetings, San Francisco, California, August 4, 2003.
Posey, Kirby G., Edward Welniak, and Charles Nelson. Income in the
American Community Survey: Comparisons to Census 2000. Presented at Joint
Statistical Meetings, San Francisco, California, August 7, 2003.
Raglin, David A., Theresa F. Leslie, and Deborah H. Griffin. Comparing
Social Characteristics between Census 2000 and the American Community
Survey. Presented at Joint Statistical Meetings, San Francisco,
California, August 3, 2003.
Stern, Sharon M. Counting People with Disabilities: How Survey Methodology
Influences Estimates in Census 2000 and the Census 2000 Supplementary
Survey. Presented at Joint Statistical Meetings, San Francisco,
California, August 7, 2003.
Alexander, Charles H. American Community Survey Data for Economic Analysis
(October 2001). Presented at the Census Advisory Committee of the American
Economic Association meeting, Washington, D.C., October 18-19, 2001.
Alexander, Charles H. Recent Developments in the American Community
Survey. Presented at the 1998 Joint Statistical Meetings, Dallas, Texas,
August 12, 1998.
Alexander, Charles H., Sharon Brown, and Hugh Knox. American Community
Survey Data for Economic Analysis (December 2001).
Presented at the Federal Economic Statistics Advisory Committee meeting,
Washington, D.C., December 14, 2001.
Alexander, Charles H., Scot Dahl, and Lynn Weidman. Making Estimates from
the American Community Survey. Presented at the 1997 Joint Statistical
Meetings, Anaheim, California, August 13, 1997.
Alexander, Charles H., and Signe Wetrogan. Integrating the American
Community Survey and the Intercensal Demographic Estimates Program.
Presented at the 2000 Joint Statistical Meetings, Indianapolis, Indiana,
August 14, 2000.
Bibliography
Butani, Shail, Charles Alexander, and James Esposito. Using the American
Community Survey to Enhance the Current Population Survey: Opportunities
and Issues. Presented at the 1999 Federal Committee on Statistical
Methodology Research Conference, Arlington, Virginia, November 15-17,
1999.
Davis, Mary Ellen, and Charles H. Alexander, Jr. The American Community
Survey: The Census Bureau's Plan to Provide Timely 21st Century Data.
Missouri Library World, Spring 1997.
DeMaio, Theresa J., and Kristen A. Hughes. Report of Cognitive Research on
the Residence Rules and Seasonality Questions on the American Community
Survey. U.S. Bureau of the Census, Statistical Research Division,
Washington, D.C., July 2003.
Love, Susan, Donald Dalzell, and Charles Alexander. Constructing a Major
Survey: Operational Plans and Issues For Continuous Measurement. Presented
at the 1995 Joint Statistical Meetings, Orlando, Florida, August 16, 1995.
Nelson, Charles, and Kathleen Short. The Distributional Implications of
Geographic Adjustment of Poverty Thresholds. U.S. Bureau of the Census,
Housing and Household Economics Statistics Division, Washington, D.C.,
December 2003.
Posey, Kirby G., and Edward Welniak. Income in the ACS: Comparisons to the
1990 Census. Presented at the American Community Survey Symposium,
Suitland, Maryland, March 1998.
Salvo, Joseph, and Arun Peter Lobo. The American Community Survey: Quality
of Response by Mode of Data Collection in the Bronx Test Site.
Presented at the 2002 Joint Statistical Meetings, New York, August 14,
2002.
Smith, Amy Symens. The American Community Survey and Intercensal
Population Estimates: Where Are the Crossroads? Technical Working Paper
31, U.S. Census Bureau, Population Division, Washington, D.C., December
1998.
Association of Public Davis, Mary Ellen. The American Community Survey
Data Products, Alexandria, Virginia, October 20, 2003.
Data Users Papers
Bibliography
Congressional Hearings and Testimony
Other Reports and Papers
Gage, Linda, State of California, Department of Finance. American
Community Survey: Research by the Data User Community. Alexandria, Va.:
October 20, 2003.
Petroni, Rita. How Do 3-Year Averages from the ACS Compare to Census 2000
Data? (Preliminary Results). Alexandria, Va.: October 20, 2003.
Salvo, Joseph, City of New York, Planning Department. American Community
Survey: Research by the Data User Community. Alexandria, Va.: October 20,
2003.
Scarr, Harry A. Deputy Director, Census Bureau. Continuous Measurement.
Association of Public Data Users, Washington, D.C.; October 16, 1994.
Barron, William Jr., Acting Director, U.S. Bureau of the Census, before
the U.S. House of Representatives, Committee on Government Reform,
Subcommittee on the Census. The Census Bureau's Proposed American
Community Survey (ACS), Serial 107-9. Washington, D.C.: June 13, 2001.
Kincannon, Charles Louis, Director, U.S. Bureau of the Census, before the
U.S. House of Representatives, Subcommittee on Technology, Information
Policy, Intergovernmental Relations, and the Census. The American
Community Survey: The Challenges of Eliminating the Long Form from the
2010 Census, Serial 108-97. Washington, D.C.: May 13, 2003.
Prewitt, Kenneth, Director, U.S. Bureau of the Census, before the U.S.
House of Representatives, Committee on Government Reform, Subcommittee on
the Census. House Hearing on ACS July 20, 2000. The American Community
Survey: A Replacement for the Census Long Form? Serial 106-246.
Washington, D.C.: July 20, 2000.
Kalton, Graham, and others. The American Community Survey: The Quality of
Rural Data, Report of a Conference. Rockville, Md.: Westat, June 29, 1998.
Nardone, Thomas, and others. Examining the Discrepancy in Employment
Growth between the CPS and the CES. Washington, D.C.: FESAC, October 17,
2003.
Bibliography
National Council on Disability. Improving Federal Disability Data.
Washington, D.C.: January 9, 2004.
ORC Macro. The American Community Survey: Challenges and Opportunities for
HUD. Calverton, Md.: September 27, 2002.
Vroman, Wayne. Comparing Labor Market Indicators from the CPS and ACS.
Washington, D.C.: Urban Institute, September 2003.
Westat Inc. The American Community Survey: A Report on the Use of
Multi-Year Averages. Rockville, Md.: April 30, 1999.
Related GAO Products
2010 Census: Cost and Design Issues Need to Be Addressed Soon.
GAO-04-37. Washington, D.C.: January 15, 2004.
Medicaid Formula: Differences in Funding Ability among States Often Are
Widened. GAO-03-620. Washington, D.C.: July 10, 2003.
Formula Grants: 2000 Census Redistributes Federal Funding Among States.
GAO-03-178. Washington, D.C.: February 24, 2003.
Major Management Challenges and Program Risks: Department of Commerce.
GAO-03-97. Washington, D.C.: January 1, 2003.
The American Community Survey: Accuracy and Timeliness Issues.
GAO-02-956R. Washington, D.C.: September 30, 2002.
Legal Authority for American Community Survey. B-289852. Washington, D.C.:
April 4, 2002.
Medicaid Formula: Effects of Proposed Formula on Federal Shares of State
Spending. GAO/HEHS-99-29R. Washington, D.C.: February 19, 1999.
Decennial Census: Overview of Historical Census Issues.
GAO/GGD-98-103. Washington, D.C.: May 1, 1998.
Poverty Measurement: Adjusting for Geographic Cost-of-Living Difference.
GAO/GGD-95-64. Washington, D.C.: March 9, 1995.
Status of the Statistical Community after Sustaining Budget Reductions.
GAO/IMTEC-84-17. Washington, D.C.: July 18, 1984.
GAO's Mission
Obtaining Copies of GAO Reports and Testimony
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
The fastest and easiest way to obtain copies of GAO documents at no cost
is through GAO's Web site (www.gao.gov). Each weekday, GAO posts newly
released reports, testimony, and correspondence on its Web site. To have
e-mail you a list of newly posted products every afternoon, go to
www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
To Report Fraud, Contact:
Waste, and Abuse in Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: [email protected] Programs Automated answering system: (800)
424-5454 or (202) 512-7470
Congressional Gloria Jarmon, Managing Director, [email protected] (202)
512-4400 U.S. Government Accountability Office, 441 G Street NW, Room 7149
Relations Washington, D.C. 20548
Public Affairs Jeff Nelligan, Managing Director, [email protected] (202)
512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
*** End of document. ***