2000 Census: Coverage Measurement Programs' Results, Costs, and
Lessons Learned (29-JAN-03, GAO-03-287).
To help measure the quality of the 2000 Census and to possibly
adjust for any errors, the U.S. Census Bureau (Bureau) conducted
the Accuracy and Coverage Evaluation (A.C.E.) program. However,
after obligating around $207 million for A.C.E. and its
predecessor program, Integrated Coverage Measurement (I.C.M.),
from fiscal years 1996 through 2001, the Bureau did not use
either program to adjust the census numbers. Concerned about the
amount of money the Bureau spent on I.C.M. and A.C.E. programs
and what was produced in return, the subcommittee asked us to
review the objectives and results of the programs, the costs of
consultants, and how best to track future coverage measurement
activities.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-03-287
ACCNO: A06024
TITLE: 2000 Census: Coverage Measurement Programs' Results,
Costs, and Lessons Learned
DATE: 01/29/2003
SUBJECT: Census
Data collection
Population statistics
Program evaluation
Cost analysis
Cost control
1990 Decennial Census
2000 Decennial Census
Census Bureau Accuracy and Coverage
Evaluation Program
Census Bureau Integrated Coverage
Management Program
2010 Decennial Census
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-03-287
A
Report to the Chairman, Subcommittee on Civil Service, Census, and Agency
Organization, Committee on Government Reform, House of Representatives
January 2003 2000 CENSUS Coverage Measurement Programs* Results, Costs,
and Lessons Learned
GAO- 03- 287
Letter
January 29, 2003 The Honorable Dave Weldon, M. D. Chairman Subcommittee on
Civil Service,
Census, and Agency Organization Committee on Government Reform House of
Representatives
Dear Mr. Chairman: To assess the quality of population data for the 2000
Census and to possibly adjust for any errors, the U. S. Census Bureau
(Bureau) conducted the Accuracy and Coverage Evaluation (A. C. E.)
program. The A. C. E. program was first included in Bureau program
documents in November 1998 and funded for fiscal years 2000 through 2002
with proposed funding through December 31, 2002. Its predecessor,
Integrated Coverage Measurement
(I. C. M.), began in May 1995 and was funded by the Bureau for fiscal
years 1996 through 1999. The Bureau obligated about $207 million to both
programs from fiscal years 1996 through 2001, 1 which was about 3 percent
of the $6.5 billion total estimated cost of the 2000 Census. However,
neither program was used to adjust the 2000 Census population count.
Concerned about the amount of money the Bureau spent on both programs as
well as what was received in return for its investment, you and former
Vice Chairman Dan Miller asked us to examine (1) the Bureau*s objectives
for the I. C. M./ A. C. E. programs and the extent to which those
objectives were met, (2) the cost of consultants and technical studies,
and (3) ways to track the costs of coverage measurement activities in
future censuses. This report responds to that request.
In October 2002, we issued a report that provides additional information
on the cost of the I. C. M./ A. C. E. programs. 2 Both reports are part of
our ongoing series on the results of the 2000 Census and the lessons
learned for planning a more cost- effective census in 2010. (See the
Related GAO
1 At the time of our report, obligated costs after fiscal year 2001 were
not final. 2 U. S. General Accounting Office, 2000 Census: Complete Costs
of Coverage Evaluation Programs Are Not Available, GAO- 03- 41
(Washington, D. C.: Oct. 31, 2002).
Products section at the end of this report for the assessments issued to
date.) Results in Brief The coverage measurement programs the Census
Bureau planned for the
2000 Census* I. C. M. and its successor, A. C. E.* did not meet their
intended objectives. Although the Bureau designed I. C. M. to measure
census coverage; generate data for apportionment, redistricting, and
federal programs using statistical sampling and estimation; and produce a
*onenumber* census based on statistical sampling and estimation, the
Bureau abandoned the program following a Supreme Court ruling that the
Census Act prohibited the use of statistical sampling to generate
population data for reapportioning the House of Representatives. 3 Because
its replacement* A. C. E.* did not provide a reliable measure of census
accuracy in time to meet legally mandated deadlines for releasing
redistricting data, the Bureau decided against using it to adjust the
census data for nonapportionment purposes.
The difficulties the Bureau encountered in trying to implement I. C. M.
and A. C. E. underscore the importance of (1) developing a coverage
measurement methodology that is both technically and operationally
feasible, (2) determining the level of geography at which coverage
measurement is intended, (3) keeping stakeholders, particularly Congress,
informed of the Bureau*s plans, and (4) adequate testing. It will be
important for the Bureau to address these lessons learned as its planning
efforts for the 2010 Census continue. Those plans currently call for a
coverage measurement program to evaluate the accuracy of the census,
whereas the issue of whether coverage measurement will be used to adjust
the numbers has not yet been resolved.
Concerning the cost of contracts, including the consultants and technical
studies related to the I. C. M./ A. C. E. programs, we identified about
$22.3 million of obligated amounts for over 170 vendors for fiscal year
1996 through fiscal year 2001 from unaudited Bureau financial management
reports. This amount does not represent the complete contract costs for
the I. C. M./ A. C. E. programs for three reasons. First, the Bureau
considered costs from earlier years to be part of its general research and
development efforts and did not assign unique project codes to identify I.
C. M./ A. C. E. 3 Department of Commerce v. United States House of
Representatives, 525 U. S. 316 (1999).
programs and related costs in its financial management system. Second,
although $182,000 of fiscal year 1996 obligated costs were identified in
the Bureau*s financial management systems as contract costs for an I. C.
M. special test, the Bureau did not consider these costs as part of the
I. C. M./ A. C. E. programs and classified these costs as general
research. We disagreed with the Bureau on this point and have included
this amount in our report as part of the I. C. M./ A. C. E. contract costs
that we could identify from Bureau records. Finally, certain costs, such
as program evaluations, were Bureau- wide in nature, and the portion
attributable to I. C. M./ A. C. E could not be separated out.
The Bureau*s ability to track future costs of coverage measurement
activities is primarily dependent on its ability to (1) ensure that its
financial management system accurately and completely captures the
accounting and reporting of project codes, (2) design its project codes to
capture coverage measurement activities as early in the planning process
as possible, even though the activities* names may change as the programs
evolve, and (3) correctly charge the project codes established. Although
the Bureau has never used the results of its coverage
measurement programs to adjust census numbers, we believe that an
evaluation of the accuracy of the census is essential given the importance
of the data, the need to know the nature of any errors, and the cost of
the census overall. Whether the results of the evaluation should be used
to adjust the census is still an open question, the answer to which should
involve discussions between the Bureau, Congress, and other stakeholders,
and be based on detailed data and a convincing demonstration of the
feasibility of the Bureau's proposed approach. Regardless of the outcome
of the decision, it is critical that it be made soon so that the Bureau
can proceed with its planning. The longer the Bureau goes without a firm
decision on the role of coverage measurement, the greater the risk of
wasted resources and disappointing results. In light of the challenges
facing the Bureau as it prepares for the next decennial census in 2010, we
recommend that the Secretary of Commerce direct the U. S. Census Bureau to
in conjunction with Congress and other stakeholders, decide soon on
whether and how coverage measurement will be used in the 2010 Census;
consider incorporating lessons learned from its coverage measurement
experience during the 2000 Census, such as (1) demonstrating both the
operational and technical feasibility of its coverage measurement methods,
(2) determining the level of geography at which coverage can be reliably
measured, (3) keeping Congress and other stakeholders
informed of its plans, and (4) adequately testing coverage measurement
prior to full implementation; and
ensure that the Bureau*s financial management systems can capture and
report program activities early in the decennial process and that project
costs are monitored for accuracy and completeness.
The Secretary of Commerce forwarded written comments from the Bureau of
the Census on a draft of this report (see app. I). The Bureau agreed with
our recommendations noting that they were important steps that should be
followed in the development of a coverage measurement methodology for the
2010 Census. However, the Bureau maintained that most of these steps such
as keeping Congress and other stakeholders informed of its plans were
followed for the 2000 Census. We disagree because, as noted in the report,
the lack of information contributed to stakeholders* skepticism
surrounding the Bureau*s plans. The Bureau also took exception to the way
we presented our conclusions concerning its ability to properly classify
certain costs associated with the development of the Bureau*s coverage
measurement programs. Our perspective on the Bureau*s position is
detailed in the *Agency Comments and Our Evaluation* section at the end of
this report. Background The Bureau puts forth tremendous effort to conduct
a complete and
accurate count of the nation*s population. However, some degree of error
in the form of persons missed or counted more than once is inevitable
because of limitations in census- taking methods. Because census results
are used, among other purposes, to apportion Congress, redraw
congressional districts, and allocate federal aid to state and local
governments, the size and demographic composition of these coverage errors
have become increasingly sensitive since the Bureau was first able to
generate detailed data on them during the 1980 Census. However, the
Bureau has never used the results of its coverage measurements to correct
estimated coverage errors.
The Bureau first attempted to measure the accuracy of the census in the
1940s when it compared the census numbers to birth and death certificates
and other administrative data using a procedure called demographic
analysis. Modern coverage measurement began with the 1980 Census when
the Bureau compared census figures to the results of an independent sample
survey of the population. Using statistical methods, the Bureau generated
detailed measures of the differences among undercounts of particular
ethnic, racial, and other groups. In the months that followed, many
lawsuits were filed, most contending that the results of the 1980 coverage
measurement should have been used to adjust the census. However, the
Bureau designed the evaluation to measure errors, not to
correct the census results, and the Director of the Census Bureau decided
against adopting the adjusted numbers, as they were deemed flawed due to
missing and inaccurate data.
The quality of the coverage measurement data improved for the 1990 Census,
and the Bureau recommended statistically adjusting the results. However,
the Secretary of Commerce determined that the evidence to support an
adjustment was inconclusive and decided not to adjust the 1990 Census. The
adjustment decision was complicated by the fact that the 1990 Census
figures had already been released when the coverage measurement results
became available in the spring of 1991. The Secretary of Commerce was
concerned that two sets of numbers* the actual census results and the
adjusted figures* could create confusion and might allow political
considerations to play a part in choosing between sets of numbers when the
outcome of the choices, such as congressional apportionment, could be
known in advance of a decision.
Scope and To determine the objectives of 2000 Census I. C. M./ A. C. E.
programs and
Methodology their results, we reviewed Bureau and other documents that
included
Federal Register notices; Census Operational Plans; reports to Congress;
internal memorandums; research and feasibility studies; and reports of the
Executive Steering Committee for Accuracy and Coverage Policy (ESCAP) I
and II, which assessed the results of the A. C. E. program and recommended
how they should be used.
To determine costs for consultants and technical studies for 2000 Census
I. C. M./ A. C. E. programs, we focused on object class code 25 from the
financial management reports to obtain contract data. With Bureau
assistance, we identified I. C. M./ A. C. E. project accounts and analyzed
amounts by fiscal year using the financial management reports generated
by the Department of Commerce*s Administrative Management System (CAMS).
We reviewed and analyzed obligated and expended data for all coverage
measurement programs that existed during the 2000 Census for
fiscal years 1991 to 2003. We did not audit financial data provided by the
Bureau.
To determine ways to track future costs, we reviewed current Bureau
financial management reports and considered established standards of
accounting, auditing, and internal controls. In addition, we met with key
Bureau officials to discuss the results of our analysis and obtain their
observations and perspectives. The limitations we encountered in the scope
of our work on this assignment are as follows.
We were unable to determine the complete contractual and technical
studies costs of the I. C. M./ A. C. E. programs because the Bureau
considered any I. C. M./ A. C. E.- related costs from fiscal years 1991
through 1995 as part of its general research and development programs
and thus did not separately track these costs. Although some costs were
tracked in fiscal year 1996, the Bureau still considered these costs as
research and development and did not include these costs as I. C. M./ A.
C. E. program costs.
We were unable to identify I. C. M./ A. C. E. portions of costs from
projects that covered the entire census, such as the 2000 Census
Evaluation program. We did not evaluate the propriety of contracts for
I. C. M./ A. C. E.
programs. Our work was performed in Washington, D. C., and at U. S. Census
Bureau headquarters in Suitland, Maryland, from June 2002 through October
2002 in accordance with generally accepted government auditing standards.
On
January 7, 2003, the Secretary of Commerce provided written comments on a
draft of this report. We address these comments in the *Agency Comments
and Our Evaluation* section, and have reprinted them in
appendix I.
Coverage In planning the 2000 Census, the Bureau developed a new coverage
Measurement measurement program, I. C. M., that was designed to address
the major
shortcomings of the 1990 coverage measurement program. However, as
Programs for the 2000
shown in table 1, much like similar programs in earlier censuses, the
Census
Bureau did not use I. C. M. and its successor program, A. C. E., to adjust
the Did Not Meet Bureau
census because of legal challenges, technical obstacles, and the inability
to resolve uncertainties in the data in time to meet the deadlines for
releasing Objectives the data.
Table 1: Coverage Measurement Programs Did Not Achieve Objectives
Objectives Program/ objectives met? Reasons
I. C. M.
Measure census coverage. No.
Program was canceled following January 1999 U. S. Supreme Court Generate
data for apportionment,
No. ruling that the Census Act prohibits the redistricting, and federal
program
use of sampling to produce counts purposes using statistical
used to apportion seats in the U. S. sampling and estimation. House of
Representatives. The Bureau then replaced I. C. M. with the A. C. E.
Produce a *one- number* census. No.
program.
A. C. E.
Measure census coverage. No.
A. C. E. results were not used because of uncertainties surrounding the
Generate data needed for
No. accuracy of the results and the inability redistricting and federal
program
to resolve them in time to meet legally purposes statistical methods.
mandated deadlines for releasing data. Source: GAO. Note: This table
reflects GAO*s analysis of U. S. Census Bureau data.
The Bureau Canceled In designing I. C. M., the Bureau*s goal was to
produce a single, consolidated I. C. M. in Response to a
count or *one- number* census and thus avoid the controversy of having
Supreme Court Ruling
two sets of census results as occurred during the 1990 Census. Thus, as
shown in table 1, the objectives of I. C. M. were to (1) measure census
coverage, (2) generate, using statistical sampling and estimation methods,
the detailed data required for apportionment, congressional redistricting,
and federal program purposes, and (3) produce a one- number census.
The Bureau*s plans for I. C. M. emerged in response to the unsatisfactory
results of the 1990 Census. Although the 1990 headcount was, at that time,
the most costly in U. S. history, it produced data that were less accurate
than those from the 1980 Census. The disappointing outcome was due in
large part to the Bureau*s efforts to count housing units that did not
mail
back their census questionnaires. The operation, known as nonresponse
follow- up, where enumerators visited and collected information from each
nonresponding housing unit, proved to be costly and error- prone when a
higher- than- expected workload and a shortage of enumerators caused the
operation to fall behind schedule. The final stages of nonresponse
followup were particularly problematic. Indeed, while enumerators finished
90 percent of the follow- up workload within 8 weeks (2 weeks behind
schedule), it took another 6 weeks to resolve the remaining 10 percent.
Moreover, in trying to complete the last portion of nonresponse follow- up
cases, the Bureau accepted less complete responses and information from
nonhousehold members such as neighbors, which may have reduced the quality
of the data.
In the years following the 1990 Census, Congress, the Bureau, several
organizations, and GAO, concluded that fundamental design changes were
needed to reduce census costs and improve the quality of the data. In
response, the Bureau reengineered a number of operations for the 2000
Census.
For example, to save time and reduce its nonresponse follow- up workload,
the Bureau planned to enumerate a sample of the last remaining portion of
nonresponse follow- up cases instead of visiting every nonresponding
household as it had done in previous censuses. To adjust for enumeration
errors, the Bureau developed I. C. M., which was intended to reconcile the
original census figures with data obtained from a separate, independent
count of a sample of 750,000 housing units using a statistical process
called Dual System Estimation. The Bureau believed that this approach
offered
the best combination of reduced costs, improved accuracy expected at
various geographic levels, and operational feasibility.
However, concerned about the legality of the Bureau*s planned use of
sampling and estimation, members of Congress challenged the Bureau*s use
of I. C. M. in court. In January 1999, the Supreme Court ruled that the
Census Act 4 prohibited the use of statistical sampling to generate
population data for reapportioning the House of Representatives. 5
A. C. E. Did Not Meet Bureau Following the Supreme Court ruling, the
Bureau planned to produce
Objectives apportionment numbers using traditional census- taking methods,
and
provide statistically adjusted numbers for nonapportionment uses of the
data such as congressional redistricting and allocating federal funds. The
Bureau initiated the A. C. E. program, which was designed to take a
national sample of approximately 300,000 housing units to evaluate
coverage errors among different population groups and statistically
correct for them. Thus, as shown in table 1, the Bureau*s objectives for
A. C. E. were to (1) measure how many people were missed in the census and
how many were erroneously included and (2) produce the detailed data
required in time for redistricting and federal program purposes.
However, while the Bureau generally conducted A. C. E. in accordance with
its plans, 6 the Bureau later determined that the A. C. E. results did not
provide a reliable measure of census accuracy and could not be used to
adjust the nonapportionment census data.
4 13 U. S. C. 195. 5 Department of Commerce v. United States House of
Representatives 525, U. S. 316 (1999). 6 See, for example, U. S. General
Accounting Office, 2000 Census: Coverage Evaluation Interviewing Overcame
Challenges, but Further Research Needed, GAO- 02- 26 (Washington, D. C.:
Dec. 31, 2001), and 2000 Census: Coverage Evaluation Matching Implemented
as Planned, but Census Bureau Should Evaluate Lessons Learned, GAO- 02-
297 (Washington, D. C.: Mar. 14, 2002).
The first decision against A. C. E. occurred in March 2001, when the
Acting Director of the Census Bureau recommended to the Secretary of
Commerce that the unadjusted census data be used for redistricting
purposes. He cited as a primary reason an apparent inconsistency between
the population growth over the prior decade, as implied by A. C. E.
results,
and demographic analysis, which estimated the population using birth,
death, and other administrative records. The inconsistency raised the
possibility of an unidentified error in either the A. C. E. or census
numbers. He reported that the inconsistency could not be resolved prior to
April 1, 2001, the legally mandated deadline for releasing redistricting
data. 7
The second decision against A. C. E. came in October 2001 when, based on a
large body of additional research, ESCAP decided against adjusting census
data for allocating federal aid and other purposes, because A. C. E.
failed to identify a significant number of people erroneously included in
the census, and other remaining uncertainties. According to Bureau
officials, it might be possible to use adjusted data to produce
intercensal population estimates for federal programs that require this
information; however, the Bureau would need to revise the A. C. E. results
before any use of the data could be considered.
The Bureau*s Experience in Although I. C. M. and A. C. E. did not meet
their formal objectives, they did
Implementing produce a body of important lessons learned. As the Bureau*s
current Coverage Measurement
approach for the 2010 Census includes coverage measurement to assess
Programs
the accuracy of the census (but not necessarily to adjust the numbers
themselves), it will be important for the Bureau to consider these lessons
Highlights Important
as its planning efforts continue. The lessons include (1) developing a
Lessons Learned
coverage measurement methodology that is both technically and
operationally feasible, (2) determining the level of geography at which
coverage measurement is intended, (3) keeping stakeholders, particularly
Congress, informed of the Bureau*s plans, and (4) adequately testing the
eventual coverage measurement program. 1. A. C. E. demonstrated
operational, but not technical feasibility. According to Bureau officials,
an important result of the A. C. E. program was that it demonstrated, from
an operational perspective
only, the feasibility of conducting a large independent field check on 7
March 1, 2001, Memorandum to Secretary Donald Evans from Acting Director
William Barron, Jr. (ESCAP I decision memo.)
the quality of the census. The Bureau canvassed the entire A. C. E. sample
area to develop an address list, collected census response data for
persons living in the sample areas on census day, and conducted an
operation to try and match A. C. E. respondents to census respondents, all
independent of the regular census operations and within required time
frames.
Our separate reviews of two of these operations* interviewing respondents
and matching A. C. E. and census data* while raising questions about the
impact on final A. C. E. results due to apparently small operational
deviations, also concluded that the Bureau implemented those two
operations largely as planned. 8 Nevertheless, while the Bureau
demonstrated that it could execute
A. C. E. field operations using available resources within required time
frames, as the Bureau has noted, feasibility also consists of a technical
component* that is, whether the A. C. E. methodology would improve the
accuracy of the census. Although the Bureau clearly stated in its
justification for A. C. E. that the effort would make the census more
accurate, as noted earlier, because of unresolved data discrepancies, its
experience in 2000 proved otherwise. Moreover, according to the Bureau,
because the A. C. E. was designed to correct a census with a net coverage
error similar to that observed in previous censuses, the Bureau commented
that applying the methodology to the historically
low levels of net error observed in the 2000 Census represented a unique
and unanticipated challenge for A. C. E. Thus, it will be important for
the Bureau to refine its coverage measurement methodology to ensure that
it is technically feasible.
8 See GAO- 02- 26 and GAO- 02- 297.
2. The level of geography at which the Bureau can successfully measure
coverage is unclear. Since the October 2001 decision to not rely on
adjusted census data for nonapportionment and nonredistricting purposes,
Bureau officials have told us that they now doubt whether
census data can reliably be improved down to the level of geography for
which A. C. E. was intended to improve the accuracy* the census tract
level (neighborhoods that typically contain around 1,700 housing units and
4,000 people). The Bureau*s current position differs from that taken in
2000, when it reported to Congress that it expected accuracy at the tract
level to be improved, on average, by A. C. E. statistically adjusting
numbers at an even lower level of geography* the census
block level. 9 Uncertainty in the level of geography at which accuracy is
to be measured or improved can affect the overall design of coverage
measurement, as well as its technical feasibility. Therefore, it will be
important for the Bureau to determine the level of geography at which it
intends to measure accuracy as it decides the role and design of
future coverage measurement programs. 3. Keeping stakeholders informed is
essential. Throughout the 1990s,
Congress and other stakeholders, including GAO, expressed concerns about
the Bureau*s planned use of sampling and statistical estimation procedures
to adjust the census. A key cause of this skepticism was the Bureau*s
failure to provide sufficiently detailed data on the effects that I. C. M.
would have at different levels of geographic detail. Information was also
lacking on the various design alternatives being considered,
their likely implications, and the basis for certain decisions. As a
result, it was difficult for Congress and other stakeholders to support
the Bureau*s coverage measurement initiatives.
9 U. S. Census Bureau, *Report to Congress* The Plan for Census 2000,* 44-
46 and Accuracy and Coverage Evaluation* Statement on the Feasibility of
Using Statistical Methods to Improve the Accuracy of the Census 2000 (June
2000), 19, fn. 19.
For example, on September 24, 1996, the House Committee on Government
Reform and Oversight issued a report that criticized the Bureau*s
initiatives for sampling and statistical estimation. Among other things,
the Committee found that the Bureau had not clarified issues of accuracy,
particularly for small geographic areas, raised by the sampling
initiative. Congress*s perspective on the process was later reflected in
its enactment of legislation in 1997 that included provisions requiring
the Department of Commerce to provide Congress with comprehensive
information on its planned use of statistical estimation within 30 days.
10
4. Adequate testing of coverage measurement methodologies is critical.
Although the Bureau conducted a dress rehearsal for the census in three
locations across the country that was intended to demonstrate the overall
design of the census, the 1998 operation did not reveal the problems that
the Bureau encountered in dealing with the discrepancies between the 2000
A. C. E. results and its benchmarks. According to Bureau officials, this
was partly because the sites were not representative of the nation at
large. Additionally, as a result of a
compromise between Congress and the administration to simultaneously
prepare for a nonsampling census, the I. C. M. was tested at only two of
the three dress rehearsal sites* an urban area and an Indian reservation*
but was not tested in a rural location as was originally planned. An
earlier test in 1995 was also not comprehensive in that it did not test a
sampling operation designed to help determine whether nonresponse follow-
up of the magnitude projected by the Bureau*s current plan could be
completed in time for the I. C. M to be done on schedule.
I. C. M./ A. C. E. From fiscal year 1996 through fiscal year 2001, the
Bureau obligated about
$207 million for I. C. M./ A. C. E. activities. 11 As shown in table 2, of
that Contractor Costs Are $207 million, we identified about $22.3 million
(11 percent) in obligated Not Complete
amounts for contracts involving more than 170 vendors. These contracts
were primarily for technical advisory and assistant services, computer
systems support, and training.
10 Pub. L. 105- 18, Title VIII, June 12, 1997. 11 At the time of our
report, obligated costs after fiscal year 2001 were not final.
Table 2: Census 2000 I. C. M./ A. C. E. Contractor Costs for Fiscal Years
1996 through 2001 (Dollars in thousands) Project description FY96 FY97
FY98 FY99 FY00 FY01 Total
A. C. E. coverage management 0 0 0 0 $1,458 0 $1,458
A. C. E. operations 0 0 0 0 249 ($ 2) 247
I. C. M. collection 0 0 $201 $41 6,272 467 6,981 Framework 3 total 0 0 201
41 7,979 465 8,686
I. C. M. procedures and training 0 $249 594 1, 065 1,766 814 4,488
I. C. M. processing 0 0 0 2, 412 287 574 3,273 Framework 5 total 0 249 594
3, 477 2,053 1, 388 7,761
I. C. M. dress rehearsal 0 0 502 20 0 (10) 512
I. C. M. special test $182 628 0 0 0 0 810
I. C. M. coverage measurement 0 409 1, 257 1, 991 (130) 999 4,526
Framework 6 total 182 1,037 1, 759 2, 011 (130) 989 5,848 Total $182
$1,286 $2, 554 $5, 529 $9,902 $2,842 $22,295
Source: GAO. Note: This table reflects GAO*s analysis of U. S. Census
Bureau financial management reports.
Although the Bureau tracked some costs of contracts for the I. C. M./ A.
C. E. programs, we found that the $22.3 million did not represent the
complete contractor costs of the programs because of the following three
factors.
First, the Bureau only tracked the contractor costs associated with
conducting the I. C. M./ A. C. E. programs, which covers the period from
fiscal year 1997 through 2003. Although life cycle costs for the 2000
Census cover a 13- year period from fiscal years 1991 through 2003, senior
Bureau officials said that the I. C. M./ A. C. E. program was not viable
for implementation until fiscal year 1997. Therefore, the Bureau
considered contractor costs from earlier years as part of its general
research and development programs, and the Bureau did not assign unique
project codes to identify I. C. M./ A. C. E. programs and related costs in
its financial management system.
Second, although $182,000 of fiscal year 1996 obligated contractor costs
were identifiable in the Bureau*s financial management system as an I. C.
M. special test, the Bureau did not consider these costs as part of the I.
C. M./ A. C. E. programs. Instead, these costs were considered general
research and development. However, because the Bureau separately
identified these costs as I. C. M. program contractor costs, we have
included the $182,000 as part of the I. C. M./ A. C. E. program contractor
costs in this report.
Finally, we were unable to identify the I. C. M./ A. C. E. portions of
costs that were part of other programs. For example, in late fiscal year
2000 and after, the Bureau did not separate A. C. E. evaluations from its
other
2000 Census evaluations in its financial management systems. Bureau
officials stated that the contracts for evaluations included overall 2000
Census and A. C. E. evaluations, and did not have a separate code
identifying A. C. E. costs. Tracking Future During the 2000 Census, the
Bureau, its auditors, and GAO, found extensive Coverage
weaknesses in the Bureau*s financial management system, the components of
which include hardware, software, and associated personnel. The
Measurement Costs
weaknesses included difficulties in providing reliable and timely
financial information to manage current government operations and
preparing financial statements and other reports. Together, they affected
the completeness, accuracy, and timeliness of data needed for informed
management decisions and effective oversight. In light of these
weaknesses, the Bureau*s ability to track future costs of coverage
measurement activities will largely depend on three factors.
First, a sound financial management system is critical. As discussed in
our December 2001 report, the Bureau*s core financial management system,
CAMS, had persistent internal control weaknesses in fiscal year 2000. 12
In its latest financial report, the Bureau indicated that these weaknesses
have continued through fiscal year 2001. 13 The Bureau expects to issue
its fiscal year 2002 financial report shortly.
Second, it would be important to set up project codes to capture
coverage measurement activities as early in the planning process as
possible. The Bureau did not set up a specific project code to identify I.
C. M. program costs until 1996 because, according to Bureau officials, 12
U. S. General Accounting Office, 2000 Census: Analysis of Fiscal Year 2000
Budget and
Internal Control Weaknesses at the U. S. Census Bureau, GAO- 02- 30
(Washington, D. C.: Dec. 28, 2001). 13 U. S. Census Bureau, 2001 Financial
Report (Suitland, Md.: May 2002).
the I. C. M. program was not viable until 1997 and all costs up to this
point were considered general research.
Finally, it would be important for Bureau personnel to correctly charge
the project codes established for the coverage measurement program
activities. During the 2000 Census, for example, while the Bureau
established a project code and a budget for the remote Alaska enumeration,
the project costs were erroneously charged to and commingled with a
project code for enumerating special populations. As a result, the actual
costs for remote Alaska enumeration were reported by the Bureau*s
financial management system as zero and are unknown,
while enumerating special population costs are overstated. Conclusions The
Bureau*s 2000 Census coverage measurement programs did not
achieve their primary objectives of measuring the accuracy of the census
and adjusting the results because of legal challenges, technical hurdles,
and questionable data. However, beyond these formal objectives, there
emerged several important lessons learned that Bureau managers should
consider because current plans for the 2010 Census include coverage
measurement. At the same time, it will also be important for the Bureau to
be capable of fully tracking the money it spends on coverage measurement
and other census activities so that Congress and other stakeholders can
hold the Bureau accountable for achieving intended results. Although the
Bureau has never used the results of its coverage
measurement programs to adjust census numbers, we believe that an
evaluation of the accuracy and completeness of the census is critical
given the many uses of census data, the importance of identifying the
magnitude and characteristics of any under- and overcounts, and the cost
of the census overall. Less clear is whether the results of the coverage
measurement should be used to adjust the census. Any Bureau decisions on
this matter should involve close consultation with Congress and other
stakeholders, and be based on detailed data and a convincing demonstration
of the feasibility of the Bureau's proposed approach. Whatever the
decision, it is imperative that it be made soon so that the Bureau can
design appropriate procedures and concentrate on the business of counting
the nation*s population. The longer the 2010 planning process proceeds
without a firm decision on the role of coverage measurement, the greater
the risk of wasted resources and disappointing results.
Recommendations for To help ensure that any future coverage measurement
efforts achieve their
Executive Action intended objectives and costs can be properly tracked, we
recommend that
the Secretary of Commerce direct the Bureau to in conjunction with
Congress and other stakeholders, come to a
decision soon on whether and how coverage measurement will be used in the
2010 Census;
consider incorporating lessons learned from its coverage measurement
experience during the 2000 Census, such as (1) demonstrating both the
operational and technical feasibility of its coverage measurement methods,
(2) determining the level of geography at which coverage can be reliably
measured, (3) keeping Congress and other stakeholders
informed of its plans, and (4) adequately testing coverage measurement
prior to full implementation; and
ensure that the Bureau*s financial management systems can capture and
report program activities early in the decennial process and ensure that
project costs are monitored for accuracy and completeness.
Agency Comments and The Secretary of Commerce forwarded written comments
from the Census Our Evaluation
Bureau on a draft of this report, which are reprinted in appendix I. The
Bureau agreed with our recommendations highlighting the steps that should
be followed in the development of a coverage measurement methodology for
the 2010 Census and acknowledged their importance. However, the Bureau
maintained that it followed most of these steps for the 2000 Census
including (1) keeping stakeholders, particularly Congress, informed of the
Bureau*s plans, (2) determining the level of geography at which coverage
measurement is intended, and (3) adequately testing coverage measurement
methodologies. The Bureau also maintained that throughout the 1990s, it
had an open and transparent process for implementing the coverage
measurement program, including the levels of geography to which its
results would be applied.
We disagree. As we stated in our report, the Bureau*s failure to provide
important information was a key cause of congressional skepticism over the
Bureau*s coverage measurement plans. In fact, Congress was so concerned
about the lack of comprehensive information on the Bureau*s proposed
approach that in July 1997, it passed a law that included
provisions requiring the Department of Commerce to provide detailed data
on the Bureau*s planned use of statistical estimation within 30 days. 14
We revised the report to include this, and provide other examples to
further
support our position that the Bureau*s I. C. M. and A. C. E. planning and
development processes were less than fully open and transparent. The
Bureau also commented that each major component of the
I. C. M./ A. C. E. program underwent *rigorous* testing in the middle of
the decade as well as during the dress rehearsal for the 2000 Census held
in 1998. We believe this overstates what actually occurred. As we noted in
the
report, the dress rehearsal failed to detect the problems that A. C. E.
encountered during the 2000 Census because the sites were not
representative of the nation. Additionally, because of an agreement
between Congress and the administration to simultaneously prepare for a
census that did not include sampling, the I. C. M. was only tested at two
of the three dress rehearsal sites* an urban area and an Indian
reservation* but was not tested in a rural location as was originally
planned. We made this and other revisions to strengthen our point.
Because the A. C. E. was designed to correct a census with a net coverage
error similar to that observed in previous censuses, the Bureau commented
that applying the methodology to the historically low levels of net error
observed in the 2000 Census represented a unique and unexpected challenge
for A. C. E. We revised the report to reflect this additional context.
The Bureau took exception to the way we presented our conclusions
concerning its ability to properly classify certain costs associated with
the development of the Bureau*s coverage measurement programs. The Bureau
noted that it decided not to separately track coverage measurement
development costs in 1994, because there was no internal or external
request for a separate cost accounting of the program.
Our report does not make interpretive conclusions or qualitative judgments
about which coverage measurement program costs the Bureau decided to
14 Pub. L. 105- 18, Title VIII, June 12, 1997.
track. Instead, the report (1) points out that we could not identify all
of the contractor costs associated with the I. C. M./ A. C. E. programs
because of the three factors described in the report, and (2) underscores
the importance of a sound financial management system for tracking,
planning, and development costs for the 2010 Census.
We are sending copies of this report to other interested congressional
committees, the Secretary of Commerce, and the Director of the U. S.
Census Bureau. Copies will be made available to others upon request. This
report will also be available at no charge on GAO*s home page at http://
www. gao. gov.
Please contact Patricia A. Dalton on (202) 512- 6806 or by E- mail at
daltonp@ gao. gov if you have any questions. Other key contributors to
this report were Robert Goldenkoff, Roger Stoltz, Carolyn Samuels, Cindy
Brown- Barnes, Ty Mitchell, and Linda Brigham.
Sincerely yours, Patricia A. Dalton Director Strategic Issues
McCoy Williams Director Financial Management and Assurance
Appendi xes Comments from the Department of
Appendi x I Commerce
Related GAO Products
2000 Census: Complete Costs of Coverage Evaluation Programs Are Not
Available. GAO- 03- 41. Washington, D. C.: October 31, 2002.
2000 Census: Lessons Learned for Planning a More Cost- Effective 2010
Census. GAO- 03- 40. Washington, D. C.: October 31, 2002.
2000 Census: Refinements to Full Count Review Program Could Improve Future
Data Quality. GAO- 02- 562. Washington, D. C.: July 3, 2002.
2000 Census: Coverage Evaluation Matching Implemented as Planned, but
Census Bureau Should Evaluate Lessons Learned. GAO- 02- 297. Washington,
D. C.: March 14, 2002.
2000 Census: Best Practices and Lessons Learned for More Cost- Effective
Nonresponse Follow- up. GAO- 02- 196. Washington, D. C.: February 11,
2002.
2000 Census: Coverage Evaluation Interviewing Overcame Challenges, but
Further Research Needed. GAO- 02- 26. Washington, D. C.: December 31,
2001.
2000 Census: Analysis of Fiscal Year 2000 Budget and Internal Control
Weaknesses at the U. S. Census Bureau. GAO- 02- 30. Washington, D. C.:
December 28, 2001.
2000 Census: Significant Increase in Cost Per Housing Unit Compared to
1990 Census. GAO- 02- 31. Washington, D. C.: December 11, 2001.
2000 Census: Better Productivity Data Needed for Future Planning and
Budgeting. GAO- 02- 4. Washington, D. C.: October 4, 2001.
2000 Census: Review of Partnership Program Highlights Best Practices for
Future Operations. GAO- 01- 579. Washington, D. C.: August 20, 2001.
Decennial Censuses: Historical Data on Enumerator Productivity Are
Limited. GAO- 01- 208R. Washington, D. C.: January 5, 2001.
2000 Census: Information on Short- and Long- Form Response Rates. GAO/
GGD- 00- 127R. Washington, D. C.: June 7, 2000.
(450133)
GAO*s Mission The General Accounting Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve
the performance and accountability of the federal government for the
American people. GAO examines the use of public funds; evaluates federal
programs and policies; and provides analyses, recommendations, and other
assistance to help Congress make informed oversight, policy, and funding
decisions. GAO*s commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of The fastest and easiest way to obtain copies of GAO
documents at no cost is
through the Internet. GAO*s Web site (www. gao. gov) contains abstracts
and fulltext GAO Reports and
files of current reports and testimony and an expanding archive of older
Testimony
products. The Web site features a search engine to help you locate
documents using key words and phrases. You can print these documents in
their entirety, including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as *Today*s Reports,* on its
Web site daily. The list contains links to the full- text document files.
To have GAO e- mail this
list to you every afternoon, go to www. gao. gov and select *Subscribe to
daily E- mail alert for newly released products* under the GAO Reports
heading.
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO
also accepts VISA and Mastercard. Orders for 100 or more copies mailed to
a single address are discounted 25 percent. Orders should be sent to:
U. S. General Accounting Office 441 G Street NW, Room LM Washington, D. C.
20548
To order by Phone: Voice: (202) 512- 6000 TDD: (202) 512- 2537 Fax: (202)
512- 6061
To Report Fraud, Contact:
Waste, and Abuse in Web site: www. gao. gov/ fraudnet/ fraudnet. htm
E- mail: fraudnet@ gao. gov Federal Programs
Automated answering system: (800) 424- 5454 or (202) 512- 7470 Public
Affairs Jeff Nelligan, managing director, NelliganJ@ gao. gov (202) 512-
4800
U. S. General Accounting Office, 441 G Street NW, Room 7149 Washington, D.
C. 20548
a
GAO United States General Accounting Office
As shown below, the two programs the Bureau employed to measure the
quality of the 2000 Census population data did not meet their objectives.
Coverage Measurement Programs Did Not Achieve Objectives Program/
objectives
Objectives met? Reasons I. C. M. Measure census coverage
Generate data for apportionment, redistricting, and federal programs using
statistical sampling and estimation
Produce a *one- number* census
No No
No Program was canceled following January
1999 U. S. Supreme Court ruling that the Census Act prohibits the use of
sampling to apportion seats in the U. S. House of Representatives. A. C.
E. Measure census coverage
Generate data needed for redistricting and other purposes using
statistical methods
No No
Uncertainties surrounding the accuracy of the A. C. E. results and the
inability to resolve them in time to meet legally mandated deadlines for
releasing data.
Source: GAO. Note: This table reflects GAO*s analysis of U. S. Census
Bureau data.
The A. C. E. program achieved results other than those laid out in the
Bureau*s formal objectives that highlight important lessons learned. They
include (1) developing a coverage measurement methodology that is both
operationally and technically feasible, (2) determining the level of
geography at which coverage measurement is intended, (3) keeping
stakeholders, particularly Congress, informed of the Bureau*s plans, and
(4) adequately testing coverage measurement methodologies. It will be
important for the Bureau to consider these as its current plans for the
2010 Census include coverage evaluation to measure the accuracy of the
census but not necessarily to adjust the results.
Of the roughly $207 million the Bureau obligated for I. C. M./ A. C. E.
programs from fiscal years 1996 through 2001, we identified about $22.3
million that was obligated for contracts involving over 170 vendors. We
could not identify any obligations prior to 1996 in part because the
Bureau included them with its general research and development efforts and
did not assign the I. C. M./ A. C. E. operations unique project codes in
its financial management system. To track these costs in the future, it
will be important for the Bureau to (1) have a financial management system
that has specific project codes to capture coverage measurement costs, (2)
establish the project codes as early in the planning process as possible,
and (3) monitor the usage of the codes to ensure that they are properly
charged. 2000 CENSUS
Coverage Measurement Programs* Results, Costs, and Lessons Learned
www. gao. gov/ cgi- bin/ getrpt? GAO- 03- 287. To view the full report,
including the scope and methodology, click on the link above. For more
information, contact Patricia A. Dalton at (202) 512- 6806 or daltonp@
gao. gov. Highlights of GAO- 03- 287, a report to
Chairman, Subcommittee on Civil Service, Census, and Agency Organization,
Committee on Government Reform, House of Representatives January 2003
To help measure the quality of the 2000 Census and to possibly adjust for
any errors, the U. S. Census
Bureau (Bureau) conducted the Accuracy and Coverage Evaluation (A. C. E.)
program. However, after obligating around $207 million for
A. C. E. and its predecessor program, Integrated Coverage Measurement (I.
C. M.), from fiscal years 1996 through 2001, the Bureau did not use either
program to adjust the census numbers. Concerned about the amount of money
the Bureau spent on I. C. M. and A. C. E. programs and what was produced
in return, the subcommittee asked us to review the objectives and results
of the programs, the costs of consultants, and how best to track future
coverage measurement activities.
The Secretary of Commerce should direct the Bureau to (1) work with
Congress and other stakeholders and soon decide on whether and
how coverage measurement will be used in 2010, (2) adopt lessons learned
from its 2000 Census experience, and (3) ensure that its financial
management systems can capture and report program activities early and
that projects* costs are monitored. The Bureau agreed with our
recommendations
but noted that for the 2000 Census, it followed the steps we identified as
lessons learned. It also took exception to how we presented our
conclusions concerning its ability to properly classify certain costs.
Page i GAO- 03- 287 Census Coverage Measurement Programs Contents Letter 1
Results in Brief 2 Background 4 Scope and Methodology 5 Coverage
Measurement
Programs for the 2000 Census Did Not Meet Bureau Objectives 7 I. C. M./ A.
C. E. Contractor Costs Are Not Complete 13 Tracking Future Coverage
Measurement Costs 15 Conclusions 16 Recommendations for Executive Action
17 Agency Comments and Our Evaluation 17
Appendix
Appendix I: Comments from the Department of Commerce 20 Related GAO
Products 23 Tables Table 1: Coverage Measurement Programs Did Not Achieve
Objectives 7 Table 2: Census 2000 I. C. M./ A. C. E. Contractor Costs for
Fiscal
Years 1996 through 2001 (Dollars in thousands) 14
Page 1 GAO- 03- 287 Census Coverage Measurement Programs United States
General Accounting Office Washington, D. C. 20548
Page 1 GAO- 03- 287 Census Coverage Measurement Programs A
Page 2 GAO- 03- 287 Census Coverage Measurement Programs
Page 3 GAO- 03- 287 Census Coverage Measurement Programs
Page 4 GAO- 03- 287 Census Coverage Measurement Programs
Page 5 GAO- 03- 287 Census Coverage Measurement Programs
Page 6 GAO- 03- 287 Census Coverage Measurement Programs
Page 7 GAO- 03- 287 Census Coverage Measurement Programs
Page 8 GAO- 03- 287 Census Coverage Measurement Programs
Page 9 GAO- 03- 287 Census Coverage Measurement Programs
Page 10 GAO- 03- 287 Census Coverage Measurement Programs
Page 11 GAO- 03- 287 Census Coverage Measurement Programs
Page 12 GAO- 03- 287 Census Coverage Measurement Programs
Page 13 GAO- 03- 287 Census Coverage Measurement Programs
Page 14 GAO- 03- 287 Census Coverage Measurement Programs
Page 15 GAO- 03- 287 Census Coverage Measurement Programs
Page 16 GAO- 03- 287 Census Coverage Measurement Programs
Page 17 GAO- 03- 287 Census Coverage Measurement Programs
Page 18 GAO- 03- 287 Census Coverage Measurement Programs
Page 19 GAO- 03- 287 Census Coverage Measurement Programs
Page 20 GAO- 03- 287 Census Coverage Measurement Programs Appendix I
Appendix I Comments from the Department of Commerce Page 21 GAO- 03- 287
Census Coverage Measurement Programs
Appendix I Comments from the Department of Commerce Page 22 GAO- 03- 287
Census Coverage Measurement Programs
Page 23 GAO- 03- 287 Census Coverage Measurement Programs
United States General Accounting Office Washington, D. C. 20548- 0001
Official Business Penalty for Private Use $300
Address Service Requested Presorted Standard
Postage & Fees Paid GAO Permit No. GI00
*** End of document. ***