[Analytical Perspectives]
[Crosscutting Programs]
[4. Strengthening Federal Statistics]
[From the U.S. Government Printing Office, www.gpo.gov]
[[Page 41]]
4. STRENGTHENING FEDERAL STATISTICS
Federal statistical programs produce key information about a range of
topics of interest to public and private decision makers, including the
economy, the population, agriculture, crime, education, energy, the
environment, health, science, and transportation. The ability of
governments, businesses, and citizens to make appropriate decisions
about budgets, employment, investments, taxes, and a host of other
important matters depends critically on the ready availability of
relevant, accurate, and timely Federal statistics.
Moreover, for Federal statistical programs to meet the needs of a wide
range of users, the underlying data systems must be viewed as credible.
In order to foster this credibility, Federal statistical programs seek
to adhere to high quality standards and to maintain integrity and
efficiency in the production of statistics. As the collectors and
providers of these basic data, Federal agencies act as data stewards--
balancing public and private decision makers' needs for information with
legal and ethical obligations to minimize reporting burden, respect
respondents' privacy, and protect the confidentiality of the data
provided to the Government. This chapter discusses the development of
standards that principal statistical programs can use to assess their
performance and presents highlights of their 2005 budget proposals.
Performance Standards
Agencies maintain the quality of their data or information products as
well as their credibility by setting high performance standards for
their activities. The statistical agencies and statistical units
represented on the Interagency Council on Statistical Policy (ICSP) have
collaborated on developing an initial set of common performance
standards for use under the Government Performance and Results Act and
in completing the Administration's new Program Assessment Rating Tool
(PART). Federal statistical agencies have agreed that there are six
conceptual dimensions within two general areas of focus that are key to
measuring and monitoring statistical programs. The first area of focus
is Product Quality, encompassing the traditional dimensions of
relevance, accuracy, and timeliness. The second area of focus is Program
Performance, encompassing the dimensions of cost, dissemination, and
mission achievement.
Statistical agencies historically have focused on measuring
performance in the area of product quality, especially the dimensions
most amenable to quantitative measurement, specifically accuracy and
timeliness. Relevance, also an accepted measure of quality, can be
either a qualitative description of the usefulness of products or a
quantitative measure such as a customer satisfaction score. Relevance is
more difficult to measure, and the indicators that do exist are more
varied.
Program performance standards form the basis for evaluating
effectiveness. They address questions such as: Are taxpayer dollars
spent most effectively? Are products made available to those who need
them? Are agencies meeting their mission requirements or making it
possible for other agencies to meet their missions? The indicators
available to measure program performance for statistical activities
currently are less well developed than those for product quality.
Product quality and program performance standards are designed to
serve as indicators when answering specific questions in the
Administration's PART process. (Please refer to Chapter 2 of this volume
for a description of the PART.) Figure 4-1 presents each principal
Federal statistical agency's assessment of the status of its current and
planned use of indicators on the six dimensions. Use of the indicators
may be for internal management, strategic planning, or annual
performance reporting. The dimensions shown in the figure reflect an
overall set of indicators for statistical activities but the specific
measures vary among the individual programs depending on their unique
characteristics and requirements. Annual performance reports and PARTs
contain these specific measures as well as additional information about
performance goals and targets and whether a program is meeting, or
making measurable progress toward meeting, its performance goals. The
examples below illustrate different ways agencies track their
performance on each dimension.
Product Quality: Statistical agencies agree that product quality
encompasses many attributes, including (but not limited to) relevance,
accuracy, and timeliness. The basic measures in this group relate to the
quality of specific products, thereby providing actionable information
to managers. These are ``outcome-oriented'' measures and are key to the
usability of information products. Statistical agencies or units
establish targets and monitor how well targets are met. In some sense,
relevance relates to ``doing the right things,'' while accuracy and
timeliness relate to ``doing things right.''
Relevance: Qualitative or quantitative descriptions of the
degree to which products are useful and responsive to users'
needs. Relevance of data products and analytic reports may be
assessed through a professional review process and ongoing
contacts with data users. Product relevance may be indicated
by customer satisfaction with product content, information
from customers about product use, demonstration of product
improvements, comparability with other data series, agency
responses to customer suggestions for improvement, new or
[[Page 42]]
customized products/services, frequency of use, or responses
to data requests from users (including policy makers). Through
a variety of professional review activities, agencies maintain
the relevance, accuracy, and validity of programs, and
encourage data users and other stakeholders to contribute to
the agency's data collection and dissemination program.
Striving for relevance requires monitoring to ensure that
information systems anticipate change and evolve to
appropriately measure our dynamic society and economy.
Accuracy: Qualitative or quantitative measures of important
features of correctness, validity, and reliability of data and
information products measured as degree of closeness to target
values. For statistical data, accuracy measures include
sampling error and various aspects of nonsampling error (e.g.,
response rates, size of revisions, coverage, edit
performance). For analysis products, accuracy may be the
quality of the reasoning, reasonableness of assumptions, and
clarity of the exposition, typically measured and monitored
through review processes. In addition, accuracy is assessed
and improved by external and internal reviews, comparisons of
data among different surveys, linkages of survey data to
administrative records, redesigns of surveys, or expansions of
sample sizes.
Timeliness: Qualitative or quantitative measure of the
timing of information releases. May be measured as time from
the collection of data or the close of the reference period to
the release of information, or customer satisfaction with
timeliness. May also be measured as how well agencies meet
scheduled and publicized release dates, expressed as a percent
of release dates met.
Program Performance: Statistical agencies agree that program
performance encompasses balancing the dimensions of cost, dissemination,
and mission accomplishment for the agency as a whole; operating
efficiently and effectively; ensuring that customers receive the
information they need; and serving the information needs of the Nation.
Costs of products or programs may be used to develop efficiency
measures. Dissemination involves making sure customers receive the
information they need via the most appropriate mechanisms. Mission
achievement means that the information program makes a difference.
Hence, three key dimensions are being used to indicate program
performance: cost (input), dissemination (output), and mission
achievement (outcome).
Cost: Quantitative measure of the dollar amount used to
produce data products and services. The development and use of
financial performance measures within the Federal Government
is an established goal, and the intent of such measures is to
determine the ``true costs'' of various programs or
alternative modes of operation at the Federal level. Examples
of cost data include full costs of products or programs,
return on investment, dollar value of efficiencies, and ratios
of cost to products distributed.
Dissemination: Qualitative or quantitative information on
the availability, accessibility, and distribution of products
and services. Most agencies have goals to improve product
accessibility, particularly through the Internet. Typical
measures include: on-demand requests fulfilled, product
downloads, degree of accessibility, customer satisfaction with
ease of use, number of participants at user conferences,
citations of agency data in the media, number of Internet user
sessions, number of formats in which data are available,
amount of technical support provided to data users, exhibits
to inform the public about information products, issuance of
newsletters describing products, usability testing of
websites, and assessing compliance with Section 508 of the
Rehabilitation Act which requires Federal agencies to make
their electronic and information technology accessible to
people with disabilities.
Mission Achievement: Qualitative or quantitative information
about the impact of or satisfaction with statistical programs.
For Federal statistical programs, this dimension responds to
the question--have we achieved our objectives and met the
expectations of our stakeholders? Under this dimension,
statistical programs document their contributions to the goals
and missions of parent departments and other agencies, the
Administration, the Congress, and information users in the
private sector. For statistical programs, this broad dimension
involves meeting recognized societal information needs and
also addresses the linkage between statistical outputs and
programmatic outcomes.
However, identifying this linkage is far from
straightforward. It is sometimes difficult to trace the impact
of information products on the public good. Such products
often are necessary intermediate inputs in the creation of a
high visibility product whose societal benefit is clearly
recognized. For example, the economic statistics produced by a
variety of agencies are directly used by the Bureau of
Economic Analysis in the calculation of the Gross Domestic
Product (GDP), which analysts use to assess changes in the
level of domestic economic activity. Similarly, statistics
from specific surveys are directly used by the Bureau of Labor
Statistics in the calculation of the Consumer Price Index
(CPI), which is widely used in diverse applications, such as
indexing pensions for retirees. As a result, a number of
statistical agencies contribute to the GDP and/or the CPI and
to the many uses of these information products. In addition,
the data produced by statistical agencies are used to track
the performance of programs managed by their parent agencies
or other
[[Page 43]]
organizations in areas such as crime, education, energy, the
environment, health, science, and transportation.
Moreover, beyond the direct and focused uses of statistical
products and programs, the statistical agencies and their
products serve a diverse and dispersed set of data users
working on a broad range of applications. Users include senior
government policy makers at the Federal, State, and local
levels, business leaders, households, academic researchers,
analysts at public policy institutes and trade groups,
marketers and planners in the private sector, and many others.
Information produced by statistical agencies often is combined
with other information for use in the decision-making process.
Thus, as with many nonstatistical programs, the relationship
between statistical program outputs and their beneficial uses
and outcomes is often complex and difficult to track.
In the absence of preferred quantitative indicators,
qualitative narratives can indicate how statistical agency
products contribute to and evaluate progress toward important
goals established for government or private programs. In
particular, narratives can highlight how statistical agencies
measure the Nation's social and economic structure, and how
the availability of the information influences changes in
policies and programs. These narratives contribute to
demonstrating mission accomplishment, particularly in response
to questions in Section I of the PART, ``program purpose and
design.'' Narratives may describe the impact of measuring
agency policy or change of policy, supporting research focused
on policy issues, furnishing information to inform debate on
policy issues, or providing in-house consulting support.
In addition to narratives, quantitative measures may be used
to reflect mission achievement. For example, customer
satisfaction with the statistical agency or unit indicates if
the agency or unit has met the expectations of its
stakeholders.
Description of Dimensions
Product Quality
Relevance: Qualitative or quantitative description of the degree to which products and
services are useful to users and responsive to their needs.
Accuracy: Qualitative or quantitative measure of important features of correctness,
validity, and reliability of data and information products measured as degree of closeness
to target values.
Timeliness: Qualitative or quantitative measure of the timing of information releases.
[[Page 44]]
Description of Dimensions--Continued
Program Performance
Cost: Quantitative measure of the dollar amount used to produce data products and services.
Dissemination: Qualitative or quantitative information on the availability, accessibility,
and distribution of products and services.
Mission Achievement: Qualitative or quantitative information about the impact of, or
satisfaction with, statistical programs.
Key to Statistical Agencies
BEA = Bureau of Economic Analysis, Department of Commerce
BJS = Bureau of Justice Statistics, Department of Justice
BLS = Bureau of Labor Statistics, Department of Labor
BTS = Bureau of Transportation Statistics, Department of Transportation
Census = Census Bureau, Department of Commerce
EIA = Energy Information Administration, Department of Energy
ERS = Economic Research Service, Department of Agriculture
NASS = National Agricultural Statistics Service, Department of Agriculture
NCES = National Center for Education Statistics, Department of Education
NCHS = National Center for Health Statistics, Department of Health and Human Services
ORES = Office of Research, Evaluation, and Statistics, Social Security Administration
SOI = Statistics of Income, Internal Revenue Service, Department of the Treasury
SRS = Division of Science Resources Statistics, National Science Foundation
Of the 14 principal Federal statistical agencies that are members of
the ICSP, four agencies have programs that have been assessed using the
PART process. These agencies' programs have received PART summary
ratings of Effective or Moderately Effective, as shown in Figure 4-2. As
additional ICSP agencies have an opportunity to undergo the PART
process, the agencies plan to use the results of the collaborative
performance standards development effort to help maintain and extend
their generally well-received assessments.
Figure 4-2. 2005 PART SUMMARY RATINGS FOR STATISTICAL PROGRAMS
------------------------------------------------------------------------
Summary Rating
------------------------------------------------------------------------
Bureau of Economic Analysis Effective
Bureau of Labor Statistics Effective
Census Bureau
Current Demographic Statistics Moderately Effective
Decennial Census Moderately Effective
Intercensal Demographic Estimates Moderately Effective
Survey Sample Redesign Effective
National Center for Education ...................................
Statistics
Statistics Effective
Assessment Effective
------------------------------------------------------------------------
Highlights of 2005 Program Budget Proposals
The programs that provide essential statistical information for use by
governments, businesses, researchers, and the public are carried out by
some 70 agencies spread across every department and several independent
agencies. Approximately 40 percent of the funding for these programs
provides resources for twelve agencies or agency units that have
statistical activities as their principal mission. (Please see Table 4-
1.) The remaining funding supports work in 60-plus agencies that carry
out statistical activities in conjunction with other missions such as
providing services or enforcing regulations. More comprehensive budget
and program information about the Federal statistical system will be
available in OMB's annual report, Statistical Programs of the United
States Government, Fiscal Year 2005, when it is published later this
year. The following highlights elaborate on the Administration's
proposals to strengthen the programs of the principal Federal
statistical agencies.
Bureau of Economic Analysis: Funding is requested to complete work
begun in 2003 to: (1) accelerate the release of some of the Nation's
most important economic statistics to dramatically increase their
usefulness to policy makers, business leaders, and other users; (2) meet
U.S. statistical obligations to international organizations on the
Special Data Dissemination Standards and complete the incorporation of
the North American Industry Classification System into BEA accounts; (3)
improve the economic accounts by acquiring monthly real-time data from
private sources to fill data gaps in current measures as well as conduct
a quarterly survey of large and volatile international services such as
telecommunications, finance, and insurance; and (4) produce more current
business investment data that include associated employment and
compensation estimates on an annual basis in order to provide data
needed to conduct analyses of tax policy, business investment, and
productivity in manufacturing and service industries.
Bureau of Justice Statistics: Funding is requested to continue
conversion of the National Crime Victimization Survey from primarily a
paper and pencil operation to a fully automated data collection process.
The BJS base program increase will provide for the maintenance of BJS's
core statistical programs, including: (1) the National Crime
Victimization Survey, the Nation's primary source of information on
criminal victimization; (2) cybercrime statistics on the incidence,
magnitude,
[[Page 45]]
and consequences of electronic and computer crime to households and
businesses; (3) law enforcement data from over 3,000 agencies on the
organization and administration of police and sheriffs' departments; (4)
nationally representative prosecution data on resources, policies, and
practices of local prosecutors; (5) court and sentencing statistics,
including Federal and State case processing data; and (6) data on
correctional populations and facilities from Federal, State, and local
governments.
Bureau of Labor Statistics: Funding is requested to support current
program operations to measure the economy through producing,
disseminating, and improving BLS economic measures, including: (1)
modernizing the computing systems for monthly processing of the Producer
Price Index (PPI) and U.S. Import and Export Price Indexes, and
producing new data outputs, such as experimental PPI's for goods and
services that will provide the first economy-wide measures of changes in
producer prices; (2) maintaining continuous updating of the Consumer
Price Index (CPI) by updating the expenditure and population weights
biennially, the superlative index annually, outlet samples on a four-
year cycle, and item samples in key categories on a two-year cycle, in
lieu of performing major revisions about every ten years; and (3)
continuing with a multi-year effort to enhance core BLS information
technology infrastructure through a central Department of Labor
appropriation.
Bureau of Transportation Statistics: Funding is requested to: (1)
develop the American Freight Data Program, a continuous source of
freight data from shippers, carriers, and receivers, to replace the
current five-year Commodity Flow Survey; (2) move the Airfare Price
Index, an input to GDP and CPI indices, from experimental to production
mode; and (3) develop more timely and comprehensive local and long-
distance travel data.
Census Bureau: Funding is requested for the Census Bureau's economic
and demographic programs and for a re-engineered 2010 Census. For the
Census Bureau's economic and demographic programs, funding is requested
to: (1) support the release of all remaining data products from the 2002
Economic Census; (2) begin planning for the 2007 Economic Census and
Census of Governments; (3) continue efforts begun in 2003 to eliminate
data gaps by measuring migration across U.S. borders; (4) improve
measurement of services by expanding key source data for critical
quarterly and annual estimates of our Nation's Gross Domestic Product;
(5) continue efforts to offer electronic reporting for almost 100
current economic surveys; and (6) support the Automated Export System
and accelerate release of trade statistics. For 2010 Census planning,
funding is requested to continue to: (1) conduct extensive planning,
testing, and development activities to support a re-engineered 2010
Census; (2) complete map feature accuracy within 7.6 meters of true GPS
location for 48 percent of all counties in the U.S., Puerto Rico, and
island areas; and (3) conduct the first full year of the American
Community Survey program to provide data on an ongoing basis rather than
waiting for once-a-decade censuses.
Economic Research Service: Funding is requested to develop an
integrated and comprehensive data and analysis framework of the food
system beyond the farm-gate to provide a basis for understanding,
monitoring, tracking, and identifying changes in food supply and
consumption patterns.
Energy Information Administration: Funding is requested to: (1)
continue the improvement of natural gas and electricity survey data; (2)
undertake development work on a liquefied natural gas storage survey and
a natural gas production survey; (3) enhance the National Energy
Modeling System's transportation modeling; and (4) revise the Voluntary
Greenhouse Gases survey to support the President's Initiative on
Greenhouse Gases.
National Agricultural Statistics Service: Funding is requested to: (1)
continue restoration and modernization of the agricultural estimates
program to ensure State, regional, and national level agricultural
estimates of sufficient precision, quality, and detail to meet the needs
of a broad customer base; and (2) support Government-wide and
departmental E-Government initiatives.
National Center for Education Statistics: Funding is requested to: (1)
support the second wave of data collection of the Early Childhood
Longitudinal Study--Birth Cohort and data release in Spring 2005; (2)
continue efforts to improve electronic data collection and data
dissemination; (3) support the ongoing data collection efforts for the
Schools and Staffing Survey, the principal collection on national and
State level indicators of teacher and school quality; (4) continue U.S.
participation in data collections, analyses, and reporting on
international assessments that compare educational performance and
progress across countries; and (5) continue support for the National
Assessment of Educational Progress (NAEP) program and its role in
benchmarking national and State performance.
National Center for Health Statistics: Funding is requested to: (1)
maintain and transform HHS' core health statistics capacity; (2)
preserve and modernize the Nation's vital statistics system; (3) fortify
and transform basic operations for the National Health and Nutrition
Examination Survey; (4) maintain and redesign systems for tracking the
health care delivery system; and (5) redesign the sample for the
National Health Interview Survey.
Science Resources Statistics Division, NSF: Funding is requested to:
(1) implement ongoing programs on the science and engineering (S&E)
enterprise; (2)
[[Page 46]]
continue implementing quality improvements to surveys on the S&E
workforce; (3) begin research on methods to implement necessary
enhancements to the Industry Research and Development survey; (4)
develop an ongoing data collection program on research instrumentation
stocks, as mandated by Congress; and (5) continue activities to
establish an ongoing data series on postdoctorates.
Statistics of Income Division, IRS: Funding is requested to: (1)
maintain and modernize core data collection systems, including several
major statistical programs for the Treasury Department, the
Congressional Joint Committee on Taxation, the Bureau of Economic
Analysis, and SOI's many other customers; (2) implement a databank
repository for SOI and IRS population file data to more efficiently
build longitudinal databases and enable sub-national estimates; (3)
examine means to more effectively mask individual records to minimize
the possibility of identification in the Individual Public Use sample
files; and (4) modernize and expedite dissemination of data and
publications, including a reengineered Internet website.
Table 4-1. 2003-2005 BUDGET AUTHORITY FOR PRINCIPAL STATISTICAL AGENCIES
(in millions of dollars)
----------------------------------------------------------------------------------------------------------------
Estimate
2003 -------------------------
Actual 2004 2005
----------------------------------------------------------------------------------------------------------------
Bureau of Economic Analysis.............................................. 66 67 82
Bureau of Justice Statistics............................................. 32 32 39
Bureau of Labor Statistics............................................... 492 518 534
Bureau of Transportation Statistics...................................... 30 31 34
Census Bureau \1\........................................................ 571 632 848
Salaries and Expenses \1\............................................. 202 213 240
Periodic Censuses and Programs........................................ 369 419 608
Economic Research Service................................................ 69 71 80
Energy Information Administration........................................ 80 81 85
National Agricultural Statistics Service \2\............................. 138 128 138
National Center for Education Statistics................................. 184 187 187
Statistics............................................................ 89 92 92
Assessment............................................................ 95 95 95
National Center for Health Statistics.................................... 126 128 150
PHS Evaluation Funds.................................................. 126 128 150
Budget Authority...................................................... 0 0 0
Science Resources Statistics Division, NSF............................... 31 32 32
Statistics of Income Division, IRS....................................... 32 36 36
----------------------------------------------------------------------------------------------------------------
\1\ Includes mandatory appropriations of $20 million for each year for the Survey of Program Dynamics and
collection of data related to the allocation to States of State Children's Health Insurance Program funds.
\2\ Includes funds for the periodic Census of Agriculture of $41, $25, and $23 million in 2003, 2004, and 2005,
respectively.