[Analytical Perspectives]
[Crosscutting Programs]
[4. Strengthening Federal Statistics]
[From the U.S. Government Printing Office, www.gpo.gov]
[[Page 53]]
4. STRENGTHENING FEDERAL STATISTICS
Federal statistical programs produce key information about a range of
topics of interest to public and private decision makers, including the
economy, the population, agriculture, crime, education, energy, the
environment, health, science, and transportation. The ability of
governments, businesses, and citizens to make appropriate decisions
about budgets, employment, investments, taxes, and a host of other
important matters depends critically on the ready availability of
relevant, accurate, and timely Federal statistics. The Federal
statistical community remains on alert for opportunities to strengthen
these measures of our Nation's performance. For example, during 2004,
Federal statistical agencies launched the first new economic indicator
survey in 40 years--the Quarterly Services Survey (Census Bureau);
expanded regional economic data from 318 to 934 American communities
(BEA); successfully adopted new collection and processing technologies
that will greatly accelerate the release of data from the National
Health Interview Survey (NCHS); and completed the first data collection
on the cyberinfrastructure of academic and biomedical facilities (NSF's
SRS).
For Federal statistical programs to effectively benefit such a wide
range of users, the underlying data systems must be viewed as credible.
In order to foster this credibility, Federal statistical programs seek
to adhere to high quality standards and to maintain integrity and
efficiency in the production of statistics. As the collectors and
providers of these basic data, Federal statistical agencies act as data
stewards--balancing public and private decision makers' needs for
information with legal and ethical obligations to minimize reporting
burden, respect respondents' privacy, and protect the confidentiality of
the data provided to the Government. This chapter discusses the
development of standards that principal statistical programs can use to
assess their performance and presents highlights of their 2006 budget
proposals.
Performance Standards
Statistical programs maintain the quality of their data or information
products as well as their credibility by setting high performance
standards for their activities. The statistical agencies and statistical
units represented on the Interagency Council on Statistical Policy
(ICSP) have collaborated on developing an initial set of common
performance standards for use under the Government Performance and
Results Act and in completing the Administration's Program Assessment
Rating Tool (PART). Federal statistical agencies have agreed that there
are six conceptual dimensions within two general areas of focus that are
key to measuring and monitoring statistical programs. The first area of
focus is Product Quality, encompassing the traditional dimensions of
relevance, accuracy, and timeliness. The second area of focus is Program
Performance, encompassing the dimensions of cost, dissemination, and
mission achievement.
Statistical agencies historically have focused on measuring
performance in the area of product quality, especially the dimensions
most amenable to quantitative measurement, specifically accuracy and
timeliness. Relevance, also an accepted measure of quality, can be
either a qualitative description of the usefulness of products or a
quantitative measure such as a customer satisfaction score. Relevance is
more difficult to measure, and the indicators that do exist are more
varied.
Program performance standards form the basis for evaluating
effectiveness. They address questions such as: Are taxpayer dollars
spent most effectively? Are products made available to those who need
them? Are agencies meeting their mission requirements or making it
possible for other agencies to meet their missions? The indicators
available to measure program performance for statistical activities
currently are less well developed.
Product quality and program performance standards are designed to
serve as indicators when answering specific questions in the
Administration's PART process. Chart 4-1 presents each principal Federal
statistical agency's assessment of the status of its current and planned
use of indicators on the six dimensions. During the past year, four
agencies (BTS, EIA, NCES, and SRS) have improved the status of their
indicators. Use of the indicators may be for internal management,
strategic planning, or annual performance reporting. The dimensions
shown in the figure reflect an overall set of indicators for statistical
activities but the specific measures vary among the individual programs
depending on their unique characteristics and requirements. Annual
performance reports and PARTs contain these specific measures as well as
additional information about performance goals and targets and whether a
program is meeting, or making measurable progress toward meeting, its
performance goals. The examples below illustrate different ways agencies
track their performance on each dimension.
[[Page 54]]
Description of Dimensions
Product Quality
Relevance: Qualitative or quantitative descriptions of the degree to which products and
services are useful to users and responsive to users' needs.
Accuracy: Qualitative or quantitative measure of important features of correctness,
validity, and reliability of data and information products measured as degree of closeness
to target values.
Timeliness: Qualitative or quantitative measure of the timing of information releases.
Program Performance
Cost: Quantitative measure of the dollar amount used to produce data products and services.
Dissemination: Qualitative or quantitative information on the availability, accessibility,
and distribution of products and services.
Mission Achievement: Qualitative or quantitative information about the effect of, or
satisfaction with, statistical programs.
Key to Statistical Agencies
BEA = Bureau of Economic Analysis, Department of Commerce
BJS = Bureau of Justice Statistics, Department of Justice
BLS = Bureau of Labor Statistics, Department of Labor
BTS = Bureau of Transportation Statistics, Department of Transportation
Census = Census Bureau, Department of Commerce
EIA = Energy Information Administration, Department of Energy
ERS = Economic Research Service, Department of Agriculture
NASS = National Agricultural Statistics Service, Department of Agriculture
NCES = National Center for Education Statistics, Department of Education
NCHS = National Center for Health Statistics, Department of Health and Human Services
ORES = Office of Research, Evaluation, and Statistics, Social Security Administration
SOI = Statistics of Income, Internal Revenue Service, Department of the Treasury
SRS = Division of Science Resources Statistics, National Science Foundation
Product Quality: Statistical agencies agree that product quality
encompasses many attributes, including (but not limited to) relevance,
accuracy, and timeliness. The basic measures in this group relate to the
quality of specific products, thereby providing actionable information
to managers. These are ``outcome-oriented''
[[Page 55]]
measures and are key to the usability of information products.
Statistical agencies or units establish targets and monitor how well
targets are met. In some sense, relevance relates to ``doing the right
things,'' while accuracy and timeliness relate to ``doing things
right.''
Relevance: Qualitative or quantitative descriptions of the
degree to which products and services are useful and
responsive to users' needs. Relevance of data products and
analytic reports may be monitored through a professional
review process and ongoing contacts with data users. Product
relevance may be indicated by customer satisfaction with
product content, information from customers about product use,
demonstration of product improvements, comparability with
other data series, agency responses to customer suggestions
for improvement, new or customized products/services,
frequency of use, or responses to data requests from users
(including policy makers). Through a variety of professional
review activities, agencies maintain the relevance, accuracy,
and validity of their products, and encourage data users and
other stakeholders to contribute to the agency's data
collection and dissemination programs. Striving for relevance
requires monitoring to ensure that information systems
anticipate change and evolve to appropriately measure our
dynamic society and economy.
Accuracy: Qualitative or quantitative measures of important
features of correctness, validity, and reliability of data and
information products measured as degree of closeness to target
values. For statistical data, accuracy may be defined as the
degree of closeness to the target value and measured as
sampling error and various aspects of nonsampling error (e.g.,
response rates, size of revisions, coverage, edit
performance). For analysis products, accuracy may be the
quality of the reasoning, reasonableness of assumptions, and
clarity of the exposition, typically measured and monitored
through review processes. In addition, accuracy is assessed
and improved by internal reviews, comparisons of data among
different surveys, linkages of survey data to administrative
records, redesigns of surveys, or expansions of sample sizes.
Timeliness: Qualitative or quantitative measure of timing of
information releases. Timeliness may be measured as time from
the close of the reference period to the release of
information, or customer satisfaction with timeliness.
Timeliness may also be measured as how well agencies meet
scheduled and publicized release dates, expressed as a percent
of release dates met.
Program Performance: Statistical agencies agree that program
performance encompasses balancing the dimensions of cost, dissemination,
and mission accomplishment for the agency as a whole; operating
efficiently and effectively; ensuring that customers receive the
information they need; and serving the information needs of the Nation.
Costs of products or programs may be used to develop efficiency
measures. Dissemination involves making sure customers receive the
information they need via the most appropriate mechanisms. Mission
achievement means that the information program makes a difference.
Hence, three key dimensions are being used to indicate program
performance: cost (input), dissemination (output), and mission
achievement (outcome).
Cost: Quantitative measure of the dollar amount used to
produce data products or services. The development and use of
financial performance measures within the Federal Government
is an established goal, and the intent of such measures is to
determine the ``true costs'' of various programs or
alternative modes of operation at the Federal level. Examples
of cost data include full costs of products or programs,
return on investment, dollar value of efficiencies, and ratios
of cost to products distributed.
Dissemination: Qualitative or quantitative information on
the availability, accessibility, and distribution of products
and services. Most agencies have goals to improve product
accessibility, particularly through the Internet. Typical
measures include: on-demand requests fulfilled, product
downloads, degree of accessibility, customer satisfaction with
ease of use, number of participants at user conferences,
citations of agency data in the media, number of Internet user
sessions, number of formats in which data are available,
amount of technical support provided to data users, exhibits
to inform the public about information products, issuance of
newsletters describing products, usability testing of web
sites, and assessing compliance with Section 508 of the
Rehabilitation Act, which requires Federal agencies to make
their electronic and information technology accessible to
people with disabilities.
Mission Achievement: Qualitative or quantitative information
about the effect of, or satisfaction with, statistical
programs. For Government statistical programs, this dimension
responds to the question--have we achieved our objectives and
met the expectations of our stakeholders? Under this
dimension, statistical programs document their contributions
to the goals and missions of parent departments and other
agencies, the Administration, the Congress, and information
users in the private sector and the general public. For
statistical programs, this broad dimension involves meeting
recognized societal information needs and also addresses the
linkage between statistical outputs and programmatic outcomes.
However, identifying this linkage is far from
straightforward. It is frequently difficult to trace the
effects of information products on the public
[[Page 56]]
good. Such products often are necessary intermediate inputs in
the creation of high visibility information whose societal
benefit is clearly recognized. For example, the economic
statistics produced by a variety of agencies are directly used
by the Bureau of Economic Analysis in the calculation of the
Gross Domestic Product (GDP), which analysts universally use
to assess changes in the level of domestic economic activity.
Similarly, statistics from specific surveys are directly used
by the Bureau of Labor Statistics in the calculation of the
Consumer Price Index (CPI), which is widely used in diverse
applications, such as indexing pensions for retirees. As a
result, a number of statistical agencies can claim credit for
contributing to the GDP and/or the CPI and to the many uses of
these information products. In addition, the statistics
produced by statistical agencies are used to track the
performance of programs managed by their parent or other
organizations related to topics such as crime, education,
energy, the environment, health, science, and transportation.
Moreover, beyond the direct and focused uses of statistical
products and programs, the statistical agencies and their
products serve a diverse and dispersed set of data users
working on a broad range of applications. Users include
government policy makers at the Federal, State, and local
levels, business leaders, households, academic researchers,
analysts at public policy institutes and trade groups,
marketers and planners in the private sector, and many others.
Information produced by statistical agencies often is combined
with other information for use in the decision-making process.
Thus, the relationship between program outputs and their
beneficial uses and outcomes is often complex and difficult to
track. Consequently, agencies use both qualitative and
quantitative indicators to make this linkage as explicit as
feasible.
In the absence of preferred quantitative indicators,
qualitative narratives can indicate how statistical agency
products contribute to and evaluate progress toward important
goals established for government or private programs. In
particular, narratives can highlight how statistical agencies
measure the Nation's social and economic structure, and how
the availability of the information influences changes in
policies and programs. These narratives contribute to
demonstrating mission accomplishment, particularly in response
to questions in Section I of the PART, ``program purpose and
design.'' Narratives may describe statistical information's
effects on measuring agency policy or change of policy,
supporting research focused on policy issues, informing debate
on policy issues, or providing in-house consulting support.
In addition to narratives, quantitative measures may be used
to reflect mission achievement. For example, customer
satisfaction with the statistical agency or unit indicates if
the agency or unit has met the expectations of its
stakeholders.
Of the 14 principal Federal statistical agencies that are members of
the ICSP, six agencies have programs that have been assessed using the
PART process. Most of these agencies' programs have received PART
summary ratings of Effective or Moderately Effective, as shown in Chart
4-2. While recognizing the strength of the Energy Information
Administration's purpose and management, EIA's PART evaluation found
that it lacks specific annual performance measures, baselines, and
targets and should consider enhancing independent expert evaluation of
its major program areas. EIA is correcting both of these shortcomings,
which should bring its PART rating into line with those of its sister
agencies. As additional ICSP agencies have an opportunity to undergo the
PART process, the agencies plan to continue to use the results of the
collaborative performance standards development effort to help maintain
and extend their generally favorable assessments.
Chart 4-2. Most Recent PART Summary Ratings for Statistical Programs
------------------------------------------------------------------------
Summary Rating
------------------------------------------------------------------------
Bureau of Economic Analysis Effective
Bureau of Labor Statistics Effective
Census Bureau
Current Demographic Statistics Effective
Decennial Census Moderately Effective
Economic Census Effective
Intercensal Demographic Estimates Moderately Effective
Survey Sample Redesign Effective
Energy Information Administration Results Not Demonstrated
National Agricultural Statistics Moderately Effective
Service
National Center for Education ...................................
Statistics
Statistics Effective
Assessment Effective
------------------------------------------------------------------------
Table 4-1. 2004-2006 BUDGET AUTHORITY FOR PRINCIPAL STATISTICAL AGENCIES
(in millions of dollars)
----------------------------------------------------------------------------------------------------------------
Estimate
2004 -------------------------
Actual 2005 \1\ 2006
----------------------------------------------------------------------------------------------------------------
Bureau of Economic Analysis \2\.......................................... 68 73 81
Bureau of Justice Statistics \3\......................................... 32 34 63
Bureau of Labor Statistics............................................... 518 529 543
Bureau of Transportation Statistics...................................... 30 30 33
Census Bureau \4\........................................................ 629 765 897
Salaries and Expenses \4\.............................................. 213 216 240
Periodic Censuses and Programs......................................... 416 549 657
Economic Research Service................................................ 71 74 81
Energy Information Administration........................................ 81 84 86
National Agricultural Statistics Service \5\............................. 128 128 145
National Center for Education Statistics................................. 187 185 208
Statistics............................................................. 92 91 91
Assessment............................................................. 95 94 117
National Center for Health Statistics \6\................................ 90 109 109
Science Resources Statistics Division, NSF............................... 31 32 32
Statistics of Income Division, IRS....................................... 36 39 39
----------------------------------------------------------------------------------------------------------------
\1\ Reflects any recissions.
\2\ 2005 estimate includes $2 million for a National Academy of Public Administration study of off-shoring.
\3\ The 2006 estimate includes funds for the Felony Arrestee Drug Use Reporting program (previously funded as
the Arrestee Drug Abuse Monitoring program within the National Institute of Justice) as well as funds for
management and administrative costs that were displayed separately in 2004 and 2005.
\4\ Includes Mandatory Appropriations of $20 million for each year for the Survey of Program Dynamics and
collection of data related to the allocation to States of State Chidren's Health Insurance Program funds.
\5\ Includes funds for the periodic Census of Agriculture of $25, $22, and $29 million in 2004, 2005, and 2006,
respectively. The 2006 Budget includes an increase of $6.5 million due to cyclical activities including
finalizing content, developing mail lists, and streamlining and upgrading processing systems in preparation
for the 2007 Census of Agriculture.
\6\ All funds from the Public Health Service Evaluation Fund. Funds for 2004 are shown comparably with 2005 and
2006. Administrative costs for NCHS that previously were displayed as part of the NCHS budget line are now
reflected in two consolidated CDC-wide budget lines for management and administrative costs.
Highlights of 2006 Program Budget Proposals
The programs that provide essential statistical information for use by
governments, businesses, researchers, and the public are carried out by
some 70 agencies spread across every department and several independent
agencies. Approximately 40 percent of the funding for these programs
provides resources for twelve agencies or units that have statistical
activities as their principal mission. (Please see Table 4-1.) The
remaining funding supports work in 60-plus agencies or units that carry
out statistical activities in conjunction with other missions such as
providing services or enforcing regulations. More comprehensive budget
and program information about the Federal statistical system will be
available in OMB's annual report, Statistical Programs of the United
States Government, Fiscal Year 2006, when it is published later this
year. The following highlights elaborate on the Administration's
[[Page 57]]
proposals to strengthen the programs of the principal Federal
statistical agencies.
Bureau of Economic Analysis: Funding is requested to: (1) make
selected improvements to the timeliness and comprehensiveness of the
Nation's international statistics on multinational corporations and
trade in services; (2) complete work to accelerate the release of gross
state product, metropolitan personal income, and county-level personal
income; (3) enhance the accuracy of BEA statistics by acquiring and
incorporating real-time data into core BEA accounts; (4) improve data on
international financial transactions by working with the Department of
the Treasury and the Federal Reserve Board to incorporate newly
developed estimates of derivatives and other financial instruments; and
(5) produce up-to-date, annual estimates of business investment spending
by industry in order to more accurately discern where high-tech and
other investments are being made in the manufacturing and service
sectors.
Bureau of Justice Statistics: Funding is requested to provide for the
maintenance of BJS's core statistical programs, including: (1) the
National Crime Victimization Survey, the Nation's primary source of
information on criminal victimization; (2) cybercrime statistics on the
incidence, magnitude, and consequences of electronic and computer crime
to households and businesses; (3) law enforcement data from over 3,000
agencies on the organization and administration of police and sheriffs'
departments; (4) nationally representative prosecution data on
resources, policies, and practices of local prosecutors; (5) court and
sentencing statistics, including Federal and State case processing data;
(6) data on correctional populations and facilities from Federal, State,
and local governments; and (7) the Felony Arrestee Drug Use Reporting
program (previously funded as the Arrestee Drug Abuse Monitoring program
within the National Institute of Justice).
Bureau of Labor Statistics: Funding is requested to support program
operations to measure the economy through producing, disseminating, and
improving BLS
[[Page 58]]
economic measures, including: (1) modernizing the computing systems for
monthly processing of the Producer Price Index (PPI) and U.S. Import and
Export Price Indexes (IPP), and producing new data outputs, such as
indexes based on the North American Industry Classification System for
the IPP; (2) maintaining continuous updating of the Consumer Price Index
(CPI) by updating the expenditure and population weights biennially, the
superlative index annually, outlet samples on a four-year cycle, and
item samples in key categories on a two-year cycle; and (3) releasing
the 2004-2014 Employment Projections and publishing the 2006-2007
edition of the Occupational Outlook Handbook.
Bureau of Transportation Statistics: Funding is requested to: (1)
enhance the Freight Data Program, a continuous source of data from
shippers, carriers, and receivers, to replace the current Commodity Flow
Survey; (2) move the Air Transportation Price Index, an input to GDP and
CPI indices, from experimental to production mode; and (3) develop more
timely and comprehensive local and long-distance travel data.
Census Bureau: Funding is requested for the Census Bureau's economic
and demographic programs, and for a reengineered 2010 Census. For the
Census Bureau's economic and demographic programs, funding is requested
to: (1) plan for the 2007 Economic Census, (2) plan and implement the
organizational phase of the 2007 Census of Governments and plan for the
employment and finance phases, (3) improve measurement of services by
expanding key source data for critical quarterly and annual estimates of
our Nation's Gross Domestic Product, (4) support improved coverage and
electronic reporting of trade statistics, (5) support the development of
a database infrastructure to integrate State administrative data and
Census Bureau data products in order to fill critical data gaps at the
State and local levels, (6) continue efforts begun in 2003 to eliminate
data gaps by measuring migration across U.S. borders, and (7) purchase
furniture and relocate operations and employees to the new headquarters
facility to avoid disruption of mission-critical operations necessary
for the successful completion of Census Bureau surveys. For 2010 Census
planning, funding is requested to continue to: (1) conduct planning,
testing, and development activities to support a reengineered 2010
Census; (2) complete map feature accuracy within 7.6 meters of true GPS
location for 700 of the Nation's counties; and (3) continue to conduct
the American Community Survey program to provide data on an ongoing
basis rather than waiting for once-a-decade censuses.
Economic Research Service: Funding is requested to support ongoing
programs and to continue the development of an integrated and
comprehensive data and analysis framework of the food system beyond the
farm-gate to provide a basis for understanding, monitoring, tracking,
and identifying changes in the food supply and consumption patterns.
Energy Information Administration: Funding is requested to continue
ongoing operations, with a focus on: (1) improving petroleum and natural
gas data security, reliability, and quality; (2) conducting the
commercial, manufacturing, and residential energy consumption surveys;
(3) implementing the enhanced Voluntary Reporting of Greenhouse Gases
program to support the President's Climate Change Initiative; and (4)
developing a program performance prototype to assess EIA's data
collection and operations costs at a more disaggregated level.
National Agricultural Statistics Service: Funding is requested to: (1)
continue restoration and modernization of the agricultural estimates
program to ensure State, regional, and national level agricultural
estimates of sufficient precision, quality, and detail to meet the needs
of a broad customer base; (2) continue development and implementation of
a locality-based agricultural county estimates/small area estimation
program; and (3) continue preparations for the 2007 Census of
Agriculture.
National Center for Education Statistics: Funding is requested to: (1)
support the ongoing data collection and analysis of the Early Childhood
Longitudinal Study Birth and Kindergarten Cohorts, which provide data to
inform child development practices and early education; (2) continue the
Integrated Education Postsecondary Data System, which collects
information on enrollment, completions, and finances from postsecondary
institutions; (3) sustain the ongoing data collection efforts for the
Beginning Postsecondary Students Longitudinal Study; (4) maintain U.S.
participation in international assessments that compare educational
achievement in the United States with that in other countries; and (5)
continue the National Assessment of Educational Progress (NAEP) program,
including funding to support the expansion of State NAEP to grade 12.
National Center for Health Statistics: Funding is requested to: (1)
increase timeliness by upgrading electronic systems for data collection
and processing; (2) expand the content of surveys, particularly those
addressing the health care delivery system; (3) redesign the sample for
the National Health Interview Survey, NCHS' largest population survey;
and (4) work collaboratively with States and other agencies on upgrading
the technology for collecting data from State birth and death
certificates.
Science Resources Statistics Division, NSF: Funding is requested to:
(1) continue to implement the results of prior methodological,
analytical, and planning activities directed toward improving the
relevance, accuracy, timeliness, and accessibility of SRS products,
including the suite of Research and Development surveys and the Survey
of Graduate Students and Postdoctorates in Science and Engineering; and
(2) lead a cross-agency effort to examine and revise current
[[Page 59]]
taxonomies used for classifying academic fields of study, including the
development of crosswalks between existing taxonomies and any potential
new taxonomy, as well as strengthen methods to enhance the
identification and description of cross-disciplinary and multi-
disciplinary fields.
Statistics of Income Division, IRS: Funding is requested to: (1)
maintain and modernize core data collection systems, including several
major statistical programs for the Department of the Treasury, the
Congressional Joint Committee on Taxation, the Bureau of Economic
Analysis, and SOI's many other customers; (2) implement a databank
repository for SOI and IRS population file data to more efficiently
build longitudinal databases and enable sub-national estimates; (3)
examine means to more effectively mask individual records to minimize
the possibility of identification in the Individual Public Use sample
files; and (4) modernize and expedite dissemination of data and
publications, including a reengineered Internet website.
[[Page 61]]