[Analytical Perspectives]
[Crosscutting Programs]
[4. Strengthening Federal Statistics]
[From the U.S. Government Printing Office, www.gpo.gov]
[[Page 37]]
4. STRENGTHENING FEDERAL STATISTICS
Federal statistical programs produce key information to inform public
and private decision makers about a range of topics of interest,
including the economy, the population, agriculture, crime, education,
energy, the environment, health, science, and transportation. The
ability of governments, businesses, and citizens to make appropriate
decisions about budgets, employment, investments, taxes, and a host of
other important matters depends critically on the ready availability of
relevant, accurate, and timely Federal statistics.
The Federal statistical community remains on alert for opportunities
to strengthen these measures of our Nation's performance. For example,
during 2006, Federal statistical agencies improved their measures of the
knowledge economy by releasing a preliminary Research and Development
Satellite Account that estimates the effect of investment in research
and development on U.S. economic growth (BEA and NSF); published, for
the first time, estimates of households experiencing identity theft
victimization and its consequences (BJS); developed procedures to ease
the reporting burden of the 2007 Economic Census by enhanced electronic
reporting, and to collect product data from all 350 service industries,
up from 80 in the last census (Census Bureau); published data on the
labor force status of persons who evacuated their homes due to Hurricane
Katrina (BLS); developed and tested quality improvements to the
Commodity Flow Survey, the most comprehensive source of nationwide data
on the transportation of goods (BTS and Census Bureau); introduced new
interactive web-based tools to facilitate access to, and use of, health
statistics information (NCHS); expanded internet data collection systems
to securely process energy survey data more quickly and obtain better
quality data (EIA); provided Internet access to forecasts of current
year farm income (ERS); offered podcasts of farm broadcast news stories
(NASS); and continued the modernization and reengineering of the
decennial census to improve its accuracy and usefulness while containing
costs (Census Bureau).
For Federal statistical programs to effectively benefit their wide
range of users, the underlying data systems must be viewed as credible.
In order to foster this credibility, Federal statistical programs seek
to adhere to high quality standards and to maintain integrity and
efficiency in the production of data. As the collectors and providers of
these basic statistics, the responsible agencies act as data stewards--
balancing public and private decision makers' needs for information with
legal and ethical obligations to minimize reporting burden, respect
respondents' privacy, and protect the confidentiality of the data
provided to the Government. This chapter discusses the development of
standards that principal statistical programs use to assess their
performance and presents highlights of their 2008 budget proposals.
Performance Standards
Statistical programs maintain the quality of their data or information
products as well as their credibility by setting high performance
standards for their activities. The statistical agencies and statistical
units represented on the Interagency Council on Statistical Policy
(ICSP) have collaborated on developing an initial set of common
performance standards for use under the Government Performance and
Results Act and in completing the Administration's Program Assessment
Rating Tool (PART). Federal statistical agencies have agreed that there
are six conceptual dimensions within two general areas of focus that are
key to measuring and monitoring statistical programs. The first area of
focus is Product Quality, encompassing the traditional dimensions of
relevance, accuracy, and timeliness. The second area of focus is Program
Performance, encompassing the dimensions of cost, dissemination, and
mission achievement.
Statistical agencies historically have focused on measuring
performance in the area of product quality, especially dimensions of
accuracy and timeliness that are most amenable to quantitative
measurement. Relevance, also an accepted measure of quality, can be
either a qualitative description of the usefulness of products or a
quantitative measure such as a customer satisfaction score. Relevance is
more difficult to measure, and the indicators that do exist are more
varied.
Program performance standards form the basis for evaluating
effectiveness. They address questions such as: Are taxpayer dollars
spent most effectively? Are products made available to those who need
them? Are agencies meeting their mission requirements or making it
possible for other agencies to meet their missions? The indicators
available to measure program performance for statistical activities
currently are less well developed.
Description of Dimensions
Product Quality
Relevance: Qualitative or quantitative descriptions of the degree to which products and
services are useful to users and responsive to users' needs.
Accuracy: Qualitative or quantitative measure of important features of correctness,
validity, and reliability of data and information products measured as degree of closeness
to target values.
Timeliness: Qualitative or quantitative measure of the timing of information releases.
Program Performance
Cost: Quantitative measure of the dollar amount used to produce data products and services.
Dissemination: Qualitative or quantitative information on the availability, accessibility,
and distribution of products and services.
Mission Achievement: Qualitative or quantitative information about the effect of, or
satisfaction with, statistical programs.
Key to Statistical Agencies
BEA = Bureau of Economic Analysis, Department of Commerce
BJS = Bureau of Justice Statistics, Department of Justice
BLS = Bureau of Labor Statistics, Department of Labor
BTS = Bureau of Transportation Statistics, Department of Transportation
Census = Census Bureau, Department of Commerce
EIA = Energy Information Administration, Department of Energy
ERS = Economic Research Service, Department of Agriculture
NASS = National Agricultural Statistics Service, Department of Agriculture
NCES = National Center for Education Statistics, Department of Education
NCHS = National Center for Health Statistics, Department of Health and Human Services
ORES = Office of Research, Evaluation, and Statistics, Social Security Administration
SOI = Statistics of Income, Internal Revenue Service, Department of the Treasury
SRS = Science Resources Statistics Division, National Science Foundation
Product quality and program performance standards are designed to
serve as indicators when answering specific questions in the
Administration's PART process. Chart 4-1 presents each principal Federal
statistical agency's assessment of the status of its current and planned
use of indicators on the six dimensions. With the exception of cost
indicators, where three agencies (ERS, NCES, and NCHS) are still
planning their
[[Page 38]]
measures, the ICSP agencies have now developed performance measures for
all six dimensions. Use of the indicators may be for internal
management, strategic planning, or annual performance reporting. The
dimensions shown in the chart reflect an overall set of indicators for
statistical activities, but the specific measures vary among the
individual programs depending on their unique characteristics and
requirements. Annual performance reports and PARTs provide these
specific measures, as well as additional information about performance
goals and targets and whether a program is meeting, or making measurable
progress toward meeting, its performance goals. The examples below
illustrate different ways agencies track their performance on each
dimension.
Product Quality: Statistical agencies agree that product quality
encompasses many attributes, including (but not limited to) relevance,
accuracy, and timeliness. The basic measures in this group relate to the
quality of specific products, thereby providing actionable information
to managers. These are ``outcome-oriented'' measures and are key to the
usability of information products. Statistical agencies or units
establish targets and monitor how well targets are met. In some sense,
[[Page 39]]
relevance relates to ``doing the right things,'' while accuracy and
timeliness relate to ``doing things right.''
Relevance: Qualitative or quantitative descriptions of the
degree to which products and services are useful and
responsive to users' needs. Relevance of data products and
analytic reports may be monitored through a professional
review process and ongoing contacts with data users. Product
relevance may be indicated by customer satisfaction with
product content, information from customers about product use,
demonstration of product improvements, comparability with
other data series, agency responses to customer suggestions
for improvement, new or customized products or services,
frequency of use, or responses to data requests from users
(including policy makers). Through a variety of professional
review activities, agencies maintain the relevance and
validity of their products, and encourage data users and other
stakeholders to contribute to the agencies' data collection
and dissemination programs. Striving for relevance requires
monitoring to ensure that information systems anticipate
change and evolve to appropriately measure our dynamic society
and economy.
Accuracy: Qualitative or quantitative measures of important
features of correctness, validity, and reliability of data and
information products measured as degree of closeness to target
values. For statistical data, accuracy may be defined as the
degree of closeness to the target value and measured as
sampling error and various aspects of nonsampling error (e.g.,
response rates, size of revisions, coverage, edit
performance). For analysis products, accuracy may be the
quality of the reasoning, reasonableness of assumptions, and
clarity of the exposition, typically measured and monitored
through review processes. In addition, accuracy is assessed
and improved by internal reviews, comparisons of data among
different surveys, linkages of survey data to administrative
records, redesigns of surveys, or expansions of sample sizes.
Timeliness: Qualitative or quantitative measure of timing
of information releases. Timeliness may be measured as time
from the close of the reference period to the release of
information, or customer satisfaction with timeliness.
Timeliness may also be measured as how well agencies meet
scheduled and publicized release dates, expressed as a percent
of release dates met.
Program Performance: Statistical agencies agree that program
performance encompasses balancing the dimensions of cost, dissemination,
and mission accomplishment for the agency as a whole; operating
efficiently and effectively; ensuring that customers receive the
information they need; and serving the information needs of the Nation.
Costs of products or programs may be used to develop efficiency
measures. Dissemination involves making sure customers receive the
information they need via the most appropriate mechanisms. Mission
achievement means that the information program makes a difference.
Hence, three key dimensions are being used to indicate program
performance: cost (input), dissemination (output), and mission
achievement (outcome).
Cost: Quantitative measure of the dollar amount to produce
data products or services. The development and use of
financial performance measures within the Federal Government
is an established goal; the intent of such measures is to
determine the ``true costs'' of various programs or
alternative modes of operation at the Federal level. Examples
of cost data include full costs of products or programs,
return on investment, dollar value of efficiencies, and ratios
of cost to products distributed.
Dissemination: Qualitative or quantitative information on
the availability, accessibility, and distribution of products
and services. Most agencies have goals to improve product
accessibility, particularly through the Internet. Typical
measures include: on-demand requests fulfilled, product
downloads, degree of accessibility, customer satisfaction with
ease of use, number of participants at user conferences,
citations of agency data in the media, number of Internet user
sessions, number of formats in which data are available,
amount of technical support provided to data users, exhibits
to inform the public about information products, issuance of
newsletters describing products, usability testing of web
sites, and assessing compliance with Section 508 of the
Rehabilitation Act, which requires Federal agencies to make
their electronic and information technology accessible to
people with disabilities.
Mission Achievement: Qualitative or quantitative
information about the effect of, or satisfaction with,
statistical programs. For Government statistical programs,
this dimension responds to the question--have we achieved our
objectives and met the expectations of our stakeholders? Under
this dimension, statistical programs document their
contributions to the goals and missions of parent departments
and other agencies, the Administration, the Congress, and
information users in the private sector and the general
public. For statistical programs, this broad dimension
involves meeting recognized societal information needs; it
also addresses the linkage between statistical outputs and
programmatic outcomes.
However, identifying this linkage is far from
straightforward. It is frequently difficult to trace the
effects of information products on the public good. Such
products often are necessary intermediate inputs in the
creation of high visibility information whose societal benefit
is clearly recognized. For example, the economic statistics
pro
[[Page 40]]
duced by a variety of agencies are directly used by the Bureau
of Economic Analysis in the calculation of the Gross Domestic
Product (GDP), which analysts universally use to assess
changes in the level of domestic economic activity. Similarly,
statistics from specific surveys are directly used by the
Bureau of Labor Statistics in the calculation of the Consumer
Price Index (CPI), which is widely used in diverse
applications, such as indexing pensions for retirees. As a
result, a number of statistical agencies can claim credit for
contributing to the GDP and/or the CPI and to the many uses of
these information products. In addition, statistics produced
by Federal agencies are used to track the performance of
programs managed by their parent or other organizations
related to topics such as crime, education, energy, the
environment, health, science, and transportation.
Moreover, beyond the direct and focused uses of statistical
products, the statistical agencies and their programs serve a
diverse and dispersed set of data users working on a broad
range of applications. Users include government policy makers
at the Federal, State, and local levels, business leaders,
households, academic researchers, analysts at public policy
institutes and trade groups, marketers and planners in the
private sector, and many others. Information produced by
statistical agencies often is combined with other information
for use in the decision-making process. Thus, the relationship
between program outputs and their beneficial uses and outcomes
is often complex and difficult to track. Consequently,
agencies use both qualitative and quantitative indicators to
make this linkage as explicit as feasible.
In the absence of preferred quantitative indicators,
qualitative narratives can indicate how statistical agency
products contribute to and evaluate progress toward important
goals established for government or private programs. In
particular, narratives can highlight how statistical agencies
measure the Nation's social and economic structure, and how
the availability of the information influences changes in
policies and programs. These narratives contribute to
demonstrating mission accomplishment, particularly in response
to questions in Section I of the PART, ``program purpose and
design.'' Narratives may describe statistical information's
effects on measuring agency policy or change of policy,
supporting research focused on policy issues, informing debate
on policy issues, or providing in-house consulting support.
In addition to narratives, quantitative measures may be used
to reflect mission achievement. For example, customer
satisfaction with the statistical agency or unit indicates if
the agency or unit has met the expectations of its
stakeholders.
Of the 14 principal Federal statistical agencies or units that are
members of the ICSP, eleven agencies have programs that have been
assessed using the PART process. All but one of these agencies' programs
have received PART summary ratings of Effective or Moderately Effective,
as shown in Chart 4-2. While recognizing the strength of the Energy
Information Administration's purpose and management, in 2004 EIA
received an initial rating of ``Results Not Demonstrated'' for two key
reasons, both of which have since been rectified. At the time of the
evaluation, EIA had recently adopted new performance measures and lacked
the necessary historical baselines and future targets; these now exist
for all measures. EIA was also critiqued for having no recurring
independent evaluation of its entire program. EIA recruited an energy
expert from the Massachusetts Institute of Technology to select and lead
a team to conduct such an evaluation, and the team completed its report
in 2006. EIA management will evaluate the team's recommendations as part
of its strategic planning process in 2007. As additional ICSP agencies
have an opportunity to undergo the PART process, the agencies plan to
continue to use the results of the collaborative performance standards
development effort to help maintain and extend their generally favorable
assessments.
Chart 4-2. MOST RECENT PART SUMMARY RATINGS FOR STATISTICAL PROGRAMS
------------------------------------------------------------------------
Summary Rating
------------------------------------------------------------------------
Bureau of Economic Analysis Effective
Bureau of Justice Statistics
Criminal Justice Statistics Effective
Program
National Criminal History Moderately Effective
Improvement Program
Bureau of Labor Statistics Effective
Bureau of Transportation Statistics Moderately Effective
Census Bureau
Current Demographic Statistics Effective
Decennial Census Moderately Effective
Intercensal Demographic Estimates Moderately Effective
Survey Sample Redesign Effective
Economic Census Effective
Current Economic Statistics Moderately Effective
/Census of Governments
Economic Research Service Effective
Energy Information Administration Results Not Demonstrated
National Agricultural Statistics Moderately Effective
Service
National Center for Education
Statistics
Statistics Effective
Assessment Effective
National Center for Health Moderately Effective
Statistics
Science Resources Statistics
Division, NSF
NSF's Infrastructure and Effective
Instrumentation component
------------------------------------------------------------------------
[[Page 41]]
Highlights of 2008 Program Budget Proposals
The programs that provide essential statistical information for use by
governments, businesses, researchers, and the public are carried out by
more than 70 agencies spread across every department and several
independent agencies. Approximately 40 percent of the funding for these
programs provides resources for 13 agencies or units that have
statistical activities as their principal mission. (Please see Table 4-
1.) The remaining funding supports work in 60-plus agencies or units
that carry out statistical activities in conjunction with other missions
such as providing services or enforcing regulations. More comprehensive
budget and program information about the Federal statistical system will
be available in OMB's annual report, Statistical Programs of the United
States Government, Fiscal Year 2008, when it is published later this
year. The following highlights elaborate on the Administration's
proposals to strengthen the programs of the principal Federal
statistical agencies.
Bureau of Economic Analysis: Funding is requested to: (1) extend the
prototype Research & Development satellite account, funded by the
National Science Foundation in 2006 and 2007, with annual updates and
extensions to BEA's Gross Domestic Product and other estimates between
2008 and 2012, and full incorporation into the economic accounts in
2013; (2) complete BEA's five-year program to improve the accuracy and
timeliness of the Nation's economic accounts by addressing data gaps and
measurement problems, expanding integration with other accounts, and
improving consistency with international standards; and (3) continue to
improve the accuracy of statistics on services, profits, compensation,
international trade in services, and off-shoring.
Bureau of Justice Statistics: Funding is requested to provide for
BJS's core statistical programs and for two initiatives: (1) a redesign
of the National Crime Victimization Survey based on anticipated
recommendations from the Committee on National Statistics of the
National Research Council; and (2) development of a national recidivism
statistical series, which will provide baseline data, as well as
representative data every 3 years, on the rates of rearrest,
reconviction, and reincarceration among released State and Federal
prisoners to provide a quantitative basis for evaluating the
effectiveness of reentry programs, post-custody surveillance, and State
policies related to parole revocation.
Bureau of Labor Statistics: Funding is requested to support the
production, dissemination, and improvement of BLS economic measures,
including: (1) the introduction of continuous updating to the housing
and geographic area samples in the Consumer Price Index (CPI), which
will improve the accuracy and timeliness of the CPI; (2) the
continuation of efforts to modernize the computing systems for monthly
processing of the Producer Price Index (PPI) and U.S. Import and Export
Price Indexes (IPP); and (3) the publication, for the first time, of
local area Employment Cost Index (ECI) and Employer Costs for Employee
Compensation (ECEC) series as deemed feasible as a result of testing
completed in 2007.
Bureau of Transportation Statistics: Funding is requested to: (1)
conduct the Commodity Flow Survey, a major national benchmark survey of
shippers; (2) release monthly statistics on the commodities and mode of
transportation used in trading with our largest partners; (3) produce a
core set of economic data and indicators, including the Government
Transportation Financial Statistics Report, multi-factor productivity
measures, the State Transit Expenditure Survey, and the Air Travel Price
Index; (4) produce and release the National Transportation Atlas Data
Base, a compendium of national geospatial transportation data; and (5)
conduct the biennial Census of Ferry Operations in the U.S.
Census Bureau: Funding is requested for the Census Bureau's ongoing
economic and demographic programs and for a re-engineered 2010 Census.
For the Census Bureau's economic and demographic programs, funding is
requested to: (1) collect and process economic census returns for the
2007 Economic Census; (2) create the universe frame and develop
organizational information for the 2007 Census of Governments, as well
as collect and process data for the employment phase, and collect and
process data from States and other sources for the finance phase; (3)
undertake an initiative to close the current gap in service sector
coverage; and (4) continue reengineering the Survey of Income and
Program Participation. For the 2010 Census program, funding is requested
to continue to: (1) conduct planning, testing, and development
activities to support a re-engineered 2010 Census, including the 2008
Census Dress Rehearsal and early operations for the 2010 Census; (2)
improve the accuracy of map feature locations for the remaining 367
counties of the total of 3,232 counties; and (3) continue to conduct the
American Community Survey to provide socio-economic data on an ongoing
basis rather than only once-a-decade.
Economic Research Service: Funding is requested to: (1) strengthen and
enhance the ERS market analysis and outlook program to provide timely
analysis of global agricultural product markets; and (2) strengthen
ERS's research and modeling capacity in the area of bio-energy with
particular emphasis given to the changing economics of livestock feeding
and the role of ethanol byproducts.
Energy Information Administration: Funding is requested to continue
ongoing operations to: (1) maintain critical energy data coverage,
analysis, and forecasting; (2) improve data reliability and statistical
accuracy through redesigning key petroleum and natural
[[Page 42]]
gas surveys; (3) initiate monthly ethanol and biofuels data collections
on a national and regional basis as mandated in Section 1508 of the
Energy Policy Act of 2005; (4) strengthen global oil and gas data and
modeling capabilities; and (5) improve the ability to assess and
forecast supply, demand, and technology trends affecting U.S. and world
energy markets.
National Agricultural Statistics Service: Funding is requested to
support printing, postage and handling of questionnaire packages,
logging returned questionnaires, capturing reported data, and conducting
telephone and personal follow-up interviews with nonrespondents for the
quinquennial Census of Agriculture via questionnaires that are scheduled
to be mailed to the Nation's agricultural producers in December 2007.
National Center for Education Statistics: Funding is requested to: (1)
conduct the National Assessment of Educational Progress, including 12th
grade reading and mathematics assessments in 2009; (2) plan for a new
high school longitudinal study that will begin with a cohort of 9th
graders in 2009 and follow them through postsecondary education and into
the workforce; (3) analyze data from international studies such as the
2007 Trends in International Mathematics and Science Study and plan for
new international assessments; (4) undertake a pilot study on the
development of postsecondary unit records, an essential restructuring of
several components of the Integrated Postsecondary Education Data
System; (5) carry out the 2007-08 Schools and Staffing Survey to obtain
information on public and private schools, principals, and teachers; and
(6) conduct the Beginning Postsecondary Student Longitudinal Survey,
which provides information on the progress of postsecondary students, as
well as the 2008 National Postsecondary Student Aid Survey.
National Center for Health Statistics: Funding is requested to: (1)
continue data collection, analysis, and dissemination for key national
health data systems, including the National Vital Statistics System,
National Health Interview Survey, National Health and Nutrition
Examination Survey, and National Health Care Survey; (2) continue gains
in timeliness by implementing systems improvements in data collection
and processing; (3) continue efforts to develop survey data that address
the health care delivery system; and (4) work collaboratively with
States and other agencies on upgrading the technology for collecting
data from State birth and death certificates.
Office of Research, Evaluation, and Statistics, SSA: Funding is
requested to: (1) continue strategic planning to modernize ORES's
processes for developing and disseminating data from the Social Security
Administration's major administrative data files for statistical
purposes; (2) support outside surveys and linkage of SSA administrative
data to surveys; (3) create a new public use file of administrative data
on earnings histories and benefits for a sample of Social Security
numbers; and 4) evaluate the analytic validity of a synthetic data file
based on data from the 1990-1993 and 1996 Survey of Income and Program
Participation (SIPP) panels matched to SSA and IRS administrative data.
Science Resources Statistics Division, NSF: Funding is requested to:
(1) implement ongoing programs on the science and engineering
enterprise; (2) continue to implement redesign and improvement
activities for a broad range of surveys, particularly the suite of
research and development (R&D) surveys; (3) support the NSF/SBE
initiative on the Science of Science and Innovation Policy to develop
the data, tools, and knowledge needed for a new science of science
policy by enhancing the comparability, scope and availability of
international data; and (4) develop data on innovation and R&D conducted
or funded by nonprofit organizations.
Statistics of Income Division, IRS: Funding is requested to: (1)
maintain and modernize tax data collection systems, including developing
interfaces with modern electronic tax return filing systems; (2)
implement a databank repository for SOI and IRS population file data to
more efficiently build longitudinal databases and enable sub-national
estimates; (3) examine means to more effectively mask individual records
to minimize the possibility of identification in the Individual Public
Use Sample files; and (4) modernize and expedite dissemination of data
and publications, including enhancement of products and features on the
www.irs.gov/taxstats website.
[[Page 43]]
Table 4-1. 2006-2008 BUDGET AUTHORITY FOR PRINCIPAL STATISTICAL AGENCIES\1\
(In millions of dollars)
----------------------------------------------------------------------------------------------------------------
Estimate
2006 -------------------------
Actual 2007 2008
----------------------------------------------------------------------------------------------------------------
Bureau of Economic Analysis ............................................. 75 75 81
Bureau of Justice Statistics \2\......................................... 50 50 62
Bureau of Labor Statistics............................................... 537 537 573
Bureau of Transportation Statistics...................................... 27 27 27
Census Bureau \3\........................................................ 822 817 1250
Salaries and Expenses \3\.............................................. 216 210 223
Periodic Censuses and Programs......................................... 606 607 1027
Economic Research Service \4\............................................ 75 75 83
Energy Information Administration........................................ 85 85 105
National Agricultural Statistics Service \5\............................. 139 140 168
National Center for Education Statistics................................. 183 183 236
Statistics............................................................. 90 90 119
Assessment............................................................. 88 88 111
National Assessment Governing Board.................................... 5 5 6
National Center for Health Statistics \6\................................ 109 109 110
Office of Research, Evaluation, and Statistics, SSA...................... 16 18 15
Science Resources Statistics Division, NSF............................... 33 33 37
Statistics of Income Division, IRS....................................... 38 41 41
----------------------------------------------------------------------------------------------------------------
\1\ Reflects any recissions.
\2\ Includes funds for management and administrative costs of $11, $11, and $17 million in 2006, 2007, 2008,
respectively that were previously displayed separately.
\3\ Includes Mandatory Appropriations of $20 million for each year for the Survey of Program Dynamics and
collection of data related to the allocation to States of State Children's Health Insurance Program funds.
\4\ 2007 funding assumes the reallocation of $350,000 provided in 2006 for a comprehensive report on the
economic development and current status of the sheep industry in the United States. Funding for that purpose
will not be needed in 2007.
\5\ Includes funds for the periodic Census of Agriculture of $29, $29, and $54 million in 2006, 2007, and 2008,
respectively. The FY 2008 Budget includes an increase of $24.7 million due to cyclical activities.
\6\ All funds from the Public Health Service Evaluation Fund. Administrative costs for NCHS that previously were
displayed as part of the NCHS budget line are now reflected in two consolidated CDC-wide budget lines for
management and administrative costs.