Nutrition Monitoring: Establishing a Model Program (Chapter Report,
07/19/95, GAO/PEMD-95-19).
GAO was asked to identify features of a model nutrition monitoring
system and examine approaches to incorporating those features in the
National Nutrition Monitoring and Related Research Program, which
consists of a combination of national surveys, federally-supported
surveillance systems operated by the states, and other research and data
collection activities. GAO identified four features of a model program.
It would have a coordinated set of activities, provide data
continuously, generate reliable inferences about important
subpopulations and small geographic areas, and support state and local
monitoring activities.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: PEMD-95-19
TITLE: Nutrition Monitoring: Establishing a Model Program
DATE: 07/19/95
SUBJECT: Nutrition surveys
Nutrition research
Monitoring
Food supply
Health research programs
Interagency relations
Data integrity
Demographic data
Data collection operations
IDENTIFIER: National Nutrition Monitoring and Related Research Program
HNIS Nationwide Food Consumption Survey
USDA Continuing Survey of Food Intakes by Individuals
USDA Thrifty Food Plan
PHS Pediatric Nutrition Surveillance System
PHS Pregnancy Nutrition Surveillance System
Special Supplemental Food Program for Women, Infants, and
Children
FDA Food Label and Package Survey
HHS Healthy People 2000 Program
High Performance Computing and Communications Program
Census Bureau Current Population Survey
WIC
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to the Ranking Minority Member, Committee on Science, House of
Representatives
July 1995
NUTRITION MONITORING -
ESTABLISHING A MODEL PROGRAM
GAO/PEMD-95-19
Features of a Model Nutrition Monitoring Program
(973397)
Abbreviations
=============================================================== ABBREV
ARS - Agricultural Research Service
BRFSS - Behavioral Risk Factor Surveillance Survey
CPS - Current Population Survey
CSFII - Continuing Survey of Food Intake by Individuals
DHKS - Diet and Health Knowledge Survey
HHANES - Hispanic Health and Nutrition Examination Survey
HHS - Department of Health and Human Services
HPCCP - High Performance Computing and Communications Program
NCST - National Council on Science and Technology
NFCS - Nationwide Food Consumption Survey
NHANES - National Health and Nutrition Examination Survey
NNMRRP - National Nutrition Monitoring and Related Research Program
OMB - Office of Management and Budget
ONDCP - Office of National Drug Control Policy
OSTP - Office of Science and Technology Policy
PATCH - Planned Approach to Community Health
PedNSS - Pediatric Nutrition Surveillance System
PHS - Public Health Service
PNSS - Pregnancy Nutrition Surveillance System
USDA - U.S. Department of Agriculture
WIC - Special Supplemental Food Program for Women, Infants, and
Children
Letter
=============================================================== LETTER
B-261325
July 19, 1995
The Honorable George E. Brown, Jr.
Ranking Minority Member
Committee on Science
House of Representatives
Dear Mr. Brown:
This report responds to your request that we identify features of a
model nutrition monitoring system and examine approaches to
incorporating those features in the National Nutrition Monitoring and
Related Research Program (NNMRRP). It builds on two earlier reports.
The first, Nutrition Monitoring: Progress in Developing a
Coordinated Program (GAO/PEMD-94-23), assessed the NNMRRP's planning
activities. The second, Nutrition Monitoring: Data Serve Many
Purposes; Users Recommend Improvements (GAO/PEMD-95-15), surveyed
users of nutrition monitoring data about how the data are used and
what changes are needed to the data collection activities. This
report describes features of a model nutrition monitoring system and
presents the strengths and limitations of various strategies for
achieving four of the model features.
As arranged with your office, we will be sending copies of this
report to the Director of the Office of Management and Budget, the
Interagency Board for Nutrition Monitoring and Related Research, the
agencies responsible for data collection, and interested
congressional committees. We will also make copies available to
others upon request.
The major contributors to this report are listed in appendix V. If
you have any questions or would like additional information, please
call me at (202) 512-3092.
Sincerely yours,
Kwai-Cheung Chan
Director of Program Evaluation
in Physical Systems Areas
EXECUTIVE SUMMARY
============================================================ Chapter 0
PURPOSE
---------------------------------------------------------- Chapter 0:1
Nutrition-related data are used to monitor food safety, study the
relationship between diet and health, inform agricultural policies,
evaluate food assistance programs, and help food industries develop
new products. However, past evaluations have criticized federal
nutrition monitoring activities as poorly coordinated and
insufficiently responsive to the needs of data users. At the request
of the former House Committee on Science, Space, and Technology, GAO
defined the features of a model monitoring program and compared
current and potential approaches to responding to the needs of data
users.
BACKGROUND
---------------------------------------------------------- Chapter 0:2
The current National Nutrition Monitoring and Related Research
Program (NNMRRP) consists of a combination of national surveys,
federally- supported surveillance systems operated by the states, and
other research and data collection activities. The 1990 National
Nutrition Monitoring and Related Research Act (P.L. 101-445)
established an Interagency Board that gave the Departments of
Agriculture (USDA) and Health and Human Services (HHS) primary
responsibility for developing and implementing a 10-year
comprehensive plan for federal nutrition monitoring.\1
In reviewing the Interagency Board's progress in coordinating the
program, an earlier GAO report found that coordination between USDA
and HHS had improved, but the 10-year plan was still inadequate. It
neither established priorities nor provided a framework for
evaluating existing or potential monitoring activities.\2 Drawing on
a survey of users of nutrition monitoring data, a second GAO report
found that the data collected by the NNMRRP support an extensive
range of purposes, but changes could increase the utility and
credibility of the data.\3 For this report, GAO first defined the
features of a model program by reviewing past evaluations of federal
nutrition monitoring and the 10-year plan, consulting experts, and
surveying users of nutrition data. Then, with the assistance of
expert advisers and studies of illustrative programs, GAO identified
and compared current and potential approaches to incorporating model
features in the NNMRRP. (See chapter 1.)
--------------------
\1 Ten-Year Comprehensive Plan for the National Nutrition Monitoring
and Related Research Program, 58 Fed. Reg. 111 (June 11, 1993), pp.
32752-806.
\2 See Nutrition Monitoring: Progress in Developing a Coordinated
Program (GAO/PEMD-94-23; May 27, 1994).
\3 See Nutrition Monitoring: Data Serve Many Purposes; Users
Recommend Improvements (GAO/PEMD-95-15; June 20, 1995).
RESULTS IN BRIEF
---------------------------------------------------------- Chapter 0:3
GAO identified four features of a model program: It would have a
coordinated set of activities, provide data continuously, generate
reliable inferences about important subpopulations and small
geographic areas, and support state and local monitoring activities.
While the NNMRRP already has some elements of a model program, other
strategies may lead to improved nutrition monitoring capabilities.
Alternate approaches to coordination, such as an independent central
office with authority over the NNMRRP agencies or a single lead
agency, may not provide any clear advantages to the current
structure, which relies on the Interagency Board. However, such
approaches do offer useful tools that could be adopted by the Board,
such as the use of formal budget reviews and the establishment of a
central contact for data users. The current source of continuous
data--the state-based surveillance systems--cannot meet the needs for
national-level information or for in-depth dietary intake
information. Alternatives are to attach modules of nutrition-related
questions to other ongoing surveys or to field a core set of
questions continuously, supplemented periodically by questions of
intermittent or emerging interest. Current approaches to providing
more information on subpopulation groups and small geographic areas
are to oversample selected groups as part of the national surveys and
to collect data on specific high-risk groups through the surveillance
systems. These approaches could be complemented by special studies
and indirect estimation. Finally, to support state and local
monitoring activities, HHS provides a combination of technical and
financial assistance for the state-based surveillance systems. One
option to provide more relevant data to localities is community-based
data collection.
GAO'S ANALYSIS
---------------------------------------------------------- Chapter 0:4
FEATURES OF A MODEL
NUTRITION MONITORING PROGRAM
-------------------------------------------------------- Chapter 0:4.1
Coordination of the data collection activities is needed to ensure
that the diverse needs for the data are met efficiently. For
example, NNMRRP's major national surveys collect complementary data,
with one focusing on health and nutritional status and the others
emphasizing food consumption. Combining these data would increase
their utility. (See chapter 2.) Continuous data collection,
"repeated regularly and frequently" (as defined in the 10-year plan),
is needed to track trends in diet-related health risks, such as those
targeted in HHS' year 2000 health objectives. In addition,
continuous data are important in evaluating the effects of policy
changes, such as replacing current food assistance programs with
block grants.
The dietary habits and nutritional problems of a subpopulation or
small geographic area may deviate from the national profile. Data on
these groups are needed to appropriately target programs that address
their nutrition-related problems, such as diabetes in some minority
groups. Finally, states and localities need relevant nutrition
monitoring data to plan programs that respond to the specific health
and nutrition concerns of their populations. For example, currently
available data have been used to initiate "5-a-Day" programs that
encourage consumption of fruits and vegetables and to establish
anemia as a priority issue for nutrition education.
CURRENT AND POTENTIAL
APPROACHES TO ACHIEVING A
MODEL PROGRAM
-------------------------------------------------------- Chapter 0:4.2
While the Interagency Board appears to have improved communication
and cooperation among the agencies, its 10-year plan did not
establish clear priorities for nutrition monitoring across agencies.
The Board has conducted a priority-setting exercise, but it has made
little progress in developing a framework to guide NNMRRP
decision-making. The two alternate approaches to coordinating
nutrition monitoring that GAO examined suggest mechanisms for
strengthening the work of the Interagency Board. Appointing an
NNMRRP administrator, for example, could provide a central contact
point for users of nutrition monitoring data. In addition, regular
budget reviews could be conducted to ensure that priority objectives
of the programs are funded. (See chapter 3.)
None of the three major national surveys are implemented
continuously. Moreover, the surveillance systems, which are the
current source of continuous data, cannot meet the needs of data
users, who depend on national-level data because of limitations in
the amount and kind of data they gather and in the populations they
target. A continuous national survey could be more efficient in
terms of costs per respondent than the current, intermittent surveys,
especially if a smaller, core set of data elements are collected
continuously and supplemented periodically by topical modules.
However, activities that are now implemented sequentially would have
to be funded and staffed to operate concurrently. Another approach
to gathering information more frequently is the inclusion of
nutrition-related questions on other ongoing surveys. (See chapter
4.)
At present, the state-based surveillance systems and oversampling in
the national surveys provide information on subpopulations and small
geographic areas. Two surveillance systems focus on low-income
pregnant women and children, but not all states participate and data
are collected only from participants in public health and nutrition
programs. Oversampling of specific subpopulations as part of a
national survey is most appropriate for groups that are
geographically clustered and that do not require different data
collection procedures than those used for the general population.
Its advantage is that the data on the group of interest can be
compared to national estimates.
Two alternate ways of gathering information on subpopulations and
small geographic areas are special studies and indirect estimation
programs. Special studies are appropriate for populations that are
geographically dispersed or require different data collection
procedures, but they can be costly relative to other approaches.
Indirect estimates, which use data from other sources to predict the
value for the group or area of interest, are likely to be less costly
than collecting new data for direct estimation. However, gauging the
bias introduced by the models that produce the indirect estimates is
difficult; thus, they may not be as useful as direct estimates. (See
chapter 5.)
Through the state surveillance systems, HHS provides assistance in
the collection, analysis, and interpretation of state and local data.
Although the data collection instruments can be modified from state
to state, the systems are organized to collect standardized
information across the participating states. As a result, they may
have limited sensitivity to state or local differences. An alternate
strategy is to design monitoring programs specifically for local
needs. However, unlike the data collected by the surveillance
systems, community-based data could probably not be aggregated to
describe statewide needs. (See chapter 6.)
RECOMMENDATIONS
---------------------------------------------------------- Chapter 0:5
GAO is not making recommendations in this report.
AGENCY COMMENTS
---------------------------------------------------------- Chapter 0:6
In their written comments on a draft of this report, officials from
USDA and HHS (1) found the report to be factual, (2) agreed with the
model program features identified by GAO, (3) agreed with the
importance of the model features discussed in the report, and (4)
provided additional detail on the progress that has been made by the
Interagency Board and its working groups toward improving NNMRRP.
Both USDA's and HHS' comments as well as comments received from three
members of the National Nutrition Monitoring Advisory Council noted
that limited resources affect the NNMRRP's ability to incorporate the
model features. (See appendixes III and IV for comments from USDA
and HHS and our response.)
THE CURRENT NUTRITION MONITORING
PROGRAM
============================================================ Chapter 1
This chapter first provides background information on the National
Nutrition Monitoring and Related Research Program. Then, the
objectives, scope, and methodology of our review of current and
potential approaches to achieving a model program are described. The
chapter concludes with an overview of the organization of the rest of
the report.
CORE DATA COLLECTION ACTIVITIES
OF THE NNMRRP
---------------------------------------------------------- Chapter 1:1
The NNMRRP is a complex system of data collection and research
activities, including national surveys, state surveillance
activities, and a variety of research programs. Over time, the
NNMRRP has developed activities focused on five content areas: (1)
food and nutrient consumption; (2) nutritional and health status; (3)
dietary knowledge, attitudes, and behavior; (4) food composition; and
(5) food supply. As shown in table 1.1, the information produced by
these activities is used for a variety of purposes, from supporting
basic research on human nutritional needs to informing policy
decisions about health, agriculture, and food programs.
Table 1.1
Selected Uses of NNMRRP Data by Content
Area
Content area Selected uses\a
------------------------- -----------------------------------------------------
Food and nutrient Assess potential exposure of children to pesticide
consumption residues
Develop standards for food stamp eligibility
Evaluate impact of farm commodity programs on demand
and pricing
Monitor trends in diet-related risks of chronic
diseases
Nutritional and health Evaluate food assistance programs
status
Research the relationship between fat consumption and
cancer
Target food fortification to foods eaten by people
with nutrient deficiencies
Update infant growth charts
Inform policies on cholesterol screening and
treatment
Dietary knowledge, Develop food labeling policies
attitudes, and behavior
Inform nutrition education programs
Food composition Support estimation of the consumption of nutrients
and food components related to chronic disease
Plan meals for military personnel and other groups
Monitor trends in nutrient availability and food
safety
Food supply Manage federal marketing and agricultural policies
--------------------------------------------------------------------------------
\a These uses were selected to illustrate the variety of purposes
supported by NNMRRP data. They are not an exhaustive list.
Table 1.2 lists specific activities managed by the Departments of
Agriculture and Health and Human Services, which have major
responsibilities for the five areas. Other agencies, including
Commerce, Defense, and the Environmental Protection Agency, also
participate in the NNMRRP.
Table 1.2
Major NNMRRP Activities by USDA and HHS
Content area USDA HHS
------------------ ------------------ ------------------
Food and nutrient Nationwide Food National Health
consumption Consumption and Nutrition
Survey Examination
Survey
Continuing Survey
of Food Intake by Total Diet Study
Individuals
Nutritional and No major role National Health
health status and Nutrition
Examination
Survey
Pregnancy
Nutrition
Surveillance
System
Pediatric
Nutrition
Surveillance
System
Dietary knowledge, Diet and Health Health and Diet
attitudes, and Knowledge Survey Survey
behavior
Behavioral Risk
Factor
Surveillance
System
Food composition National Nutrient Food Label and
Data Bank Package Survey
Food supply Food and Nutrition No major role
Supply Series
----------------------------------------------------------
USDA ACTIVITIES
-------------------------------------------------------- Chapter 1:1.1
USDA has major responsibilities for collecting information in all of
the content areas except nutritional and health status. USDA gathers
data on food and nutrient consumption through two national
surveys--the Nationwide Food Consumption Survey (NFCS) and the
Continuing Survey of Food Intake by Individuals (CSFII). In the
past, NFCS has gathered nationally-representative information on the
food consumption behavior of households and individuals. It provides
detailed data on household costs for food. One of the major uses for
these data is the development of the Thrifty Food Plan, which is the
basis for calculating food stamp benefits. Implemented decennially,
NFCS suffered from severe response rate problems (less than 40
percent) in 1987-88. As a result, the individual food consumption
portion of the NFCS is expected to be dropped in the future.
Since the mid-1980s, CSFII has supplemented NFCS by providing regular
information on individual dietary intake. The data collected in its
most recent implementation (1994-96) will be used to describe both
general and low-income populations. The Diet and Health Knowledge
Survey (DHKS), which collects data on dietary knowledge, attitudes,
and behavior, is a follow-up to CSFII. Together, CSFII and DHKS are
intended to inform policies relating to food production and
marketing, food safety, food assistance, and nutrition education.
The USDA activities focused on food composition and food supply are
not surveys. For food composition, USDA gathers data from food
industries and other sources on the nutrient content of foods. These
data support the dietary surveys by translating the foods consumed
into their nutrient components. Food supply is estimated by
deducting data on exports, year-end inventories, and nonfood use from
data on production, imports, and beginning inventories.
HHS ACTIVITIES
-------------------------------------------------------- Chapter 1:1.2
HHS is responsible for the National Health and Nutrition Examination
Survey (NHANES) and the state-based surveillance systems. These data
collection activities provide information on the content areas (1)
food and nutrient consumption, (2) nutritional and health status, and
(3) dietary knowledge, attitudes, and behavior. Like NFCS and CSFII,
NHANES collects data from a nationally-representative sample.
However, NHANES' unique contribution is its use of physical
examinations and clinical and laboratory tests as well as traditional
survey methods to gather information. NHANES' data support research
on the relationship between diet and health and inform health policy
decisions, such as the promotion of cholesterol screening. After two
earlier implementations in the 1970s, NHANES has just completed its
third administration (1988-94). NHANES has been supplemented by the
Hispanic Health and Nutrition Examination Survey (HHANES) in the
early 1980s, follow-up surveys of respondents to NHANES I, and
follow-up matches of the records from NHANES II to the National Death
Index and other vital statistics records.
The state-based surveillance systems were set up to provide quick
information to states to use in planning and managing nutrition and
health programs. They include the Pediatric Nutrition Surveillance
System (PedNSS), Pregnancy Nutrition Surveillance System (PNSS), and
Behavioral Risk Factor Surveillance System (BRFSS). Participating
states collect the data for these systems with technical and other
kinds of assistance from HHS. Both PedNSS and PNSS rely on data from
clinic records from publicly funded health, nutrition, and food
assistance programs, primarily the Special Supplemental Food Program
for Women, Infants, and Children (WIC). PedNSS monitors nutritional
status among low-income, high-risk children, while PNSS focuses on
low-income, high-risk pregnant women, measuring nutrition-related
problems and behavioral risk factors associated with low birthweight.
In contrast to PedNSS and PNSS, data for BRFSS are gathered through
telephone interviews, with respondents (adults 18 years and over)
sampled through random digit dialing. In addition to a core set of
questions on various health risk factors, BRFSS includes optional
modules for the assessment of dietary fat and fruit and vegetable
consumption.
Other major HHS monitoring activities include the Total Diet Study,
which analyzes nutrient and contaminant levels in the food supply;
the Food Label and Package Survey, which monitors nutrition labeling
practices; and the Health and Diet Survey, which assesses dietary
knowledge and practices as they relate to health problems.
STRUCTURE OF THE NNMRRP
---------------------------------------------------------- Chapter 1:2
Although the United States has one of the most comprehensive
monitoring programs in the world, several problems with nutrition
monitoring activities have been identified over the past two decades.
Of key concern has been the lack of coordination and compatibility of
different data collection activities. This encompasses differences
across surveys in methods for assessing dietary intake and
nutritional status, sampling designs, population descriptors and
other measures, and the timing and reporting of results.
To improve the coordination of federal nutrition monitoring
activities and the quality of the data collected, the Congress passed
the National Nutrition Monitoring and Related Research Act of 1990
(P.L. 101-445). The act established an Interagency Board, jointly
chaired by USDA and HHS, to coordinate activities across the various
agencies involved in nutrition monitoring. The Interagency Board was
charged with developing a strategic plan that would establish a
comprehensive nutrition monitoring and related research program.
This plan--known as the 10-year comprehensive plan--was published in
the Federal Register on June 11, 1993. It outlines a set of planning
activities, including a general time frame and lead agencies for each
activity. The activities are organized around six objectives, which
are to
-- provide for a comprehensive NNMRRP through continuous and
coordinated data collection;
-- improve the comparability and quality of data across the NNMRRP;
-- improve the research base for nutrition monitoring;
-- develop and strengthen state and local capacity for continuous
and coordinated nutrition monitoring data collection that
complements national nutrition surveys;
-- improve methodologies to enhance comparability of NNMRRP data
across federal, state, and local levels; and
-- improve the quality of state and local nutrition monitoring
data.
In addition, an Advisory Council of experts from outside the federal
government was created to guide the Interagency Board on scientific
and technical matters. (Chapter 3 contains more information on the
Interagency Board and its activities.)
OBJECTIVES, SCOPE, AND
METHODOLOGY
---------------------------------------------------------- Chapter 1:3
OBJECTIVES
-------------------------------------------------------- Chapter 1:3.1
This is the third and final report in a series responding to a
request from the former House Committee on Science, Space, and
Technology. The first report reviewed past evaluations of federal
nutrition monitoring and examined the progress of the NNMRRP since
the passage of the 1990 act.\1 It concluded that (1) a coherent
program for nutrition monitoring was not yet in place and (2)
although there has been progress in coordinating the program, the
10-year plan is incomplete because it does not include a framework
for evaluating current and potential activities or detailed plans for
achieving the objectives.
Based on a survey of users of nutrition monitoring data, the second
report described the purposes for which nutrition data are used and
summarized respondents' suggestions for improving NNMRRP
activities.\2 These suggestions, which addressed such issues as the
timing of the surveys, their coverage of subpopulations, and the ease
with which the data could be used, were consistent with the concerns
raised by past evaluations of federal nutrition monitoring and
indicated a continued need to address the long-standing problems of
federal nutrition monitoring.
Completing our response to the Committee's request, this report
builds on our earlier work to meet two objectives: (1) define a
model nutrition monitoring program and (2) compare the current system
with potential options for implementing the components of a model
program in the NNMRRP.
--------------------
\1 Nutrition Monitoring: Progress in Developing a Coordinated
Program (GAO/PEMD-94-23).
\2 Nutrition Monitoring: Data Serve Many Purposes; Users Recommend
Improvements (GAO/PEMD-95-15).
SCOPE AND METHODOLOGY
-------------------------------------------------------- Chapter 1:3.2
DEFINING THE FEATURES OF
A MODEL PROGRAM
------------------------------------------------------ Chapter 1:3.2.1
Before defining the features of a model program, we first limited our
review to three of the five content areas covered by the NNMRRP: (1)
food consumption and dietary intake; (2) health and nutritional
status; and (3) knowledge, attitudes, and behavior. These three
variables were selected for both substantive and methodological
reasons. Substantively, the selected elements provide the data that
support the planning and evaluation of interventions that directly
affect health, such as nutrition education and food assistance
programs. A substantive argument could also be made for the
inclusion of food composition data, which are used to translate the
information on what foods are eaten into estimates of nutrient
intake. However, because an earlier GAO project focused on the
NNMRRP's food composition activities, we did not include this in our
review.\3
Methodologically, the three selected variables are linked because
they rely on data obtained from individuals through surveys and
physical examinations. In contrast, food composition information is
based on chemical analyses of foods, and food supply is determined
from macroeconomic data. Our focus on the components of the NNMRRP
that rely on surveys facilitates the comparison of current and
potential approaches to achieving a model program.
To identify features of a model nutrition monitoring program, we used
four sources: reviews of previous evaluations of federal nutrition
monitoring activities, review of the objectives and related
activities outlined in the 10-year comprehensive plan developed by
the Interagency Board, consultation with expert advisers, and our
survey of data users. These sources are detailed below, and
supporting material is provided in appendix II. (See chapter 2 for a
description of the features.)
Past Evaluations of Federal Nutrition Monitoring. Past evaluations
by such groups as the National Academy of Public Administrators, the
Joint Nutrition Monitoring Evaluation Committee, and the National
Research Council identified several concerns about the federal
nutrition monitoring program. Because these evaluations also
informed the Interagency Board's development of its 10-year plan, we
used them as the starting point for our identification of features of
a model nutrition monitoring program. Our report, Nutrition
Monitoring (GAO/PEMD-94-23), discusses these past evaluations and
NNMRRP progress in addressing their recommendations. (Table II.1 in
appendix II lists the criticisms identified in these evaluations and
categorizes them by the features of a model program that they
suggest.)
The 10-Year Comprehensive Plan. As described above, the NNMRR Act
required the Interagency Board to develop a plan for the program.
The plan outlined six objectives, three with a federal focus and
three emphasizing state and local monitoring, and listed 68 planned
activities. These were reviewed for responsiveness to the model
features suggested by the past evaluations. (See table II.2 in
appendix II for examples of the 68 activities listed in the plan.)
Consultation With Expert Advisers. To assist us at critical decision
points in this project, we organized three panels. (The members of
each panel are listed in appendix I.) The Core Policy Panel consisted
of nationally-known experts in nutrition and nutrition monitoring
policy. These panelists were consulted throughout the project. In
addition, they helped us develop a framework of purposes for
nutrition monitoring data that guided our survey of data users.
The Methodology Panel included renowned experts in such fields as
sampling, survey design, dietary assessment, and nutritional
epidemiology. This panel met to help us identify promising
approaches to critical elements of a nutrition monitoring system. In
addition, the panelists assisted us on issues related to their areas
of specialization.
The Data Users Panel consisted of users of the nutrition monitoring
data, chosen to reflect the broad range of purposes that the data
must serve, including the support of state and local nutrition
programs, academic research, food industry research, and the
development and evaluation of federal food assistance programs. As
with the Methodology Panel, this panel was convened once to help us
identify promising approaches to nutrition monitoring. Individual
panelists were consulted later about specific issues related to their
expertise.
Through the process of reviewing materials and participating in panel
meetings, the expert advisers generated several suggestions for
possible changes to the NNMRRP, examples of which are given in table
II.3 in appendix II. In addition, they reviewed a draft of this
report.
Suggestions From the Survey of Data Users. Our survey of users of
nutrition monitoring data focused on primary users of 14 of the
NNMRRP surveys and surveillance systems. Primary users were defined
as those who directly access the data rather than use information
that has already been processed and interpreted by others in reports
and other documents. Since there is no single list of primary users
of NNMRRP data, we obtained lists of known and potential data users
such as people who had requested the data from NNMRRP agencies,
attendees at nutrition-related workshops, state and local government
officials working in nutrition, and members of associations for
nutrition professionals.\4
A major portion of the survey was dedicated to determining how the
respondents used the data. We also asked whether changes are needed
to better meet the respondent's information and data quality needs.
If the respondent indicated a need for change, we asked for
suggestions in the following categories: (1) data elements
collected, (2) data collection methods, (3) units of analysis, (4)
time of data collection, (5) population group coverage, (6)
geographic area coverage, and (7) ease of use. These comments were
analyzed to identify major themes for each of three groups of data
collection activities--USDA surveys, HHS surveys, and HHS state-based
surveillance systems. In appendix II, table II.4 identifies the
themes associated with the features of a model program.
--------------------
\3 See Food Nutrition: Better Guidance Needed to Improve Reliability
of USDA's Food Composition Data (GAO/RCED-94-30; Oct. 25, 1993).
\4 See Nutrition Monitoring (GAO/PEMD-95-15) for details of the
survey.
COMPARING APPROACHES TO
ATTAINING THE MODEL
FEATURES
------------------------------------------------------ Chapter 1:3.2.2
From the features that were identified, we selected four as the focus
of the second objective--the comparison of current and potential
approaches to achieving the model program. We focused on features
that reflect long-standing concerns about federal nutrition
monitoring, that encompass other desired characteristics, and that
generate debate about how they should be addressed.
For each of the selected features, we identified current activities
of the NNMRRP through interviews with staff in NNMRRP agencies,
attendance at meetings of the Interagency Board and the Advisory
Council, and reviews of program documents. Potential approaches were
identified through literature review, analysis of our survey results,
and consultation with the expert advisers. We did not identify the
universe of potential approaches. Instead, our search was focused on
those approaches deemed promising and feasible by our expert
advisers.
To assess each of the potential approaches, we first identified
programs from the same set of sources that helped us define the
features of a model program--that is, the literature, our survey of
data users, and expert advisers. Where possible, we limited our
consideration to programs that had some linkage to nutrition
monitoring. For example, for separate studies of subpopulation
groups, we looked at the experience with the Hispanic Health and
Nutrition Examination Survey in the early 1980s. To describe the
strengths and weaknesses of the programs illustrating the potential
options, we reviewed program documents and related literature and
interviewed managers and staff.
This review was conducted between October 1993 and December 1994 in
accordance with generally accepted government auditing standards.
STRENGTHS AND LIMITATIONS OF
OUR METHODOLOGY
-------------------------------------------------------- Chapter 1:3.3
This report describes the results of a systematic examination of
current and potential approaches to selected features of a model
nutrition monitoring program. The strength of the review is its
reliance on multiple sources of information. In addition to
surveying users of nutrition monitoring data, consulting with
experts, and reviewing both technical literature and program
documents, we also interviewed officials and program staff in
nutrition monitoring programs and in programs illustrating the
alternatives. The major limitation of our work is its prospective
nature. Because we were examining potential changes to the NNMRRP,
hard evidence of the costs or effectiveness of the options was not
available. Instead, the strengths and weaknesses of the options
relative to the current nutrition monitoring system are supported
primarily by logic and stated in tentative terms. Given this
limitation, the report makes no recommendations for specific changes
to the NNMRRP.
ORGANIZATION OF THE REPORT
---------------------------------------------------------- Chapter 1:4
In response to the first objective of the project, chapter 2
describes the model program and provides information on the selection
of four model features as the focus of the report. (The sources used
to develop the model are described above.) The second
objective--comparing current and potential approaches to each model
feature--is addressed in chapters 3-6, organized by the four model
features. Specifically, chapter 3 examines coordination options;
chapter 4 compares alternate approaches to providing continuous data;
chapter 5 discusses different methods of supporting inferences about
subpopulation groups and small geographic areas; and chapter 6
reviews approaches to assisting state and local monitoring
activities. Appendix I provides additional detail on the expert
advisers to the project, and appendix II describes the sources for
the model features. Agency comments on a draft of the report are in
appendixes III and IV.
A MODEL NUTRITION MONITORING
PROGRAM
============================================================ Chapter 2
This chapter responds to the first objective of our review--the
definition of a model nutrition monitoring program. First, the major
features identified from the sources described in chapter 1 are
outlined. From this set of features, we selected four as the focus
for our response to the second objective of the review--the
comparison of current and other approaches to achieving the model
characteristics. The chapter describes our selection process and
details the importance of the four features that are the subject of
the rest of the report.
FEATURES OF A MODEL PROGRAM
---------------------------------------------------------- Chapter 2:1
Depending on the purposes that the data serve, the specific elements
of a model nutrition monitoring program change. For example,
researchers and program managers interested in food safety need
detailed information on dietary intake, including specific brand
names of the foods consumed. In contrast, a nutrition educator may
place a higher priority on information about dietary knowledge,
attitudes, and behaviors. However, at a more general level, some
common ideal characteristics can be identified. Focusing on this
general level, we used the sources described in chapter 1 to identify
a number of features of a model program. Table 2.1 lists these
features and the sources that support them.
Table 2.1
Features of a Model Nutrition Monitoring
Program and Their Sources
Feature Source\a
-------------------------------------------------- ------------------
Coordinated system that responds efficiently Past evaluations
to the diverse needs of the data users
10-year plan
Expert advisers
Continuous or more frequent collection of data Past evaluations
10-year plan
Expert advisers
Survey of data
users
Support for reliable inferences about Past evaluations
subpopulation
groups and small geographic areas 10-year plan
Expert advisers
Survey of data
users
Assistance to states and localities Past evaluations
10-year plan
Expert advisers
Survey of data
users
Strong research base, including improved methods Past evaluations
for assessing dietary intake and nutritional
status 10-year plan
Expert advisers
Survey of data
users
High response rates and low respondent burden Past evaluations
Survey of data
users
Timely processing and dissemination of survey Past evaluations
information
10-year plan
Survey of data
users
On-going evaluation of the system's content and Expert advisers
methods, including a review of the information
needs
Collection of both household and individual data Expert advisers
Survey of data
users
Maintenance of data comparability over time Expert advisers
Longitudinal data (collection of information on Expert advisers
the
same individuals or households over time) Survey of data
users
----------------------------------------------------------------------
\a For more information on these sources, see the discussion in
chapter 1 and the tables in appendix II.
All of the features identified are clearly important elements of a
comprehensive nutrition monitoring program. However, we selected
four characteristics as the focus of our review: (1) a coordinated
set of activities that (2) provides data on a continuous basis, (3)
supports inferences about important population groups, and (4)
assists state and local monitoring activities. These features
encompass other desired characteristics, respond to long-standing
concerns about federal nutrition monitoring, and generate debate
about how they can be achieved.
The first criterion for selection was to focus on the most general
concerns. With these features, other desirable characteristics can
be considered even though they are not emphasized. Specifically, a
mechanism for evaluating the NNMRRP's content and methods, including
a review of the information needs of the data users, is described in
chapter 3 as an element of a coordinated program. The theme of
evaluating options in relation to the needs for data also underlies
the discussion of the alternatives for the other features. Three
other features--the comparability of data over time, the collection
of longitudinal data, and the timeliness of data release--are related
to the continuous collection of data and, as such, are considered
briefly in chapter 4.
The four features were also selected because they respond to
long-standing criticisms of federal nutrition monitoring activities.
Concerns about coordination, the continuity and timeliness of the
data, the availability of information on subpopulations, and the role
of states and localities were raised as early as 1977 by witnesses
before a House subcommittee.\1 In contrast, the concerns with
response rates and the level of the data (individual or household)
can be traced to the problems with the last NFCS (described in
chapter 1), which USDA has taken steps to avoid in the future.
The selected features are also not easily addressed; that is, there
is debate about the best approach to achieving each feature. For
example, some expert advisers stated that assistance to states and
localities should focus on data analysis and interpretation, while
others argued for a larger state and local role in data collection.
Similarly, to provide information on subpopulation groups, national
surveys could be supplemented by such means as the surveillance
systems and oversampling or the national surveys could be abandoned
and their resources dedicated to special studies of specific groups.
In contrast to the debates about how best to achieve these and the
other selected features, there has been consensus about the kind of
dietary intake methodology that will be used by the NNMRRP surveys
and ongoing research to improve these methods through automation and
other means.
--------------------
\1 Hearings before the House Subcommittee on Domestic and
International Scientific Planning, Analysis and Cooperation,
Committee on Science and Technology, July 26, 27, and 28, 1977.
THE SELECTED FEATURES
---------------------------------------------------------- Chapter 2:2
An ideal federal nutrition monitoring program would have a
coordinated set of activities that provides data on a continuous
basis, covers important population groups, and supports state and
local monitoring activities. Coordination is the key both to the
efficiency of the system and its responsiveness to the needs of data
users. A continuous flow of data would ensure that the information
on the nation's nutritional status was up-to-date and would also
enable the tracking of dietary behavior and nutritional status over
time. Information on population groups that are vulnerable or
growing rapidly is needed to plan, manage, and evaluate programs
intended to prevent or ameliorate nutritional problems. Because
state and local governments are often the location of such programs,
they need assistance in either interpreting available data or
collecting their own information. The importance of these features
is further explained in the following sections.
A COORDINATED NUTRITION
MONITORING PROGRAM
-------------------------------------------------------- Chapter 2:2.1
The coordination of nutrition monitoring activities has implications
for both the utility of the information produced and the costs of the
program as a whole. The utility of the information is constrained
when data from different data collection activities cannot be easily
combined. For example, research on the relationship between diet and
health could be strengthened if CSFII's data on dietary intake could
be combined with NHANES' data on health and nutritional status.
However, because of differences in the sampling designs and nutrition
measures, combining the data from the two surveys is difficult and
controversial. Similarly, poor integration of the data collected by
the state-based surveillance systems with the national surveys
presents a barrier to meaningful comparisons of state and national
populations.
To the extent that the lack of coordination results in unnecessary
duplication, a fragmented system can also increase the costs of
nutrition monitoring. For example, the current NNMRRP includes two
surveys focused on dietary knowledge, attitudes, and behaviors--one
operated by USDA and the other by HHS. The Interagency Board plans
to review these surveys for duplication. While some overlap can be
useful as a quality check on data from different sources, unnecessary
redundancy in the system uses resources that could be used for
currently unmet data needs--such as the need for information on
specific subpopulations at risk for nutrition-related problems.
The need for a coordinated system was supported by three of the four
sources used to identify features of a model program. In addition to
a general concern about the lack of coordination, past evaluations of
federal nutrition monitoring have criticized the incompatibility of
the data gathered by different enterprises. Specifically, these
evaluations have called for compatible methods of assessing dietary
intake, a core set of standardized measures for the major surveys,
compatible sampling techniques in the national surveys, and
integrated reporting. Another criticism related to coordination
focused on the absence of a systematic process for determining the
needs for nutrition monitoring data across the different data
collection activities. Although needs are assessed for individual
activities, no comprehensive assessment of needs in relation to the
total system of activities has taken place.
Coordination is also a major theme of the 10-year plan. Four of the
six objectives discussed in the plan focus on coordinating data
collection and improving the comparability of the data. Of the
specific activities listed for each objective, some of those focused
on coordination are: coordinating the planning for coverage,
tracking, and reporting of findings from surveys and surveillance
systems; identifying ways to increase comparability within a dietary
method to improve the quality and usefulness of data; and
establishing a mechanism for improved coordination among federal
agencies that collect and use survey information about knowledge,
attitudes, and behavior to assess gaps and duplications in existing
surveys.
The third source of support for the need for coordination comes from
meetings of the advisory panels. The panelists noted that the
different agencies involved in nutrition monitoring have different
missions and priorities and, hence, coordination is difficult. Their
major suggestions for improving coordination were to give
coordination responsibility to a single lead agency, to coordinate
from an interagency body with permanent staff and enforcement
authority, to locate nutrition monitoring in statistical agencies
within the user departments or within a central statistical agency,
to centralize the congressional appropriations process for nutrition
monitoring activities, and to ensure informed review of data
collection plans by qualified staff at the Office of Management and
Budget.
Our survey queried respondents about changes to specific data
collection activities, rather than the NNMRRP as a whole, so
coordination was not a major theme of the comments provided by the
data users.
THE CONTINUOUS COLLECTION OF
DATA
-------------------------------------------------------- Chapter 2:2.2
The Interagency Board defines continuous data as data collection that
is "repeated regularly and frequently."\2 Two consequences follow
when data are not regularly available. First, because the kinds of
foods that are available and the eating patterns of the American
people change rapidly, the data become outdated quickly. Compounded
by delays between data collection and data release, long intervals
between administrations of the surveys diminish the relevance of the
data to the current situation and, hence, their utility in program
planning and management. For example, if any of the policy changes
currently being considered are implemented-- such as the
consolidation of food assistance programs--up-to-date information at
regular intervals before and after the change will be needed to
monitor any positive or negative effects of such policy changes on
the population.
A second consequence of pauses in data collection is that potential
efficiencies of a continuous survey operation are lost. Each
implementation of a national survey requires extensive planning,
including reviews of the needs of data users, development and testing
of data collection procedures, and all the steps involved in
approving a contract. An ongoing data collection operation could
streamline some of these processes. In addition, when surveys are
not in the field continuously or even at dependable intervals, they
may attempt to meet as many of the needs of data users as possible
when they are administered. For example, the low response rates of
the 1987-88 NFCS have been partially attributed to the burden on
survey respondents resulting from its attempt to obtain both
household and individual data with one interview. In contrast, an
ongoing survey could consist of a core set of questions and rotating
modules of questions that address the needs of specific users.
The continuous collection of data was a theme in all four of our
sources for the features of a model program. Past evaluations of
nutrition monitoring have called for the continuous collection and
timely release of nutrition-related data. Although the 10-year plan
does not list specific activities focused on the continuous
collection of data, two of the plan's objectives indicate the
Interagency Board's concern with the timeliness of the data collected
at the federal and state and local levels. The expert advisers also
emphasized the need for regularly available data and suggested the
following mechanisms for the ongoing collection of the information:
-- continuous national nutrition surveys,
-- addition of nutrition-related modules to existing surveys,
-- reliance on program data that are already collected, and
-- collection of longitudinal data (that is, data collected from
the same sample over time).
Finally, responses to our survey of the users of nutrition data not
only stated a desire for continuous data, but also indicated that
data that are collected at regular and frequent intervals serve some
important purposes. For example, data that are currently available
on a regular basis have been used to measure progress toward the
Healthy People 2000 objectives and to evaluate policies such as the
fortification of infant formula.\3
--------------------
\2 Ten-Year Comprehensive Plan, 58 Fed. Reg. 32806 (1993).
\3 The Healthy People 2000 objectives, published in 1990, are targets
for improving the health of the population by the year 2000. The
targets are phrased in specific terms and with reference to baseline
information so that progress toward the objectives is measurable.
Nutrition-related objectives address such issues as overweight,
breast-feeding, consumption of fruits and vegetables, iron
deficiency, and others.
SUPPORT FOR RELIABLE
INFERENCES ABOUT
SUBPOPULATION GROUPS AND
SMALL GEOGRAPHIC AREAS
-------------------------------------------------------- Chapter 2:2.3
The need for continued and improved information on subpopulation
groups and small geographic areas is supported by several arguments.
First, information is needed on specific populations known to be at
risk for nutrition-related problems, such as Native Americans or
homeless persons, in order to identify their needs and develop and
target assistance programs. Second, some subpopulations, including
Hispanics and the elderly, are growing rapidly. Their dietary
patterns or nutritional needs may be different from the population as
a whole; thus, information about these groups is needed to monitor
their needs and to understand their effect on estimates of the
prevalence of various nutritional problems in the overall population.
Finally, the samples for the three major NNMRRP surveys are designed
to yield national estimates. However, much of the planning for
health and nutrition programs is conducted at the state and local
levels. Hence, states and localities also need information on
nutrition-related indicators for their populations.
For the reasons given above, past evaluations of federal nutrition
monitoring have criticized the program for not covering specific
population groups and geographic areas. Although none of the overall
objectives of the 10-year comprehensive plan focus on the need for
information on subpopulation groups, several of the activities do.
For example, one planned action is to develop and implement a plan
for improved coverage of groups at nutritional risk. Our advisory
panels also discussed the importance of information on
subpopulations, noting differences in the kinds of foods consumed in
different regions of the country and by different ethnic and racial
groups as well as differences in nutritional needs at different ages.
Their suggestions for improving the availability of data on
subpopulation groups and small geographic areas included different
sampling strategies for different populations, contracts with states
and localities to gather information on geographically-based
populations, and indirect estimation to support inferences about
subpopulation groups and small geographic areas.
The availability of data on important subpopulations and states and
localities was one of the changes to the current data collection
activities requested by respondents to our survey of data users. The
users also emphasized the importance of information on subpopulations
by describing the uses supported by currently available data on
subpopulation groups such as determining dietary needs of the
elderly, assessing differences between blacks and whites in the
effect of obesity on diabetes, informing policies on the
fortification of infant and toddler foods, and targeting a blood
pressure screening program to the Mexican-American population.
ASSISTANCE TO STATES AND
LOCALITIES FOR NUTRITION
MONITORING
-------------------------------------------------------- Chapter 2:2.4
State and local governments are interested not only in the
applicability of federal nutrition monitoring data to their
jurisdictions, but also in having federal assistance in collecting
and interpreting their own data. The major justification for the
emphasis on state and local monitoring is the wide range of uses that
states and localities have for nutrition data. The examples in table
2.2, drawn from responses to our survey of data users, illustrate the
utility of existing NNMRRP data for state and local governments.
While NNMRRP data collection systems meet some of the state and local
needs for nutrition monitoring data, state and local officials have
called for additional technical assistance in analyzing and
interpreting existing sources of data and for federal support in
collecting their own data.
Table 2.2
Examples of State and Local Uses of
Nutrition Data\a
Purpose Specific use
---------------------- ----------------------------------------------
Problem identification Determine nutritional habits of school-age
children
Identify which nutrition problems are most
prevalent among WIC participants
Assess fruit and vegetable consumption and
need for intervention
Program planning or Plan cardiovascular interventions for state
policy-making residents with diabetes
Establish anemia as a priority issue for
nutrition education
Develop a program targeted toward the
prevention of obesity in children
Seek iron fortification of flour used in
ethnic and large county bakeries
Launch a "5-a-Day" program to encourage
consumption of fruits and vegetables
Select ethnic target groups for Preventive
Health Block Grants
Map anemia rates to identify highest risk
areas for targeting resources
Program evaluation or Monitor progress toward Healthy People 2000
program management objectives, such as improvement in rates of
breast-feeding
Assess access to food assistance
Justify procurement of hematological equipment
Expand nutrition technical assistance to
county health departments
----------------------------------------------------------------------
\a Provided by state and local officials who responded to our survey.
In addition to the support provided by the survey of data users, our
other sources indicated the importance of building state and local
capacity for nutrition monitoring. For example, past evaluations
recommended assisting state and local nutrition monitoring
activities. These recommendations are mirrored in the Interagency
Board's 10-year comprehensive plan, which clearly signals the
importance of states and localities in the NNMRRP by devoting three
of its six objectives to strengthening state and local monitoring
activities.
The expert advisers also noted the role of states and localities in
nutrition monitoring, but disagreed about the responsibility states
should have for data collection. Some argued for state-based data
collection that feeds into a federal system, and others argued for
less state responsibility for data collection, but for increased
consideration of state needs in federal data collection activities.
Their suggestions for assisting states and localities include
-- providing financial assistance to states to determine their own
data needs,
-- creating a federal-state partnership in which states can provide
funds for some extra sampling or extra questions on federal
surveys,
-- developing standardized modules of interest for state data
collection activities,
-- assisting state collection of data on subpopulations, and
-- providing technical assistance in data interpretation.
APPROACHES TO ACHIEVING THE
SELECTED MODEL FEATURES
---------------------------------------------------------- Chapter 2:3
The rest of the report describes the strengths and limitations of
current and potential approaches to achieving the selected model
features. These approaches are listed in table 2.3. The alternate
strategies were selected from the suggestions generated by the expert
advisers using the criteria of responsiveness to criticisms of the
current approach and feasibility. For example, the current approach
to coordination--the Interagency Board--is criticized for its lack of
authority over the member agencies. In contrast, an independent
central authority could have influence over the NNMRRP agencies. The
options were considered feasible if they were already used in other
programs with similar issues (such as the lead agency approach for
other cross-agency programs), past activities of the NNMRRP (such as
the special study approach for information on subpopulations), or
related current activities by NNMRRP agencies (such as indirect
estimation).
Table 2.3
Features of a Model Program and Current
and Potential Approaches
Model feature Current approach Potential option
------------------ ------------------ ------------------
Coordinated system Interagency Board Coordination
of activities and Working Groups through a central
authority
Coordination from
a single lead
agency
Continuous State-based Continuous
collection of data surveillance nationally-
systems representative
survey
Addition of
nutrition
questions to
existing ongoing
surveys
Support for Oversampling of Regular, separate
reliable selected groups in studies of
inferences about the national subpopulations
subpopulation surveys
groups and small
geographic areas
State-based Development of
surveillance indirect
systems estimation
programs
Assistance to State-based Community-based
states and surveillance nutrition
localities systems monitoring
----------------------------------------------------------
APPROACHES TO COORDINATING THE
NUTRITION MONITORING PROGRAM
============================================================ Chapter 3
As described in chapter 2, a model program would have a mechanism for
coordinating the various nutrition monitoring activities to maximize
the utility of the data and minimize the costs of its collection.
This chapter first reviews the status of current NNMRRP activities to
improve coordination. Then, two other possible coordination
mechanisms-- coordination by a central authority and coordination by
a single lead agency--are examined. (Table 3.1 provides an overview
of the strengths and limitations of the various approaches to
coordination.)
Table 3.1
Approaches to Coordinating Nutrition
Monitoring Activities
Approach Strength Limitation
------------------------ -------------------------- --------------------------
Current: Interagency Improved communication and Difficulties setting
Board and Working Groups cooperation among the priorities across
agencies agencies
Compilation of a Inadequate assessment of
coordinated budget report the needs for data by
different kinds of users
Difficulties in
disaggregating the costs
of the nutrition component
of multipurpose programs
Limited authority, staff,
and financial resources
for coordination
Potential
Coordination through a Increased ability to Conflict between research
central authority establish priorities concerns and policy
concerns
Alignment of agency
priorities with Difficulties in
overarching goals through disaggregating the costs
centralized budget of multipurpose programs
reviews
Limited staff and
A central contact point financial resources for
for state and local coordination
governments, interest
groups, Congress, and
other federal agencies
Coordination from a Improved communication and Difficulties setting
single lead agency cooperation among the priorities across
agencies agencies
Compilation of a Difficulties in
coordinated budget report disaggregating the costs
of multipurpose programs
A central contact point
for state and local Limited authority, staff,
governments, interest and financial resources
groups, Congress, and for coordination
other federal agencies
Decreased attention to the
data needs of other
agencies
--------------------------------------------------------------------------------
CURRENT APPROACH: THE
INTERAGENCY BOARD
---------------------------------------------------------- Chapter 3:1
As described in chapter 1, the NNMRRP meets multiple needs for
nutrition-related data. Yet, historically, these needs have not been
met by an integrated program with the capacity for evaluating data
needs and making adjustments as those needs change. Instead, a
fragmented system of activities developed over the decades as new
needs for nutrition data were identified. For example, in the early
1930s, USDA developed its first national survey of household food
consumption because data on the food supply provided no information
about the distribution of food at the household and individual
levels. Similarly, the nutrition component was added to the National
Health Examination Survey in the early 1970s in response to a need
for more information about hunger, and state-based surveillance
systems were established in recognition of the primary role of states
in providing services to populations at risk of nutritional problems.
To address concerns about the lack of coordination across the
agencies involved in nutrition monitoring, the National Nutrition
Monitoring and Related Research Act of 1990 required the Secretaries
of Agriculture and Health and Human Services to implement a
coordinated program of nutrition monitoring. The act specified
several tools: an Interagency Board, the development of a
comprehensive plan for the program, a council of outside advisers,
and an integrated budget.
The Interagency Board created by the act has the difficult task of
coordinating numerous data collection and analysis activities across
several agencies that have traditionally had separate and distinct
missions and operations. The Board has two chairpersons, one
selected by the Secretary of Agriculture and one by the Secretary of
Health and Human Services. For USDA, the chair is the Under
Secretary for Research, Education, and Economics. The HHS chair is
the Assistant Secretary for Health. Membership on the Board includes
representatives of various agencies in USDA and HHS, as well as the
Bureau of the Census, Agency for International Development, Bureau of
Labor Statistics, Department of Defense, Department of Veterans
Affairs, and Environmental Protection Agency, among others. The
Executive Secretary for the Board rotates between USDA and HHS every
2 years. To facilitate coordination, the Interagency Board
established working groups focused on survey comparability, food
composition, and federal-state linkages and information
dissemination.
The Secretaries and the Interagency Board were charged with
developing a 10-year strategic plan, which was published in the
Federal Register on June 11, 1993. The plan outlines a set of
planning activities, including a general time frame and the lead
agencies for each activity. The activities are organized around six
objectives, listed in chapter 1 (see p. 14). The Interagency Board
clearly recognizes the need for improved coordination since four of
these objectives focus on either the coordination of the data
collection activities or the comparability of the data.
To advise the Board on the development and implementation of the
NNMRRP, the act established the National Nutrition Monitoring
Advisory Council. The members of the Council represent academic
institutions and other interested parties drawn from outside the
federal government. The act also required the Interagency Board to
submit annually a coordinated budget for nutrition monitoring.
STRENGTHS
-------------------------------------------------------- Chapter 3:1.1
Both the concern that preceded passage of the act and the structure
it created appear to have improved communication and cooperation
among the agencies. The Board and its working groups provide
mechanisms for communication and joint decision-making. Specific
actions that demonstrate the increased coordination include the
development of
-- common population descriptors for use in conducting and
reporting the 1994-96 CSFII and the next NHANES,
-- a marketing and distribution plan for NNMRRP reports,
-- an automated dietary intake interview that would facilitate
timely data release and linkage across CSFII and NHANES, and
-- a common set of questions on food security (a concept that
addresses the certainty about having enough to eat) to be used
in the Current Population Survey.
In addition, a jointly funded research project explored the possible
linkage of CSFII and NHANES sampling plans. Alternate sampling
designs were evaluated using the criteria of (1) ability to satisfy
the separate objectives of the two surveys, (2) benefits in overall
costs or analytic power, and (3) feasibility, especially in terms of
burden on the survey respondents. The draft report from the
contractor on the project emphasized the compromises one or both
surveys would have to make to link their sample designs.\1 For
example, while the combination of the NHANES and CSFII into a single
survey could yield a rich database of information on diet and health,
the likely increase in respondent burden could reduce response rates
and response quality. Another alternative--using linked samples for
NHANES and CSFII--could decrease CSFII's precision because NHANES'
sampling design is determined by the survey's reliance on mobile
examination centers to conduct physical exams of the respondents.\2
The Interagency Board concluded that the two surveys should remain
independent, although work on improving the comparability of the data
should continue.
As required by the act, the Interagency Board submits an annual
budget for the NNMRRP to the Congress. The budget report includes
costs allocated for data collection, related research, information
dissemination and exchange, and technical assistance; however, these
different types of costs are not distinguished. Instead, the funds
dedicated to nutrition monitoring and related activities are reported
only by agency, not by type of activity. The budget report for
fiscal years 1994-96 indicated that a total of $157.7 million was
dedicated to nutrition monitoring or related research in 1994. Of
that, $30 million was reported by the Centers for Disease Control and
Prevention (which has responsibility for the HHS surveys and
state-based surveillance systems) and $9.3 million was reported by
the Human Nutrition Information Service (which had responsibility for
NFCS and CSFII).\3 The remainder was accounted for by agencies whose
primary involvement in nutrition monitoring is related research. For
example, HHS' National Institutes of Health reported $25.9 million
dedicated to NNMRRP activities and USDA's Agricultural Research
Service accounted for $50.7 million.
The budget report is useful in communicating to the Congress a
general sense of the cost of the NNMRRP across agencies. However,
funds for nutrition monitoring cannot always be disaggregated from
other purposes of the data collection and research programs. As a
result, the budget report contains only approximate amounts dedicated
to nutrition monitoring and related research. Moreover, with the
recent incorporation of the office responsible for NFCS and CSFII
into ARS, determining which funds are dedicated to the monitoring
activities and which are used for related research will be more
difficult.
--------------------
\1 An Evaluation of Linked Survey Designs for NHANES and CSFII,
prepared for NCHS by Westat, July 29, 1994.
\2 HHS maintains only three of these expensive mobile units and, as a
result, has a sampling design that minimizes the geographic
dispersion of respondents. However, HHS is reviewing alternatives to
the mobile examination centers.
\3 The Human Nutrition Information Service and its functions have
been merged with the Agricultural Research Service (ARS).
LIMITATIONS
-------------------------------------------------------- Chapter 3:1.2
The literature on the development of objectives to increase
accountability for program results indicates that (1) objectives
should be written in terms that can be used to judge progress toward
achieving them and (2) implementation plans and specific measures of
progress should be developed for the goals and objectives.\4 In the
10-year plan, the objectives are stated in general, global terms so
that it is not clear when an objective can be considered achieved.
For example, the first objective is to "provide for a comprehensive
NNMRRP through continuous and coordinated data collection." Neither
the objective itself, nor the text following it clearly defines what
the terms "comprehensive" or "coordinated" mean. Similarly, the
activities are too vague to be considered implementation plans for
the objectives; for example, no activity directly relates to the
development of continuous data collection.
As another illustration, the activity, "identify ways to increase
comparability within a dietary method to improve the quality and
usefulness of data," specifies neither the degree of quality required
nor the uses that will be facilitated. Without more concrete,
measurable objectives, there is little accountability for the program
because progress toward the objectives cannot be assessed. (Other
examples of activities listed in the 10-year plan can be found in
table II.2 in appendix II.)
In addition, the plan did not describe how activities would be ranked
by importance or addressed within current fiscal constraints.
However, since the plan was published, the Interagency Board drafted
an approach for ranking the 68 activities listed in the plan into
three categories of priorities: (1) essential (mandatory,
legislatively required), (2) necessary (critical but not mandatory),
and (3) beneficial. An interagency implementation group, involving
around 60 agency representatives, applied the approach. Twenty-five
of the activities were ranked in the high-priority category, 33 were
ranked as next most important, and only 10 were ranked as beneficial
but not critical. While this is an important first step in setting
priorities, the Interagency Board has not yet linked the top-ranked
tasks to the costs and benefits of completing them. Such a framework
could be used to understand the trade-offs in selecting one or
another approach to each task.
Since an assessment of the effectiveness of the different approaches
to the objectives requires information on the uses of the data,
another obstacle to the Interagency Board's implementation of a
coordinated system is the absence of a comprehensive assessment of
how nutrition monitoring data are used across the federal government
and by data users in other settings. For example, although USDA and
HHS held a joint workshop to assess the needs of users of dietary
intake data in August 1994, the workshop included only
representatives of federal agencies.
Finally, the ability of the Interagency Board to coordinate the
NNMRRP is limited by the lack of resources dedicated to coordination.
The Interagency Board has no staff, although two people--one from
USDA and one from HHS--have been given primary responsibility for
organizing NNMRRP activities. The NNMRR Act gives the Secretaries
the option of appointing an administrator for the program, but so far
they have chosen not to exercise that option.
--------------------
\4 Performance Measurement: An Important Tool in Managing for
Results (GAO/T-GGD-92-35; May 5, 1992).
POTENTIAL OPTIONS
---------------------------------------------------------- Chapter 3:2
While recognizing the progress made by the Interagency Board, we also
considered other coordination mechanisms. Two options for improving
coordination--through a central authority or by a lead agency--are
reviewed in detail in this section. In addition, other options
discussed by our expert advisers are briefly presented.
COORDINATION THROUGH A
CENTRAL AUTHORITY
-------------------------------------------------------- Chapter 3:2.1
The suggestion of coordination through a central authority came in
response to the lack of enforcement power held by the Interagency
Board over the member agencies. The kind of central, coordinating
agency envisioned by the expert advisers is most clearly exemplified
by the executive offices in the White House. Therefore, to examine
the advantages and disadvantages of having a central authority
provide coordination, we reviewed the literature (including prior GAO
reports and congressional hearings) and interviewed agency officials
in three White House offices that have coordination responsibilities:
Office of Management and Budget (OMB), Office of Science and
Technology Policy (OSTP), and Office of National Drug Control Policy
(ONDCP). As indicated by the brief descriptions of each office
provided in table 3.2, the coordination tools used by the White House
offices are similar to those used by NNMRRP's Interagency Board,
including interagency committees and working groups, development of
plans, and review of budgets.
Table 3.2
Responsibilities and Coordination Tools
of Three White House Offices
Office Responsibility Tool
-------- ----------------------------- -----------------------------
OMB Ensure coordination and Review and approve agency
cooperation where federal budgets
agency lines of authority
overlap Review and approve data
collection instruments
Assist in the development of
efficient coordinating
mechanisms
Expand interagency
cooperation
OSTP Coordinate research and National Council on Science
development activities in and Technology (NCST), which
other agencies is chaired by the President
and includes Cabinet
Review and participate in the Secretaries and selected
annual formulation of the agency heads
budgets related to science,
research, and development NCST subcommittees and
working groups
Review of agency budgets for
interagency programs
Commissioning studies of
controversial issues
ONDCP Through the National Drug Interagency working groups
Control Strategy, establish
policies, objectives, and Review of agency plans for
priorities for federal drug implementing the National
control programs Drug Control Strategy
Coordinate federal budget for Review and certification of
drug activities drug control budgets
----------------------------------------------------------------------
POTENTIAL STRENGTHS
------------------------------------------------------ Chapter 3:2.1.1
One potential advantage of elevating the coordination of nutrition
monitoring to a high-level central authority is the increased
participation of high-level officials in coordination activities.
This participation could, in turn, increase the ability of the
coordinating body to establish priorities. Currently, some of the
agencies participating in the Interagency Board are represented by
administrators or directors, while other agencies are represented by
staff members who have no authority to change agency activities or
establish priorities. In contrast, the political visibility of a
program under White House management could encourage agencies to send
representatives in positions of authority, capable of establishing
priorities and committing resources to support them.
The political visibility that comes with the participation of a
high-level central authority may also contribute to the effectiveness
of the various coordination tools. For example, the Interagency
Board compiles budgets obtained from each of the agencies involved in
nutrition monitoring without reviewing them for consistency with the
activities identified in the 10-year comprehensive plan. In
contrast, the White House offices use the budget review process to
bring activities of the agencies into line with overarching policy
goals. For example, ONDCP can threaten to decertify an agency's drug
budget if it is not consistent with the National Drug Control
Strategy. Decertification has no practical ramifications, but it
sends a politically important message about the priority given to
drug control activities by the White House.
An additional potential advantage of having a central authority
provide coordination is that it can be a central location for
assistance to all data users. For example, ONDCP has a Bureau of
State and Local Affairs that works with state and local government
agencies involved in drug control activities. The Bureau serves as a
clearinghouse for information about state and local activities and
uses conferences to increase communication with and among state and
local officials. In addition, the Bureau can communicate the
concerns of state and local governments and community groups to the
federal agencies involved in drug control programs. For nutrition
monitoring, such an office could provide a central contact point for
users in a variety of settings, including federal, state, and local
governments, food industry, and health care organizations.
POTENTIAL LIMITATIONS
------------------------------------------------------ Chapter 3:2.1.2
Although the increased political visibility of a high-level central
coordination office may facilitate the development of priorities, it
may also increase the potential for political pressure on the data
collection and research. A conflict between ONDCP and HHS over the
data collection and reporting of drug data illustrates this issue.
HHS was concerned about the degree of ONDCP's involvement in how the
data were collected and reported, while ONDCP expected HHS to meet
its data needs. Political influence on the scientific agenda is also
a concern for OSTP, where priorities may change with changing
administrations even if scientific issues remain the same.
In addition, although White House offices have more influence over
the budgets reported by the different agencies, the budgets developed
for other programs that cut across agency jurisdictions share the
limitation of the coordinated NNMRRP budget. Specifically, agency
activities may serve multiple purposes, thus making it difficult to
determine how much of the overall costs are dedicated to the
interagency program. For example, Coast Guard patrol boats serve
drug interdiction purposes, but are also used in search and rescue
missions that are not related to drug control. Thus, the Coast Guard
can only estimate the portion of its resources dedicated to
supporting national drug control efforts. Ambiguity about the
portion of a multipurpose program that serves the interagency purpose
makes it difficult to monitor the costs of the program.
Although central authorities like the White House offices are
financed separately from the agencies they oversee, resource
limitations persist. For example, HHS staff attributed part of their
conflict with ONDCP to a lack of technical and substantive expertise
among ONDCP staff. Similarly, a past director of OSTP identified
limited staff resources as a reason why long-term planning received
less attention than short-term problems.
COORDINATION THROUGH A LEAD
AGENCY
-------------------------------------------------------- Chapter 3:2.2
Another possible approach to coordinating the NNMRRP is locating the
responsibility within a single agency. To investigate this approach,
we discussed the option with our expert panels and used program
documents and evaluations conducted by GAO and the Congressional
Budget Office to examine the experience of the High Performance
Computing and Communications Program (HPCCP).\5 Like the NNMRRP, the
HPCCP involves multiple agencies with different strengths and
missions. Unlike the NNMRRP, the oversight of the HPCCP is located
in OSTP, which delegated the responsibility for coordinating the
activities across the agencies to a single agency, the National
Library of Medicine. While not a major player in high-performance
computing, the National Library of Medicine was seen as an
independent, unbiased participant with interest in and knowledge
about the technology. The Library of Medicine's role has been to
pull together materials for program reports, convene meetings, and
provide a clearinghouse.
--------------------
\5 The purpose of HPCCP is to further the development and
dissemination of U.S. supercomputer and high-speed computer network
technologies.
POTENTIAL STRENGTHS
------------------------------------------------------ Chapter 3:2.2.1
HPCCP's National Coordination Office shares two of the strengths of
the Interagency Board: It appears to have facilitated communication
among the agencies and coordination of individual activities. In
addition, it has provided the Congress with a budget that looks at
the costs of high- performance computing activities across agencies.
Moreover, the National Coordination Office has the added advantage of
providing the Congress and the public with a central contact point
for information on high-performance computing.
POTENTIAL LIMITATIONS
------------------------------------------------------ Chapter 3:2.2.2
Just as the HPCCP shares the NNMRRP's strengths, it also has some of
its limitations. Specifically, no mechanism is in place to set
priorities across different agency activities. In a recent review,
GAO suggested that the HPCCP needs
"an explicit technical agenda, identifying and prioritizing
specific technology challenges and establishing a framework of
expected costs and results . . . [to] clarify the program's
goals and objectives, focus efforts on critical areas, and serve
as a baseline for measuring program progress and results."\6
In addition, HPCCP did not have uniform guidelines for which research
activities should be included in the budgets submitted by the
different agencies, mirroring the difficulty the other coordination
mechanisms have in tracking funds used for multipurpose programs.
The National Coordination Office also shares the NNMRRP's lack of
budget and staff resources for coordination.
An additional concern about the lead agency model was raised by our
expert advisers: If responsibilities for nutrition monitoring were
located in one agency, nutrition monitoring might become a monopoly,
serving the needs of only one agency. Its current dispersion across
agencies allows the different components to serve different purposes.
However, safeguards--such as that used for the HPCCP when it was
located in an agency that did not have a large investment in
high-performance computing, relative to some of the other
agencies--could address this concern.
--------------------
\6 High Performance Computing and Communications: New Program
Direction Would Benefit From a More Focused Effort (GAO/AIMD-95-6;
Nov. 4, 1994), p. 22.
OTHER COORDINATION
SUGGESTIONS
-------------------------------------------------------- Chapter 3:2.3
In addition to coordination by an interagency body with permanent
staff and enforcement authority or a single lead agency, other
suggestions for improving coordination of the NNMRRP were to locate
nutrition monitoring in statistical agencies within the user
departments, centralize the congressional appropriations process for
nutrition monitoring activities, and ensure informed review of data
collection plans by qualified staff at the Office of Management and
Budget.
The first idea--locating nutrition monitoring in statistical agencies
within the departments that use the data--was intended to focus the
program on the quality of the data collected. However, a concern
about this suggestion was that it could result in decreased
responsiveness to the needs of the data users. We did not pursue it
because the major surveys are already located in statistical or
research branches of HHS and USDA. (NHANES is operated by the
National Center for Health Statistics, and NFCS and CSFII were
recently relocated to ARS.)
Similarly, OMB already has responsibility for reviewing the data
collection activities of the agencies, although OMB staff reported
that they rely on the agencies to describe coordination efforts.
Finally, we did not review the advantages and disadvantages of
centralizing the congressional appropriations process for nutrition
monitoring activities because its relevance to the utility and
efficiency of the data collection activities was not clear.
CONCLUSIONS
---------------------------------------------------------- Chapter 3:3
Lack of coordination across program activities has implications for
both the effectiveness and the efficiency of the NNMRRP as a whole.
The Interagency Board has made progress toward coordinating
activities across the different agencies. However, the other
approaches to coordination suggest mechanisms that could further
strengthen the work of the Interagency Board.
First, because its responsibilities are shared by USDA and HHS, the
Interagency Board does not provide a central contact point for users
of nutrition monitoring data. A central contact could be established
if the Secretaries of USDA and HHS used their option to appoint an
administrator of the program or if responsibility for responding to
requests for information about the program was assigned to a single
agency, as it was for high-performance computing.
Second, the Interagency Board does not review the agency budgets it
compiles for consistency with the overarching priorities of the
program. Before such a review could occur, specific objectives and
priority activities would need to be identified. Then, the
Interagency Board could work with OMB to secure funding of NNMRRP
priorities.
APPROACHES TO THE CONTINUOUS
COLLECTION OF DATA
============================================================ Chapter 4
Evaluating programs intended to reduce diet-related chronic disease,
tracking progress toward health objectives, and monitoring changes in
our diets are the kinds of activities that require continuous data.
The advantages and disadvantages of the current and alternate
approaches to collecting continuous data are summarized in table 4.1
and detailed below.
Table 4.1
Approaches to Providing Continuous Data
Approach Strength Limitation
------------------------- ------------------------- --------------------------
Current: state-based Quick availability of No in-depth food intake or
surveillance systems data on specific issues health data
Connection between data Uneven quality and
elements and program completeness of data
decisions across states
Efficient relative to Limited to populations
national surveys because participating in publicly
of the reliance on funded programs (PedNSS
program records (PedNSS and PNSS) or living in
and PNSS) or telephone households with telephones
interviews (BRFSS) (BRFSS)
Incomplete participation
of states in PedNSS and
PNSS
Potential
A continuous, nationally- Up-to-date information on Costs of conducting all
representative survey the population survey processes
concurrently
Increased flexibility to
respond to new issues by Difficulties in
adding modules to a core identifying a core set of
set of data elements data elements
Increased efficiency as Inflexibility of core data
start-up costs diminish elements
Addition of nutrition Efficient means of Dependent on flexibility
questions to existing gathering data at of surveys
ongoing surveys frequent intervals
--------------------------------------------------------------------------------
CURRENT APPROACH: THE
STATE-BASED SURVEILLANCE
SYSTEMS
---------------------------------------------------------- Chapter 4:1
Although the provision of continuous data is one of the objectives
identified in the Interagency Board's 10-year plan, not one of the
three major NNMRRP surveys--NFCS, CSFII, or NHANES--is implemented
continuously. In fact, planned future administrations of two of the
surveys have been postponed, potentially compromising the ability to
monitor trends in diet-related health risks over time and evaluate
the effect of any changes in food assistance policy.
NFCS has been administered at approximately 10-year intervals since
the 1930s. Its next implementation was planned to begin in 1996, but
is now tentatively scheduled for 1998, depending on funding. CSFII,
originally intended to provide continuous data on dietary intake, has
had three separate administrations: 1985-86, 1989-91, and 1994-96.
After a 1-year pause in 1997, it is expected to resume for another
3-year period in 1998. Since the National Health Examination Survey
gained a nutrition component in 1971, NHANES has been fielded three
times: 1971-75, 1976-80, and 1988-94. Like NFCS, its future
implementation is uncertain because of budget constraints. Its
planned implementation in 1997 is now expected to be postponed.
Because of the lack of certainty about the implementation of the
national nutrition surveys, the state-based surveillance systems are
currently the primary source of continuous data in the NNMRRP. As
described in chapter 1, PedNSS and PNSS rely on data from clinic
records from publicly funded health, nutrition, and food assistance
programs and BRFSS collects information through telephone interviews,
with respondents (adults 18 years and over) identified through random
digit dialing. While a valuable source of quick information for
state and local program managers, the surveillance systems do not
meet the needs of researchers or program decisionmakers who require
either national data or in-depth food intake data.
STRENGTHS
-------------------------------------------------------- Chapter 4:1.1
Compared to the national surveys, one of the strengths of the
state-based surveillance systems is that they not only provide data
continuously, but they are also able to process and report the data
relatively quickly. For PedNSS and PNSS, information is collected as
part of the process of receiving services from WIC and other publicly
funded health, nutrition, and food assistance programs. Because they
depend on program records, PedNSS and PNSS do not burden respondents
the way surveys dependent on interviews do. The information is
transmitted from the records of local health and nutrition programs
to the state, which then forwards the records to HHS for analysis.
Similarly, the data collected by the states for BRFSS are sent to HHS
for processing. According to HHS officials, all three systems report
data back to the states within a year and generally in less than 9
months. HHS has also helped states conduct their own analyses by
distributing a standardized software package.
Another strength of the state-based surveillance systems is that the
data they collect are directly linked to program decisions. Although
PedNSS and PNSS include only a few indicators of nutritional
deficiencies and behaviors, they are selected to support state data
needs for program planning and management. For example, the data are
used to target resources for the WIC program. Similarly, BRFSS data
have been used to inform decisions about nutrition education
programs, such as campaigns to encourage the consumption of five
servings of fruits and vegetables a day.
LIMITATIONS
-------------------------------------------------------- Chapter 4:1.2
In contrast to the national surveys, the surveillance systems do not
permit examination of diverse diet-health associations across the
entire population. Instead of collecting extensive biochemical,
anthropometric, and interview data, they focus on a narrow range of
variables relevant to specific programs or nutritional risks. For
example, PedNSS collects clinical data on weight and height, monitors
infant feeding practices, and assesses anemia. PNSS collects
information on anemia and behaviors associated with low-birthweight
babies. BRFSS asks respondents to report on their consumption of
fat, fruits, and vegetables. This focus limits the breadth of uses
that can be supported by the data; however, as noted above, it also
limits the burden placed on respondents.
While the systems are currently limited in the amount of dietary data
they collect, HHS is exploring other methods of gathering these data.
For example, with HHS support, the University of Texas examined the
use of bar code data to look at dietary patterns. They concluded
that the technology is not yet ready for use, but through a
partnership with food manufacturers could be a promising method for
the future. In addition, with USDA, HHS is evaluating the
feasibility of collecting additional dietary data in the clinics that
provide the PedNSS and PNSS records.
Another concern about surveillance systems is the quality and
completeness of the data across the different states. For example,
for PedNSS, error can be introduced by variations in practice in
weighing infants, such as with or without the baby's winter clothes,
or by clerical errors in entering the data in states that do not have
automated data systems. PNSS, which attempts to collect a wider
range of information than PedNSS, suffers from missing data on
several variables, such as the pregnancy risk factors of smoking and
alcohol consumption. However, HHS provides technical assistance to
help states standardize their data collection procedures. In fact,
by flagging biologically implausible values for the physical
measures, HHS analyses of the surveillance data help identify clinics
that may have poor procedures.
The surveillance systems are also limited because, within the
participating states, only certain groups of the population are
covered. As described earlier, PedNSS and PNSS primarily provide
information on mothers and children participating in the WIC program.
As a result, data may not be available on other populations that are
potentially at risk, such as homeless people or older children not
eligible for WIC. In contrast, BRFSS has a wider target population,
collecting data from randomly selected adults 18 years and over.
However, neither adults in households without telephones nor children
are covered by BRFSS. Since there is evidence that some health risk
factors, such as smoking, are associated with living in a household
without a telephone, this could affect estimates of the extent of
diet-related risk factors as well.
Finally, not all states participate in the PedNSS and PNSS
surveillance systems and, therefore, they are not good sources of
national-level data on their populations of interest. In 1993, 38
states participated in PedNSS and only 20 states participated in
PNSS. (In contrast, BRFSS has good state coverage; in 1993, only
Wyoming did not participate.)
POTENTIAL OPTIONS
---------------------------------------------------------- Chapter 4:2
In addition to the state-based surveillance systems, we reviewed two
other approaches to providing continuous data. The first option is a
national nutrition-related survey that is operated continuously. To
describe the strengths and weaknesses of this approach, we consulted
with our expert advisers and interviewed managers of the current
national nutrition monitoring surveys about the continuous operation
of the surveys. The second option is the inclusion of
nutrition-related questions on continuous non-NNMRRP surveys. The
strengths and weaknesses of this alternative were explored by
examining the NNMRRP's recent experience developing food insecurity
questions for the Current Population Survey (CPS). In addition, our
expert advisers and the data users who responded to the survey made
other suggestions, which are also briefly presented.
A CONTINUOUS NNMRRP SURVEY
-------------------------------------------------------- Chapter 4:2.1
An ongoing national survey is a possible approach to providing
continuously updated information that addresses some of the
limitations of the state-based surveillance systems. One of the
current national surveys, the CSFII, was developed in response to
calls for the continuous collection of individual data on food
intake. It has not been implemented regularly in the past; however,
current plans are for the ongoing implementation of CSFII as a 3-year
survey followed by a 1-year pause for planning and development before
the next 3-year period.
POTENTIAL STRENGTHS
------------------------------------------------------ Chapter 4:2.1.1
An ongoing survey has the potential to yield continuously updated,
timely data for monitoring the nutritional status of the nation's
population. The increased timeliness of the data could decrease the
current pressure from collaborating federal agencies to include
components on each implementation of the periodic surveys. For
example, in contrast to the recent implementation of NHANES, which
attempted to meet as many data needs as possible within a single
survey, the ongoing implementation of NHANES could contain a core set
of data items that would be collected continuously, supplemented at
intervals by rotating modules. Such a streamlining would also reduce
the burden on respondents.
Moreover, the surveys could become more flexible by distinguishing
between variables that change rapidly, variables that need regular
but not continuous monitoring, and variables of emerging policy
importance. The core set of items in continuous implementation could
gather information on rapidly changing variables. Topical modules on
issues that do not change as rapidly could be included on a regular
schedule. Finally, as new issues arise, additional questions could
be added.
In addition to increased timeliness and flexibility, a continuous
survey operation could be more efficient than periodic surveys
because current costly start-up activities, such as planning,
designing sampling strategies, and training interviewers, would be
diminished. Thus, the data could be collected for less cost per
respondent. However, as described below, without concomitant
streamlining of the survey, overall costs could increase.
POTENTIAL LIMITATIONS
------------------------------------------------------ Chapter 4:2.1.2
While an ongoing survey could save money on start-up costs, it could
increase costs overall because activities that are now funded and
staffed sequentially would require a continuous flow of resources to
be conducted concurrently. For example, as NHANES is currently
implemented, staff change their activities as the survey moves
through the phases of planning, implementation, analysis, and
dissemination. If NHANES was implemented continuously, all of these
activities would be going on at the same time. According to both
USDA and HHS officials, the primary constraints on the continuous
operation of their surveys are the need for dependable funding and
sufficient staff resources. As described above, the absence of
dependable funding has affected the frequency with which the national
surveys are now implemented.
The development of a survey that continuously collects data on a core
set of items and intermittently collects data on other issues raises
two difficult issues. First, the definition of the core items is
complex. An expert panel convened by the Federation of American
Societies for Experimental Biology was charged with identifying a set
of core indicators to assess the nutritional status of
difficult-to-sample populations. The report summarizing the work of
the panel noted that (1) the suitability of an indicator changes with
the purposes for the data and (2) information on the determinants and
the consequences of each indicator is also needed.\1 The panel ended
by identifying three sets of indicators--minimal, intermediate, and
comprehensive--without recommending specific measures for the
indicators.
The second issue with a continuous survey is the potential
inflexibility of the core items once they are selected and
implemented. While opportunities to test new methods are enhanced
because the survey is in the field continually, making changes to the
data collection procedures can be difficult because of the pressure
to ensure that the measures are consistent over time. Without such
assurance, changes in an indicator such as obesity may be the result
of changes in how obesity is measured rather than changes in the
prevalence of the condition itself. Our expert advisers suggested a
survey with built-in periods of transition to allow the survey to
incorporate new methods and new data elements as they emerge.
--------------------
\1 S.A. Anderson, ed., "Core Indicators of Nutritional State for
Difficult-to-Sample Populations," Journal of Nutrition, 120:11S (Nov.
1990), 1559-99.
ADDING NUTRITION QUESTIONS
TO EXISTING CONTINUOUS
SURVEYS
-------------------------------------------------------- Chapter 4:2.2
A third approach to obtaining frequently up-dated information is the
addition of nutrition-related questions to existing continuous
surveys. The potential of existing surveys to provide data regularly
on some variables is demonstrated by plans that included NNMRRP
questions on food security in the April 1995 Current Population
Survey conducted by the Census Bureau.\2 Food security is a concept
intended to go beyond the idea of hunger to measure the availability
of food for a family or individual.
The food security questions were developed by an interagency working
group cochaired by HHS' National Center for Health Statistics and
USDA's Food and Consumer Service. While the working group developed
questions, USDA reserved space on the CPS for 1995. The question
development process included determining how the data would be used
and soliciting input from both federal and other data users. The
food security module contains both core questions and supplemental
ones. The Census Bureau will include the complete module. NNMRRP
surveys could include the smaller core set of questions. If the
initial implementation of the questions yields useful data, USDA
plans to continue to support the inclusion of the food security
questions annually in the CPS.
The same strategy of piggy-backing nutrition-related questions on
continuous surveys could be used with other surveys, such as the
Survey of Income and Program Participation or the National Health
Interview Survey. The latter has the added advantage of collecting
health data that could be linked to nutritional indicators.
--------------------
\2 The CPS is a monthly survey of about 60,000 households across the
country.
POTENTIAL STRENGTHS
------------------------------------------------------ Chapter 4:2.2.1
The major advantage of this approach is the efficiency with which
data on specific issues can be gathered. USDA will pay for the cost
that the food security questions add to the CPS without having the
responsibility or the cost of fielding and managing the survey
itself.
POTENTIAL LIMITATIONS
------------------------------------------------------ Chapter 4:2.2.2
A potential disadvantage of this approach is that, just as core
questions on nutritional status could be inflexible, existing
continuous surveys can be hard to change because of their momentum.
Moreover, existing surveys have their own set of constraints and
limitations. For example, while CPS could accommodate the addition
of questions about food security, it probably could not accommodate a
module obtaining data on an individual's dietary intake over the last
24 hours. A 24-hour recall instrument requires considerable training
to administer and adds substantially to the burden on the respondent.
For these reasons, opportunities to piggy-back nutrition-related
questions on other surveys may be limited.
OTHER OPTIONS FOR CONTINUOUS
DATA
-------------------------------------------------------- Chapter 4:2.3
In addition to the approaches reviewed, other actions were suggested
by our expert advisers and data users who responded to our survey.
Specifically, longitudinal surveys were suggested as an efficient way
to collect data over time because a new sample does not have to be
selected every time a survey is fielded. Since original respondents
are followed up in subsequent administrations of the survey,
longitudinal surveys can also be useful for tracking individual-level
changes in food consumption behavior. However, longitudinal surveys
have their own costs, including the need to collect additional data
so that respondents can be found for later surveys and the likelihood
of attrition as respondents either drop out or cannot be found.
Another suggestion was the collection of survey data using automated
survey technology. Because direct entry of the data into an
automated system can speed the processing and aggregation of the
survey, it can also accelerate the release of the data for analysis.
However, it does not affect the regularity with which the data are
collected in the first place. HHS already uses automated data
collection for NHANES. USDA is planning to automate the next
administration of the CSFII.
CONCLUSIONS
---------------------------------------------------------- Chapter 4:3
The funding constraints that have caused the postponement of two of
the national surveys jeopardize the availability of periodic data on
the population as a whole. Although the state-based surveillance
systems provide continuous information, they are inadequate to meet
the need for data on the population at large or on in-depth nutrition
and health status. Moreover, if approved, proposals to collapse
funds for the WIC program with other food assistance programs into a
block grant for the states could affect the major source of data for
two of the state-based surveillance systems, PedNSS and PNSS.
Although the NNMRRP has pursued such creative solutions as including
food security questions on the CPS, the availability of up-to-date
information could worsen. To decide how best to meet the needs for
continuous data in the future, the NNMRRP would first need to analyze
the purposes that require frequently collected data and the current
mechanisms for supporting those purposes. Within this framework, the
strengths and limitations of the different approaches to increasing
the frequency with which important indicators are measured could be
weighed according to which purposes are supported and which are
diminished.
APPROACHES TO SUPPORTING RELIABLE
INFERENCES ABOUT SUBPOPULATIONS
============================================================ Chapter 5
Subpopulations can be defined by geographic location as well as by
age and sex (such as infants or elderly women), physiological
characteristics (such as pregnancy), ethnicity or race (such as
Hispanic or Native American), income, and the intersection of any of
these groups (such as low-income children). As described in chapter
2, information on subpopulations is needed to appropriately target
and evaluate programs that address nutrition-related programs. As
summarized in table 5.1, this chapter describes current approaches
and some potential options to responding to the calls for better
information on subpopulation groups and small geographic areas in the
NNMRRP. Other issues of federal assistance to states and localities
for nutrition monitoring are discussed in chapter 6.
Table 5.1
Approaches to Supporting Inferences
About Subpopulations
Approach Strength Limitation
------------------------- ------------------------- --------------------------
Current
Oversampling of selected Less costly than a Appropriate only for
groups in the national separate survey special populations that
surveys are covered by the
Ability to compare the national sampling frame
subpopulation to the (such as children in low-
general population income households)
Primarily appropriate for
populations that are
geographically clustered
or large
Costs of screening for
members of the group
State-based surveillance Collection of data on No in-depth food intake or
systems high-risk populations: health data
PedNSS (low-income
children) and PNSS (low- Uneven quality and
income pregnant women) completeness of data
across states
Collection of data
representative of the Incomplete participation
state population of of states in PedNSS and
adults, 18 years or older PNSS
(BRFSS)
Potential expansion to
include programs that
serve other at-risk
populations
Potential
Special studies Ability to tailor data Inability to compare to
collection instruments the rest of the population
and content to the unless the special study
subpopulation is conducted in tandem
with a national survey
Collection of data on using comparable
subpopulations that are instruments
not adequately covered by
the sampling frame used Costs of tailoring data
for national surveys collection instruments,
developing the sampling
frame, and screening
Difficulties in developing
a sampling frame that
allows for generalization
Development of indirect Efficient means of Difficult to assess the
estimation programs generating estimates for quality of the estimate
subpopulations
Diminished confidence in
Historical use in other the estimates because they
areas (estimates of are based on models rather
population, crop yields) than direct observation
Response to limitations
of other sources of
information, such as cost
and relevance
--------------------------------------------------------------------------------
CURRENT APPROACHES
---------------------------------------------------------- Chapter 5:1
Subpopulation groups are covered in two ways by current NNMRRP
activities. First, the three national surveys use oversampling of
certain groups to ensure the selection of enough respondents to
support subpopulation estimates. NHANES focuses on racial and ethnic
subpopulations, while CSFII and NFCS have included subpopulations
defined by income, reflecting USDA's focus on food assistance to
low-income populations. The second way in which data on
subpopulations are gathered is through the state-based surveillance
systems--PedNSS, PNSS, and BRFSS.
OVERSAMPLING
-------------------------------------------------------- Chapter 5:1.1
Oversampling includes members of subpopulation groups in a sample at
a rate greater than their proportion in the population. The purpose
of oversampling is to ensure that data will be collected on enough
group members to support inferences about the group as a whole.
Oversampling is already used in the major NNMRRP surveys. NHANES III
(1988-94)
oversampled non-Hispanic blacks and Mexican-Americans, as well as
persons 60 years or older and children 1-5 years old. NHANES staff
indicated that two groups (Hispanics and persons 75 years and older)
are likely to be important groups in the next implementation of the
survey because both are growing and have significant health-related
issues to study. In addition to estimates of the general population,
the current administration of CSFII (1994-96) is expected to produce
estimates for low-income populations through oversampling. Also,
staff of both USDA and HHS surveys stated that the national surveys
could oversample a state's population, but that the state would have
to finance the added costs.
STRENGTHS
------------------------------------------------------ Chapter 5:1.1.1
Oversampling has two major strengths. First, because it can be used
in conjunction with a national survey, it has efficiencies of scale.
Specifically, the planning and implementation costs are diminished
because they are part of a larger survey. Second, data on both the
group and the rest of the population are collected at the same time
and with the same survey procedures, facilitating the comparison of
the two population groups.
LIMITATIONS
------------------------------------------------------ Chapter 5:1.1.2
Oversampling implies that the subpopulation group of interest is
included in the sampling frames used to identify participants in the
national surveys.\1 However, not all special populations are well
covered by a national sampling frame. For example, homeless
individuals, persons who live in institutions, and American Indians
and Alaska Natives living on reservations would not be included in
the national household sampling frame.
Even for those individuals who are included in the sampling frame,
oversampling may not be appropriate because of the costs incurred in
screening for members of the group. Screening is the process of
asking questions at sampled households to identify whether they
represent (or include representatives of) the subpopulation of
interest. Screening adds to the costs of the survey because enough
households have to be sampled and screened to identify the smaller
number of households or individuals that meet the definition of the
subpopulation. To reduce screening costs, oversampling is most
effective for subpopulations that are geographically clustered or
fairly well represented in the general population, such as persons
with low income who are often clustered by neighborhood. In
contrast, oversampling is not appropriate for groups that are few in
number or geographically dispersed, such as pregnant women.
A possible response to some of the limitations of oversampling is the
use of multiple sampling frames. For some subpopulations,
alternative frames or lists may be available that can be used in
conjunction with the national sampling frame. Samples can be
selected from both the subpopulation frame and the general population
frame, and weighted estimates can then compensate for the fact that
some group members could be selected from two different sources. For
example, to oversample for the frail elderly, the elderly individuals
identified in the sample drawn from the national sampling frame could
be supplemented by samples drawn from lists of elderly who
participate in congregate meals programs. This approach is useful
because, relative to screening in the general population, it is an
inexpensive way to identify members of the subpopulation.
--------------------
\1 The sampling frame is the group of units from which a sample is
actually drawn. So, for example, both NHANES and CSFII sample from a
sampling frame of households in the 50 states and the District of
Columbia.
STATE-BASED SURVEILLANCE
SYSTEMS
-------------------------------------------------------- Chapter 5:1.2
Although identified as a limitation in the previous chapter, the
focus of the state-based surveillance systems on particular
subpopulations can also be seen as a strength. As described in
chapters 1 and 4, states use BRFSS to collect data on the health
behaviors of their adult population. PedNSS and PNSS are sources of
information on the nutritional status of low-income mothers and
children. In addition, HHS is exploring opportunities to expand the
program to other populations, such as schoolchildren. However, the
other disadvantages of the systems, such as the limited amount of
nutritional data they collect and the incomplete participation of the
states, diminish their utility as a source of information on
subpopulations.
POTENTIAL OPTIONS
---------------------------------------------------------- Chapter 5:2
The two major alternate approaches to providing subpopulation data
are special studies and indirect estimation. Special studies are
those that use a separate sampling frame from a national survey and
are not necessarily conducted at the same time as a national survey.
To describe the strengths and weaknesses of special studies as an
approach to collecting information on subpopulations, we examined
HHS' past experience with the Hispanic Health and Nutrition
Examination Survey (HHANES) and discussed the option with our expert
advisers.
Indirect estimation uses data that are not direct observations of the
group of interest to develop inferences about the subpopulation. Our
review of this approach relied on reviews of technical literature,
interviews with USDA and Bureau of Census staff responsible for
indirect estimation programs, and consultation with methodological
experts on our panel of advisers and in HHS. These and the current
approaches address the suggestions made by the expert advisers and
data users who responded to our survey.
SPECIAL STUDIES: THE HHANES
MODEL
-------------------------------------------------------- Chapter 5:2.1
An alternate approach to covering subpopulations is conducting a
special study. A model of this approach is the Hispanic Health and
Nutrition Examination Survey conducted in 1982 by HHS. HHS developed
HHANES in response to recommendations made by the National Academy
for Public Administration, which identified the Hispanic population
as growing, likely to have low income, and potentially at risk for
health and nutrition problems. HHANES was conducted as a separate
study rather than integrated into the national survey because NHANES
II was already completed and funding was not available to conduct
HHANES as part of a national survey.
POTENTIAL STRENGTHS
------------------------------------------------------ Chapter 5:2.1.1
One of the advantages of conducting HHANES as a separate study was
the opportunity to change the content of the survey instrument to
address issues of special relevance to the Hispanic population. For
example, unlike NHANES II, HHANES gathered information on health
services use and gallstone disease. In addition, HHS took steps to
address the cross-cultural issues of applying HHANES to different
Hispanic populations. The survey instrument was translated into the
idiomatic Spanish of each of the three Hispanic groups surveyed
(Cuban-Americans, Mexican-Americans, and Puerto Ricans) and an
appropriate plan for reaching out to the respondents was developed.
For example, unlike the regular NHANES, which relies primarily on
press releases and the formal leaders (such as mayors) of the places
they have sampled to communicate the importance of participation in
the survey, HHANES used informal leaders (such as church leaders) and
Spanish-language media. The opportunity to tailor a special survey
to the population of interest is an advantage of the approach, but it
can be complicated for a population whose members speak multiple
distinct languages and have varying degrees of assimilation into the
U.S. population.
Although not demonstrated by HHANES, another advantage of a special
study is the ability to study populations that are not appropriately
studied through oversampling. As described above, this includes
groups that are not included in the national sampling frame, such as
homeless persons, and groups that are geographically dispersed or not
well represented in the general population, such as pregnant and
lactating women.
POTENTIAL LIMITATIONS
------------------------------------------------------ Chapter 5:2.1.2
Unlike oversampling, in which the subpopulation is surveyed at the
same time and usually with the same procedures as the population as a
whole, the data collected by special studies may not be comparable to
the population as a whole. For example, differences found between
HHANES data on Hispanic groups and NHANES data on the nation could be
the result of national changes in health and nutrition status during
the gap between the two surveys rather than actual differences
between the groups. Of course, conducting special studies in tandem
with a national survey is possible. In fact, HHS is considering
conducting special subpopulation surveys at the same time as the next
NHANES.
Special studies that are conducted in addition to the national
surveys clearly add to the overall cost of data collection. Specific
costs for surveys of ethnic populations include bilingual
interviewers, the translation of the survey instrument, and outreach
to the group. Other costs are the development of a separate sampling
frame and screening potential respondents to identify those that
belong to the group of interest. Finally, if a special study is
conducted at the same time as the national survey, the burden on the
survey support facilities, such as the laboratories that analyze
blood and urine specimens, is increased, which may slow down the
processing and release of the data.
If a special study is conducted because the group of interest is not
well covered by the national sampling frame, it needs a sampling
frame that allows for generalization to the subpopulation group,
which may be difficult to construct. For example, to use survey
results to draw conclusions about the population of people who are
without homes, one needs to sample from a complete list of the
members of that group. For the homeless population, such a list
would be very costly to construct. As a result, other means--such as
sampling shelters--would have to be used. The survey results,
though, would probably not be generalizable to the overall population
of homeless people because these other means are likely to be
incomplete.
INDIRECT ESTIMATION
-------------------------------------------------------- Chapter 5:2.2
Indirect estimation (also known as small area or synthetic
estimation) refers to procedures that use values of the variable of
interest from an area or time other than the area and time of
interest. For example, to develop an indirect estimate of the
prevalence of iron-deficiency anemia in a county, the national
estimate of iron-deficiency anemia can be adjusted based on the
county's demographic profile. Both USDA and HHS have experience with
indirect estimation. Since the early part of the century, USDA has
had a program to develop indirect estimates of crop yields. Although
some information is available from state surveys of nonprobability
samples of farmers, the USDA program adjusts these less dependable
estimates so that they aggregate to the more reliable regional and
national estimates that are based on a survey of a national
probability sample of farmers. HHS has no regular program to produce
indirect state estimates, but since 1968, it has supported the
occasional development and evaluation of indirect estimates from
National Health Interview Survey data. In addition, HHS produced
state estimates from the National Natality and National Fetal
Mortality Surveys conducted in 1980 using demographic data from the
states to adjust the national estimates. Indirect estimation models
range from the relatively straightforward adjustments of national or
regional estimates to match local demographic profiles to more
complex models.
POTENTIAL STRENGTHS
------------------------------------------------------ Chapter 5:2.2.1
The major strength of indirect estimation is that, compared to
increasing survey sample sizes to obtain data for direct estimates at
state and local levels, it is far less costly. In addition, it is a
means of extending the usefulness of costly national survey data to
inform decisions made at state or local levels of government.
Moreover, it is an established approach. Indirect estimates of such
variables as state and local populations, employment and
unemployment, and crop yields are already used by the federal
government in formulas for determining eligibility and benefit
amounts for federal programs. Some state governments have also used
indirect estimation to conduct analyses for economic and other types
of programs.
Indirect estimation also responds to limitations of the data on which
direct estimates might be based. Program records, such as those used
by PedNSS and PNSS, have the advantage of timeliness since they are
usually collected continuously, but their relevance may be limited
because the data are collected for specific administrative purposes,
not just for nutrition monitoring. Sample survey data, however, have
the advantage of relevance, but the data can be costly to obtain at a
level of detail that will support estimates for states, localities,
or other subpopulation groups. In contrast, indirect estimation is
an approach to producing timely, relevant, and detailed information
without a major increase in cost.
POTENTIAL LIMITATIONS
------------------------------------------------------ Chapter 5:2.2.2
The major limitation of indirect estimation is the difficulty of
determining the quality of the estimate. The best way is to compare
it to the true value for the population. For example, comparing an
indirect estimate of a county's population to census data on the
county's population would enable an assessment of the bias of the
estimate. However, since indirect estimation is used when data are
not available for direct estimation, such a comparison is usually not
feasible. Moreover, the quality of the estimates yielded by a model
changes for different populations and for different times. In other
words, even if a model yields estimates that prove to be unbiased in
comparison to direct observations for the same year, estimates from
the same model for subsequent years may be biased if the
relationships between the variables change over time.
This limitation has an implication for the use of indirect estimates.
Specifically, indirect estimates may be difficult to defend in the
political arena because they are based on models rather than direct
observations. However, in the absence of direct data from other,
more expensive approaches, indirect estimation is preferable to no
information at all.
Although indirect estimation has been used successfully with other
federal surveys, there are constraints on the development of indirect
estimation programs for the major nutrition monitoring surveys.
According to USDA and HHS staff, major obstacles include the lack of
staff resources to support the program and the complexity of the
surveys. Despite these concerns, both agencies have long experience
with indirect estimation in other arenas, which could be applied to
nutrition monitoring.
CONCLUSIONS
---------------------------------------------------------- Chapter 5:3
Information on subpopulation groups and small geographic areas is
used to identify nutrition-related problems that are associated with
specific populations and to target programs to those most at risk.
Different strengths and weaknesses are associated with the four
approaches to meeting this need that we reviewed. Surveillance
systems and indirect estimation programs are likely to be less
expensive than oversampling or special surveys. However,
oversampling and special surveys can yield more detailed information
than the surveillance systems that rely on program records and more
dependable estimates than indirect estimation models. Of the four
approaches, only indirect estimation has not been a part of NNMRRP
activities at one time or another. From programs in other areas,
this approach appears to be a potentially efficient means of
expanding the information available on subpopulations and small
geographic areas. However, before the Interagency Board can
determine what priority to give to this promising approach, a
complete picture of what needs would be met by each option is
required.
APPROACHES TO ASSISTING STATES AND
LOCALITIES
============================================================ Chapter 6
State and local governments are interested not only in the
applicability of federal nutrition monitoring data to their
jurisdictions, but also in having federal assistance in collecting
and interpreting their own data. This chapter follows on the
previous discussion about NNMRRP support for reliable inferences
about state and local populations to describe current and potential
options for assisting states and localities in their own monitoring
activities.
The NNMRRP currently assists states through the state-based
surveillance systems, which provide technical and other assistance to
states. The alternative that we examined, community-based nutrition
monitoring, is a response to states and localities that are
interested in building their capacity to collect their own data. The
strengths and weaknesses of these two approaches to assisting states
and localities are summarized in table 6.1 and detailed below.
Table 6.1
Approaches to Assisting States and
Localities
Approach Strength Limitation
------------------------- ------------------------- --------------------------
Current: state-based Balance between federal Limited flexibility for
surveillance systems need for standardization state and local
and state need for governments
tailoring
Limited support for local
Development of states' use of the data
capacity for data
collection and
interpretation
Potential for
improvements on current
foundation
Potential: community- Development of local Technical and financial
based nutrition capacity for collecting resources required to
monitoring and using data tailored assess needs or implement
to community concerns responses to needs
Limited federal Tension between federal
investment interest in
standardization and local
interest in flexibility
--------------------------------------------------------------------------------
Other suggestions by our expert panelists--such as providing
financial and technical assistance, developing standardized modules
of interest for state data collection activities, and assisting state
collection of data on subpopulations--can be implemented as part of
either of the two approaches.
CURRENT APPROACH: THE
STATE-BASED SURVEILLANCE
SYSTEMS
---------------------------------------------------------- Chapter 6:1
The strengths and limitations of the surveillance systems as sources
of continuous data and of information on state and other population
groups have been discussed in chapters 4 and 5. However, they have
additional strengths and limitations related to their ability to meet
state and local needs for assistance.
STRENGTHS
-------------------------------------------------------- Chapter 6:1.1
Surveillance systems currently balance the federal interest in
information that is collected in a standard format across states and
the states' needs for flexibility. For example, with BRFSS, HHS
supplies states with standardized modules of questions, training in
collecting the data, and support in analysis and reporting. In
addition, changes to the survey content are made in consultation with
state participants in the surveillance system, and states have the
opportunity to add their own questions to the survey.
The surveillance systems have also played a role in building state
capacity for data collection and analysis. Specifically, some states
have used their experience with BRFSS to implement their own
telephone surveys using the random digit dialing procedure. In
addition, standardized software developed by HHS for PNSS has enabled
states to generate their own reports.
Yet another strength of the surveillance systems is the foundation of
federal-state partnership it provides for future improvements of
federal technical assistance to state and local nutrition monitoring
activities. For example, as mentioned in previous chapters, HHS is
exploring ways to use the surveillance systems to gather additional
nutritional data, such as more in-depth information on dietary
intake, and to cover new populations, such as schoolchildren.
Additional technical assistance in data analysis and interpretation
could also be provided through the surveillance system structure.
LIMITATIONS
-------------------------------------------------------- Chapter 6:1.2
While the surveillance systems could be expanded and improved to
further respond to the interest of states and localities in receiving
more technical assistance, they are limited in their flexibility
because of the federal interest in standardization across states. In
addition, users of surveillance system data who responded to our
survey identified some concerns about the surveillance systems.
Specific issues included the availability of the data to localities,
timeliness of HHS' processing of the data, and the formats of the
reports that HHS provided. Recommendations for improving the
surveillance systems included increasing local access to data,
reducing the time it takes HHS to process the data, simplifying
reports for local users, and providing additional technical and
financial assistance in data collection and interpretation.
POTENTIAL OPTION:
COMMUNITY-BASED NUTRITION
MONITORING
---------------------------------------------------------- Chapter 6:2
The federal-state linkage forged by the surveillance systems could be
further extended to support community-based nutrition monitoring. To
explore the strengths and limitations of this approach, we reviewed
the literature on two models of community-based programs. HHS has
funded Planned Approach to Community Health (PATCH) projects, which
used the BRFSS survey instrument as the basis for a community needs
assessment. The survey data, in combination with interviews with
knowledgeable informants in the community, were used to plan health
promotion programs. HHS provided technical assistance in the needs
assessment portion of the projects and small awards of a few thousand
dollars for project activities.
Researchers at Cornell University developed a similar model
specifically for nutrition monitoring that was pilot-tested in three
New York counties with funding from the State Department of Health
and technical assistance from the university. In their approach, a
coalition of interested community members first articulates potential
information needs. Then, the group selects specific needs on the
basis of the likelihood that the data will be used by the community.
To meet the selected information needs, feasible sources of routinely
available data are identified. According to the model, data
collection should depend on procedures that are already in place,
from such sources as program participation records, patient charts,
and school health screenings. In its reliance on program records,
this approach is similar to PedNSS and PNSS. However, there is no
expectation that the same issues will be targeted or the same sources
of data used in each community. To facilitate the final component of
the system--the communication of monitoring results--a network of
users must be cultivated and the appropriate vehicle for
communicating the results must be used. Drawing on the experience of
these projects and HHS' PATCH program, the following strengths and
limitations of community-based nutrition monitoring were identified.
POTENTIAL STRENGTHS
-------------------------------------------------------- Chapter 6:2.1
Evaluations of both types of community-based programs found evidence
that local capacity for collecting and using data was developed. By
involving community members in assessing needs, PATCH built skills in
identifying health risks and cultivated community and organizational
supports for health promotion programs. In the New York counties
that pilot-tested the Cornell model of community-based nutrition
monitoring, data were collected and compiled to describe access to
food and nutrition services and nutritional health of county
residents. To disseminate the information and guide decisions based
on the data, interagency coalitions were formed. These coalitions
brought together local nutrition-related professionals, so that
nutrition interventions were better coordinated and information was
shared.
The limited federal investment in the community-based programs is
another strength. HHS supported PATCH with small grants and
technical assistance, while the community-based nutrition monitoring
projects were sponsored by the state and received technical
assistance from the state's land grant institution.
Moreover, there is some evidence that the activities put in place by
these projects will continue. For example, Cornell reports that
"At the end of the [3-year] pilot stage, one county obtained
local funding to continue their monitoring activities, including
the issue-based coalitions; in the other counties, local
nutrition councils are coordinating continued monitoring
efforts."\1
The program has also spread to other counties, some of which have
initiated monitoring activities without outside funding.
--------------------
\1 "Community-Based Nutrition Monitoring," Division of Nutritional
Sciences, Cornell University, and Division of Nutrition, N.Y. State
Department of Health, Mar. 1993.
POTENTIAL LIMITATIONS
-------------------------------------------------------- Chapter 6:2.2
The experience with community-based programs indicates that
communities require a set of resources--specifically, technical
skills and dedicated personnel--to fully benefit from the projects.
One evaluation of PATCH found that the projects were most effective
in communities that already had human services and health
professionals who were involved in community health promotion
efforts. In addition, the projects that had directors seemed to have
the greatest implementation successes. Both kinds of projects
required considerable time and effort to collect and interpret data.
Moreover, while HHS provided technical assistance in using the BRFSS
survey in the local needs assessments, additional assistance was
needed to help communities set priorities based on the data.
The reliance on BRFSS was a limitation for PATCH because BRFSS did
not necessarily include the issues of primary concern for the
community. For example, one PATCH project was specifically
interested in water quality, which is not addressed in BRFSS. In
contrast, the Cornell model emphasized local sources of data. Based
on the initial experience with PATCH, HHS no longer expects
communities to use BRFSS for its needs assessment.
CONCLUSIONS
---------------------------------------------------------- Chapter 6:3
Data from the HHS surveillance systems are already extensively used
in state and local program planning. In addition, the systems are
flexible and are exploring ways to increase the information they
collect on dietary intake. However, they seem less responsive to the
needs of localities than to the needs of states. Recent efforts to
build local capacity for data collection and interpretation indicate
that community-based programs are a promising approach to responding
to local needs for nutrition information. Before the Interagency
Board can decide what priority to place on community-based nutrition
monitoring, however, it must first identify the objectives of the
NNMRRP that would be furthered and the importance of these objectives
relative to others that are competing for program resources.
LIST OF EXPERTS
=========================================================== Appendix I
This appendix lists the expert advisers who assisted with this
project. As described in chapter 1, the advisers were organized into
three panels--core policy panel, methodology panel, and data users
panel.
CORE POLICY PANEL
------------------------------------------------------- Appendix I:0.1
Johanna Dwyer, D.Sc., R.D., Francis Stern Nutrition Center, New
England Medical Center, and Tufts University Schools of Medicine and
Nutrition
Jean-Pierre Habicht, M.D., Ph.D., Division of Nutritional Sciences,
Cornell University
Catherine Woteki, Ph.D.\1
--------------------
\1 Dr. Woteki withdrew from the panel when she was appointed to the
Office of Science and Technology Policy in the White House. During
her participation in our work, she was the Director of the Food and
Nutrition Board of the National Academy of Sciences.
METHODOLOGY PANEL
------------------------------------------------------- Appendix I:0.2
Norman Bradburn, Ph.D., Director, National Opinion Research Center
Marilyn Buzzard, Ph.D., Director, Nutrition Coordinating Center,
University of Minnesota
Ricardo O. Castillo, M.D., M.P.H., Co-Director, Pediatric
Gastroenterology, Stanford University Medical Center
Alan R. Kristal, Dr.P.H., Fred Hutchinson Cancer Research Center and
Department of Epidemiology, University of Washington
James Lepkowski, Ph.D., Institute for Social Research, University of
Michigan
Cheryl Ritenbaugh, Ph.D., Department of Family and Community
Medicine, University of Arizona
Laura Sims, Ph.D., Department of Nutrition and Food Science,
University of Maryland
DATA USERS PANEL
------------------------------------------------------- Appendix I:0.3
Elizabeth Barnett, Ph.D., North Carolina Department of Environment,
Health, and Natural Resources
Doris Disbrow, Dr.P.H., R.D., Center for Health Education
Pamela Haines, Dr.P.H., R.D., Department of Nutrition, University of
North Carolina
Jay Hirschman, M.P.H., Senior Analyst, Food and Consumer Service,
U.S. Department of Agriculture
Karen J. Morgan, Ph.D., Senior Director, Nutrition and Consumer
Affairs, Nabisco Brands
Barbara Petersen, Ph.D., Technical Assessment Systems
SOURCES FOR THE MODEL FEATURES
========================================================== Appendix II
The model features that are the focus of this report were derived
from four sources. First, we reviewed previous evaluations, from
which 13 historical criticisms of federal nutrition monitoring were
identified. Second, we considered the objectives and related
activities outlined in the 10-year comprehensive plan developed by
the Interagency Board. Third, through an iterative process of panel
meetings and review of written materials, a number of suggestions for
improvements to the NNMRRP were generated by the expert advisers.
Fourth, the data users who responded to our survey provided written
comments describing what changes to specific data collection
activities would increase their use of the data.
PAST EVALUATIONS OF FEDERAL
NUTRITION MONITORING
-------------------------------------------------------- Appendix II:1
Table II.1 lists the criticisms identified in past evaluations and
categorizes them by the features of a model program that they
suggest. (See Nutrition Monitoring (GAO/PEMD-94-23), pp. 5-6.)
Table II.1
Features of a Model Program Suggested by
Past Evaluations
Feature Past criticism
--------------------------------------- ---------------------------------------
Coordinated system that responds Lack of coordination among nutrition
efficiently to the diverse needs of the monitoring activities
data users
Information needs of users not
systematically determined
Lack of compatibility in methods for
assessing dietary intake
Core set of standardized measures not
yet developed for major surveys
Compatible sampling techniques not used
for national surveys
Reporting by national surveys not
integrated
Continuous or more frequent collection Data not collected continuously
of data
Support for reliable inferences about Specific population groups not covered
subpopulation groups and small by major surveys
geographic areas
Specific geographic areas not
represented by major surveys
Assistance to states and localities State and local data needs not fully
addressed
Improved methodology for assessing Improvement needed in methodology for
dietary intake and nutritional status assessing dietary intake and
nutritional status
Low respondent burden and high response Sampling problems with NFCS
rates
Timely processing and dissemination of Need for more timely dissemination of
data survey information
--------------------------------------------------------------------------------
THE 10-YEAR COMPREHENSIVE PLAN
-------------------------------------------------------- Appendix II:2
Table II.2 provides examples from the 68 activities listed in the
10-year comprehensive plan that correspond to the desirable features
identified from the review of previous evaluations of federal
nutrition monitoring. (Other desirable features suggested by the
10-year plan activities but not addressed in this report focus on
strengthening the food composition and food supply data systems.)
Table II.2
Examples of Planned NNMRRP Activities
Associated With Model Features
Feature Planned activity\a
--------------------------------------- ---------------------------------------
Coordinated system that responds Coordinate the planning for coverage,
efficiently to the diverse needs of the tracking, and reporting of findings
data users from surveys and surveillance systems
Identify ways to increase comparability
within a dietary method to improve the
quality and usefulness of data
Establish a mechanism for improved
coordination among federal agencies
that collect and use survey information
about knowledge, attitudes, and
behavior to assess gaps and
duplications in existing surveys
Continuous or more frequent collection No specific activities listed\b
of data
Support for reliable inferences about Develop and implement a plan for
subpopulation groups and small improved coverage of subgroups of the
geographic areas population at nutritional risk or at
risk for underconsumption or
overconsumption of nutrients and food
components
Evaluate the feasibility of alternate
statistical methodologies for creating
state and local estimates
Assistance to states and localities Provide assistance for the development
and maintenance of state structure,
staff, and programs to support their
participation in NNMRRP
Expand the coverage of current state
and local nutrition monitoring
activities in selected population
groups through technical assistance and
grant awards
Improved methodology for assessing Conduct research to develop, improve,
dietary intake and nutritional status and validate laboratory measures of
nutritional status
Develop and evaluate procedures for
determining usual intakes of foods and
nutrients from surveys employing 24-
hour recall measures of dietary intake
Low respondent burden and high response No specific activities listed\c
rates
Timely processing and dissemination of No specific activities listed\d
data
--------------------------------------------------------------------------------
\a See the Ten-Year Comprehensive Plan, 58 Fed. Reg. 111 (June 11,
1993), pp. 32752-806.
\b Although no specific activities are planned to ensure the
continuous collection of data, two of the six objectives include
continuous data and one of the activities mentions coordinating the
timing of surveys.
\c USDA has separated the household and individual portions of the
NFCS survey in order to reduce respondent burden and improve response
rates. In addition, USDA signed interagency agreements with the
Bureau of Census for assistance in designing and conducting the
individual and household surveys, as well as for research on
improving methods for collecting household food use data.
\d Although none of the activities identified under the objectives
specifically address the timeliness of data processing and
dissemination, part of the overarching goal stated for the plan is
"efficiently disseminating and exchanging information with data
users." In addition, the plan discusses possible mechanisms for
increasing awareness and dissemination of the data and states that
the Interagency Board plans to establish a central clearinghouse.
CONSULTATION WITH EXPERT
ADVISERS
-------------------------------------------------------- Appendix II:3
The expert advisers to the project (listed in appendix I) provided
guidance by reviewing materials and participating in panel meetings.
Through this process, several suggestions were generated, and their
advantages and disadvantages were discussed. The suggestions
outlined below in table II.3 do not indicate consensus among the
panelists.
Table II.3
Advisory Panel Suggestions Associated
With Model Features
Feature \Suggested approach\
--------------------------------------- ---------------------------------------
Coordinated system that responds A single lead agency
efficiently to the diverse needs of the
data users
An interagency body with permanent
staff and enforcement authority
Location in statistical agencies within
the user departments or within a
central statistical agency
Centralization of the congressional
appropriations process for nutrition
monitoring activities
Informed review of data collection
plans by qualified people at OMB
Continuous or more frequent collection Continuous national nutrition surveys
of data
Addition of modules to existing surveys
Reliance on program data that are
already collected
Collection of longitudinal data
Support for reliable inferences about Surveys of population subgroups instead
subpopulation groups and small of national-level surveys
geographic areas
Different sampling strategies for
different populations (for example,
list-based sampling for populations
that are rare in a broad-based sample)
Contracts with states and localities to
gather information on geographically-
based populations
Indirect estimation to support
inferences about population subgroups
and small geographic areas
Assistance to states and localities\a Financial assistance to states to
determine their data needs
A federal-state partnership in which
states can provide funds for some extra
sampling or extra questions on federal
surveys
Federal development of standardized
modules of interest for state data
collection activities
Federal assistance to state collection
of data on subpopulations
Technical assistance in data
interpretation
Improved methodology for assessing Continued or increased research on
dietary intake and nutritional status nutrition monitoring methods
Postmarket surveillance data (such as
that gathered through bar code
scanners)
Automated collection of dietary data
Reliance on program records
Low respondent burden and high response Provide more feedback to respondents
rates about survey results
Timely processing and dissemination of Assistance to data users in
data interpreting and analyzing the data
Development of core analysis data sets
that are focused on specific issues or
groups
A data clearinghouse
Automated data collection
--------------------------------------------------------------------------------
\a The panelists disagreed about the responsibility states should
have for data collection, with some arguing for state-based data
collection that feeds into a federal system and others arguing for
less state responsibility for data collection, but increased
consideration of state needs in federal data collection activities.
In addition to comments that could be categorized by the model
features suggested by the historical criticisms, the expert advisers
also had other suggestions for improving the system, including
-- develop an ongoing evaluation of the system's content and
methods, including a review of the information needs;
-- continue to collect both household and individual level data;
-- have a system to maintain data comparability over time; and
-- use longitudinal designs to measure change.
SUGGESTIONS FROM THE SURVEY OF
DATA USERS
-------------------------------------------------------- Appendix II:4
The comments that users of nutrition monitoring data provided in
response to our survey were analyzed to identify major themes for
each of three groups of data collection activities--USDA surveys, HHS
surveys, and HHS state-based surveillance systems. Table II.4
identifies the themes associated with the features of a model
program. More detailed summaries can be found in Nutrition
Monitoring (GAO/PEMD-95-15).
Table II.4
Major Themes in the Suggestions Made by
Data Users
HHS surveillance
Model feature USDA surveys HHS surveys systems
------------------ ------------------ ------------------ --------------------
Coordinated system \a \a \a
that responds
efficiently to the
diverse needs of
the data users
Continuous or more Continuous or more Continuous or more \b
frequent frequent data frequent data
collection of data collection collection
Support for Increased sample More detail on More detail on
reliable sizes and racial, ethnic, subpopulation groups
inferences about broadened and age groups in the reporting of
subpopulations coverage the data
geographic areas Data that can
More detail on support estimates Increased ability to
racial, ethnic, for small look at substate
age, and income geographic areas geographic divisions
groups
Refined geographic
coverage,
specifically state
and substate data
Assistance to \c \c Simplified reports
states and that are more
localities readily used at the
local level
Additional technical
and financial
assistance in data
collection and
interpretation
Better controls on
the quality of the
data collected
Improved More specificity More information More data on dietary
methodology for and detail about on health habits intake
assessing dietary foods and better and outcomes
intake and data on
nutritional status food composition More detailed data
on food
Improved questions consumption
about dietary
behavior Improved dietary
intake methods
More information
about health and
demographic
variables
Higher quality
dietary recall
data
Reduced respondent Reduced respondent \d \d
burden and burden and
improved response improved response
rates rates
Timely processing Improved Improved Improved timeliness
and dissemination timeliness and timeliness and of HHS' return of
of data documentation of documentation of the data to states
the data the data
Dissemination of Dissemination of
the data in the data in
formats that formats that
facilitate access facilitate access
and analysis and analysis
--------------------------------------------------------------------------------
\a Coordination was not a major theme of the comments provided in
response to our survey of data users probably because the survey
queried respondents about changes to specific data collection
activities rather than NNMRRP as a whole.
\b Users of surveillance system data were unlikely to identify
continuous data as a major theme since these systems collect data
continuously.
\c Because our survey asked for comments about the two data
collection systems most frequently used by the respondent, state and
local users were more likely to comment on the surveillance systems.
As a result, their interest in changing the national surveys to
better meet their needs was not gauged.
\d The HHS systems have not suffered the same response rate problems
that USDA's 1987-88 NFCS experienced.
The themes reflect the strengths and weaknesses of the different data
collection activities. For example, response rates were probably a
major issue to users of USDA surveys because of the response rate
problems of NFCS. Similarly, assistance to states and localities was
a theme from the users of the surveillance systems, many of whom
manage state and local programs. In addition to the themes that
correspond to the features of a model program, other major issues in
the suggestions from the data users were the collection of
longitudinal data (from users of USDA survey data) and the collection
of both individual and household data (from users of data from USDA
and HHS surveys).
(See figure in printed edition.)Appendix III
COMMENTS FROM THE DEPARTMENT OF
AGRICULTURE
========================================================== Appendix II
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
The following are GAO's comments in response to the June 13, 1995,
letter from the Department of Agriculture.
1. What USDA finds inconsistent with the report reflects the
inconsistency of NNMRRP. The 10-year plan, a natural location for a
statement of priorities within expected funding levels, specified
neither priorities nor expected funding levels. As noted in our
report, the later priority-setting exercise was an improvement but
was also limited in its utility because it did not specify whether
data needs and feasibility were considerations in ranking the
objectives.
2. The central contact for data users that is suggested by the
alternate approaches to coordination entails much more than a
clearinghouse for data products. For example, the Office of National
Drug Control Policy acts not only as a source of information for
state and local drug enforcement interests, but also as a conduit of
information about state and local concerns to the federal agencies.
3. USDA makes several recommendations on how the Board can be
improved. These recommendations are consistent with the strengths
that we identified in the other approaches to coordination.
4. The Consumer Expenditure Survey also provides information about
the cost of food used at the household level. However, because of
the detail about the expenditures on specific foods gathered by the
NFCS, we have amended the description of the NFCS.
5. Following the classification provided by the NNMRRP document, The
Directory of Federal and State Nutrition Monitoring Activities, CSFII
and DHKS focus on food and nutrient consumption. Table 1.1 indicates
that data on food and nutrient consumption are used for health- as
well as food-related purposes.
6. The sentence has been changed to reflect this information.
7. Because the coordinated NNMRRP budget reports amounts dedicated
to nutrition monitoring by agency rather than by purpose, the
accounting system used by ARS is not relevant. All NNMRRP
expenditures by ARS will be reported together, whether they are
dedicated to surveys or related research.
8. The fact that the plan was intentionally vague does not diminish
the degree to which the lack of detail limits its utility as a
statement about the specific objectives of the program, the
activities planned to meet the objectives, and the resources needed
to support the plans.
9. While technical expertise can be supplied by staff, only members
with organizational authority can make decisions about agency
priorities and resources. USDA's comment regarding p. ES-7
(responded to in comment 3 above) suggests the importance of having
members with budget decision-making authority.
10. The statement has been changed to reflect this information.
(See figure in printed edition.)Appendix IV
COMMENTS FROM THE PUBLIC HEALTH
SERVICE
========================================================== Appendix II
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
The following are GAO's comments on the June 16, 1995, letter from
the Public Health Service.
1. As PHS states, the 10-year plan does plan to conduct evaluations
in 1997 and 2002. However, neither these evaluations nor the Board's
annual summaries of progress are a strategy for weighing the costs
against the benefits of current or new activities. A framework for
comparing existing and potential monitoring activities in such terms
as their feasibility, ability to meet data needs, and other
considerations is still needed.
2. The paragraph referred to (now on pp. 34-35) is not about the
progress of the working groups, but about the guidance provided by
the 10-year plan for the program as a whole. While the progress
reports indicate specific examples of improvements within working
groups, the 10-year plan does not constitute a comprehensive
proposal. Such a proposal would include specific objectives,
detailed plans for how to meet the objectives, and cost estimates and
funding requests to support the plans. The availability of resources
affects the ability to meet an objective, but is not a reason for not
developing specific plans in the first place.
3. In contrast to such coordinating bodies as the Interagency Board
and the National Library of Medicine, the central authorities that we
reviewed have personnel who are not detailed from the agencies.
While the potential disadvantage that PHS suggests (that is,
unfamiliarity with NNMRRP) can be easily overcome, the advantage of
independence from the interests of any single agency is difficult to
achieve in the other approaches.
4. The sentence has been changed to clarify that, with current
levels of state participation, the PedNSS and PNSS can not be used to
provide nationally representive data of the low-income mothers and
children receiving assistance through the WIC program.
5. This section has been changed to reflect the lack of coverage of
American Indians and Alaska Natives living on reservations by the
national household sampling frame.
MAJOR CONTRIBUTORS TO THIS REPORT
=========================================================== Appendix V
PROGRAM EVALUATION AND
METHODOLOGY DIVISION
--------------------------------------------------------- Appendix V:1
John Oppenheim, Assistant Director
Leslie Riggin, Project Manager
L� Xu�n Hy, Adviser
Venkareddy Chennareddy, Referencer
Elizabeth W. Scullin, Communications Analyst
RELATED GAO PRODUCTS
============================================================ Chapter 1
Nutrition Monitoring: Data Serve Many Purposes; Users Recommend
Improvements (GAO/PEMD-95-15; June 20, 1995).
High Performance Computing and Communications: New Program Direction
Would Benefit From a More Focused Effort (GAO/AIMD-95-6; Nov. 4,
1994).
Nutrition Monitoring: Progress in Developing a Coordinated Program
(GAO/PEMD-94-23; May 27, 1994).
Food Nutrition: Better Guidance Needed to Improve Reliability of
USDA's Food Consumption Data (GAO/RCED-94-30; Oct. 25, 1993).
*** End of document. ***