Nutrition Monitoring: Data Serve Many Purposes, But Users Recommend
Improvements (Letter Report, 06/20/95, GAO/PEMD-95-15).
The National Nutrition Monitoring and Related Research Program consists
of a network of surveys, surveillance systems, and research activities
that serves various purposes. It provides researchers and
decisionmakers with data for assessing the safety of the nation's food
supply, targeting food assistance to low-income families, and studying
the relationship between diet and disease. The program has been
criticized, however, for the lack of coordination among the various
activities and its poor coverage of populations at risk of nutritional
problems. GAO surveyed users of nutrition-monitoring data. This report
(1) describes the users and the major uses of nutrition-monitoring data
and (2) summarizes user satisfaction with nutrition-monitoring
activities and the changes that users believe are likely to increase
their use of, or confidence in, the data.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: PEMD-95-15
TITLE: Nutrition Monitoring: Data Serve Many Purposes, But Users
Recommend Improvements
DATE: 06/20/95
SUBJECT: Nutrition research
Nutrition surveys
Data collection operations
Monitoring
Food supply
Data integrity
Interagency relations
Health research programs
Demographic data
IDENTIFIER: Special Supplemental Food Program for Women, Infants, and
Children
USDA Continuing Survey of Food Intakes by Individuals
Hispanic Health and Nutrition Examination Survey
National Health and Nutrition Examination Survey
HNIS Nationwide Food Consumption Survey
National Health Interview Survey
National Nutrition Monitoring and Related Research Program
WIC
HANES
PHS Pregnancy Nutrition Surveillance System
PHS Pediatric Nutrition Surveillance System
PHS Navajo Health and Nutrition Survey
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to the Ranking Minority Member, Committee on Science, House of
Representatives
June 1995
NUTRITION MONITORING - DATA SERVE
MANY PURPOSES; USERS RECOMMEND
IMPROVEMENTS
GAO/PEMD-95-15
Nutrition Monitoring Data Serve Many Uses
(973403)
Abbreviations
=============================================================== ABBREV
ARS - Agricultural Research Service
BRFSS - Behavioral Rish Factor Surveillance Survey
CDC - Centers for Disease Control and Prevention
CSFII - Continuing Survey of Food Intake by Individuals
DHKS - Diet and Health Knowledge Survey
FDA - Food and Drug Administration
HHANES - Hispanic Health and Nutrition Examination Survey
HHS - Department of Health and Human Services
HNS - Health and Nutrition Survey
NCCDPHP - National Center for Chronic Disease Prevention and Health
Promotion
NCHS - National Center for Health Statistics
NFCS - Nationwide Food Consumption Survey
NHANES - National Health and Nutrition Examination Survey II and
III
NHEFS - National Health and and Nutrition Examination Survey I
NHIS - National Health Interview Survey
NNMRR - National Nutrition Monitoring and Related Research
NNMRRP - National Nutrition Monitoring and Related Research Program
NTIS - National Technical Information Service
PedNSS - Pediatric Nutrition Surveillance System
PHS - Public Health Service
PNSS - Pregnancy Nutrition Surveillance System
USDA - U.S. Department of Agriculture
WIC - Special Supplemental Food Program for Women, Infants, and
Children
Letter
=============================================================== LETTER
B-260632
June 20, 1995
The Honorable George E. Brown, Jr.
Ranking Minority Member
Committee on Science
House of Representatives
Dear Mr. Brown:
The National Nutrition Monitoring and Related Research Program
(NNMRRP) is a network of surveys, surveillance systems, and research
activities designed to serve multiple purposes. It provides
researchers and decisionmakers with data for assessing the safety of
the nation's food supply, targeting food assistance to low-income
families, and studying the relationship between diet and disease,
among other uses. However, past evaluations of federal nutrition
monitoring have criticized it on several counts, including the lack
of coordination among the various activities and its poor coverage of
populations at risk of nutritional problems. Through the National
Nutrition Monitoring and Related Research Act of 1990 (P.L.
101-445), the Congress established objectives for addressing these
problems.
This report is the second in a series of three responding to your
request for information on the NNMRRP. In our first report,
published last year, we detailed the activities that make up the
NNMRRP, the history of concerns about the data collection systems,
and agency progress toward meeting the objectives of the NNMRR Act.\1
In this report, we summarize the results of our survey of users of
nutrition monitoring data. Specifically, the objectives of this
report are to (1) describe users and major uses of nutrition
monitoring data and (2) summarize the satisfaction of users with
selected nutrition monitoring activities and the changes that users
identified as likely to increase their use of or confidence in the
data. The survey results presented here serve as a foundation for
our follow-up report on the features of a model nutrition monitoring
program.\2
--------------------
\1 Nutrition Monitoring: Progress in Developing a Coordinated
Program (GAO/PEMD-94-23; May 27, 1994).
\2 See GAO/PEMD-95-19, forthcoming.
RESULTS IN BRIEF
------------------------------------------------------------ Letter :1
The data users who responded to our survey were located in a variety
of settings, including governmental, academic, and business. These
users reported that they provide analyses to the general public as
well as to decisionmakers in their organizational settings and to
other audiences. They also reported using nutrition monitoring data
for an extensive range of purposes, from identifying
nutrition-related problems and designing programs to address the
problems to informing basic research.
Although most of the data users who responded to our survey were
satisfied with the degree to which the data meet their information
and data quality needs, a majority also suggested changes that would
increase their use of or confidence in the data. Their
recommendations include the need for improved dietary intake methods,
more continuous data collection, better coverage of subpopulations
and small geographic areas, improved timeliness and documentation of
the data, and increased dissemination of the data in formats that
facilitate access and analysis.
BACKGROUND
------------------------------------------------------------ Letter :2
The U.S. nutrition monitoring system has included more than 70
separate data collection activities conducted by several different
federal agencies. Major components of the system include the
national health and nutrition surveys administered by the National
Center for Health Statistics (NCHS) within the Centers for Disease
Control and Prevention (CDC), the state-based surveillance systems
managed by the National Center for Chronic Disease Prevention and
Health Promotion (NCCDPHP), and national surveys operated by the
Agricultural Research Service (ARS). Table 1 lists the data
collection activities addressed in our survey.
Although the system has been praised for being comprehensive, it has
also been criticized for the redundancy of some of the monitoring
activities, the prolonged data collection and delays in data release,
the poor coverage of subpopulations, and the lack of compatibility in
data assessment and sampling methods across different surveys.\3 In
response to these concerns, the National Nutrition Monitoring and
Related Research Act of 1990 required the Secretaries of the
Departments of Agriculture (USDA) and Health and Human Services (HHS)
to implement a coordinated program of nutrition monitoring and
established an Interagency Board to facilitate the process. The
Board developed preliminary plans for meeting the goals of the
legislation and published them in a 10-year comprehensive plan in
June 1993.\4
Table 1
Data Collection Systems in Our Survey\a
Time period
Agency Data collection system covered Short name
---------- ------------------------------------ -------------- --------------
HHS/PHS/ National Health and Nutrition 1982-84, 1986, NHEFS
CDC/NCHS Examination Survey I: 1987, 1992
Epidemiological Follow-up
National Health and Nutrition 1976-80 NHANES II
Examination Survey II
National Health and Nutrition 1988-94 NHANES III
Examination Survey III
Hispanic Health and Nutrition 1982-84 HHANES
Examination Survey
National Health Interview Survey 1986 NHIS-Vitamin
Supplement on Vitamin
and Mineral Supplements
National Health Interview Survey 1987, 1992 NHIS-Cancer
Supplement on Cancer Epidemiology
and Cancer Control
HHS/PHS/ Behavioral Risk Factor Surveillance Continuously BRFSS
CDC/ System since 1984
NCCDPHP
Pregnancy Nutrition Surveillance Continuously PNSS
System since 1988
Pediatric Nutrition Surveillance Continuously PedNSS
System since 1973
HHS/PHS/ Navajo Health and Nutrition Survey 1991, 1992 Navajo HNS
IHS
HHS/PHS/ Health and Diet Survey Biannually Health and
FDA since 1982 Diet
USDA/ARS Nationwide Food Consumption Survey Every 10 years NFCS
since 1936,
1977-78, 1987-
88
Continuing Survey of Food Intake by 1985-86, 1989- CSFII
Individuals 91
Diet and Health Knowledge Survey 1989, 1990, DHKS
1991, 1993
--------------------------------------------------------------------------------
\a For more information on the scope and design of these systems, see
Directory of Federal and State Nutrition Monitoring Activities,
prepared by the Interagency Board for Nutrition Monitoring and
Related Research, HHS Pub. No. (PHS) 92-1255-1 (1992).
--------------------
\3 See GAO/PEMD-94-23 for a discussion of these concerns and the
NNMRRP activities intended to address them.
\4 Ten-Year Comprehensive Plan for the National Nutrition Monitoring
and Related Research Program, 58 Fed. Reg. 111 (June 11, 1993), pp.
32752-806.
SCOPE AND METHODOLOGY
------------------------------------------------------------ Letter :3
To obtain information on the current uses of nutrition monitoring
data and identify the kinds of changes that are needed to increase
the utility of the data, we conducted a survey of potential users of
nutrition monitoring data. The survey focused on the 14 NNMRRP data
collection activities listed in table 1, selected because they are
major activities or because they addressed a major concern, such as
the need for data on subpopulations. These activities collect three
kinds of nutrition data: nutritional and health status; food
consumption and dietary intake; and dietary knowledge, attitudes, and
behavior. Two other areas of nutrition monitoring--food composition
and food supply and demand--were not addressed by the survey.
Because we found no comprehensive list of people who use these data,
we developed a complex sampling plan to obtain lists of potential
users from a variety of sources.\5 Our focus was on obtaining the
views of primary data users, defined as those who have conducted
analyses in the past 5 years rather than relied on information
already processed and interpreted by others. We limited our focus to
these users because we expected them to have a greater familiarity
with the strengths and limitations of each data collection system.
(The data collection and sampling design are detailed in appendix I.)
Our sample design cast a wide net with the intention of obtaining
information from a variety of users. However, because we aggregated
samples of different sizes from multiple lists, the survey results
cannot be used to characterize the average user in general or the
typical user in each of the organizational settings. Moreover, we
asked users to consider their experiences with individual data
systems only, rather than with the NNMRRP as a whole. Users first
identified which of the 14 data collection activities they had used
in the last 5 years and then focused on the two they used most
frequently.
We conducted our review between December 1993 and December 1994 in
accordance with generally accepted government auditing standards.
--------------------
\5 Out of a total of 1,585 potential data users sampled from the
lists, 1,180 (or 74.5 percent) responded; an industry group helped us
find another 10 respondents. Among all respondents, 93 provided
insufficient information and 123 pooled their responses with those of
another respondent. Among the rest of the respondents, 344 were not
users of nutrition monitoring data, 190 were secondary users (using
information that had already been analyzed), and 440 were primary
users.
PRINCIPAL FINDINGS
------------------------------------------------------------ Letter :4
DATA ARE USED IN A VARIETY
OF SETTINGS FOR MULTIPLE
PURPOSES
---------------------------------------------------------- Letter :4.1
The 440 primary users who responded to our survey worked in federal,
state, and local government; academic institutions; for-profit
businesses, such as food industries; and other settings, such as
hospitals. These users also represented a variety of occupations.
While data users in federal or academic settings were more likely to
identify themselves as engaged in basic research, those in state or
local settings were more likely to indicate program planning and
management as their primary occupation. (As noted above, these data
users who responded to our survey are not necessarily representative
of users in general or of the users in each organizational setting.)
Some use of almost every data collection system was reported in each
of the organizational settings. (The exception is the Navajo HNS,
which was reported as used by only a small number of respondents in
the federal government and "other" category.) While state and local
government respondents were more likely to use the state-based
surveillance systems, federal government and academic respondents
were more likely to report using the national surveys. (Appendix IV
presents more information on users, the data sets they use, and their
organizational setting.)
Across the different settings, the nutrition monitoring data
supported a variety of uses from identifying nutritional problems to
planning programs to address the problems, evaluating food- and
nutrition-related programs and policies, informing basic and
methodological research, and supporting state and local surveillance
activities. Table 2 provides specific examples of the decisions
respondents stated they made based on the data. (Tables showing the
percent of respondents indicating a specific purpose for which they
used a data collection system are provided in appendix II.)
Table 2
Respondents' Examples of Decisions Made
Based on the Data
Category Reported use
-------------------------------------- ----------------------------------------
Problem identification Determine prevalence of high blood
cholesterol in U.S. adults
Calculate exposure estimates for
regulatory issues involving food
additives
Assess damage from Exxon Valdez oil
spill
Policy-making or program planning Refocus on diabetes in minority
populations by the American Diabetes
Association
Support goals and activities for
improved nutrition status of population
in state cancer plan
Develop national guidelines for
screening and management of iron-
deficiency anemia
Confirm need for addition of calcium to
infant and toddler foods
Decide the size of target populations
for new pharmaceutical products
Place breast-feeding coordinators in
areas of greatest need
Policy or program management and Increase funding for Healthy Heart
evaluation Programs
Modify year 2000 objectives for blood
pressure to include Mexican Americans
Document the need for use of iron-
fortified infant formula and then
document the success of the policy
implementation
Calculate sales tax consequences of
cashing out food stamps
Conclude that children have too much fat
in their diets, but the excess is not
caused by participation in child feeding
programs
Research related to nutrition Implement a clinical trial to prevent
diabetes through diet modification
Plan study of unusually high anemia
levels in Alaskan Natives, which led to
new cause of iron-deficiency anemia
(bacterial)
Determine which foods to include on a
food-frequency questionnaire for Puerto
Rican elderly
Support of monitoring activities Choose knowledge and attitude indicators
by states and localities for a state survey because reference
values from national surveys are
available
Revise weighing and measuring policy to
increase accuracy in clinics
Use data in community needs assessment
for counties to develop plans for
services
--------------------------------------------------------------------------------
The primary users who responded to the survey also identified the
customers for their analyses. As shown in table 3, users in each
organizational setting identified a range of end users of the data.
In general, customers in their own organizational setting were most
commonly indicated, but the general public was also frequently
identified as a customer for the primary users' analyses of the
nutrition data.
Table 3
End Users of Most Frequently Used Data
Systems
Feder Academ Busine Other
End users al State Local ic ss \a
-------------------------- ----- ----- ----- ------ ------ -----
Federal government 79% 14% 0 17% 22% 18%
State government 38 81 32% 19 14 23
Local government 26 80 75 7 3 23
Universities 55 50 11 70 14 40
Hospital or health care 28 50 25 17 16 50
Researchers 68 34 14 74 38 48
For-profit business 27 10 11 14 68 20
Nonprofit, noncharitable 26 49 18 15 16 45
Charitable organization 13 24 18 8 5 18
Media 43 40 21 18 22 30
General public 50 53 43 30 30 53
Other\b 9 10 11 7 11 8
----------------------------------------------------------------------
\a Other settings include hospitals, nonprofit organizations, and
other charitable organizations.
\b Other end users include trade associations, labor groups, tribal
governments, community action agencies, medical practitioners,
minority groups, regulators, university students, and so on.
DESPITE GENERAL
SATISFACTION, USERS
SUGGEST CHANGES
---------------------------------------------------------- Letter :4.2
The majority of the primary users responding to our survey reported
that the data collection systems meet their information and data
quality needs to at least a moderate extent. However, despite this
and the evidence that the data are used for a variety of purposes, a
majority of respondents stated that changes are needed to increase
their confidence in or substantially increase their use of the data.
A somewhat higher proportion of users of USDA data systems than of
HHS systems indicated a need for change. (Users' satisfaction is
summarized in table IV.3.)
We asked primary users to identify what changes are needed in the
systems that they use most frequently. Common themes in their
comments were
-- continuous or more frequent data collection;
-- more detailed information on racial, ethnic, and age groups;
-- data that can support estimates for small geographic areas;
-- improved timeliness and documentation of the data; and
-- increased dissemination of the data in general and in formats
that facilitate access and analysis.
Specific comments from the users that illustrate these themes are
provided in table 4. More detailed summaries are in appendix III.
Table 4
Comments Illustrating Users' Suggestions
for Change
Data
Category Comment system
---------------------------- ---------------------------------------- --------
Frequency of data collection Consider sampling subsets of variables NHANES
in NHANES and NFCS more frequently and
conducting full survey occasionally--
i.e.,
every 5-10 years.
NFCS could be done every 5 years since N FCS
food availability changes
so much.
Coverage of racial, ethnic, Include very young and old and enough NHANES
and age groups minorities to make conclusions about the
different groups.
Expand sample size to provide sufficient BRFSS
number of minority respondents.
Need up-to-date analysis for all age NFCS
groups. Had to use the different
databases because one did not provide
all age groups.
Coverage of geographic areas More specific regional coverage would be NHANES
highly useful for assessing
the diffusion of dietary and other
health behaviors.
Geographic area coverage should be more CSFII
specific to allow analysis
and interpretation of data for
individual states.
It would be valuable to be able to BRFSS
provide county-level data for use by
local health departments.
Timeliness and documentation Would like faster turnaround from CDC to PedNSS
states for annual PedNSS tables.
If results could be published more NHANES
frequently, it would help us
see how well interventions are working.
Need more documentation, especially of CSFII
what was done in the
survey, how it was done, and how the
statistical analyses were done.
Dissemination Data can be made more accessible and NHANES
more timely using modern
technology and user-friendly systems.
Put on CD-ROM, include software that DHKS
facilitates use, establish bulletin
board with updates as new data become
available.
--------------------------------------------------------------------------------
CONCLUSIONS
------------------------------------------------------------ Letter :5
The NNMRRP data systems provide an important resource, serving a wide
set of purposes in a variety of settings. Moreover, the data users
are mostly satisfied with the quality of the data and the degree to
which their data needs are met. Despite this evidence of
satisfaction with the data systems, those who responded to our survey
had numerous suggestions for improving the data collection activities
of the NNMRRP. These suggestions are consistent with many of the
past criticisms of the nutrition monitoring system. Understanding
who uses the data and for what purposes is essential to developing
and implementing an effective nutrition monitoring system. Our study
begins this process and provides a useful framework of purposes for
nutrition monitoring data.
AGENCY COMMENTS
------------------------------------------------------------ Letter :6
We provided the Board and responsible agencies with summary survey
data so they could begin revising their data collection activities as
we continued with our analyses and prepared this report. A draft of
this report was then sent to USDA, HHS, and members of the National
Nutrition Monitoring Advisory Council for review and comment. USDA
and HHS provided written comments, which are included in appendixes
VI and VII.
In general, officials from these Departments agreed with our
principal findings and conclusions. USDA noted that our survey
results will be useful as they plan future monitoring activities, and
HHS indicated that our report provides a good overview of the user
survey. HHS officials, however, thought that our report did not
sufficiently describe all of the major federal uses of nutrition
monitoring data, and they provided further detail about these uses.
Both HHS and USDA also presented additional information about actions
taken that respond to concerns raised by survey respondents regarding
information and data quality needs. Technical comments made by HHS
and USDA officials and the members of the Advisory Council that
reviewed the report have been incorporated where appropriate.
As arranged with your office, we will be sending copies of this
report to the Director of the Office of Management and Budget, the
Interagency Board on Nutrition Monitoring and Related Research, the
agencies
responsible for data collection, and to other interested parties. We
will also make copies available to others upon request. If you have
any questions or would like additional information, please call me at
(202) 512-3092. Major contributors to this report are listed in
appendix VIII.
Sincerely yours,
Kwai-Cheung Chan
Director of Program Evaluation
in Physical Systems Areas
QUESTIONNAIRE DESIGN, SAMPLING
PLAN, AND ANALYSIS DECISIONS
=========================================================== Appendix I
QUESTIONNAIRE DESIGN
This section describes how we developed our questionnaire and
provides an overview of the questionnaire content.
SELECTING DATA COLLECTION
SYSTEMS
------------------------------------------------------- Appendix I:0.1
Our survey queried respondents about only 14 of the approximately 70
data collection activities listed in the Directory of Federal and
State Nutrition Monitoring Activities. (See table 1 on p. 3.) All
14 systems met our criteria of focusing on (1) dietary, nutritional,
and health status; (2) food consumption; or (3) dietary knowledge,
attitudes, and behavior. Information about the food composition
databases or activities for monitoring the food supply was not
gathered. This allowed us to concentrate on survey-based data
collection activities.
An additional criterion was that the data collection system be an
ongoing program. For example, periodic surveys like the Nationwide
Food Consumption Survey were included, while one-shot evaluations of
food assistance programs were excluded. We made three exceptions to
this criterion. We chose the NHIS-Vitamin and Mineral Supplements
and NHIS-Cancer Epidemiology and Cancer Control because of their
large size (nearly 11,800 and 45,000 interviews, respectively). We
also collected information on the Navajo Health and Nutrition Survey
because of the need for data on subpopulation groups expressed in
public comments to a draft of the 10-year comprehensive plan.
DEFINING USERS
------------------------------------------------------- Appendix I:0.2
Most of the questions in the survey were directed only to primary
users of the data from the 14 selected activities. We defined a
primary data user as one who directly accesses these data. This
includes those who request analyses from others as well as those who
access the data systems themselves. In contrast, secondary users are
those who use nutrition monitoring information that has already been
processed and interpreted by others in reports, articles,
publications, or other documents. We chose this definition to target
the questionnaire to respondents with firsthand experience with the
design and content of the data collection activities and the
strengths and limitations of the data.
STRUCTURING THE
QUESTIONNAIRE
------------------------------------------------------- Appendix I:0.3
We sent our survey to both known and potential users. Primary users
of the data could not be identified in advance, so in the first
section, we screened out secondary users and nonusers of the 14 data
collection systems. Then we asked the remaining respondents--the
primary data users--to identify the data collection system they used
most frequently and the next most frequently in the past 5 years.
A major portion of the survey was dedicated to determining how the
respondents used the data. Through literature reviews and a series
of expert panel meetings, we developed an inventory of the uses of
federal nutrition monitoring data. As shown in appendix II, specific
uses were categorized in five main areas: (1) problem
identification, (2) policy-making and program planning, (3) policy
and program evaluation and management, (4) research related to
nutrition, and (5) support of state and local nutrition monitoring
activities. Respondents also had the opportunity to record up to
five additional purposes for which they used the data. To ascertain
the validity of the uses they identified, we asked them to list at
least one report, article, or other document produced with the data.
We also obtained information on the extent to which the data
collection systems met the respondent's information and data quality
needs. We asked whether changes are needed to better meet their
needs for the data. Of those indicating a need for changes, we asked
for their suggestions on improving the (1) data elements collected,
(2) data collection methods, (3) units of analysis, (4) time of data
collection, (5) population group coverage, (6) geographic area
coverage, and (7) ease of use.
SAMPLING PLAN
This section details the sampling approach and provides information
on the sources from which we obtained names of people to survey.
OVERVIEW
------------------------------------------------------- Appendix I:0.4
We had no way of identifying all the users of the federal nutrition
monitoring data, so we chose a nonrandom sampling approach to
maximize the heterogeneity of the individuals surveyed. From a
variety of sources, we obtained lists of known and potential users of
the data and also of contacts in organizations likely to contain data
users.
We mailed out a total of 1,614 surveys. Addresses were incorrect for
29, so the sample size was reduced to 1,585. We received 1,180
responses, or 74.5 percent. An additional 10 responses came from a
confidential industry mailing list, increasing our total responses to
1,190. Of those, 344 were nonusers, 190 were secondary users, and
440 were primary users. In addition, 123 indicated that their
responses were included in with other respondents, and 93 did not
provide useful information because they were ill, retired, or failed
to complete the questionnaire. Primary users tended to respond
early, and nonusers tended to respond only after one or two
follow-ups.
IDENTIFYING USERS AND
POTENTIAL USERS
------------------------------------------------------- Appendix I:0.5
We selected users both as individuals likely to use nutrition
monitoring data and as members of organizations likely to contain one
or more data users. We asked the former to answer only for their own
uses and the latter to direct the survey to the most appropriate user
within their organization, who would also answer only for his or her
own use. We did not distinguish between individual and
organizational respondents in our analyses.
INDIVIDUALS
----------------------------------------------------- Appendix I:0.5.1
We identified actual and potential individual users of the 14
nutrition monitoring activities from a variety of sources, including
lists maintained by federal agencies of people who had requested
data, referrals of likely users from the Interagency Board and other
federal contacts, lists of people attending workshops and
conferences, and professional association membership lists. Tables
I.1-I.3 identify the sources of our lists and the number of people
surveyed.
Table I.1 provides the number individuals we surveyed, by data
collection activity, who requested data from the National Technical
Information Service (NTIS) or directly from the federal agencies
administering any of the 14 systems. The two largest groups were the
NHANES III and Cancer Risk Survey data requesters.
Table I.1
Data Requesters Included in Our Sample
Source Number
agency Data system surveyed
---------- ---------------------------------------------- ----------
USDA/HNIS CSFII 1989 7
CSFII 1990 1
NFCS 1987-88 7
NFCS (household) 5
NFCS (low income) 9
NFCS (household and individual intakes) 6
HHS/NCHS NHANES I 16
NHANES II 24
NHANES III 58
NHANES I Epidemiological Follow-up 13
NHANES I Follow-up Group Members 12
Hispanic HANES 7
NHANES (unspecified) 9
Cancer Risk Survey 42
Vitamin and Mineral Survey 5
HHS/FDA Health and Diet Survey 6
NTIS CSFII 10
Nutrient Data Base 8
NHANES I 2
NHANES II 9
NHANES I Epidemiological Follow-up 7
NHIS 10
NFCS 1977-78 6
NFCS 1987-88 10
----------------------------------------------------------------------
To ensure that we had full coverage of federal government users, we
asked the Interagency Board for the names of directors of agency
divisions mentioned in the 10-year comprehensive plan. Through other
referrals, we added the names of 26 potential users within those
agencies. From lists of attendees at three federally-sponsored,
nutrition-related workshops, we identified another set of known or
likely users that we surveyed. Members of associations for nutrition
professionals were another source of potential users we surveyed.
Finally, we obtained lists of local government officials working in
nutrition. (Table I.2 provides the source and application of these
additional potential users that we surveyed.)
Table I.2
Additional Sources of Nutrition Data
Users
Number
Source Affiliation surveyed
------------------ ---------------------------------------- --------
Likely federal Division directors (list provided by the 23
users Interagency Board)
Potential federal users identified 26
through referrals
Nutrition-related Dietary Consensus Conference 58
workshop
attendees
Food Insecurity Conference 61
USDA Household User Group 14
Association American Dietetic Association\a 203
members
American Institute of Nutrition\b 268
Society for Nutrition Education\c 107
Lists of local CityMatCH members (Urban Maternal and 144
government Child Health directors)
officials
National Association of County Health 27
Officials\d
----------------------------------------------------------------------
\a From more than 65,000 members, we identified a subgroup of 2,030
employed in education and research, and then we drew a 1-in-10
sample.
\b From nearly 3,000 members, we drew a 1-in-10 sample.
\c We drew a 1-in-3 sample of members working in higher education,
industry, public health, and county extension education.
\d From their National Directory of Local Health Departments, we
chose the directors from 30 counties containing 1 million or more
inhabitants, or 23.6 percent of the 1990 U.S. population (according
to the Census Bureau).
ORGANIZATIONS
----------------------------------------------------- Appendix I:0.5.2
To capture any state and local officials we may have missed, we
targeted organizations that were likely to have at least one or more
state and local nutrition monitoring data users. We asked
organizations to direct the survey to the most appropriate or
experienced officials, who would respond only for their own use of
the data, not for the organization as a whole. (See table I.3.)
To cover the fragmented groups in nutrition research and policy
analysis, we built in some redundancy within the sampling plan.
Respondents who received more than one survey, however, were counted
only once in our analyses.
Table I.3
Additional State and Local Nutrition
Data Users
Number
Affiliation Respondents surveyed
----------------------- ----------------------------------- --------
Association of State Child health and nutrition 45
and Territorial Public officials
Health Officials
Chronic disease officials 42
Health education officials 53
Maternal and Child Maternal and child health directors 52
Health Association
Center on Budget and WIC directors 44
Policy Priorities
State Government Yellow Health department directors 52
Book
Surveillance system BRFSS 47
contacts
PedNSS 39
PNSS 18
----------------------------------------------------------------------
LIMITATIONS OF THE SAMPLING
PLAN
------------------------------------------------------- Appendix I:0.6
While our approach allowed us to cast a wide net and contact as many
data users as possible, it also has some limitations. The major one
is our inability to determine the degree to which the survey
respondents are representative of primary data users in general.
With a nonprobability sample, we cannot generalize beyond our
respondents to the universe of all users of the 14 data collection
activities the questionnaire addresses. In addition, we cannot make
any inferences about the extent of use across groups. Our ability to
identify primary users within groups varied, so differences in
reported use may be a function of our sample design rather than of
actual differences in use. For example, we were able to target
federal users of the data more closely than users in other sectors,
but it would be inappropriate to compare the extent of their use to
that by other groups of respondents.
A further limitation of our survey design is that we asked
respondents to comment on individual data collection activities (such
as DHKS) and not on the specific survey components within each
activity (such as the 1989, 1990, 1991, or 1992 DHKS). Some users
who are familiar with only a specific survey component and not all
the survey components may have made suggestions to us for changes
that have already been addressed by the agencies in later versions of
the data collection activity.
ANALYSIS DECISIONS
CONSTRUCTION OF ANALYSIS
GROUPS
------------------------------------------------------- Appendix I:0.7
Our six groups of primary users--federal, state, local, academic,
business (for-profit), and other--were constructed according to
respondents' self-reports. Healthcare (hospital, nursing home) was
the dominant group within the "other" category, which also included
nonprofit businesses and charitable organizations. Overall, the
groups were diverse, and no single subgroup dominated any group.
The federal group in our sample consisted of at least 23 different
agencies. Those with the largest number of respondents were the
National Center for Health Statistics (17) and the Human Nutrition
Information Service (11). They are responsible for the main data
systems, and they provided an insider's view to the strengths and
weaknesses of these systems. The state group included 45 states that
responded and the District of Columbia and Guam. The largest number
of respondents from one state was eight, or 6 percent, so no one
state had a large influence on the whole group. The local group had
one user each from 28 different counties.
The academic group came from 67 different cities, and some cities,
such as Boston and Chicago, were represented by more than one school.
The largest number of respondents from one school was five, or 5
percent. Five of the 37 in the business group did not give their
addresses; the others came from 26 different cities, and the largest
number from one city was three. The 40 respondents in the "other"
group came from 33 cities, and the largest number from one city was
three.
USES OF NUTRITION MONITORING DATA
========================================================== Appendix II
The data collected by the nutrition monitoring systems are not only
used across a variety of organizational settings, but they also
support a range of uses. With the assistance of our expert
consultants, we developed an inventory of the purposes that federal
nutrition monitoring data serve. Specific purposes were categorized
in five overarching areas: (1) problem identification, (2)
policy-making and program planning, (3) policy and program evaluation
and management, (4) research related to nutrition, and (5) support of
state and local nutrition monitoring activities.
For the two data collection activities used most frequently (see
table II.1), respondents to our survey were asked to indicate the
purposes the data had served. The data users were also asked to
write in specific decisions that the data had supported. Although
our respondents presented a variety of purposes and decisions, one
noted that the data were not timely enough, or sufficiently on
target, to truly inform decisions. We classified the written
comments into the five major categories of purposes.
Table II.1
Two Most Frequently Used Data Collection
Activities
Numbe Perce Numbe Perce Numbe Perce
Data collection activity r nt r nt r nt
---------------------------- ----- ----- ----- ----- ----- -----
NHEFS 34 8 20 6 54 12
NHANES II 46 10 55 16 101 23
NHANES III 54 12 36 11 90 20
HHANES 11 2 17 5 28 6
NIHS-Vitamin 2 1 1 0 3 1
NHIS-Cancer 9 2 4 1 13 3
BRFSS 64 15 21 6 85 19
PNSS 29 7 44 13 73 17
PedNSS 68 15 28 8 96 22
Health and Diet 8 2 5 1 13 3
NFCS 69 16 41 12 110 25
CSFII 43 10 46 14 89 20
DHKS 3 1 20 6 23 5
======================================================================
Total 440 101\a 338\b 99\a 440\c
----------------------------------------------------------------------
\a Because of rounding, total percentages do not add up to 100.
\b Some respondents had used only one data system; thus, the number
of the second most frequently used system was less than 440.
\c Total sample; not column total.
PROBLEM IDENTIFICATION
The expert panels distinguished seven kinds of problems that the data
might be used to identify. (See table II.2.) As shown in the table,
problem identification was a commonly indicated purpose supported by
the data. However, there is some variation in the kinds of problems
examined by the different data sets. For example, NHANES data--with
its emphasis on health--is used to examine chronic degenerative
diseases, as well as deviations in nutritional status.
Table II.2
Respondents' Indication of Use of Data
for Problem Identification
Data collection activity 1 2 3 4 5 6 7
---------------------------- ---- ---- ---- ---- ---- ---- ----
NHEFS 70% 48% 6% 52% 6% 4% 14%
NHANES II 49 45 11 55 9 2 10
NHANES III 55 47 21 57 6 5 14
HHANES 65 42 4 54 4 0 8
NHIS-Cancer 31 39 0 39 8 0 8
BRFSS 55 41 1 74 1 0 13
PNSS 14 86 15 80 0 1 6
PedNSS 17 88 16 80 0 1 5
Health and Diet 39 39 8 62 8 23 15
NFCS 29 44 22 56 18 11 43
CSFII 30 44 23 61 21 14 42
DHKS 36 27 23 77 32 23 27
Median percent 38 44 13 59 7 3 14
----------------------------------------------------------------------
\a For each of the two nutrition monitoring data collection
activities that they used most frequently, respondents were asked to
indicate if they had used the data to identify or estimate the risk,
incidence, prevalence, duration, or cost of any of the following
problems:
1. Chronic degenerative diseases and their relationship to diet and
nutritional status;
2. Nutritional deficiency diseases and health-related issues;
3. Hunger and food insecurity, including its relationship to diet and
its periodicity;
4. Deviations in nutritional status (e.g., obesity) and diet quality
across the life-cycle and across population groups;
5. Food safety problems over which consumers have little control
(e.g., contaminants);
6. Food safety problems over which consumers have some control (e.g.,
microbiological problems resulting from food preparation, handling,
or consumption activities); and
7. Other food quality problems (availability, accessibility, and
composition).
POLICY-MAKING OR PROGRAM PLANNING
Within the overarching objective of policy-making or program
planning, the expert panel identified seven purposes. (See table
II.3). As shown in the table, most of the program purposes reported
were fairly common uses of nutrition monitoring data sets. One
exception was "to compare cost-effectiveness" of different kinds of
policy or program interventions, which is not surprising since the
nutrition data do not provide this information. Many of the uses
that respondents wrote in response to the request for four specific
decisions informed by the data appeared to fit under policy-making or
program planning.
Table II.3
Respondents' Indication of Use of Data
for Policy-making or Program Planning
Data collection activity 1 2 3 4 5 6 7
---------------------------- ---- ---- ---- ---- ---- ---- ----
NHEFS 51% 30% 42% 20% 2% 19% 42%
NHANES II 58 36 44 28 4 23 43
NHANES III 64 52 55 31 8 27 61
HHANES 58 46 46 36 4 27 54
NHIS-Cancer 46 31 31 31 0 15 77
BRFSS 72 58 47 53 4 46 78
PNSS 85 70 65 85 7 60 89
PedNSS 86 69 66 80 11 60 89
Health and Diet 77 54 54 46 8 39 54
NFCS 66 44 52 38 8 22 48
CSFII 76 51 57 40 6 23 51
DHKS 74 48 73 52 0 23 57
Median percent 69 50 53 39 5 25 56
----------------------------------------------------------------------
\a For each of the two nutrition monitoring data collection
activities that they used most frequently, respondents were asked to
write in four specific decisions informed by the data, many of which
fit within the following purposes:
1. Define and quantify the extent and distribution of a food- or
nutrition-related problem or the risk of the problem;
2. Assess the importance of a problem or risk of the problem relative
to other problems;
3. Identify determinants of a food- or nutrition-related problem or
risk of the problem;
4. Identify policy and programmatic responses to the problem or risk
of the problem;
5. Compare cost-effectiveness of responses;
6. Justify the selection of a response (prevention, intervention to
mitigate, or intervention to deal with the consequences); and
7. Serve as a basis for targeting prevention or intervention
resources or both.
PROGRAM EVALUATION AND MANAGEMENT
The inventory of purposes served by nutrition monitoring data
included nine program evaluation and management purposes. (See table
II.4.) As shown in the table, two of the more commonly indicated
purposes in this area were measuring changes in deficiency diseases
and assessing achievement of specific dietary objectives. For
example, several of the uses described by the survey respondents
focused on monitoring or modifying Healthy People 2000 objectives.
In contrast, measuring changes in food safety problems was one of the
least common purposes indicated in the survey.
Table II.4
Respondents' Indication of Use of Data
for Program Evaluation or Management
Data collection activity 1 2 3 4 5 6 7 8 9
------------------------- --- --- --- --- --- --- --- --- ---
NHEFS 47% 8% 31% 35% 8% 33% 12% 15% 8%
NHANES II 37 8 23 43 7 46 14 22 13
NHANES III 46 13 33 42 7 50 13 28 18
HHANES 39 8 23 27 0 31 8 23 12
NHIS-Cancer 25 8 8 33 0 42 17 33 8
BRFSS 44 1 26 46 4 77 1 41 38
PNSS 16 7 63 66 1 78 3 40 64
PedNSS 20 7 75 73 0 79 3 30 63
Health and Diet 33 0 46 39 8 77 25 69 46
NFCS 19 17 20 50 13 51 28 42 27
CSFII 19 17 19 55 13 57 26 48 25
DHKS 36 18 14 46 10 57 23 57 30
Median percent 35 8 24 45 7 54 14 37 26
----------------------------------------------------------------------
\a For each of the two nutrition monitoring data collection
activities that they used most frequently, respondents were asked if
they had used the data for the following:
1. Measure changes in chronic degenerative diseases, their
relationship to diet and nutritional status, and the risk of such
diseases;
2. Measure changes in hunger and food insecurity;
3. Measure changes in deficiency diseases and health-related issues;
4. Measure changes in deviation in nutritional status and diet
quality;
5. Measure changes in food safety problems;
6. Measure achievement of specific dietary objectives;
7. Evaluate food supply and nutrient supplements and fortificants;
8. Measure changes in food- and nutrition-related behaviors and their
precursors and determinants; and
9. Assess targeting and coverage of food- and nutrition-related
programs.
RESEARCH RELATED TO NUTRITION
The expert panelists included seven research purposes for nutrition
monitoring data in the inventory. (See table II.5.) Across the
different data sets, respondents commonly indicated that the data
were used to increase basic research knowledge of the determinants of
problems and options for intervention.
Table II.5
Respondents' Indication of Use of Data
for Research
Data collection activity 1 2 3 4 5 6 7
---------------------------- ---- ---- ---- ---- ---- ---- ----
NHEFS 27% 14% 54% 44% 35% 19% 10%
NHANES II 29 26 50 48 33 22 9
NHANES III 40 31 53 45 36 21 7
HHANES 54 44 62 50 42 39 8
NHIS-Cancer 39 23 54 23 23 8 0
BRFSS 33 44 40 33 18 8 5
PNSS 41 62 52 40 33 13 3
PedNSS 35 51 44 35 32 13 4
Health and Diet 15 15 39 39 23 8 0
NFCS 28 23 43 44 27 36 23
CSFII 26 30 51 49 40 41 22
DHKS 27 18 64 68 36 46 19
Median percent 31 28 52 44 33 20 8
----------------------------------------------------------------------
\a For each of the two nutrition monitoring data collection
activities that they used most frequently, respondents were asked to
indicate if they had used the data for the following:
1. Improve sampling and statistical methods for gathering data from
people at different life-cycle stages or for minority or other
subpopulations, especially those at risk of food- or
nutrition-related problems;
2. Improve methods for informing decisionmakers so analysis results
are timely, pertinent, and understandable;
3. Increase basic research knowledge of the determinants of problems
and options for intervention;
4. Increase basic research knowledge of the relationships between
food, nutrition, and health;
5. Identify and stimulate needed research and development on
monitoring methods;
6. Conduct other kinds of basic research (e.g., on the distribution
of human nutrient requirements, databases on food cost and food
preparation, individual variability); and
7. Conduct food composition research and improve food composition
databases.
A few data users conduct food composition research; however, the food
composition databases were not included as a focus of the survey.
Specific research uses supported by the data included identifying
foods for food-frequency questionnaires, establishing cut-points for
defining research subjects, and developing survey instruments.
SUPPORT OF MONITORING ACTIVITIES
BY STATES AND LOCALITIES
Under the general goal of supporting the monitoring activities of
states and localities, the expert panelists identified two specific
purposes for the nutrition monitoring data: supporting state and
local surveillance activities and supporting technical assistance.
At least some portion of the users of each of the data collection
activities identified one of these purposes as a way in which they
use the data. (See table II.6.) Of the two, the latter was more
commonly indicated. This pattern also appears in the respondents'
comments, many of which focus on identifying the need for technical
assistance and improving the quality of data collection. (See
appendix III.)
Table II.6
Respondents' Indication of Use of Data
for Activities by States and Localities
Data collection activity 1 2
---------------------------------------- ------------- -------------
NHEFS 10% 10%
NHANES II 5 13
NHANES III 12 26
HHANES 8 19
NHIS-Cancer 8 15
BRFSS 12 54
PNSS 22 63
PedNSS 19 62
Health and Diet 15 15
NFCS 11 16
CSFII 8 16
DHKS 14 23
Median percent 12 18
----------------------------------------------------------------------
\a For each of the two nutrition monitoring data collection
activities that they used most frequently, respondents were asked to
indicate if they had used the data for the following:
1. Support state and local surveillance of and responses to food- and
nutrition-related crises; and
2. Support development and provision to states and localities of
technical assistance in data collection, analysis, and
interpretation.
SUMMARIES OF THE SUGGESTIONS MADE
BY USERS
========================================================= Appendix III
SYSTEMS UNDER NCCDPHP
The National Center for Chronic Disease Prevention and Health
Promotion administers three surveillance systems that collect
information on health and nutritional status: the Pediatric
Nutrition Surveillance System, the Pregnancy Nutrition Surveillance
System, and the Behavioral Risk Factor Surveillance System. The
three systems are overseen by the National Center, but are actually
implemented by the states that participate in the surveillance
programs. Table III.1 describes each system's target population and
data collection methods.
Table III.1
CDC's NCCDPHP Surveillance Systems
Type and source of
System Target population sample Data collection method
------ ---------------------- ---------------------- ------------------------
PedNSS Low-income, high-risk Participants in Clinic staff record data
children publicly-funded at checkups (body
prenatal nutrition and measurements, blood test
food assistance results, and demographic
programs data)
PNSS Low-income, high-risk Participants in Clinic staff record
pregnant women publicly-funded health status, blood
prenatal nutrition and test results, risk
food assistance behaviors, and
programs demographic data
BRFSS Adults, age 18 and Random telephoning of Telephone interviews
over households (body measurements, risk
behaviors, food choices)
--------------------------------------------------------------------------------
These surveillance systems vary in their purposes, methods of data
collection, and types of respondents, yet we found common themes in
the recommendations made by primary users of the systems. For all
three systems, users suggested providing
-- more data on dietary intake,
-- better controls on the quality of the data collected,
-- more detail on subpopulation groups in the reporting of the
data,
-- increased ability to look at substate geographic divisions,
-- improved timeliness of CDC's return of the data,
-- simplified reports that are more readily used at the local
level, and
-- additional technical and financial assistance in data collection
and interpretation.
User recommendations specific to each of the systems are presented in
the tables below.
COMMENTS ON PEDNSS AND PNSS
----------------------------------------------------- Appendix III:0.1
Because PedNSS and PNSS collect data on several similar issues, they
are listed together in table III.2. In addition to these comments,
some respondents complimented CDC on the quality of PedNSS,
specifically for the automated system and for its coordination of the
data collection with the WIC Program.
Table III.2
User Suggestions for Improving PedNSS
and PNSS
Type of change Comment
---------------------------- --------------------------------------------------
Data elements Collect more data on
Infant feeding practices, particularly breast-
feeding or type of formula (and provide
better analysis)
Pregnancy risk information (PNSS)
Dietary intake (food frequency, 7-day records,
"usual" intake)
Food security and hunger (PedNSS)
Demographics
Other indicators--blood lead levels, serum
cholesterol, immunizations, height and
weight at 2 and 3 years, household smoking
(PedNSS) and physical activity and
risk behaviors (PNSS)
Data collection methods Improve data collection quality control (training,
uniform reporting, better software, and
standardization of measurements)
Use other sources of information (vital records,
private physicians)
Maintain cultural sensitivity
Streamline and simplify questions
Stop changing the data requirements (PNSS only)
Use more biochemical measures
Develop methods to obtain data from more sources
than public clinics (e.g., scannable forms that
private physicians could complete)
Units of analysis Maintain records by individual child, not by
clinic visit (PedNSS only) to avoid duplication of
records
Time of data collection Facilitate analyses of changes over time by
linking all records to the individual child
(PedNSS only)
Population group coverage Expand beyond participants in WIC and other
publicly-funded programs to include non-low-
income women and children
Collect and report more data by subgroup (race,
ethnicity, age, sex, income)
Geographic area coverage Enable reporting by substate divisions
Improve national estimates by including all states
(currently, states choose whether they
participate)
Ease of use Provide
Improved timeliness of reporting
Simplified report format and content
Reports that are more accessible for local users
Improved flexibility of the PedNSS automated
system and exportability of the data
An automated system for PNSS that is similar to
PedNSS
Technical training and funding assistance to
states to implement systems
--------------------------------------------------------------------------------
COMMENTS ON BRFSS DATA
----------------------------------------------------- Appendix III:0.2
User recommendations for BRFSS are listed below in table III.3.
Table III.3
User Suggestions for Improving BRFSS
Type of change Comment
---------------------- --------------------------------------------------------
Data elements Collect more data on
Dietary intake (in general and to link to behavior)
Specific dietary elements (fat, food groups, fiber,
nutritional supplements, alcohol, ethnic foods)
Households (number of adult smokers)
Ethnicity (state-specific)
Improve
Questions on dietary fat to obtain a better measure
Correspondence of health indicators with health
objectives (percent of fat, salt intake, label reading)
Add questions on cholesterol, diabetes, and disease risk
Make questions more culturally sensitive and relevant
Data collection Address data validity and other quality control issues
methods (translation for non-English-speakers, applicability to
adults in households without telephones, nonuniformity
across states)
Develop a method to gather more complete dietary data
Units of analysis Maintain records by individual, with more data on the
individual's household
Time of data Collect nutrition data
collection
Continually or at least every 2 years
At times most representative of year-round habits (make
seasonal adjustments)
Population group Provide more detail on
coverage
Subpopulations in general (increase sample)
Racial and ethnic groups (and those specific to a state)
Specific age groups
High-risk populations
Include populations without telephones
Geographic area Improve national estimates by including all states
coverage
Increase sample sizes within states for better estimates
Provide information for substate divisions (counties,
cities, rural areas), which will assist in planning and
evaluating community interventions
Ease of use Provide
Improved timeliness of data (not only for state and
local users, but also for researchers, who must obtain
permission from each state)
An automated system for state analyses of data
Improved documentation
Facilitated access to the data for nonstate users
Technical assistance in data interpretation (especially
dietary fat data)
--------------------------------------------------------------------------------
SYSTEMS UNDER NCHS
Users of four of the data collection activities run by the National
Center for Health Statistics commented on these systems in our
survey: the National Health and Nutrition Examination Survey, the
NHANES I Epidemiological Follow-up Study, the Hispanic Health and
Nutrition Examination Survey, and the National Health Interview
Survey on Cancer Epidemiology and Cancer Control. Table III.4
provides some summary information about these surveys.\1
Table III.4
CDC's NCHS Data Collection Activities
Activity Target population Type and source of sample Data collection method
---------------- -------------------------- -------------------------- --------------------------
NHANES Civilian, Stratified, multistage, In-person interviews,
noninstitutionalized probability cluster sample including a single 24-
population age 2 months of households; hour recall and physical
and older oversampling of children, examinations
elderly, African-
Americans, and Mexican-
Americans
NHEFS All persons between 25 and Same as for NHANES, with In-person interviews,
74 years old who completed tracing of age group of physical measurements,
a medical examination at interest review of hospital and
NHANES I in 1971-75 other records
HHANES Civilian, Stratified, multistage, In-person interviews and
noninstitutionalized probability cluster sample physical examinations
Hispanics (Mexican- of the target populations
Americans, Cubans, Puerto
Ricans) age 6 months-74
years residing in
households in three
defined U.S. geographic
areas
NHIS-Cancer Civilian, Stratified, multistage, In-person interviews
noninstitutionalized U.S. cluster sample, including
population age 18 years one randomly selected
and older person 18 years or older
in each NHIS household;
oversampling of Hispanics
and African-Americans in
the last implementation
(1990)
----------------------------------------------------------------------------------------------------
Our survey asked respondents to identify themselves as users of
NHANES II, which was conducted from 1976 to 1980, or NHANES III,
which started in 1988 and was completed in 1994. (NHANES I was
conducted between 1971 and 1975.) Because the surveys are very
similar in their design, no distinction is made between suggestions
made by users of NHANES II and those made by users of NHANES III in
the discussion below. HHANES differs from NHANES in its focus on
three Hispanic subpopulations, but is otherwise similar in
methodology. The data collected by NHEFS, unlike that for NHANES and
HHANES, allow for the study of changes over time through follow-up
surveys (in 1982-84, 1986, 1987, and 1992) with all persons between
25 and 74 years of age who had completed a medical examination for
NHANES I. All three use both food-frequency questions and 24-hour
recall to collect dietary intake data.
In contrast, NHIS-Cancer relies solely on food-frequency questions.
Even though it is not one of the major nutritional data collection
systems, NHIS-Cancer was included in our survey because it measured
nutrition variables such as frequency of eating selected food items,
vitamin and mineral supplement intake, and knowledge of the
relationship between diet and cancer.
Although the target populations and methods used for the NCHS data
collection activities vary, some common themes emerged in the
analysis of the comments from the users of the different systems.
Users suggested providing
-- more information on health habits and outcomes;
-- more detailed data on food consumption;
-- improved dietary intake methods, whether food-frequency
questions or 24-hour recalls;
-- a focus on the individual unit of analysis, with information
linking the individual to the family or household unit;
-- continuous or more frequent data collection;
-- more detailed information on racial, ethnic, and age groups;
-- data that can support estimates for smaller geographic areas;
-- improved timeliness and documentation of the data; and
-- increased dissemination of the data in general and in formats
that facilitate access and analysis.
The specific comments made under these general themes and on other
subjects are detailed in table III.5.
Table III.5
User Suggestions for Improving NCHS Data
Collection Systems
Type of change Comment
---------------------- --------------------------------------------------------
Data elements Collect more information on
Health-related habits (physical activity, smoking,
alcohol use)
Medical history
Health outcomes in general (arthritis, skin diseases,
food allergies, cancer, and for elderly, hearing loss)
Cause of death (NHIS-Cancer)
Dietary intake
Demographics (occupation as a source of nondietary
exposure to cancer)
Environmental risk factors
Nonrespondents
Data collection Conduct more research on data collection methods, in
methods particular
Measuring for race, ethnicity, and age (minorities,
adolescents)
Validating portion size (absolute amounts and percent of
calories
Using biochemical analyses (larger samples)
Improve automation and processing
Standardize techniques
Use multiple measures (telephone and in-person
interviews and mailed questionnaires)
Expand the use of food-frequency questions
Include Hispanic foods and newer versions of common
foods
Translate questionnaires for non-English-speaking
persons and use fully bilingual interviewers
Obtain dietary data on more than a single day (multiple
24-hour data or 3-day records)
Units of analysis Retain individual as most important unit for nutrition
issues (NHANES, HHANES, NHEFS)
Link individual data to family or household unit (NHIS-
Cancer)
Account for non-Hispanics in household (HHANES)
Time of data Continuously collect nationally representative NHANES
collection data while collecting subpopulation data\a
Shorten NHANES to conduct more frequently if not
continuously (NHIS is a model for continuous
collection.)
Shorten cycles of surveys to produce more frequent
updates
Conduct more methods research and data analysis between
surveys
Increase frequency for nutritionally vulnerable groups
Conduct longitudinal follow-up on chronic diseases
Account for seasonality
Population group Provide
coverage
Better and more coverage of racial, ethnic, and age
groups
Clearer criteria on definition of race
Comparable age-sex groups for racial and ethnic groups
Generalizability (HHANES)
Geographic area Provide
coverage
More specific regional coverage
More refinement of geographic detail (rural; urban;
standard metropolitan statistical areas; Alaska, Hawaii,
Puerto Rico, and Indian reservations)
Use small-area estimation models
Provide state-level estimates
Ease of use Provide
More timely release of data
More and better documentation of complex sample designs
On-line documentation Improved advertising of
availability of different data
Training in using complex sample designs
Data in common statistical package format
Occupation data coded for risk categories
Anthropometric data using 15th and 85th percentiles as
well as 25th and 75th
--------------------------------------------------------------------------------
\a One of the criticisms of HHANES is that the data were collected at
a different time from the NHANES data, and thus, the health and
nutritional status of the Hispanic groups cannot be compared to that
of the nation as a whole.
SYSTEMS UNDER USDA
Our survey asked users of three USDA data collection activities to
comment on changes to the surveys that would increase their use of
the data. The three USDA surveys addressed are the Nationwide Food
Consumption Survey (now called the Household Food Consumption
Survey), the Continuing Survey of Food Intakes by Individuals, and
the Diet and Health Knowledge Survey. Table III.6 describes each
activity's target population and data collection methods.
Table III.6
USDA's Data Collection Activities
Data
collection
Activity Target population Type and source of sample method
---------- -------------------------- -------------------------- ------------
NFCS Households in the 48 Stratified, multistage, Personal
contiguous states and area probability sample interview
individuals residing in with oversampling for low- with the
those households income households household
food
manager,
including a
7-day record
of household
food use;
personal
interview
with
household
members on
dietary
intake,
including 3
consecutive
days of
dietary
intake data
collected
with one 24-
hour recall
and a 2-day
record
CSFII Individuals in the 48 Stratified, multistage, Personal
contiguous states area probability sample interviews
with oversampling for with
individuals in low-income household
households members on
dietary
intake,
including 3
consecutive
days of
dietary
intake data
collected
with one 24-
hour recall
and a 2-day
record
DHKS Main meal planner or Same as CSFII Computer-
preparer in households assisted
that participated in CSFII telephone
interviews
(supplemente
d with in-
person
interviews
for
respondents
without
telephones)
--------------------------------------------------------------------------------
The focus of NFCS is on household use of food, including food costs,
food preparation, and food consumption. NFCS data are intended to
inform policies related to food production and marketing, food
safety, food assistance, and nutrition education. CSFII is intended
to complement NFCS in two ways. First, it provides a more frequent
source of information than the decennial NFCS, and second, it focuses
on individual, rather than household, food consumption. DHKS, a
follow-up to CSFII, is intended to support analyses of the
relationship between dietary intake and knowledge and attitudes about
dietary guidance and food safety.
Although NFCS and CSFII vary in their target populations and
purposes, they are similar in their sampling approach (national with
oversampling for low-income population) and the focus on food
consumption. Their similarities are reflected in the common themes
in the recommendations made by the primary users of the two data
collection systems. DHKS users had somewhat different concerns about
data elements and data collection methods, but their comments were
otherwise consistent with remarks made about the other two systems.
The major themes in the comments about the data systems were to
provide
-- more specificity and detail about foods and better data on food
composition;
-- improved questions on dietary behavior;
-- more information about health and demographic variables;
-- reduced respondent burden and improved response rates;
-- higher quality dietary recall data in general and, specifically,
more nonconsecutive days of 24-hour recall;
-- individual data and information on the individual's household;
-- continuous or more frequent collection;
-- longitudinal component;
-- increased sample size and broadened coverage;
-- more detail on racial, ethnic, age, and income groups;
-- refined geographic area coverage, specifically state and
substate data;
-- more rapid release of the data;
-- improved documentation; and
-- dissemination of the data in alternative forms (for example,
CD-ROM, formatted for use with statistical packages).
Table III.7 provides more user suggestions and other issues from our
survey on USDA systems.
Table III.7
User Suggestions for Improving USDA Data
Collection Systems
Type of change Comment
---------------------- --------------------------------------------------------
Data elements Collect more data on
Food eaten away from home
Food shopping access, prices, and behavior; food
preparation methods and facilities; food
storage; and safety
Use of salt, condiments, nutritional supplements,
specific foods (dairy, caffeine, water, fruits,
processed, seafood)
Improve
Quality and completeness of food composition data (newer
products, brand names, reliability of data)
Questions to assist linking diet and behavior (nutrition
knowledge and opinion, exercise, barriers and motivation
to change, participation in food programs)
Health data (by measuring rather than self-reporting
height, weight, health status)
Bring questions in line with current theory (DHKS);
standardize questions from year to year
Data collection Streamline the instrument to reduce burden on
methods respondents
Use automation to improve response rates (also
telephones, home bar scanners)
Collect more days of recall data and more nonconsecutive
days
Use two periods of household records (shorten the 7
days) to measure better the usual food use
Ensure questionnaires are answered completely
Use multiple measures (telephone and in-person
interviews and mailed questionnaires)
Units of analysis Focus on individual data (NFCS)
Focus on household data and individuals (CSFII)
Time of data Need continuous survey or at least collect data more
collection frequently
Collect NFCS data every 5 years and CSFII data in the
interim
Collect longitudinal data to track changes in individual
consumption (NFCS and CSFII)
Population group Increase coverage of subpopulations and racial, ethnic,
coverage and age groups
Increase sample size
Need clearer criteria for definition of race
Focus on high-risk groups
Integrate CSFII with NHANES sample
Geographic area Need
coverage
More refinement of geographic detail (regions,
localities, areas of low density)
State-level estimates (allow states to collect their own
data and feed into national survey)
Use small-area estimation models
Provide specific estimates for major population centers
Ease of use Collect data more frequently and allow more rapid access
to both published reports and raw data
Provide
User-friendly documentation
More detailed data on sampling design variables
Clear documentation on data tape and file format to
facilitate combining record types
Documentation on changes in format in food composition
database, codebook, and recipe file
Survey protocol and operations manual
More technical assistance (to nonnutritional
researchers)
On-line documentation
Data in common statistical package format
Lists of surveys and sources for both data and technical
assistance in professional journals
--------------------------------------------------------------------------------
--------------------
\1 Our survey also asked respondents if they had used and had
comments on a fifth NCHS survey--the National Health Interview Survey
on Vitamin and Mineral Supplements. Only three respondents
identified themselves as primary users of NHIS-Vitamin, and none of
these had comments.
CHARACTERISTICS OF THE USES OF
DATA COLLECTION ACTIVITIES
========================================================== Appendix IV
The tables in this appendix are based on the 440 responses from those
who have used at least one of the 14 data systems in the past 5
years. Since some respondents pooled their answers, each of these
440 responses may represent one or more than one user. The first two
tables describe some characteristics of the users in our sample.
Table IV.1 shows the occupations that users identified themselves
with, by organizational setting. To construct table IV.2, we asked
them what data collection activites they have used at all in the past
5 years.
Table IV.1
Main Occupation of Respondents\a
Federal State Local Academic Business Other\b Average
------ -------- --------- --------- --------- --------- -------- --------
Sample 112 125 28 98 37 40
size
Occupation
--------------------------------------------------------------------------------
Servic 6% 14% 32% 16% 14% 33% 15%
e
deliv
ery
Basic 47 6 0 57 35 28 32
resea
rch
Applie 29 10 7 24 49 13 21
d
resea
rch
Progra 13 66 71 1 5 15 29
m
manag
ement
and
plann
ing
Other 7 6 0 2 16 3 5
--------------------------------------------------------------------------------
\a Column percentage totals exceed 100 percent because some users
identified more than one main occupation.
\b Other settings include hospitals, nonprofit organizations, and
other charitable organizations.
Table IV.2
Respondents' Use of Data Systems Within
the Past 5 Years\a
Federal State Local Academic Business Other\b Average
--- --------- --------- --------- --------- --------- --------- ---------
Sam 112 125 28 98 37 40
ple
si
ze
Data system
--------------------------------------------------------------------------------
NHE 31% 14% 36% 39% 30% 45% 30%
FS
NHA 50 17 36 41 59 58 39
NES
II
NHA 57 17 39 29 30 48 35
NES
II
I
HHA 38 10 11 21 24 23 22
NES
NHI 9 8 7 10 3 23 10
S-
Vi
ta
mi
n
NHI 11 5 14 16 14 20 12
S-
Ca
nc
er
BRF 13 71 39 15 11 33 33
SS
PNS 13 47 50 6 3 33 25
S
Ped 14 58 68 9 5 30 30
NSS
Nav 2 0 0 0 0 5 1
ajo
HN
S
Hea 13 7 11 10 19 23 12
lth
an
d
Di
et
NFC 44 20 36 67 65 53 44
S
CSF 46 10 7 55 46 40 35
II
DHK 28 10 7 23 27 30 20
S
--------------------------------------------------------------------------------
\a Column percentage totals exceed 100 percent because most users
checked two data systems.
\b Other includes hospitals, nonprofit organizations, and other
charitable organizations.
Tables IV.3-IV.5 show three aspects of respondents' satisfaction with
each data collection activity: first, whether it provided for their
information needs; second, whether it met their data quality needs;
and third, whether they thought changes were needed to either
increase their confidence in or substantially increase their use of
the data system.
Table IV.3
Users' Response to How Well the Current
Data Meet Their Information Needs
Data
collecti
on Little or Sample
activity none Some Moderate Great Very great size
-------- ---------- ---------- ---------- ---------- ---------- ----------
NHEFS 2% 10% 45% 33% 10% 54
NHANES 3 14 37 31 15 101
II
NHANES 2 16 34 28 19 90
III
HHANES 0 7 37 33 22 28
NHIS- 0 23 46 15 15 13
Cancer
BRFSS 4 30 42 19 6 85
PNSS 4 23 27 41 6 73
PedNSS 6 21 31 35 6 96
Health 8 8 33 33 17 13
and
Diet
NFCS 3 15 34 38 10 110
CSFII 6 13 37 36 8 89
DHKS 13 4 52 22 9 23
Median 4 15 37 33 10
--------------------------------------------------------------------------------
Table IV.4
Users' Response to How Well the Current
Data Meet Quality Needs
Data
collecti
on Little or Sample
activity none Some Moderate Great Very great size
-------- ---------- ---------- ---------- ---------- ---------- ----------
NHEFS 4% 18% 36% 34% 8% 54
NHANES 2 8 32 36 22 101
II
NHANES 4 11 33 34 19 90
III
HHANES 0 7 22 37 33 28
NHIS- 0 8 23 62 8 13
Cancer
BRFSS 6 22 42 25 5 85
PNSS 7 14 38 35 6 73
PedNSS 9 17 35 33 6 96
Health 0 17 42 25 17 13
and
Diet
NFCS 8 13 43 32 4 110
CSFII 5 15 42 31 7 89
DHKS 4 9 52 26 9 23
Median 4 14 37 34 8
--------------------------------------------------------------------------------
Table IV.5
Users' Response to Whether Changes Would
Increase Their Confidence in or Use of
the Data
Da
ta
co
ll
ec
ti
on
ac
ti
vi Probably Probably No basis to Sample
ty No not Uncertain yes Yes judge size
-- ------- ---------- ---------- ---------- ----- ---------------- ------
NH 8% 16% 24% 25% 27% 0 54
E
F
S
NH 14 17 14 28 26 1% 101
A
N
E
S
I
I
NH 13 17 16 34 16 5 90
A
N
E
S
I
I
I
HH 19 15 4 15 44 4 28
A
N
E
S
NH 0 23 0 46 31 0 13
I
S
-
C
a
n
c
e
r
BR 5 20 16 30 28 1 85
F
S
S
PN 4 26 16 26 24 3 73
SS
Pe 6 26 11 27 29 2 96
d
N
S
S
He 25 17 17 8 25 8 13
a
l
t
h
a
n
d
D
i
e
t
NF 3 12 13 30 38 4 110
CS
CS 8 9 10 36 33 3 89
F
I
I
DH 9 9 22 26 35 0 23
KS
Me 8 17 15 28 29 3
d
i
a
n
--------------------------------------------------------------------------------
LIST OF EXPERTS
=========================================================== Appendix V
This appendix lists the expert advisers who assisted on this project.
The advisers were organized into three panels: core policy panel,
methodology panel, and data users panel.
CORE POLICY PANEL
------------------------------------------------------- Appendix V:0.1
Johanna Dwyer, D.Sc., R.D., Francis Stern Nutrition Center, New
England Medical Center and Tufts University Schools of Medicine and
Nutrition
Jean-Pierre Habicht, M.D., Ph.D., Division of Nutritional Sciences,
Cornell University
Catherine Woteki, Ph.D.\1
--------------------
\1 Dr. Woteki withdrew from the panel when she was appointed to the
Office of Science and Technology Policy in the White House. During
her participation in our work, she was the Director of the Food and
Nutrition Board of the National Academy of Sciences.
METHODOLOGY PANEL
------------------------------------------------------- Appendix V:0.2
Norman Bradburn, Ph.D., Director, National Opinion Research Center
Marilyn Buzzard, Ph.D., Director, Nutrition Coordinating Center,
University of Minnesota
Ricardo O. Castillo, M.D., M.P.H., Co-Director, Pediatric
Gastroenterology, Stanford University Medical Center
Alan R. Kristal, Dr. P.H., Fred Hutchinson Cancer Research Center
and Department of Epidemiology, University of Washington
James Lepkowski, Ph.D., Institute for Social Research, University of
Michigan
Cheryl Ritenbaugh, Ph.D., Department of Family and Community
Medicine, University of Arizona
Laura Sims, Ph.D., Department of Nutrition and Food Science,
University of Maryland
DATA USERS PANEL
------------------------------------------------------- Appendix V:0.3
Elizabeth Barnett, Ph.D., North Carolina Department of Environment,
Health, and Natural Resources
Doris Disbrow, Dr. P.H., R.D., Center for Health Education
Pamela Haines, Dr. P.H., R.D., Department of Nutrition, University
of North Carolina
Jay Hirschman, M.P.H., Senior Analyst, Food and Consumer Service,
U.S. Department of Agriculture
Karen J. Morgan, Ph.D., Senior Director, Nutrition and Consumer
Affairs, Nabisco Brands
Barbara Petersen, Ph.D., Technical Assessment Systems
(See figure in printed edition.)Appendix VI
COMMENTS FROM THE DEPARTMENT OF
AGRICULTURE
=========================================================== Appendix V
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)Appendix VII
COMMENTS FROM THE PUBLIC HEALTH
SERVICE
=========================================================== Appendix V
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
The following are GAO's comments on the letter from the Public Health
Service dated May 1, 1995.
GAO COMMENTS
1. Our analysis of the responses to the open-ended questions
involved, first, sorting responses by data collection activity and
focus of the comment (such as data element, population coverage, ease
of use). These responses were then aggregated to identify major
themes. The comments made by FDA users were not identified as a
major theme across the many users of the different systems and, thus,
were not reported separately. However, detailed summaries of the
responses were provided to the responsible agencies for their use.
2. We have included a reference to the Directory of Federal and State
Nutrition Monitoring Activities for those readers who are interested
in more information on the data collection systems. (See p. 3.)
MAJOR CONTRIBUTORS TO THIS REPORT
======================================================== Appendix VIII
PROGRAM EVALUATION AND METHODOLOGY
DIVISION
John Oppenheim, Assistant Director
Leslie Riggin, Assignment Manager
L� Xu�n Hy, Project Manager
James Joslin, Social Science Analyst
Venkareddy Chennareddy, Referencer
Elizabeth W. Scullin, Communications Analyst
*** End of document. ***