Nutrition Monitoring: Progress in Developing A Coordinated Program
(Letter Report, 05/27/94, GAO/PEMD-94-23).

The United States has one of the most comprehensive nutrition monitoring
programs in the world today. The main goal of nutrition monitoring is to
accurately measure and survey the dietary and nutritional status of the
U.S. population, as well as the quality, quantity, and safety of the
food it eats. Observing trends in the health of the population and
linking nutritional intake to health status are important elements of
effective monitoring. Data from current monitoring, conducted primarily
by the Departments of Agriculture (USDA) and Health and Human Services
(HHS), serve a multitude of users in government, academia, and private
industry. However, several recurring problems have plagued the
consistency, quality, and cost of nutrition monitoring during the past
two decades. As a result, USDA and HHS have been required to develop and
implement a 10-year comprehensive plan for the National Nutrition
Monitoring and Related Research Program. GAO concludes that (1) a
coherent, consistent system or program for nutrition monitoring is not
yet in place and (2) although the current 10-year plan reflects some
progress in planning for a program, several important aspects of the
plan remain incomplete. GAO urges Congress to continue closely
monitoring the development of the program to ensure its success.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  PEMD-94-23
     TITLE:  Nutrition Monitoring: Progress in Developing A Coordinated 
             Program
      DATE:  05/27/94
   SUBJECT:  Nutrition surveys
             Nutrition research
             Food supply
             Program evaluation
             Interagency relations
             Health research programs
             Health statistics
             Data integrity
             Data collection operations
             Statistical methods
IDENTIFIER:  National Nutrition Monitoring and Related Research Program
             HNIS Nationwide Food Consumption Survey
             National Health and Nutrition Examination Survey
             USDA Continuing Survey of Food Intakes by Individuals
             HANES
             
**************************************************************************
* This file contains an ASCII representation of the text of a GAO        *
* report.  Delineations within the text indicating chapter titles,       *
* headings, and bullets are preserved.  Major divisions and subdivisions *
* of the text, such as Chapters, Sections, and Appendixes, are           *
* identified by double and single lines.  The numbers on the right end   *
* of these lines indicate the position of each of the subsections in the *
* document outline.  These numbers do NOT correspond with the page       *
* numbers of the printed product.                                        *
*                                                                        *
* No attempt has been made to display graphic images, although figure    *
* captions are reproduced. Tables are included, but may not resemble     *
* those in the printed version.                                          *
*                                                                        *
* A printed copy of this report may be obtained from the GAO Document    *
* Distribution Facility by calling (202) 512-6000, by faxing your        *
* request to (301) 258-4066, or by writing to P.O. Box 6015,             *
* Gaithersburg, MD 20884-6015. We are unable to accept electronic orders *
* for printed documents at this time.                                    *
**************************************************************************


Cover
================================================================ COVER


Report to the Chairman, Committee on Science, Space, and Technology,
House of Representatives

May 1994

NUTRITION MONITORING - PROGRESS IN
DEVELOPING A COORDINATED PROGRAM

GAO/PEMD-94-23

Nutrition Monitoring


Abbreviations
=============================================================== ABBREV

  CDC - Centers for Disease Control and Prevention
  CSFII - Continuing Survey of Food Intakes by Individuals
  DHKS - Diet and Health Knowledge Survey
  EPA - Environmental Protection Agency
  ERS - Economic Research Service
  FDA - Food and Drug Administration
  GAO - General Accounting Office
  HHS - Department of Health and Human Services
  HNIS - Human Nutrition Information Service
  NCCDPHP - National Center for Chronic Disease Prevention and Health
     Promotion
  NCHS - National Center for Health Statistics
  NFCS - Nationwide Food Consumption Survey
  NHANES - National Health and Nutrition Examination Survey
  NNMRRP - National Nutrition Monitoring and Related Research Program
  NNMS - National Nutrition Monitoring System
  USDA - Department of Agriculture

Letter
=============================================================== LETTER


B-249872

May 27, 1994

The Honorable George E.  Brown, Jr.
Chairman, Committee on Science, Space,
 and Technology
House of Representatives

Dear Mr.  Chairman: 

The United States has one of the most comprehensive nutrition
monitoring programs in the world today.  Data from the current
monitoring activities, conducted primarily by the Departments of
Agriculture (USDA) and Health and Human Services (HHS), serve a
multiplicity of users in government, academia, and private industry. 
However, several problems have been identified over the past two
decades concerning the consistency, quality, and cost of the various
nutrition monitoring activities.  These problems eventually led to
passage of the National Nutrition Monitoring and Related Research Act
of 1990 (P.L.  101-445), which requires USDA and HHS to develop and
implement a 10-year comprehensive plan for the National Nutrition
Monitoring and Related Research Program (NNMRRP).  NNMRRP is intended
to enhance the benefits of current and future nutrition monitoring
activities. 

You asked us to perform the following work:  (1) an examination of
current monitoring activities and planning efforts for NNMRRP, to
include a review of statistical weighting issues associated with the
1987-88 Nationwide Food Consumption Survey (NFCS) data; (2) a
definition of a model program or system, and viable options for such
a program; and (3) a comparison of the methodological strengths and
weaknesses and the potential costs associated with a model program
and other viable options. 

This interim report covers the first component of the work requested
by the Committee.  It provides a descriptive overview of current
monitoring activities, summarizes the major findings and
recommendations of previous studies of nutrition monitoring
activities, reviews the act and planning activities by HHS and USDA,
and evaluates statistical weighting issues associated with the
1987-88 NFCS conducted by USDA.  Our discussion also considers the
validity of the data released to the public from NFCS, given concerns
raised regarding the low rate of response (34 percent) to that
survey.  We plan to complete the other parts of our work and report
on alternative approaches to nutrition monitoring early in 1995. 


   BACKGROUND
------------------------------------------------------------ Letter :1

Nutrition monitoring is a critical governmental activity.  Its
principal goal is to accurately measure and survey the dietary and
nutritional status of the U.S.  population, as well as the quality,
quantity, and safety of the food it consumes.  Observing trends in
the health of the population and linking nutritional intake to health
outcomes are important elements of effective monitoring.  Nutrition
monitoring should provide information on a regular basis about the
kinds and amounts of foods eaten by Americans; shifts in people's
knowledge about, and preferences for, certain foods (both of which
influence food choices); the composition of the foods eaten,
including their content of essential nutrients, as well as the
presence of any contaminants that may affect food quality or safety;
and the availability of food for consumption--which may in turn be
affected by such factors as food production practices, commodity
prices, and government farm support policies. 

These activities are clearly important in and of themselves, but the
monitoring program also serves to provide data for several public
policy uses.  For example, data from the program are used in
determining benefits in food assistance programs (such as food
stamps).  Data are also used to formulate national nutrition and
health policies (such as national initiatives that seek to lower the
fat content of diets or to educate the public about cholesterol),
devise food labeling regulations (such as determining serving sizes
and defining criteria for the qualification of nutrient content and
health claims), and evaluate nutrition and health programs (such as
studies to determine factors affecting participation in food programs
and studies of the relationship of calcium intakes to increased risk
of osteoporosis, hypertension, and colon cancer).  Finally, the data
are used to determine the adequacy and safety of the food supply. 
The FDA uses the data to assess the need for appropriate and safe
levels of food fortification and the levels of dietary exposure to
food additives and contaminants.  In addition, pesticide residue
estimates are calculated by the Environmental Protection Agency (EPA)
using data from the monitoring program.  Appendix IV contains more
detail on how the data are used. 

Over 70 different federal data collection activities, developed over
the past six decades, presently comprise the nation's nutrition
monitoring, surveillance, and research activities.  However, three
nationwide surveys constitute the heart of the program:  the HHS
National Health and Nutrition Examination Survey (NHANES), the USDA
Nationwide Food Consumption Survey (NFCS), and the USDA Continuing
Survey of Food Intakes by Individuals (CSFII).  A series of NHANES
studies have been conducted since 1971 to collect health and
nutrition data through direct physical examinations and interviews of
individuals.  NHANES III, the largest and most recent survey, is
being conducted in two 3-year segments over the period 1988-94. 
NFCS, conducted every 10 years, was last conducted in 1987-88.  It is
designed to collect data on household food use and individual food
consumption.  CSFII, while originally intended to be an annual
supplement to NFCS data, has been administered twice (1985-86 and
1989-91), with the most recent survey started in 1994.  (See table
I.1.)


   PRINCIPAL FINDINGS
------------------------------------------------------------ Letter :2


      DIFFICULTIES WITH THE
      CURRENT SET OF ACTIVITIES
---------------------------------------------------------- Letter :2.1

Even though a considerable amount of information is provided by
current federal nutrition monitoring activities, they do not
constitute a well-integrated system.  They are, instead, a kind of
patchwork of federal activities that have evolved over a 60-year
period.  Integrated approaches that specify measures and collect data
to answer the public policy needs of federal, state, and local
governments are not currently in place.  Those activities that do
exist have not been jointly planned so as to collect consistent and
comparable data. 

The agencies involved in nutrition monitoring have been criticized in
the past for not coordinating their data collection activities.  Our
review of the literature also revealed specific concerns that
included the cost of redundant activities, the difficulty of
comparing data across surveys, and data gaps (for example, the lack
of information on some population subgroups that are at high risk for
nutrition-related health problems).  In addition, issues concerning
data quality, response rates, frequency of data collection,
timeliness of reporting, and dissemination of nutrition monitoring
information have been raised.  These problems have led to a
reexamination of the relationship between the NHANES and USDA
surveys, and the collection of data from other federal nutrition
monitoring activities, so that these data will be most helpful to
users and amenable to integration into the new NNMRRP. 


      THE 10-YEAR PLAN
---------------------------------------------------------- Letter :2.2

As already noted, the National Nutrition Monitoring and Related
Research Act required that a 10-year comprehensive plan be developed
for establishing and implementing the coordinated NNMRRP.  This plan
was published in June of 1993 and is currently the centerpiece of HHS
and USDA planning efforts.  It lays out a broad set of activities
that are important and necessary for addressing known problems, and
when implemented, such planned activities should go a long way
towards achieving the goals and objectives for improving the
nutrition monitoring system. 

The plan, however, is weak in several respects.  It does not
establish priorities and does not provide a framework for
maintaining, deleting, or adding new monitoring activities.  The plan
relies largely on what already exists without placing monitoring
activities in a new coordinated context and critically evaluating
which activities are essential, which should be eliminated or
modified, and which new ones are needed.  In addition, no attention
is given in the plan to assessing the likely costs and the
feasibility of implementing monitoring activities.  Further, many of
the specific planning details normally present in a comprehensive
plan are deferred to future planning by working groups or committees. 
In sum, the plan lacks important ingredients essential for the
successful implementation of a new NNMRRP designed to resolve the
problems of the past. 

On the positive side, the enactment of the legislation and the
development of the 10-year comprehensive plan have led to a number of
agency activities that address at least some of the identified
weaknesses in nutrition monitoring.  (See table 1.) Perhaps most
encouraging is the fact that coordination between the two major
departments involved, HHS and USDA, has improved in recent years,
with the establishment of better collaboration and communication as
well as mechanisms to facilitate continued improvements.  Not only
did the agencies successfully work together in drafting the plan, but
the creation of the Interagency Board for Nutrition Monitoring and
Related Research has also provided a forum for discussion, review,
and implementation of NNMRRP.  The Interagency Board, for example,
has recently proposed options for prioritizing planning activities
set forth in the 10-year plan and prepared reports on the progress of
the coordinated program.  Furthermore, various workshops have also
been conducted that have linked federal with state officials and with
other nutrition monitoring users from the academic and industry
sectors, and additional studies have been undertaken to assess survey
sampling designs and methodologies as well as improve compatibility
between data collection and reporting activities. 



                                     Table 1.
                     
                      Criticisms of Nutrition Monitoring and
                                 Agency Responses

Criticism of nutrition
monitoring                    Agency response
----------------------------  --------------------------------------------------
Lack of coordination among    Interagency Board and Working Groups created\a and
nutrition monitoring          progress made towards coordinating the activities
activities                    of HHS and USDA

Lack of compatibility in      Common methods being assessed by HHS and USDA for
methods for assessing         gathering information on dietary intake
dietary intake

Specific population groups    NCHS initiated study to identify and evaluate
not covered by major surveys  design approaches for sampling population
                              subgroups in the next NHANES; supplement planned
                              for CSFII (1994-96) to increase collection of
                              dietary intake information on infants and children

Specific geographic areas     State-based surveillance systems expanded in
not represented by major      recent years as a source of some geographically-
surveys                       specific data; CSFII (1994-96) includes Alaska and
                              Hawaii, which were previously excluded from the
                              NFCS and CSFII surveys

Reporting by national         Revised directory of federal and state nutrition
surveys not                   monitoring activities published in 1992; first
integrated                    chartbook of selected findings from NNMRRP
                              published in 1993; and third scientific report on
                              the dietary and nutritional status of the U.S.
                              population currently being prepared under contract
                              (guidelines for reporting dietary intake data
                              developed by NCHS/HNIS working group and being
                              used by agencies in preparing data for the report)

Core set of standardized      Working group on comparability developed common
measures not yet developed    set of population descriptors in 1992 (which were
for major surveys             incorporated into the CSFII 1994-96
                              questionnaires); two other working group efforts
                              begun to assess similarities and differences of
                              survey questions on nutrition and nutrition-
                              related health, as well as on nutrition knowledge,
                              attitudes, and behavior; CDC/NCHS also funded
                              study to evaluate what core nutrition and health
                              indicators to include in NHANES and to determine
                              the feasibility of applying a core unit in
                              alternative settings, such as nursing homes,
                              schools, and so on

Compatible sampling           Study of NFCS/CSFII and NHANES sampling designs
techniques not used for       conducted in 1991;\b NCHS and HNIS funded contract
national surveys              to consider ways in which the next implementations
                              of NHANES and CSFII can link sampling

Information needs of users    Interagency Board provides forum for identifying
not systematically            federal agency data needs; workshop convened by
determined                    HNIS to review the objectives of the NFCS/CSFII
                              surveys and, in part, to determine whether the
                              surveys meet the needs of users;\c in planning
                              NHANES, NCHS formally solicited input from various
                              federal agencies and other researchers on which
                              topics to include in the survey

Data not collected            CSFII intended to be operated continuously; to
continuously                  date, CSFII has been conducted at periodic
                              intervals (next series to begin 1998)

Improvement needed in         Several workshops and studies conducted to assess
methodology for assessing     the tradeoffs and the strengths and weaknesses of
dietary intake and            different methods (for example, the appropriate
nutritional status            number and type of recalls to use)

Need for more timely          NCHS released NHANES III, Phase I (1988-91) data
dissemination of survey       on topics of public health importance (for
information                   example, cholesterol levels of the population)
                              beginning in 1993; HNIS has automated certain data
                              collection and processing activities, which may
                              help speed release of the CSFII 1994-96 data

Sampling problems with NFCS   HNIS has separated the household and individual
                              portions of the NFCS survey in order to reduce
                              respondent burden and improve response rates (NFCS
                              will become the Household Food Consumption Survey,
                              scheduled to be conducted in the late 1990s, and
                              CSFII will provide individual level data); in
                              addition, HNIS signed interagency agreements with
                              the Bureau of the Census for assistance in
                              designing and conducting the individual and
                              household surveys, as well as for research on
                              improving methods for collecting household food
                              use data

State and local data needs    Through state-based CDC/NCCDPHP surveillance
not fully addressed           systems, CDC assisting states in collecting some
                              nutrition-related data through program records of
                              participants in maternal and child health
                              programs, as well as telephone interviews with
                              randomly selected residents
--------------------------------------------------------------------------------
\a Three formal working groups on survey comparability, food
composition data, and federal- state relations and information
dissemination were established in 1989 and operate under the guidance
of the Interagency Board for Nutrition Monitoring.  They meet
regularly through the year to provide oversight for implementing
planning activities in the 10-year plan and to facilitate better
communications and coordination among agencies. 

\b Research Triangle Institute, "Sampling Designs and Population
Descriptors of Nationwide Food Consumption Surveys and National
Health and Nutrition Examination Surveys," report prepared for HHS
and USDA, July 1991. 

\c "Report of a Workshop to Review the Nationwide Food Consumption
Survey Conducted by USDA," seminar convened by HNIS at USDA
headquarters, Washington, D.C., July 29-30, 1991. 

Important methodological problems remain, however.  Many geographic
areas and specific population groups are not covered by the major
surveys.  Data collection and reporting by national surveys remain to
be integrated.  No core set of standardized measures has yet been
developed for the major surveys, and other measures are still not
interlocking across the three national surveys.  There is still a
need for compatible sampling techniques and for data that are
collected continuously.  A further weakness continues to exist in the
methodologies for assessing dietary intake and for measuring the
nutritional status of the nation's population.  In addition, state
and local information needs are not yet being fully addressed in
survey designs. 

Finally, we found that the methodology used to design weighting
equations for the 1987-88 NFCS data would have been technically
correct (that is, within the constraints of standard survey analysis)
if the data had been missing at random.  However, with no nonresponse
data available, it is impossible to know whether the missing data are
randomly distributed.  Therefore, there is no way to truly test the
accuracy of the weighting, which must therefore remain uncertain
until data are collected on the characteristics of the
nonrespondents. 

Warnings on the data are provided by USDA, and backup documentation
is delivered upon request to users.  However, given that the data
from the last NFCS may be problematic, we believe that they should be
used as a last resort and only after all other usable data sources
have first been identified.  Several major users of the data have
decided not to use the 1987-88 NFCS data.  For example, EPA could not
reliably estimate dietary exposure to pesticide residues using the
NFCS data because the sampling error ranged from 70 percent to up to
175 percent of the estimate for various subpopulation groups.  A
telephone survey we performed of NFCS users revealed a general
awareness of the data problems as well as a conservative approach to
their use.  (See appendix IV.)


   CONCLUSIONS
------------------------------------------------------------ Letter :3

We conclude that (1) a coherent, consistent system or program for
nutrition monitoring is not yet in place, and (2) although the
current 10-year plan reflects some progress in planning for a
program, several important aspects of the plan are incomplete.  We
therefore believe the Committee should continue to closely monitor
the development of NNMRRP in order to ensure its success. 

USDA and HHS provided written comments on a draft of our report. 
(See appendixes V and VI.) Officials from these departments agreed in
general with our principal findings and conclusions.  They did,
however, provide additional detail about activities under way within
their departments that they believe reflect further progress in
meeting the goals of NNMRRP.  We have incorporated these comments in
the report where appropriate. 

As agreed with your office, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30 days
from its date of issue.  At that time, we will send copies to
interested congressional committees and government agencies, and we
will make copies available to others upon request. 

If you have any questions or would like additional information,
please call me at (202) 512-2900 or Kwai-Cheung Chan, Director of
Program Evaluation in Physical Systems Areas, at (202) 512-3092. 
Major contributors to this report are listed in appendix VII. 

Sincerely yours,

Eleanor Chelimsky
Assistant Comptroller General


OVERVIEW OF THE NATIONAL NUTRITION
MONITORING SYSTEM
=========================================================== Appendix I

Nutrition monitoring refers to a broad range of activities designed
to periodically and systematically assess the dietary and nutritional
status of the American people, the conditions in this country that
affect the nutritional status of individuals, and the relationship
between diet and health.  The primary nutrition monitoring activities
consist of assessments of dietary and nutritional status obtained
through three nationwide surveys.  Supporting activities include
continuous updating of food composition data, as well as research on
human nutritional requirements and nutritional assessment methods. 


   USES OF SYSTEM DATA
--------------------------------------------------------- Appendix I:1

The current set of activities identified as nutrition monitoring,
often referred to as the National Nutrition Monitoring System (NNMS),
provide data for a broad range of goals and purposes.  \1 These cover
problem identification, policy making and program planning, program
evaluation, and related research areas. 

Data from NNMS have been used to develop the Dietary Guidelines for
Americans and the nutrition and related health objectives included in
Healthy People 2000, as well as to evaluate progress towards
achievement of the 1990 Health Objectives for the nation.\2 Other
uses of data from the system include the development of the
Recommended Dietary Allowances and the identification of areas of
nutrition research that are needed to increase the knowledge base and
revise standards pertaining to human nutrient requirements. 

Data from NNMS are also important for defining the prevalence of
nutrition-related health problems, developing practical program
strategies, and determining what changes, if any, are occurring over
time following the implementation of program interventions.  For
example, the National Institutes of Health have used NNMS data in
establishing clinical and population guidelines for the detection,
evaluation, and treatment of hypertension and high cholesterol, as
well as in assessing what progress has been made in lessening both of
these risk factors following the creation of the National High Blood
Pressure and Cholesterol Education Programs.  Similarly, the
Occupational Safety and Health Administration and the Environmental
Protection Agency (EPA) have used NNMS data in establishing
regulations aimed at combating elevated blood lead levels in the
population and in tracking the extent to which reductions have
occurred.  Furthermore, NNMS data have been analyzed by the Food and
Drug Administration (FDA) to identify how, and what types of, food
should be fortified with nutrients such as iron, zinc, and vitamin A. 

In addition, nutrition monitoring data have been important in
developing the Thrifty Food Plan, which forms the basis for
determining benefit levels for food stamp recipients, and in
evaluating various USDA food assistance programs (for example,
determining the factors that affect program participation, as well as
the extent to which participation affects food consumption and
expenditures).\3 NNMS also provides information to estimate the
impact that consumer demand and spending have on commodity supplies
and prices, which in turn are important factors in the management of
government farm commodity policies and programs.  Furthermore, food
consumption data are used by FDA to estimate dietary exposure to food
additives, toxicants, and contaminants, as well as by EPA to estimate
dietary exposure to pesticide residue levels.  FDA also uses the data
to develop food labeling regulations, such as establishing the
reference standard for determining serving sizes for nutrition
labeling purposes and defining the criteria for the qualification of
nutrient content and health claims. 

Beyond federal needs for information, state and local governments
require access to the data on food consumption to aid them in the
areas of resource allocation in public health policy, budget
justification in programming, identification of problems for
legislative or regulatory intervention, and determination of
evaluation needs.  Scholarly research is also conducted with the data
to better understand the relationship of diet to health, as well as
to understand the relationship between food supply and demand. 


--------------------
\1 Since enactment of the National Nutrition Monitoring and Related
Research Act of 1990 (P.L.  101-445), the system has been called the
National Nutrition Monitoring and Related Research Program (NNMRRP). 

\2 USDA and HHS, Nutrition and Your Health:  Dietary Guidelines for
Americans, 3rd ed.  (Washington, D.C.:  1990); and HHS, Healthy
People 2000:  National Health Promotion and Disease Prevention
Objectives (Washington, D.C.:  Public Health Service, 1990). 

\3 Further discussion of the Thrifty Food Plan is presented in
appendix IV. 


   NUTRITION MONITORING ACTIVITIES
--------------------------------------------------------- Appendix I:2

NNMS is defined to include all data collection and analysis
activities of the federal government associated with five traditional
categories: 

  nutritional and health status measurements;

  food consumption measurements;

  food composition measurements and nutrient data banks;

  dietary knowledge, behavior, and attitude assessments; and

  food supply and demand determinations. 

The Directory of Federal and State Nutrition Monitoring Activities
lists more than 70 separate survey, surveillance, and research
activities conducted by 22 different agencies of the federal
government charged with covering these areas.\4 (See table I.1.) Only
about one half of these activities, however, represent specific
monitoring of food and nutritient content, food consumption,
nutritional status, or relationships between diet and health or food
supply and demand.  Of the major ongoing nutrition monitoring
activities described in table I.1, there are three national surveys
that make up the core of NNMS:  (1) the Nationwide Food Consumption
Survey (NFCS), (2) the Continuing Survey of Food Intakes by
Individuals (CSFII)--both sponsored by USDA--and (3) the National
Health and Nutrition Examination Survey (NHANES) supported by HHS. 



                                    Table I.1
                     
                      Principal Federal Nutrition Monitoring
                                    Activities

Nutrition
monitoring           Monitoring
component            activity            Agency              Description
-------------------  ------------------  ------------------  -------------------
1. Nutritional and   National Health     HHS/CDC/NCHS        Nationwide data
health status        and Nutrition                           gathered on the
                     Examination Survey                      health and
                     (NHANES)                                nutritional status
                                                             of the population
                                                             through physical
                                                             examinations,
                                                             clinical and
                                                             laboratory tests,
                                                             and traditional
                                                             survey methods;
                                                             third NHANES
                                                             currently underway
                                                             and scheduled for
                                                             completion in 1994

                     Pregnancy           HHS/CDC/NCCDPHP/    Participating
                     Nutrition           states              states use data on
                     Surveillance                            low-income, high-
                     System                                  risk pregnant women
                                                             who participate in
                                                             government
                                                             nutrition and food
                                                             assistance
                                                             programs, in order
                                                             to monitor
                                                             nutrition-related
                                                             problems and
                                                             behavioral risk
                                                             factors associated
                                                             with low birth
                                                             weight

                     Pediatric           HHS/CDC/NCCDPHP/    Participating
                     Nutrition           states              states use data on
                     Surveillance                            low-income, high-
                     System                                  risk children who
                                                             participate in
                                                             government health,
                                                             nutrition, and food
                                                             assistance
                                                             programs, in order
                                                             to monitor
                                                             nutritional status
                                                             among children

2. Food consumption  Nationwide Food     NFCS/USDA/HNIS      Information
                     Consumption Survey                      collected from
                     (NFCS)                                  nationwide sample
                                                             of households and
                                                             individuals within
                                                             households on food
                                                             consumption
                                                             behavior (including
                                                             where foods are
                                                             purchased and
                                                             consumed and what
                                                             they cost) and the
                                                             nutritional content
                                                             of diets; NFCS
                                                             conducted every 10
                                                             years since 1936

                     Continuing Survey   USDA/HNIS           Designed to
                     of Food Intakes by                      supplement NFCS and
                     Individuals                             collect on a more
                     (CSFII)                                 regular basis
                                                             information on food
                                                             consumption and
                                                             nutritional content
                                                             of diets; CSFII
                                                             conducted twice
                                                             since the mid-
                                                             1980's

                     Total Diet Study    HHS/FDA             Key foods are
                                                             purchased from
                                                             stores and
                                                             restaurants,
                                                             prepared for
                                                             consumption, and
                                                             then analyzed to
                                                             determine nutrient
                                                             and contaminant
                                                             levels in the food
                                                             supply and in
                                                             representative
                                                             diets of specific
                                                             population groups

                     National Health     HHS/CDC/NCHS        Dietary intake
                     and Nutrition                           information on
                     Examination Survey                      individuals
                     (NHANES)                                collected as part
                                                             of overall NHANES
                                                             survey design

3. Food composition  National Nutrient   USDA/HNIS           Data compiled from
and nutrient data    Data Bank                               various sources on
                                                             the nutrient
                                                             composition of
                                                             foods; used as a
                                                             basis with food
                                                             consumption data
                                                             (from NFCS, CSFII,
                                                             and NHANES) to
                                                             estimate nutrient
                                                             intake

                     Food Label and      HHS/FDA             Survey of retail
                     Package Survey                          packaged foods
                                                             conducted to
                                                             monitor nutrition
                                                             labeling practices

4. Dietary           Diet and Health     USDA/HNIS           Telephone survey
knowledge,           Knowledge Survey,                       conducted to assess
behavior, and        follow-up to CSFII                      individuals'
attitudes                                                    knowledge and
                                                             attitudes about
                                                             dietary guidance,
                                                             food safety, and
                                                             other food and
                                                             nutrition issues

                     Health and Diet     HHS/FDA             Telephone survey
                     Survey                                  conducted to assess
                                                             public knowledge,
                                                             attitudes, and
                                                             practices
                                                             concerning food and
                                                             nutrition as they
                                                             relate to health
                                                             problems

                     Behavioral Risk     HHS/CDC/NCCDPHP/    Participating
                     Factor              states              states conduct
                     Surveillance                            telephone surveys
                     System                                  to assess personal
                                                             health practices
                                                             that are related to
                                                             leading causes of
                                                             death; optional
                                                             modules included
                                                             for the assessment
                                                             of dietary fat and
                                                             for fruit and
                                                             vegetable
                                                             consumption

5. Food supply and   Food and Nutrition  USDA/ERS/HNIS       Used to estimate
demand               Supply Series                           the levels of foods
                                                             and nutrients
                                                             available for
                                                             consumption in the
                                                             U.S. food supply by
                                                             deducting data on
                                                             exports, year-end
                                                             inventories, and
                                                             nonfood use from
                                                             data on production,
                                                             imports, and
                                                             beginning
                                                             inventories
--------------------------------------------------------------------------------

--------------------
\4 Prepared by the Interagency Board for Nutrition Monitoring and
Related Research, HHS Publication No.  (PHS) 92-1255-1 (1992). 


      THE NATIONWIDE FOOD
      CONSUMPTION SURVEY (NFCS)
------------------------------------------------------- Appendix I:2.1

The first national survey of household food consumption and dietary
levels, called the Consumer Purchases Study, was conducted jointly in
1935-36 by several federal agencies.  Since this survey, six national
surveys (with different names) have been conducted by USDA, roughly
every 10 years.  The early surveys--in 1935-36, 1942, 1948 (urban
only), and 1955--were designed to measure food used by the household
as a whole, the costs of that food, and the dietary levels of
household members. 

The 1965-66 survey, called the Household Food Consumption Survey, was
the first to include data collection in all four seasons and on food
intake by individual members of households as well as on food used by
the household as a whole.  Collection of information on the intake of
individuals was included because of the emerging interest in diet and
health.  The 1977-78 and 1987-88 Nationwide Food Consumption Surveys
(NFCS) continued to include both components:  (1) household food use
where the household food manager is asked to recall, with the aid of
a food list, the kinds and amounts of food used from home food
supplies during the previous 7 days and the cost of those foods; and
(2) individual intakes where each household member is asked to recall
the kinds and amounts of foods eaten at home and away during the
previous day and to keep a record of the food eaten on the day of the
interview and the following day (1-day recall/2-day record). 

NFCS was a multistage, stratified area probability sample that
targeted households in the 48 contiguous states and individuals
residing in those households.  In 1977-78 and 1987-88, NFCS included
two samples:  a basic sample of all households and a low-income
sample of households with incomes at or below 130 percent of the
poverty threshold--a level consistent with eligibility for the Food
Stamp Program.  In 1987-88, the basic sample was 4,589 households and
the low-income sample was 2,584 households.  The basic sample
included 10,172 individuals. 

USDA has replaced NFCS with two separate surveys--the Household Food
Consumption Survey (planned for the late 1990s) and the Continuing
Survey of Food Intakes by Individuals (CSFII), which is described in
the next section. 


      CONTINUING SURVEY OF FOOD
      INTAKES BY INDIVIDUALS
      (CSFII)
------------------------------------------------------- Appendix I:2.2

CSFII was initially designed to supplement data from NFCS annually. 
However, it has only been conducted in 1985-86, 1989-91, and is
currently underway for 1994-96 (as the third in a series).  The
survey has now replaced the individual intake component of NFCS.  The
surveys provide information on diets of individuals in the United
States, the diets of population groups of concern such as the
low-income population, and an indication of changes taking place in
the dietary status of Americans. 

The 1985-86 survey targeted persons (women aged 19 to 50 and their
children aged 1 to 5, and men aged 19 to 50) in households with all
levels of income as well as a separate sample with low income.  The
1989-91 survey was redesigned to provide data in 3-year time periods
and targeted all-income households and low-income households, as well
as individuals of all ages and both sexes.  Both surveys used
multistage, stratified area probability samples.  The first study
included the collection of six 1-day recalls at about 2-month
intervals during a 1-year period.  The first 1-day recall was
collected with an in-person interview; subsequent interviews were
done by telephone whenever possible.  The second survey included the
collection of 3 days of intake data.  In both surveys, respondents
were asked to recall the kinds and amounts of foods eaten at home and
away from home during the previous day.  In the second survey,
respondents were also asked to keep a record of foods eaten on the
day of the interview and on the following day (1-day recall and 2-day
record).  Both surveys used the Nutrient Data Bank, developed by
HNIS, to derive nutrients ingested by individuals. 

The Diet and Health Knowledge Survey (DHKS) was also initiated in
1989 as a follow-up to the CSFII.  It measures attitudes and
knowledge about diet and health among Americans.  DHKS and CSFII
together are designed to examine relationships between individuals'
attitudes and knowledge about food and nutrition and the same
individuals' food choices and nutrient intakes. 

The 1994-96 CSFII/DHKS differs from the 1985-86 and 1989-91 surveys
in several important ways.  Specifically, it features

  a target population of individuals in all 50 states, rather than
     the 48 contiguous states;

  the collection of 2 nonconsecutive days of food intake, rather than
     3 consecutive days (as in 1989-91), or 6 nonconsecutive days as
     in 1985-86;

  the use of two in-person 24-hour recalls, rather than 1-day
     recall/2-day record as in 1989-91 or a combined
     in-person/telephone 24-hour recall for 6 days as in 1985-86;

  oversampling of the low-income population, rather than a separate
     low-income survey;

  a larger sample in selected sex-age categories-- specifically,
     young children and elderly;

  subsampling within households, rather than the collection of
     information from all members of a household;

  collection of DHKS data from adults aged 20 and older, rather than
     from only main meal planners/preparers; and

  additional questions on attitudes and knowledge about using food
     labels. 


      THE NATIONAL HEALTH AND
      NUTRITION EXAMINATION SURVEY
      (NHANES)
------------------------------------------------------- Appendix I:2.3

Since 1960, the National Health Examination Survey has collected data
through interviews and direct physical examinations.  Since 1971,
when a nutrition component was added, the survey has been called the
National Health and Nutrition Examination Survey (NHANES).  The
survey is conducted under the direction of the National Center for
Health Statistics, which is part of the Centers for Disease Control
and Prevention (CDC).  NHANES III, the current, most recent, and
largest survey, is the seventh in the series of surveys using a
sample of approximately 40,000 individuals in communities throughout
the country.  The major goals of NHANES III are (1) to estimate the
national prevalence of selected diseases, risk factors, and health
conditions; (2) to assess the health and nutritional status of the
nation's population, as well as specific population subgroups, and
estimate changes over time; and (3) to provide information on the
interrelationships of health and nutrition variables. 

Survey participants are randomly selected.  The interview includes
demographic, socioeconomic, dietary, and health-related questions. 
Dietary intake data are collected from individuals using a single
24-hour recall.\5 The physical examination component consists of
medical and dental examinations, physiological measurements, and
laboratory tests administered by medical personnel. 

Examinations and interviews are conducted in specially-equipped
mobile examination centers that travel to survey sites throughout the
country.  The survey team consists of a physician, a dentist, medical
and health technicians, and dietary and health interviewers.  A large
staff of interviewers conduct the household interview.  The sample
for the survey is selected to be representative of the U.S. 
population aged 2 months and older.  In order to produce reliable
statistics for children, the elderly, blacks, and Mexican Americans,
these groups are oversampled for the survey. 

Data collection for NHANES III began in September 1988 and will
conclude in 1994; the survey team will have traveled to 88 locations
across the country by the time they complete their data collection. 
The survey is being conducted in two 3-year segments, with data
collected and analyzed at the end of the segments as well as for the
full survey.  The National Center for Health Statistics (NCHS) is
planning the next NHANES to begin in 1997. 


--------------------
\5 Nonrandom replicate recalls are also collected to adjust
population distributions of nutrients, and two additional 24-hour
recalls were collected by telephone for all examined persons aged 50
years and older in 1989-91 to estimate usual dietary intake in older
persons. 


PREVIOUS REVIEWS OF THE NUTRITION
MONITORING SYSTEM
========================================================== Appendix II

There is a substantial literature on nutrition monitoring, including
six comprehensive studies by expert panels:  an evaluation by a panel
of the National Academy of Public Administration on improving the
Health and Nutrition Examination Survey (1980); a study of food
consumption data sources by the National Research Council (1981); a
report on national survey data on food consumption by the National
Research Council (1984); a progress report from the Joint Nutrition
Monitoring Evaluation Committee, Department of Health and Human
Services (HHS) and Department of Agriculture (USDA), (1986); a report
on approaches to assessing nutrient adequacy by the National Research
Council (1986); and an update report on nutrition monitoring prepared
by the Life Sciences Research Office, Federation of American
Societies for Experimental Biology, for HHS and USDA (1988).  In
addition, the General Accounting Office (GAO) has reviewed the
nutrition monitoring system many times and made a number of
recommendations in consequence.  We summarize these reports,
including their findings, in the next two sections. 


   SIX EXPERT-PANEL EVALUATIONS
-------------------------------------------------------- Appendix II:1


      NATIONAL ACADEMY OF PUBLIC
      ADMINISTRATION (1980)
------------------------------------------------------ Appendix II:1.1

An evaluation panel of the National Academy of Public Administration
focused attention on improving the Health and Nutrition Examination
Survey (now NHANES).  Published in 1980, many of their conclusions
and recommendations are still relevant today.\1 The study emphasized
the importance of NHANES as the only means by which strictly
standardized physical examinations of a representative sample of the
population are conducted.  However, the evaluation panel pointed to
the essential need for a core set of standardized measures that would
be repeated in every national population survey for planning health
services and allocating public and private resources to health
programs, facilities, and education. 

Once the need for standardized measures had been met, the panel
concluded that other parts of the survey could assess selected
conditions of special national interest, which might change from
survey to survey.  The evaluation found that the survey should be
repeated every 5 years, with the midpoint coinciding as closely as
possible with that of NFCS.  Additionally, during the interval
between the national population surveys, the panel recommended that
one or more surveys should be conducted of special population
subgroups, such as ethnic, geographic, or age groups.  (This
recommendation was partially fulfilled by the Hispanic Health and
Nutrition Examination Survey conducted in 1982-84.)

The report stressed the importance of interlocking data on the
nation's health, nutrition, diet, and food utilization between the
Health and Nutrition Examination Survey and NFCS.  Therefore, the
panel recommended that comparable planning, scheduling, sampling,
field procedures, and coding would render the Health and Nutrition
Examination Survey data and the NFCS interview data complementary for
given socioeconomic and demographic groups.  Finally, the evaluation
panel stressed the need for care in both operations and reporting to
assure reliable and valid data that would be released within 12 to 15
months of the conclusion of the survey, with preliminary high
priority data being released before fieldwork was completed. 


--------------------
\1 National Academy of Public Administration, Panel to Review the
Health and Nutrition Examination Survey, Improving the Health and
Nutrition Examination Survey:  An Evaluation by a Panel of the
National Academy of Public Administration (Hyattsville, Md.: 
HHS/NCHS, 1981). 


      NATIONAL RESEARCH COUNCIL
      (1981)
------------------------------------------------------ Appendix II:1.2

The Food and Drug Administration (FDA) contracted with the Food and
Nutrition Board of the National Academy of Sciences to develop
recommendations for integrating sources of data on food consumption
with other data on nutrition and health status.  A Committee on Food
Consumption Patterns was formed to study existing data sources and
design a system to meet the needs of FDA and other agencies involved
in monitoring the food consumption and nutritional and health status
of the population.  The committee concluded that available
information from different sources was not adequate and could not be
integrated because of differences in sampling designs, data
collection methods, and measures.  \2 They recommended a proposed
system that would include

  continuous collection, processing, and analysis of food consumption
     data;

  a review of the precision of food identification and coding in the
     collection of food consumption data;

  the identification and transfer of existing sources of health
     status data to the proposed system;

  the use of other aggregate government and commercial databases to
     supplement information needed for an overall monitoring system;
     and

  interagency discussions to develop a coordinated system. 


--------------------
\2 National Research Council, Committee on Food Consumption Patterns,
Assessing Changing Food Consumption Patterns (Washington, D.C.: 
National Academy Press, 1981). 


      NATIONAL RESEARCH COUNCIL
      (1984)
------------------------------------------------------ Appendix II:1.3

In response to a request from the Human Nutrition Information Service
(HNIS) of USDA (and supported by NCHS as well), a Coordinating
Committee on Evaluation of Food Consumption Surveys was established
under the auspices of the Food and Nutrition Board in the National
Research Council's Commission on Life Sciences.  The purpose of the
review was to consider ways in which food consumption data from NFCS
and NHANES were used and to make recommendations on survey design
that would facilitate wider application of survey data. 

The report found that both surveys were important to a multiplicity
of users in government, the academic community, and industry, and
concluded that the present system of two separate national surveys
should continue.\3 The committee recommended that, although the two
surveys should continue to collect dietary intake data, a common
methodological core should be developed and implemented. 
Furthermore, the National Research Council reported that the two
surveys could be better linked through compatible sampling and common
population descriptors.  Finally, the review concluded that the NFCS
Individual Dietary Intake component and NHANES should be redesigned
as continuous survey processes with continuous data reporting to
ensure timely data release and reporting.  They recommended the
Household Food Use components of the NFCS should continue on a
regular, intermittent basis, unless future study demonstrated that
some other design (for example, continuous basis) would be more
advantageous. 


--------------------
\3 National Research Council, Commission on Life Sciences, Food and
Nutrition Board, National Survey Data on Food Consumption:  Uses and
Recommmendations (Washington, D.C.:  National Academy Press, 1984). 


      JOINT NUTRITION MONITORING
      EVALUATION COMMITTEE (1986)
------------------------------------------------------ Appendix II:1.4

The Joint Nutrition Monitoring Evaluation Committee was established
as a federal advisory group (sponsored by USDA and HHS) to report on
the nutritional status of the U.S.  population.  The committee
examined the importance of different food components in the diet and
their relation to nutritional and health status, relying heavily on
the dietary and health measurements derived from the NFCS and NHANES
data.  It was the first time that a systematic effort was made to
integrate and interpret data from these two major surveys.\4

The committee found that differences between the surveys concerning
sample design and population descriptors made comparability difficult
and also that more valid methods for measuring dietary intake and
health were needed to improve assessments of nutritional status.  The
committee further concluded that data gaps existed for several key
food components in the diet, which prevented adequate assessments of
their importance for public health. 

The committee was also charged with issuing recommendations to
improve the National Nutrition Monitoring System.  Such
recommendations were made in four general areas, as follows: 

  improve the information exchange between data users and gatherers
     by establishing a means to learn more about the information
     needs of users and how nutrition monitoring data are used, and
     by increasing the identifiability and availability of nutrition
     monitoring information for users;

  increase the use of existing data collected under the system by
     conducting more in-depth analyses for policy making and program
     management, providing better documentation of data files,
     improving the comparability of data so that information from
     different data sources can be integrated (for example, by
     developing "core questionnaires," compatible sampling schemes,
     and similar definitions of terms), and improving the timely
     release of data;

  improve the methods and techniques for gathering information for
     assessing nutritional status by expanding efforts to study the
     factors that influence nutritional status, improving coverage of
     low-income population groups in the monitoring system,
     developing nutrition indicators to monitor changes in food
     consumption and nutritional status, and increasing research to
     improve methods for assessing dietary intake and nutritional
     status; and finally,

  increase resources for the monitoring system to implement the
     foregoing recommendations. 


--------------------
\4 HHS, Public Health Service, NCHS, and USDA, Food and Consumer
Service, HNIS, Nutrition Monitoring in the United States:  A Progress
Report From the Joint Nutrition Monitoring Evaluation Committee
(Hyattsville, Md.:  July 1986). 


      NATIONAL RESEARCH COUNCIL
      (1986)
------------------------------------------------------ Appendix II:1.5

USDA asked the National Research Council to develop criteria and
approaches for using survey data on dietary intakes (particularly the
NFCS data) in order to estimate nutrient adequacy in the U.S. 
population.  The study, undertaken by the Subcommittee on Criteria
for Dietary Evaluation of the National Research Council, examined
various methods used by USDA and other researchers for assessing
nutrient adequacy.\5 The subcommittee concluded that the use of fixed
standards alone, such as the Recommended Dietary Allowances, did not
take into account normal variability in nutrient requirements among
individuals and could lead to imprecise estimates of nutrient
adequacy.  An alternative, probability-based approach for
interpreting nutrient intake proposed by the subcommittee would
incorporate information on the distribution of usual dietary intake
among individuals. 

To facilitate the further development and use of this approach, the
subcommittee recommended a number of design changes to NFCS, as well
as improvements in other nutrition monitoring research activities. 
The subcommittee recommended that consideration be given to (1)
having a sufficient number of days of intake data, (2) using a single
method for obtaining dietary intake data, (3) collecting intake data
on independent rather than on consecutive days, and (4) collecting
information on dietary supplements as well as on food intake.  The
subcommittee also recommended continued research on dietary intake
methods and the design of sampling strategies, as well as the
development of methods to improve the reference tables on nutrient
composition of foods. 


--------------------
\5 Subcommittee on Criteria for Dietary Evaluation, Coordinating
Committee on Evaluation of Food Consumption Surveys, Food and
Nutrition Board, Commission on Life Sciences, National Research
Council, Nutrient Adequacy:  Assessment Using Food Consumption
Surveys (Washington, D.C.:  National Academy Press, 1986. 


      UPDATE REPORT ON NUTRITION
      MONITORING (1989)
------------------------------------------------------ Appendix II:1.6

Like the report by the Joint Nutrition Monitoring Evaluation
Committee, this report was focused on an update of the dietary and
nutritional status, as well as nutrition-related health conditions,
of the U.S.  population.\6

This report placed special emphasis on using nutrition monitoring
data to examine two topics:  the relationship between diet and
cardiovascular disease and the assessment of iron nutrition
deficiency in the population.  It also addressed the strengths and
weaknesses of existing data and the methodological issues associated
with combining data from different components of the nutrition
monitoring system. 

The committee identified several problems and limitations in the
existing nutrition monitoring system and made the following
recommendations for improvement: 

  improve comparability of nutrient composition data and coding used
     in different dietary surveys;

  test the impact of methodological differences on survey results;

  use a common core of sociodemographic descriptors (for example,
     age, race, income, and education) in all NNMS surveys;

  increase similarities in NNMS data reporting;

  investigate methods for assessing population groups currently
     excluded from NNMS, such as the homeless, migrant workers,
     military personnel, Native Americans, and those residing in
     institutions;

  improve coverage of some groups at nutritional risk, such as
     infants, pregnant women, lactating women, preschool children,
     adolescents, and the elderly;

  improve measures of usual dietary intake in NHANES;

  collect information for assessing the impact of knowledge and
     attitudes on patterns of food consumption and nutrient intake;

  obtain quantitative information on vitamin and mineral supplement
     use to better estimate total nutrient intake;

  improve estimates of alcohol consumption;

  improve survey response rates and analyze nonresponse;

  educate data users on the proper use of data from complex surveys;
     and lastly,

  be responsive to the needs of state and local data users. 


--------------------
\6 This report built on the framework of the report from the Joint
Nutrition Monitoring Evaluation Committee.  Prepared by the Life
Sciences Research Office of the Federation of American Societies for
Experimental Biology and sponsored jointly by USDA and HHS, the
report is entitled Nutrition Monitoring in the United States:  An
Update Report on Nutrition Monitoring (Hyattsville, Md.:  September
1989). 


   GAO REPORTS
-------------------------------------------------------- Appendix II:2

We have issued several reports on nutrition monitoring, surveillance,
and research, dating back more than 20 years.  An early report was
our 1971 review recommending that the Health and Nutrition
Examination Survey (now called NHANES) and Household Food Consumption
Survey (now called NFCS) be merged.\7 In 1977, we revisited NFCS and
found need for improvement and expansion.  Specifically, our report
found the sample size to be too small to provide useful information
in evaluation of food assistance programs and in identifying
nutritional problems of low-income families.\8 We therefore
recommended that the Congress approve the requests for funds for an
additional survey of low-income families and that the survey
methodology be validated. 

In 1978, we reviewed a joint proposal developed and submitted to the
Congress by the Departments of Health, Education, and Welfare (now
HHS) and USDA for a comprehensive Nutritional Status Monitoring
System.  This proposal recognized that there was no adequate
surveillance system and proposed to institute one.  In our report, we
established a set of criteria for an effective nutrition surveillance
system, which were designed to (1) promptly identify nutritional
needs; (2) pinpoint, within rather narrow geographic boundaries, the
nutritional needs of specific target groups; (3) predict future areas
of nutritional concern; and (4) provide data that federal agencies
could use to monitor the effectiveness of programs to improve the
nutrition, health, and food consumption of various population
groups.\9

We found that a considerable amount of data was being collected that,
to varying degrees, satisfied the above criteria, but that a number
of weaknesses existed that prevented the programs from functioning as
an effective nutrition surveillance system.  Specifically, the system
(1) was not always sufficiently specific to identify problems by
narrow geographic area or did not always include important population
groups; (2) did not produce information in a timely manner; and (3)
did not provide information adequate for evaluating the effectiveness
of programs designed to improve nutritional health.  Many of the
components included in the proposal by the Department of Health,
Education, and Welfare and USDA were never adopted.  We criticized
the proposal for its lack of specificity and agreement between the
Departments, the lack of consensus on collaboration for a decennial
survey, the insufficient consideration of program evaluation, and
inadequacies in coordination mechanisms. 

Later in the same year, we issued a report on the future of the
National Nutrition Intelligence System.\10 The report reiterated
problems identified in earlier reviews, including untimely data
reporting, insufficient geographic specificity, omission of important
population groups, fragmentation and lack of integration for a
coordinated system between Departments, and a lack of evaluation
capability.  However, the report pointed out the positive action
taken by both USDA and the Department of Health, Education, and
Welfare in their jointly developed proposal for a comprehensive
system of nutrition intelligence in response to the Food and
Agriculture Act of 1977. 

We again reviewed the NFCS in 1991 and found that the most serious
data quality problem in the 1987-88 NFCS resulted from the low
response rate of only 34 percent of the households in the basic
sample.\11 We consequently again made the recommendations concerning
data validity, and suggested that a detailed sampling plan be
developed and that better internal controls be developed to avoid
problems with future surveys.  We also recently reviewed USDA's
procedures for evaluating the quality of its food composition data
and found that specific quality assurance criteria needed to be
developed to ensure data reliability.\12


--------------------
\7 Letter report to the Department of Health, Education, and Welfare
and the Department of Agriculture, July 30, 1971. 

\8 Nationwide Food Consumption Survey:  Need for Improvement and
Expansion, GAO/CED-77-56 (Washington, D.C.:  March 25, 1977). 

\9 Joint Proposal for a Nutrition Surveillance System, GAO/CED-78-145
(Washington, D.C.:  June 29, 1978). 

\10 Future of the National Nutrition Intelligence System,
GAO/CED-79-5 (Washington, D.C.:  November 7, 1978). 

\11 Nutrition Monitoring:  Mismanagement of Nutrition Survey Has
Resulted in Questionable Data, GAO/RCED-91-117 (Washington, D.C.: 
July 1991). 

\12 Food Nutrition:  Better Guidance Needed to Improve Reliability of
USDA's Food Composition Data, GAO/RCED-94-30 (Washington, D.C.: 
1993). 


      SUMMARY
------------------------------------------------------ Appendix II:2.1

As is evident from this series of reviews, several problems with
federal nutrition monitoring activities have been repeatedly
identified over the past two decades.  Although these problems have
thus been known for some time, many still exist today.  The problem
area on which there has been greatest consensus is the lack of
coordination between, and compatibility of, different data collection
activities, particularly the USDA and HHS surveys.  This encompasses
differences across surveys in methods for assessing dietary intake
and nutritional status, sampling designs, population descriptors and
other measures, and the timing and reporting of results.  In
addition, attention has been focused on the need for more valid
methods of measurement, better coverage of population subgroups and
geographic areas, and improved survey response rates. 


RECENT LEGISLATIVE CHANGES
========================================================= Appendix III


   THE 1990 LEGISLATION
------------------------------------------------------- Appendix III:1

Since the late 1970's, USDA and HHS have jointly issued three
separate plans to strengthen the nation's nutrition monitoring
system.\1 The lack of progress in implementing these planning
efforts, along with growing recognition of the need to improve
nutrition monitoring, led to enactment of the National Nutrition
Monitoring and Related Research Act of 1990 (P.L.  101-445).  As
noted earlier, the legislation requires the Secretaries of the
Departments of Agriculture (USDA) and Health and Human Services (HHS)
to prepare and implement a comprehensive 10-year plan to assess the
dietary and nutritional status of the U.S.  population, support
research and development of nutrition monitoring, foster national
nutrition education, and establish dietary guidelines.  This 10-year
coordinated effort is called the National Nutrition Monitoring and
Related Research Program (NNMRRP). 

The legislation delineates several activities, to be conducted
periodically and systematically, that are meant to result in timely
information about the role and status of factors bearing on the
eating habits and health of the people of the United States.  While
the law places responsibility for coordinating NNMRRP with USDA and
HHS, these activities cut across the jurisdictions of other federal
departments and agencies, including Labor, Commerce, Defense,
Veterans Affairs, and the Environmental Protection Agency (EPA).  To
facilitate coordination of nutrition monitoring activities and assist
in implementing the program, the law requires the establishment of an
interagency board.  In addition, it establishes a National Nutrition
Monitoring Advisory Council, consisting of independent experts (drawn
from outside the federal government), to provide technical advice on
the development and implementation of the program. 


--------------------
\1 "Proposal:  A Comprehensive Nutritional Status Monitoring System,"
1978; "Joint Implementation Plan for a Comprehensive National
Nutrition Monitoring System," 1981; and "Operational Plan for the
National Nutrition Monitoring System," 1987. 


      THE 10-YEAR COMPREHENSIVE
      PLAN
----------------------------------------------------- Appendix III:1.1

USDA and HHS published a draft of the Ten-Year Comprehensive Plan for
the National Nutrition Monitoring and Related Research Program for
public comments in the Federal Register in October 1991.  The plan
was subsequently revised, based on 53 sets of written comments
received, and issued in final form in June of 1993. 

The plan provides an overview of the history of nutrition monitoring
and extensive listings of past and current monitoring activities.  It
lays out a set of planning activities that are organized around three
overall national objectives (achieve continuous and coordinated data
collection, improve the comparability and quality of data across
NNMRRP, and improve the research base for nutrition monitoring) and
three objectives addressing state and local nutrition monitoring
efforts (develop and strengthen state and local capacities for
continuous and coordinated data collection, improve methodologies to
enhance comparability of NNMRRP data across federal, state, and local
levels, and improve the quality of state and local nutrition
monitoring data).  In addition, the plan also emphasizes the need for
better nutrition monitoring information about selected population
subgroups and for more efficient and effective dissemination to, and
exchange of information with, data users.  Since both of these areas
cut across other nutrition monitoring components, they are not
discussed separately in the plan but rather are addressed in the
sections devoted to the national and state objectives.  Within each
section, planned activities and agency responsibilities are described
according to five component areas--nutrition and related health
measurement; food and nutrient consumption; knowledge, attitudes, and
behavior assessments; food composition and nutrient data bases; and
food supply determinations. 

The plan recognizes the problems identified within the nutrition
monitoring system over the past two decades and provides a broad,
comprehensive set of activities to address them as well as other
requirements of the act.  Many of the activities are important and
necessary for strengthening the nutrition monitoring system.  A
general time frame (specified by year) for when planned activities
will be initiated and completed, as well as when expected products
will be issued, is included in the plan.  In addition, the plan
identifies agency responsibility in terms of lead, collaborating, or
contributing agency, for each planned activity.  Taken together,
these planned activities, when implemented, should go a long way
towards achieving the goals and objectives for improving the
nutrition monitoring system. 

One weakness of the plan, however, is that while existing problems
are identified as issues, they are often responded to with plans for
more planning under NNMRRP.  That is, instead of the provision of
specific solutions, there are calls to develop further plans, review
existing programs and procedures, assess needs, and evaluate
alternative methods.  The plan also provides little evidence of the
extent to which listed activities can or will be implemented in the
future.  Given the broad scope of activities included in the plan,
several of which involve large research and data collection efforts,
it is not clear that what is planned for can be accomplished. 
Further, no attention is given in the plan to prioritizing the myriad
activities or assessing the likely costs and feasibility of
implementing them.  Instead, USDA and HHS acknowledge, in their
foreword to the plan, that completion of planned activities will be
heavily influenced by the availability of financial resources.  They
estimate that 20 to 40 percent additional funding will be needed to
carry out the plan, amounting to over $200 million in additional
appropriations through fiscal year 1998. 


THE 1987-88 NATIONWIDE FOOD
CONSUMPTION SURVEY:  QUALITY AND
USES OF THE DATA
========================================================== Appendix IV


   BACKGROUND
-------------------------------------------------------- Appendix IV:1

In this appendix, we review statistical weighting issues in the
1987-88 NFCS data set, as identified by the Joint Nutrition
Monitoring Evaluation Committee.  As we reported earlier, the most
serious data quality problem in the 1987-88 Nationwide Food
Consumption Survey resulted from the low response rate for the basic
sample.\1 Only 34 percent of the households in the basic sample
provided individual data.  This raised the question of whether the
data were representative of the U.S.  population.  USDA contracted
with a group of statisticians to develop statistical weights to
adjust for the high nonresponse rate.  These weights are now included
with the data tapes provided to users. 


--------------------
\1 Nutrition Monitoring:  Mismanagement of Nutrition Survey Has
Resulted in Questionable Data, GAO/RCED-91-117 (Washington, D.C.: 
July 26, 1991). 


   IMPACT OF NONRESPONSE ON
   DIETARY DATA FROM THE 1987-88
   NFCS
-------------------------------------------------------- Appendix IV:2

In 1991, an expert panel was convened by HNIS to assess the integrity
of the 1987-88 data.  The panel's report focused solely on the
1987-88 NFCS and examined the statistical design and survey
execution.\2 The report placed particular emphasis on issues related
to nonresponse, reviewed analyses of nonresponse conducted by HNIS,
identified additional analyses needed to evaluate further the
potential for nonresponse bias in NFCS, and reviewed and identified
critical issues relating to the implications of potential nonresponse
bias for possible inclusion by HNIS in formal publications of survey
results and research analyses. 

The report concluded that it is not possible to definitively
establish the presence or absence of nonresponse bias in the 1987-88
NFCS data.  The possibility of nonresponse bias is suggested,
however, by the analyses reviewed in the report.  According to the
panel, it is impossible to determine the extent to which nonresponse
bias might influence the interpretation of analyses using these data. 

The report also concluded that it is questionable whether any
weighting system could rectify the nonresponse and possible
noncoverage (day of the week and month of the year) of the survey. 
Because of the high level of nonresponse, the report includes an
opinion that no weighting procedure could give users the confidence
that the low response rate had been successfully dealt with.  The
report to HNIS warned that HNIS should include strongly worded
cautionary statements concerning the potential for nonresponse bias
in all publications of the NFCS 1987-88 data, as well as with all
public releases of information and data.  We note that HNIS continues
to make the 1987-88 data available to the public and does provide a
notice to those users receiving the data set emphasizing its
nonrepresentativeness.  In addition, a nonresponse evaluation report
is distributed by HNIS to data users who request it. 


--------------------
\2 Life Sciences Research Office, Federation of American Societies,
Impact of Nonresponse on Dietary Data From the 1987-88 Nationwide
Food Consumption Survey (Bethesda, Md.:  April 1991). 


   USES OF THE 1987-88 NFCS DATA
-------------------------------------------------------- Appendix IV:3

In ongoing work on nutrition monitoring data uses and users, we are
making a thorough study of NFCS data utilization.  Here we want
merely to mention two critical uses that underscore the importance of
high quality data from NFCS.  First, the Food and Nutrition Service,
USDA, makes benefit determinations for the Food Stamp Program based,
in large part, on NFCS data.  Second, the Environmental Protection
Agency (EPA) establishes safe levels of pesticide residues for food
based on data from NFCS.  These uses make clear the importance of
nutrition monitoring data not only to the understanding of health and
nutrition, but also to food assistance and safety programs. 


      DETERMINING FOOD STAMP
      BENEFIT LEVELS
------------------------------------------------------ Appendix IV:3.1

While eligibility for food stamps is based primarily on income and
assets, benefits are based on the cost of the USDA Thrifty Food
Plan--a nutritionally adequate diet required to feed a family of four
consisting of two adults and two children--adjusted for household
size.  The food stamp benefit, or allotment, is the difference
between the cost of the Thrifty Food Plan for an eligible household
and one third of its countable income.  About 26 million people are
recipients of food stamps. 

The cost of the Thrifty Food Plan originates from its predecessor,
the Economy Food Plan.  The Economy Food Plan was based on the 1955
Household Food Consumption Survey data and was used to develop
official poverty thresholds.  The Thrifty Food Plan replaced the
Economy Food Plan in 1975.  In the development of the Thrifty Food
Plan in 1975 and its subsequent revision in 1983, the target cost for
each was linked to the updated cost of the plan that was being
replaced.  Thus, the cost of the current Thrifty Food Plan is based
on the inflation-adjusted cost of the original plan.  An inflation
adjustment is made monthly by HNIS using Bureau of Labor Statistics
food inflation data.  This adjustment has been applied to a food
basket of 31 food groups based on consumption patterns by low-income
households in the 1977-78 NFCS adjusted to meet cost and nutritional
standards.  The Thrifty Food Plan uses the consumption patterns of
low-income households provided by the NFCS low-income sample.  This
survey collected data on the food consumption behavior of low-income
households.  The plan uses the NFCS household and individual intake
data components. 

According to a USDA official, questions about the reliability and
credibility of the 1987-88 NFCS data set led to the internal decision
not to revise the most recent food plan based on 1977-78 NFCS data. 
Given that these are 15 years old, they may not be representative of
the consumption behaviors of today's low-income households. 
According to this same official, there is no other source of
household food consumption data available for revising the food plan. 
Therefore, if current methods are used, the plan cannot be revised
until the data from the planned 1996 survey are available.  This
suggests that USDA will continue to determine food stamp benefits on
data collected in 1977-78 through the year 2000. 


      PESTICIDE RESIDUE LEVELS FOR
      FOOD
------------------------------------------------------ Appendix IV:3.2

To establish safe levels of pesticide residues for food, EPA
estimates dietary exposure to pesticide residues using data from
NFCS.  EPA registers pesticides and establishes maximum allowable
pesticide residues (called tolerances) for food in accordance with
the Federal Insecticide, Fungicide, and Rodenticide Act and the
Federal Food, Drug, and Cosmetic Act.  EPA registers pesticides for
specific uses and takes regulatory action--such as denying,
canceling, or restricting a pesticide's use--if a pesticide presents
a significant health or environmental risk.  To determine potential
health effects, EPA conducts risk assessments of pesticide products,
based largely on laboratory tests and field trials of pesticides. 

In 1986, EPA began estimating dietary exposure to pesticide residues
for tolerance assessments using a computerized system and data
collected from the 1977-78 NFCS.  In addition to individual food
intake information, EPA uses data such as the individual's age,
gender, weight, race, and place of residence.  These data, combined
with residue data (usually tolerance levels), allow EPA to estimate
exposure for 22 distinct subpopulations who, because of their diets,
may be exposed to unsafe pesticide levels. 

Because they are based on a sample of the population, EPA's pesticide
exposure estimates are subject to sampling error.  Our analysis
indicated that EPA's ability to adequately base tolerance assessments
on exposure estimates for the five smallest subpopulations--namely,
nursing and non-nursing infants, nursing females, pregnant females,
and non-Hispanic others (such as Asians and Native Americans)--could
be compromised because the sampling error for these subpopulations,
based on the 1987-88 NFCS data, can range from nearly 70 to up to 175
percent of the estimate.\3

EPA was unable to update food consumption data with the 1987-88
survey results to reflect the current eating habits of Americans. 
The next major update is scheduled to occur after the 1996 survey has
been conducted.  This represents another example of an important
government program affected by data from NFCS.  In this case, a major
government program is using noncurrent data and may continue to do so
through the year 2000. 


--------------------
\3 For the complete report on this issue see Pesticides:  Food
Consumption Data of Little Value to Estimate Some Exposures,
GAO/RCED-91-125 (Washington, D.C.:  May 22, 1991). 


   USERS' AWARENESS OF 1987-88
   NFCS DATA LIMITATIONS
-------------------------------------------------------- Appendix IV:4

The weaknesses of the 1987-88 NFCS data were well-documented before
we prepared this report.  However, as already noted, use of the data
continues, and the Committee asked us to consider whether users were
aware of the data's limitations.  We contacted users, identified for
us by HNIS, of both the Household Data and the Individual Intake Data
from the 1987-88 NFCS.  HNIS identified 61 users of the data tapes as
of December 1992.  We were given a list of users who obtained data
tapes directly from HNIS or who purchased the data through the
National Technical Information Service.  We contacted 31 of those
listed and conducted an informal telephone survey using a
standardized interview schedule from April through May of 1993.  We
asked data users in government, academia, and industry the following
questions: 

1.  Can you please describe your use of the 1987-88 NFCS data set? 

2.  What do you consider to be the strengths and weaknesses of the
data? 

3.  Are you aware that, of the households selected in the basic
sample, 34 percent responded to the survey? 

4.  In your analysis of the data, do you break the data out for
subgroups?  If so, what groups?  Do you utilize the weighting
formulae provided with the data by HNIS? 

5.  Are you comfortable with the results of your analyses using this
data set?  Why or why not? 


   RESULTS OF TELEPHONE SURVEY
-------------------------------------------------------- Appendix IV:5

Government uses, according to our respondents, included determining
whether the nutrient needs of children are being met and identifying
the barriers to a nutritious diet as well as the shopping habits of
households.  The Food and Drug Administration (FDA) uses the amount
of food commonly consumed as a standard to determine serving sizes
for labeling.  FDA also uses NFCS to select core food samples for its
Total Diet Study, a yearly monitoring program, and for assessment of
the adequacy and safety of the U.S.  food supply.  Another agency
conducted a study to identify the shopping habits, preparation
skills, and dietary guidance of low-income persons in order to
determine how many achieved the Recommended Daily Allowance of
nutrients for the nation's population.  The Federal Trade Commission,
which regulates food advertising, looks at how new types of
advertising may change behavior.  For example, the Commission looks
at how consumption of cereal items and fat, as well as cholesterol
claims, change after advertisers air messages linking fat consumption
and heart disease.  Any change in consumption, such as a reduction in
fat, should be detectable in NFCS. 

Nonfederal government, academic, and food industry uses of the NFCS
data identified by our respondents included the study of low-income
households; food consumption patterns of various U.S.  populations;
exposure to pesticides; the prevalence of high fat diets in
low-income populations; the impact of employment, age, and gender on
eating patterns and nutrient profiles; nutritional adequacy profiles
for food consumed away from home and money spent on food away from
home; and finally, demand functions for dairy products, product
development, serving sizes, labeling, nutrient composition data for
product menu/recipe development and planning, product promotions,
dietary guidelines, and consumption of meat products. 

Our respondents identified as the main strength of the NFCS data the
fact that it is the only large household food use data set available. 
In addition, the broad variety of foods, with commodities delineated
and the dollar value of the food items listed, is a unique data
source.  Information is also available in NFCS on household income,
food expenditures, and demographics.  Time and again, users called
the survey results the "only data around" to approximate a national
probability sample with household level information on food use. 
Overwhelmingly, respondents indicated that the chief weakness of the
data was the low response rate.  Only one user was unaware of this
issue. 

Most of our respondents utilized the weights provided with the data
by HNIS, and seemed to interpret their results with caution.  When
large aggregated data were analyzed, it appears our respondents were
less concerned that the results might be skewed.  However, when
breaking the data out into smaller cells (such as Hispanics, black
males, young persons, or alternatively, infrequently consumed foods),
the level of comfort with any result declined for our users.  Several
of them tried to match samples with data in CSFII or other national
data sets, to compare for similar trends. 


   USING THE WEIGHTED DATA FROM
   THE 1987-88 NFCS
-------------------------------------------------------- Appendix IV:6

The original 1987-88 basic survey sample was designed as a
self-weighting, stratified, multistage, area probability sample,
representative of the households in the 48 coterminous states.  There
were 60 strata.  Two primary sampling units were selected from each
stratum, with replacement.  Area segments were selected from each
selected sampling unit; then housing units were selected from each
selected area segment.  Every household member of each selected
housing unit was to be interviewed. 

The survey was planned to start in April 1987 and continue to March
of 1988.  The total number of area segments was 1,000.  The total
number of estimated sample housing units was 8,800, with each quarter
including 2,200.  Four interpenetrating samples were to be drawn--one
for each quarterly survey.  However, due to the high nonresponse rate
of the first quarter, the sample sizes of successive quarters were
increased, and the survey was extended to August of 1988.  After
March of 1988, the households interviewed were housing units from the
four prior quarterly sample housing units that had not yet been
interviewed. 

After sample housing units were selected, the interviewers contacted
and screened them, and then made appointments with eligible
households to conduct interviews.  The interviews could not be
conducted until at least 7 days after screening.  During the
interview, the interviewer obtained the household food use over the
last 7 days and 1-day dietary recall from each member of the
household, and left a 2-day dietary record for each member of the
household to complete.  The final sample for the 1987-88 NFCS basic
survey deviated from the original design and was not a self-weighting
sample. 

As has been stated earlier, in spite of efforts to raise them, the
NFCS response rates were very low.  A standard practice in survey
research is to develop weights to be applied to the data in analyses
which attempt to compensate for the nonresponse.  Weights were
developed to compensate for the NFCS nonresponse.  However, given the
high level of nonresponse in the survey, we concur with the report
that suggests there may be nonresponse bias in the data despite the
weighting adjustments.\4

We reviewed general weighting adjustment methods for NFCS, including
sample-based adjustments, population-based adjustments, a regression
adjustment procedure, combinations of sample- and population-based
adjustments, and the combination of sample-based and regression
adjustments.  The weighting adjustment methods chosen depend on the
survey design and what kinds of information are available. 
Generally, combination methods are better.  The regression adjustment
method was used to develop weights released with the NFCS data. 

In the combined method, subgroups called weighting classes are used
for the weighting adjustments and are formed using a combination of
control variables.  The most important issue is to find weighting
classes in which a missing-at-random assumption or a "similarity
assumption" holds as accurately as possible.  The missing-at-random
assumption states that nonresponding units within weighting classes
are a random sample of all units within the class.  Similarity is a
weaker assumption requiring only that responding and nonresponding
units within a class be, on average, alike.  Usually, the choice of
classes is decided by experience and best judgment. 

The two major categories of missing units are "not-at-home" and
"refusal." The probability of not-at-home depends on a wide variety
of situations.  For example, a family with small children may be easy
to find because they are at home more often than families without
small children, whereas single people may not be easy to find. 
Alternatively, older or widowed "shut-ins" may be easy to find, while
young married, childless couples may not.  Other significant factors
are housing unit location, household head's age, and employment
status. 

The probability of refusal depends on the patience of respondents,
the number of surveys they have been exposed to, the skill of the
survey staff, and other factors.  Different survey subjects have
different degrees of patience.  For example, some studies suggest
that higher income and busy people usually have a lower level of
patience with surveys.  The poor design of a questionnaire, which can
cause heavy respondent burden, as well as poor interviewing skills,
may increase the probability of refusal. 

Since unit nonresponse is affected by many different circumstances,
it is hard to find weighting classes in which data are likely to be
missing at random.  However, it is easier to find the weighting
classes in which data for responding and nonresponding units are
similar.  Usually, units with the same characteristics--such as age,
sex, and level of urbanization--yield similar data. 

We conclude that the weighting techniques used for the 1987-88 NFCS
data are technically correct within the constraints of standard
survey analysis, assuming the data are missing at random.  However,
it is virtually impossible to determine whether the missing-at-random
assumption is valid for these data, especially in view of the high
level of nonresponse.  Thus, results of analyses that use these data
are questionable even when the regression adjustment weights are
employed. 

There are other adjustments that could have been made that require
weaker assumptions.  Even these adjustment procedures are suspect
since there is no way of knowing whether the findings are
substantially biased due to the high level of nonresponse.  Thus,
these data should be used only as a last resort, when no other data
are available.  Even when used as a last resort, they should be used
with caution and an awareness of the potential problems inherent in
the data.  Further, any findings from these data must contain a
caution to the readers of the results concerning the potential for
bias due to nonresponse. 



(See figure in printed edition.)Appendix V

--------------------
\4 Life Sciences Research Office, Federation of American Societies,
Impact of Nonresponse on Dietary Data From the 1987-88 Nationwide
Food Consumption Survey (Bethesda, Md.:  April 1991). 


COMMENTS FROM THE DEPARTMENT OF
AGRICULTURE
========================================================== Appendix IV



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)




(See figure in printed edition.)APPENDIX VI
COMMENTS FROM THE DEPARTMENT OF
HEALTH AND HUMAN SERVICES
========================================================== Appendix IV



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)


MAJOR CONTRIBUTORS TO THIS REPORT
========================================================= Appendix VII

PROGRAM EVALUATION AND METHODOLOGY
DIVISION

James Joslin
John E.  Oppenheim
Leslie Riggin
Randall H.  Wold

DENVER REGIONAL OFFICE

Maricela Camerena
Art Gallegos

